A citation that opens the right manual can still highlight the wrong sentence. Bind each citation to an immutable source revision, a versioned extracted-text representation and an explicit coordinate system. Verify the quote and its surrounding context before showing the highlight. If these checks disagree, return an unresolved citation—not the first plausible match. This establishes locator integrity; it does not establish that the sentence supports the answer or that the reader may access it.
This matters when a parser upgrade changes whitespace, a browser counts UTF-16 units while a backend counts Unicode code points, or a repeated heading appears on several pages. More retrieval tuning cannot repair that interface contract. The buyer decision is whether your evidence viewer can prove which span it displayed after ingestion changes, rather than merely showing a credible document name.
Define three identities before choosing offsets
Keep a stable business document ID and revision policy, but also hash the exact raw source bytes. Separately hash the extracted, normalized text encoded with a declared encoding. Record the parser package, configuration and relevant model revisions, plus a normalization-profile identifier. The fixture below uses a deliberately simple plain-text adapter, not a PDF parser.
The source digest detects a changed artifact even when its extracted text happens to be identical. The text digest detects changes introduced by extraction. The profile identifies the procedure that produced those bytes. These fields are complementary: a hash without a protected manifest does not authenticate who supplied the document, and a matching hash grants no permission.
Document applicability comes first. The revision-resolution workflow decides which manual applies to an asset. The anchor then answers a narrower question: does this citation still select the recorded span in that exact representation? Do not migrate a historical citation to the newest manual simply because its title is the same.
Use W3C semantics without inventing a W3C rejection rule
The Web Annotation Data Model defines TextQuoteSelector with exact text and optional prefix and suffix, and TextPositionSelector with a zero-based inclusive start and exclusive end. Selection uses Unicode code points in logical text order, not encoded code units. Quote and position selection must use the same normalized text. Avoid splitting a grapheme cluster when choosing boundaries.
The specification discusses normalization such as removing markup and decoding character entities. That is not a complete instruction to choose NFC, NFKC, case folding or your own whitespace policy. Specify those application choices separately. The Web Annotation Vocabulary supplies the selector terms; it does not supply your source-digest manifest, authorization service or evidence-approval policy.
There is an important policy distinction. When prefix, exact and suffix still match multiple sequences, W3C says the selection SHOULD be treated as matching all matches. Our single-span citation application instead rejects ambiguity. That is an intentionally stricter application rule, not a claim that W3C mandates rejection. Requiring quote and position to agree is likewise our integrity policy. Do not describe an application sidecar as a complete interoperable JSON-LD annotation without implementing and testing its serialization.
Pin normalization and keep the original
Unicode Standard Annex #15 distinguishes canonical from compatibility equivalence. NFC composes canonically equivalent sequences where possible. NFKC also applies compatibility mappings; a ligature can become multiple letters. Neither form is a universal solution for evidence preservation. Retain the raw source and make the chosen transformation explicit.
Our profile decodes UTF-8 strictly, changes CRLF to LF, then applies NFC. It does not trim, collapse whitespace, case-fold, remove markup or expand compatibility ligatures. The runtime Unicode database is pinned in the recorded profile. A parser that dehyphenates line endings or removes page headers needs additional explicit rules and migration tests.
Do not add a normalized offset directly to a raw-source offset. Composition and expansion change lengths; reordering and extraction can also change correspondence. Preserve an alignment map where you need original-page highlighting. For PDFs, text positions alone do not identify page boxes. The PDF evidence contract explains why cell, header and page provenance must survive separately.
Executed fixture: three coordinates for one heading
A local synthetic fixture was executed with Python 3.14.9 and its Unicode 16.0.0 database. The input begins with a supplementary-plane tool emoji, a space, decomposed Müller, CRLF and the heading Prüfung. Later it repeats Prüfung and contains Office with the single ffi ligature. These are test strings, not operational instructions or customer evidence.
After CRLF-to-LF and NFC normalization, the first Prüfung occupies the half-open code-point span [9, 16). Its start is byte offset 13 in UTF-8 and code-unit offset 10 in UTF-16. The visible heading is the same, but the numeric coordinates are not interchangeable. The NFC output retains ffi; a separate NFKC assertion confirms its expansion to ffi. An assertion also confirms composition of the umlaut and removal of CR characters.
| Coordinate system | Heading start | Consumer contract |
|---|---|---|
| Unicode code points | 9 | Recorded normalized span [9, 16) |
| UTF-8 bytes | 13 | Not a code-point position |
| UTF-16 code units | 10 | Viewer conversion required |
The stored anchor contains raw-source SHA-256, normalized-text SHA-256, parser plain-text-fixture-v1, profile utf8-strict;crlf-to-lf;NFC;unicode-16.0.0, start 9, end 16, exact Prüfung, prefix Müller followed by LF, and suffix LF followed by A:. The record is a frozen Python dataclass in the test. Production immutability additionally requires controlled storage and versioning; a frozen object is not a security boundary.
The resolver checks the injected access decision first, then source digest, parser/profile and text digest. It finds all exact-plus-context candidates. It rejects zero or multiple candidates, and requires the unique candidate to agree with the stored half-open position. It never repairs an anchor by silently choosing the first occurrence. An authorized explicit migration may create a new reviewed anchor, leaving the old record intact.
Resolution states are part of the API contract
| State | Meaning | Response |
|---|---|---|
| DENIED | Current access refused | Return no protected evidence |
| REVISION_MISMATCH | Raw bytes changed | Resolve approved revision explicitly |
| PROFILE_MISMATCH | Parser or normalization changed | Version and review migration |
| EXTRACTION_MISMATCH | Normalized text digest differs | Reject representation |
| QUOTE_NOT_FOUND | Exact text plus context absent | Return unresolved citation |
| AMBIGUOUS | Multiple context matches | Reject under single-span policy |
| POSITION_MISMATCH | Position disagrees with quote | Reject; inspect coordinate adapter |
| RESOLVED | Identity, quote and span agree | Evaluate semantic support separately |
The executed test returned RESOLVED for the unchanged anchor. Nine deliberately changed conditions were rejected: revoked access, changed source bytes, changed parser, changed normalization profile, changed extracted text, removed disambiguating context, altered quote, byte-based offsets and UTF-16-based offsets. These are deterministic mutation checks, not a measured production citation accuracy rate. The access check is an injected Boolean; it does not test an identity provider or real policy engine.
The repeated-heading mutation removes prefix and suffix while retaining the original position. Our policy still returns AMBIGUOUS: a position does not override a quote selector that lacks the required disambiguation. A different application may choose another documented policy, but it must not accidentally switch behavior when offsets stop matching.
Bridge the backend and viewer explicitly
Have the resolver return a representation identifier, coordinate-system label and verified span, not just two unnamed integers. A browser adapter must convert code-point positions into its actual text-node boundaries and UTF-16 offsets. Validate the exact substring again after constructing the range. A code-point offset over one concatenated string is not automatically a DOM Range offset within one node.
The diagram separates these layers; the local fixture does not execute a DOM Range adapter, PDF overlay, OCR model or bidirectional-layout test. Before production release, add tests for multiple text nodes, combining sequences, emoji sequences, non-breaking spaces, ligatures, repeated headers and parser-specific reading order. Test selection boundaries as well as string equality. A correct backend anchor can still become a wrong visual highlight if the viewer uses a different extraction.
Do not anchor a translated paraphrase as though it were an exact quote from an English source. Store the source-language span and identify any translation as a derived presentation. The cross-language retrieval evaluation tests whether German questions retrieve the applicable English evidence; anchor verification tests the selected literal span after that decision. Neither replaces semantic support review.
Operational costs and security boundaries
Hashing entire documents on every click can be unnecessary work. At ingestion, compute immutable content-addressed manifests and protect their provenance; at resolution, verify that the object and manifest are the expected revision. Cache only within the correct tenant and authorization scope. A cache hit must not bypass a current access decision. Measure the real storage, lookup and extraction cost before choosing a cache duration; this fixture supplies no latency benchmark.
Exact quotes and context snippets may contain sensitive information. Apply source-equivalent access controls and an explicit retention policy to anchors, logs and diagnostic artifacts. Prefer reason codes and opaque identifiers in operational telemetry instead of logging full rejected quotes. A denial response should not reveal the protected document's contents or whether a secret phrase exists.
Strict rejection reduces silent mis-highlighting but increases unresolved citations after legitimate parser upgrades. Budget for versioned representations, controlled migration and user-facing unavailable states. A fail-open fuzzy search looks convenient but can move a quote to a different warning or condition. Offer a clearly labeled document-level fallback only where access and product policy permit it, never a fabricated verified highlight.
Acceptance checklist and next step
Ingestion: preserve original bytes, extraction configuration and normalized text; pin the Unicode database; record source and text digests in a protected manifest. Keep applicability and extraction provenance separate.
Resolution: recheck access; enforce revision/profile identity; compare quote, context and half-open positions; return explicit unresolved states. Treat an empty or invalid span as a failure, not a request for the first match.
Viewer: label coordinate units; convert and test the rendered range; preserve the link to original-page evidence; test mobile and desktop interaction on your actual viewer. Backend fixture success is not viewer acceptance.
Release: retain the executed mutation fixture and runtime versions; add representative documents and authorized migration tests; evaluate answer support independently. The script and result manifest for this article are retained as publication evidence, not presented as a deployed product capability.
If your RAG answers link to convincing documents but users cannot reproduce the cited passage, start with one manual family and a small set of disputed highlights. I can help your engineering team define the representation contract, build the migration tests and separate locator failures from retrieval and answer-quality failures. The useful deliverable is a resolver that explains why it refused a highlight—not a more confident-looking citation badge.
