A production RAG system should answer only when the authorised evidence is current, applicable and sufficient for the requested decision. Retrieval similarity is not that decision's probability of correctness. Separate the answerability gate from a post-generation claim-support check, and evaluate both against the coverage they remove and the escalation work they create. A system that refuses everything can avoid unsupported answers while failing its business purpose.
This article proposes an implementation and a small synthetic policy fixture, not a customer deployment result. The buyer decision is where to place the runtime answer, clarify and escalate boundary—not which chunker wins or whether a release's average evaluation score looks acceptable.
Why the closest passage can still be the wrong evidence
Consider a constructed maintenance-document question: which inspection interval applies to assembly B under the currently approved revision? A near-identical passage for assembly A, an obsolete revision for B, and a current paragraph naming B without its interval can all look relevant. None establishes the requested answer. Do not turn the language model's background knowledge into an unlabelled substitute for missing controlled documentation.
The Sentence Transformers semantic-search documentation describes embedding queries and corpus entries and retrieving high-similarity neighbours; its semantic_search utility defaults to cosine similarity. That is a ranking operation. It does not establish that a particular score is the probability that every required claim can be answered from authorised, applicable sources.
A reranker may improve relevance without solving this distinction. Record its score and model revision as features, not as a universal confidence percentage. Changing the embedding model, language mix or candidate pool can change score distributions. Even an apparently precise threshold needs evaluation against an explicitly defined outcome.
Define answerability before fitting a score
Use a case record containing intent ID, question, permitted scope, asset or product identity, requested effective time, source IDs and revisions, required evidence units, acceptable alternatives, label, reviewer and reason. Label a case answerable only if the evidence available to that user supports the requested scope. For multi-clause answers, distinguish interchangeable passages from requirements that must all be present.
Keep two labels: corpus answerability, established by reviewers over the authorised snapshot; and context sufficiency, assessed over what actually survives retrieval, reranking and prompt construction. A missing mandatory warning in the final context is a pipeline failure even when the corpus contains it. Do not relabel that question unanswerable to make the retrieval system look better.
Create separate missing-evidence, contradictory, stale, wrong-revision and ambiguous-intent cases. A domain owner must define document authority and applicability; a newer timestamp alone need not supersede an approved instruction. Unresolved conflicts should remain visible rather than being averaged into a score. Use the chunking evaluation framework to repair evidence loss upstream instead of using refusal as a permanent ingestion workaround.
Authorisation is a separate hard boundary. Filter before retrieval and prompt construction; a confidence model cannot override denial. Neither the user message nor an unrestricted log should reveal that a protected document exists. An authorised support route may handle the denial, but it must not receive protected excerpts by default.
Route the request before generating an answer
The following is a proposed policy, not a vendor-prescribed threshold. Apply hard boundaries before probabilistic acceptance. An answerability estimate can govern uncertain evidence sufficiency only after access and applicability checks have passed.
| Observed condition | Route | Required control |
|---|---|---|
| Scope is denied | Deny outside answer policy | Generic response; no protected source disclosure |
| Intent lacks a necessary identifier | Clarify | Ask for the identifier without suggesting secret values |
| Required evidence is missing | Escalate or explain limitation | Name the missing evidence type, not an invented answer |
| Revision is stale or sources conflict | Escalate | Source owner resolves applicability and authority |
| Evidence passes the sufficiency gate | Generate candidate answer | Use only approved evidence and preserve citations |
| Generated claims fail support review | Withhold and escalate | Do not expose the rejected candidate as an answer |
Clarification is appropriate when the user can resolve ambiguity. Asking users to rephrase cannot repair a missing manual revision. An escalation needs an owner, response target and bounded queue; otherwise “human review” is an unimplemented dependency. If the queue is unavailable, return an honest limitation with a permitted next step, not a silent fallback to always-answer.
The diagram separates evidence admission from answer release. A candidate may have all necessary evidence and still omit a qualifier, transpose a value or introduce an unsupported claim. The final support check must compare claims and cited spans, including applicability and units. Automated judges can assist, but they also miss errors. High-impact procedures need the relevant human and operational approval; this gate is not permission to actuate equipment.
Calibrate the target, not the cosine score's name
Start with a transparent rule baseline and a similarity-only baseline. If a learned classifier is justified, its target should be reviewer-labelled context sufficiency for the defined scope. Candidate features might include required-unit coverage, revision checks, contradictory-evidence flags and ranking signals. These are design choices to validate, not features proven sufficient by this article.
Separate model training, probability calibration, policy-threshold selection and untouched acceptance testing. Group paraphrases and shared document families to avoid leakage; use a temporal holdout where future changes are the operational challenge. Freeze index snapshot, permissions fixture, retriever, reranker, prompt, generator and judge versions. Do not tune the threshold after reading acceptance failures and then call the same set untouched.
The scikit-learn calibration documentation explains that calibrated probabilities should align with observed positive frequencies and warns against fitting calibration on the classifier's training predictions. Inspect reliability diagrams and bin counts. A lower Brier score alone does not establish better calibration because discrimination also contributes. Calibration requires representative data; it does not certify an individual answer or survive arbitrary distribution shift.
On the threshold-selection set, compare candidate policies by selective error, coverage and escalation capacity. Choose the admissible policy with the useful coverage your service needs, subject to predeclared risk and workload constraints. Do not prescribe one numerical threshold for every tenant, language and document class. If no policy satisfies the constraints, repair evidence or narrow scope rather than quietly raising the tolerated error.
A synthetic fixture that exposes the trade-off
The following eight access-eligible cases and scores are deliberately invented test inputs. They are not measured retrieval or model outputs. The answerability labels represent reviewed corpus truth; the routes and support flags are scripted to exercise bookkeeping, including a false refusal and a generation error. A Python fixture was executed to verify the counts below; no retriever, classifier, LLM or production workload was run.
| Case | Label and constructed condition | Similarity input | Scripted initial route |
|---|---|---|---|
| A1 | Answerable; current complete evidence | 0.93 | Answer; supported candidate |
| A2 | Answerable; cross-language wording | 0.63 | Answer; supported candidate |
| A3 | Answerable; gate misses sufficient evidence | 0.57 | Escalate; false refusal |
| A4 | Answerable; generator adds unsupported claim | 0.88 | Answer; support check then blocks |
| U1 | Not answerable; required clause missing | 0.92 | Escalate |
| U2 | Not answerable; only stale revision | 0.95 | Escalate |
| U3 | Not answerable; unresolved contradiction | 0.91 | Escalate |
| U4 | Not answerable as asked; identifier ambiguous | 0.87 | Clarify |
For this arithmetic exercise, forced answers for U1–U4 are stipulated unsupported. A3 would have a supported candidate if answered. The similarity baseline answers at or above 0.85; that value is an arbitrary fixture input, not a recommendation. All policies see the same eight eligible cases. Denial tests belong in a separate security suite and in the total operational workload ledger, not in this answerability denominator.
| Policy | Answer coverage | Unsupported among answers | Unnecessary non-answer | Escalate / clarify |
|---|---|---|---|---|
| Always answer | 8 / 8 | 5 / 8 | 0 / 4 | 0 / 8 and 0 / 8 |
| Similarity baseline | 6 / 8 | 5 / 6 | 2 / 4 | 2 / 8 and 0 / 8 |
| Bounded, before support check | 3 / 8 | 1 / 3 | 1 / 4 | 4 / 8 and 1 / 8 |
| Bounded, after support check | 2 / 8 | 0 / 2 | 2 / 4 | 5 / 8 and 1 / 8 |
Answer coverage is released answers divided by eligible requests; the pre-support row instead counts candidates. Selective error is answers with at least one unsupported claim divided by answers, not claims divided by claims. Unnecessary non-answer uses the four corpus-answerable cases as denominator, including A4: withholding its bad candidate protects the user but still loses an answerable opportunity. Escalation and clarification each use all eight eligible requests. With no answers, selective error is undefined—not zero.
The final row is not evidence of a successful production policy. Its support checker is scripted, and two released answers cannot estimate operational risk. The example shows why zero observed error must be reported alongside coverage, false refusal and reviewer workload. In a real worksheet, retain numerator, denominator, segment, run manifest and uncertainty interval; analyse by independent intent or document family rather than treating paraphrases as independent evidence.
Stress-test what the aggregate hides
Add irrelevant but semantically close context, a conflicting revision and a required clause pushed beyond the context budget. Keep each perturbation paired with its original case. Track whether the failure occurs at retrieval, admission, generation or support review. The Ragas noise-sensitivity metric evaluates incorrect response claims relative to reference evidence and relevant or irrelevant context. It can help diagnose noise sensitivity; it is not an answerability label or a calibrated runtime refusal policy.
Report German and English queries, cross-language requests, document families and high-impact procedures separately. The cross-language retrieval guide explains why fluent German output does not prove that the English evidence survived. Watch for a policy that improves blended selective error by disproportionately refusing a difficult language segment.
Measure gate latency, support-check latency, full-response latency, evaluator timeouts and review backlog. Log structured reason codes and evidence identifiers under the same access and retention controls as the source material. A timeout must not become approval. One bounded retry may be appropriate for a transient checker failure; repeated generation until an answer passes changes both the evaluated policy and cost distribution.
Release checklist and next implementation step
Before release, verify that source applicability rules have named owners; the corpus and final-context labels are distinct; access denial never reveals protected content; every policy route has an executable fallback; and threshold selection is isolated from acceptance data. Retain numerator-level results for coverage, unsupported answers, unnecessary refusal and escalation by segment.
Then connect these fixtures to the RAG evaluation CI gate. Canary the runtime policy and retain rollback of the complete bundle. Reassess after corpus, permission, model, language-distribution or scoring changes. More conservative thresholds can reduce answer coverage and overload specialists; more elaborate judges add latency, cost and another failure dependency. Neither trade-off disappears behind a confidence label.
Bring one bounded question family, the applicable document revisions, your permission model and the service owner's error and escalation constraints. I can help your team build the labelled evidence fixture and a reproducible answerability-policy comparison. The useful deliverable is an auditable answer boundary with explicit failure handling—not a reassuring score beside every response.
