Annex IV Evidence Reconciliation: Find Stale Test Reports Before Release

A complete documentation folder can still describe the wrong AI release. Before promotion, reconcile the assessed purpose, system revision, test suite and monitoring plan against immutable evidence. Automate missing-reference and version checks; leave evidence adequacy and legal sufficiency to accountable reviewers. A green checksum is not a conformity assessment.

This is a proposed engineering control for technical-documentation workflows, not legal advice, an official Annex IV template or a compliance certificate. Qualified counsel must determine system classification, actor responsibilities, applicable provisions and dates, simplified-documentation options and interactions with product legislation before a real coverage map is adopted.

Why file presence is the wrong release test

Consider a hypothetical inspection assistant moving from release r16 to r17. The folder contains a purpose statement, a signed test report and a monitoring plan. Every link opens. Yet the report evaluates r16 with eval-7, while the candidate requires eval-8; the monitoring plan still covers purpose-2 rather than purpose-3. File counting says complete. Evidence reconciliation says the package does not support this candidate.

This differs from building the logging and technical-documentation architecture. Logging records operation. Reconciliation checks relationships between independently maintained evidence objects. It also differs from the LLM release bundle: that bundle pins runtime dependencies for reproducibility and rollback; the documentation package explains the assessed system, controls, limitations and review basis. They should share identity, not be treated as interchangeable objects.

Ground the coverage map in reviewed scope

Article 11 in the EUR-Lex consolidation states that technical documentation for a high-risk AI system is drawn up before placing it on the market or putting it into service and kept up to date. The engineering implication is a version relationship, not a requirement to use the particular JSON structure proposed here.

The Commission's Annex IV reference connects intended purpose and system version, relevant software versions, validation and testing, lifecycle changes and post-market monitoring. Point 2(g) includes test logs and dated, signed test reports. A digest in your manifest does not replace a responsible person's signature or establish that the test was adequate.

Source review on 8 October 2026 checked the consolidation and the amending Official Journal instrument. The Commission's Article 11 explainer explicitly warns that its displayed text has not been updated for amendments. Use that page for orientation, not as the current legal rulebook. Consolidations are documentation aids; authentic Official Journal acts and qualified review govern legal interpretation. Do not infer that every cited provision already applies to every deployment.

Have legal/compliance reviewers maintain a versioned scope inventory. Each potentially relevant element receives an applicability decision, rationale and accountable review reference. Simplified forms and combined product documentation are legal-review questions, not excuses to silently delete rows. A general-purpose model card can support a system dossier but is not itself the dossier for a high-risk AI system.

A machine-readable map that cannot hide missing rows

Use an independently controlled scope inventory and an evidence manifest. Otherwise an author can make the checker pass by deleting the inconvenient row. The scope inventory defines expected element IDs; the manifest supplies their evidence. Version both and retain their digests with the check result.

The following JSON is a deliberately incomplete synthetic teaching map, not a legal coverage determination. Short field names remain stable across translations. The SHA-256 marker must be replaced with a computed digest; a real schema must reject the marker. The example lists only one applicable element and omits most of Annex IV.

{"system_id":"inspection-assist","release_id":"r17","purpose_revision":"purpose-3","policy_ref":"reviewed-scope-1","rows":[{"element":"IV-2g","applicability":"applicable","scope_rationale":"Tests for the assessed inspection workflow","owner":"evaluation-owner","system_id":"inspection-assist","release_id":"r17","purpose_revision":"purpose-3","evidence_type":"test_report","artifact_ref":"evidence/report-r17.json","sha256":"REPLACE_WITH_COMPUTED_SHA256","review_status":"pending","review_ref":"review-42","missing_reason":""}]}

For an excluded element, use applicability=not_applicable, an explicit na_reason, owner and reviewed scope decision. A missing field is unknown, not not-applicable. For unavailable evidence, retain the row with a missing_reason and assigned remediation owner; recording the reason must not turn a missing report into a passing one.

In production, allow multiple evidence references per element and one artifact to support several elements. The teaching fixture uses one reference per row to expose the checks clearly. Real references need immutable object versions, access constraints, evidence type, creation and review provenance, and an explicit connection to the assessed configuration. Do not require an unchanged architecture document to be rewritten for every build: a reviewed carry-forward binding can establish its applicability to a new release without altering the original artifact.

Reconcile identity, bytes and meaning in separate passes

First, validate the manifest and policy schemas before following references. Reject duplicate element IDs, unknown applicability values, malformed digests and unreviewed scope revisions. Compare expected element IDs with actual rows. A syntactically valid empty array must not pass a non-empty reviewed scope.

Second, resolve evidence through an allowlisted repository adapter, not arbitrary URLs supplied in a manifest. Missing object, denied access and backend timeout are separate findings; all remain unresolved for the release decision. Record which authorised service and reviewer could access the evidence. Avoid creating an SSRF path or exporting confidential reports into an unrestricted build log.

Third, hash the exact bytes retrieved from the pinned object version. Compare that digest with the reviewed manifest. Do not regenerate the expected digest automatically after a mismatch: that would bless changed evidence. Parse the report or its controlled sidecar and compare system_id, release_id and purpose_revision against the candidate and row binding. A matching filename is not a match.

Fourth, compare the evaluation-suite revision with the suite approved for this candidate, including dataset and evaluator revisions in a real implementation. A newer suite is not automatically mandatory for every historical release. The failure is disagreement with the reviewed candidate policy. Retain the superseded report as history; do not overwrite it to make the folder look current.

Fifth, bind the monitoring plan to the same assessed purpose and relevant operational configuration. The checker can detect a revision mismatch; it cannot decide whether the plan covers the right populations, risks, metrics or escalation duties. That requires technical and domain review. If purpose or authority has changed, route facts into the separate substantial-modification review handoff, not an automatic legal classification.

Finally, emit STRUCTURALLY_RECONCILED only when structural findings are empty. Always retain technical_adequacy=NOT_ASSESSED and legal_sufficiency=NOT_ASSESSED for this checker. The result is input to an authorised release decision, not permission to deploy.

Decision table: act on the evidence state

A hold here means holding candidate promotion under the proposed internal policy. It is not an instruction to stop a running industrial system. Runtime safety, incident response and any legal suspension duties remain separate approved processes.

What the synthetic validator actually caught

A local Python fixture was executed for this article using synthetic in-memory JSON artifacts and SHA-256 checks. It returned the expected findings in 12 asserted cases: a clean baseline plus dangling evidence, changed bytes, wrong release, superseded suite, detached monitoring purpose, deleted row, unexplained exclusion, pending review, duplicate row, unresolved scope and wrong system. These are implementation tests, not customer results or evidence that the legal mapping is complete.

The baseline returned {"state":"STRUCTURALLY_RECONCILED","issues":[],"legal_sufficiency":"NOT_ASSESSED","technical_adequacy":"NOT_ASSESSED"}. Removing the test report returned DANGLING_REFERENCE and MISSING_REASON. Adding a single whitespace byte to its still-valid JSON returned BYTES_CHANGED. Changing the monitoring plan's purpose while recomputing its digest still returned PURPOSE_REVISION_MISMATCH. Integrity and contextual consistency are independent checks.

To reproduce the essential algorithm, initialise findings and a seen-elements set; visit every row, check its owner, scope rationale, evidence type and review reference, and reject duplicate or unreviewed elements. Handle a reviewed exclusion separately. For applicable rows, retrieve bytes, compare SHA-256, parse the controlled metadata, compare the three identity fields and the required suite. After the loop, add COVERAGE_MISSING for every expected element absent from the seen set. Return findings together with both NOT_ASSESSED fields, even on success.

The fixture deliberately trusts its local policy and review strings. A production service must authenticate those records, verify reviewer authority and signature provenance, handle malformed inputs, and check the entire evidence graph. A string saying reviewed is not proof of review. The exercised fixture also does not implement network access control, retention, revocation or concurrency protection.

Close the gap between check time and promotion

Freeze a release-check tuple containing candidate digest, scope-policy digest, documentation-manifest digest and evidence object versions. Bind human review and the final promotion transaction to that tuple. Compare-and-swap the current release reference; if any bound input changes, invalidate the result and rerun the affected checks. Otherwise a correct check can authorise a different package minutes later.

Carry-forward evidence needs its own accountable statement linking the old artifact to the new candidate and explaining why it remains valid. Do not implement that as a global ignore-release-mismatch flag. Separate storage integrity from trust: hashes detect changes against a trusted expectation, but an attacker who can rewrite both report and manifest can manufacture consistency. Protect the manifest, scope inventory and review ledger with separation of duties and appropriate signing/access controls.

Costs, limitations and the legal-review checklist

Hashing and fetching large artifacts adds I/O and release latency. Pin immutable versions and reuse verified content-addressed objects, while separately rechecking current reviewer access and authorisation. A cached digest does not prove that a reviewer can still inspect the evidence. Keep the control at release and relevant change boundaries rather than adding document retrieval to each inference request.

Strict identity checks also create false holds when unchanged evidence remains valid. Reviewed carry-forward decisions reduce needless retesting, but cost reviewer time and must not become blanket exceptions. Track unresolved-evidence age, recurring missing owners, retrieval failures and reopened reviews. Do not report a percentage of filled rows as a compliance score.

Before adoption, confirm with qualified reviewers: the applicable system and actor scope; the current legal version and application provisions; expected Annex IV elements and justified exclusions; any simplified or combined documentation route; required report signatures and review authority; and retention, confidentiality and evidence-access duties. Then have technical reviewers judge whether the tests and monitoring actually cover the intended purpose. Structural consistency cannot answer those questions.

Start with one candidate, not a new governance platform

Choose one pending release, its purpose statement, one test report and its monitoring plan. Freeze their identities, agree the reviewed scope inventory and run the failure injections above before making the check a promotion gate. The useful first deliverable is a reviewer who can explain every mismatch—not another green dashboard.

I help platform and compliance teams turn these evidence relationships into testable release controls. Bring the candidate manifest, documentation index and one stale report to a technical evidence review; qualified counsel remains responsible for the legal assessment. The result should be a smaller, explicit remediation queue and a release decision bound to the evidence actually reviewed.