A model change is not reproducible merely because a deployment succeeded. If you cannot reconstruct the exact base-model revision, adapter, tokenizer, prompt, data snapshot, evaluator and approval that produced a live response, you have no dependable rollback boundary. Treat those inputs as one immutable LLM release bundle, promote the bundle—not a mutable endpoint—and rehearse the rollback.
The operational problem: configuration is part of the model
Teams often record a model name and a Git commit, then discover during an incident that a hosted alias moved, a tokenizer changed, a retrieval corpus was refreshed, or an evaluator threshold was edited after the result. The same prompt can therefore have a different operational meaning. OpenAI explicitly notes that LLM output is non-deterministic and behaviour changes between snapshots and model families; it recommends continuously measuring application performance with evals.
This complements operating self-hosted inference against SLOs rather than GPU utilisation and semantic data-contract evidence for industrial AI: performance and data lineage are not enough when the deployed model configuration itself is ambiguous.
Hugging Face makes a related reproducibility point in its model-card guidance: metadata can state the base model, relation such as adapter or fine-tune, datasets and structured evaluation results. A model card is useful documentation, but it is not a deployable evidence object. The release bundle closes that gap by pinning the runtime inputs and the decision that promoted them.
Architecture: make the bundle the unit of promotion
Start with a versioned API-contract boundary. A release controller resolves immutable artifact references, runs the frozen evaluation suite, applies policy thresholds, records an accountable approval, and exposes one release ID to inference, tracing and incident tooling. The inference gateway accepts only approved release IDs; it must not silently resolve “latest”.
Diagram: source repository + prompt registry + dataset snapshot + evaluator registry → manifest builder → signed release bundle → evaluation gate → approval record → deployment registry → inference gateway → trace.release_id → rollback target.
The manifest should be content-addressed and stored in a write-once or append-only evidence store. A signature proves which service account assembled it; it does not prove quality. Quality comes from the evaluation evidence and the accountable promotion decision.
Minimum release-manifest evidence
The following fields are the practical minimum. Add domain-specific dependencies—for example retrieval-index snapshot, tool schema version or policy pack—when they can change a result.
| Evidence item | Why it must be pinned | Failure if it is not |
|---|---|---|
| Base model provider, model revision and region | Provider aliases and regional availability can change | A rollback points to a different model |
| Adapter/quantisation artifact SHA-256 | Weights or runtime variants alter behaviour and latency | You cannot recreate the candidate |
| Tokenizer and chat template revision | Token boundaries and formatting affect context and tool calls | Token counts, truncation and outputs drift |
| Prompt/template commit and tool-schema version | Prompt text and schemas are executable configuration | A valid-looking response breaks a downstream action |
| Dataset or evaluation-suite snapshot digest | A moving benchmark makes a passing score meaningless | A regression is hidden by changed test data |
| Evaluator version, grader configuration and thresholds | Measurement logic is a release dependency | Promotion criteria cannot be audited |
| Approval identity, timestamp, change reason and rollback release ID | A release is a decision, not only an artifact | Incident response has no accountable path |
Promotion flow with gates
1. Build the candidate manifest from immutable references; reject floating tags, unversioned prompts and datasets without a digest.
2. Run deterministic checks first: JSON/tool-schema compatibility, dependency resolution, secrets-free configuration and manifest completeness.
3. Run the frozen evaluation suite by segment. Keep safety, structured-output, German-language, long-context and business-critical cohorts separate; an aggregate average can hide a harmful regression.
4. Enforce explicit thresholds and a tolerance rule for cost and latency. A candidate that improves average quality but violates a critical segment floor does not promote.
5. Store evaluation outputs, reviewer notes and the signed decision against the bundle. Deploy a canary only by release ID; trace that ID on every request, tool call and output.
6. Run a rollback drill before calling the process reliable: move a small canary back to the prior approved bundle, confirm routing and cache invalidation, then compare telemetry and audit records.
Decision table: how strict should the gate be?
| Situation | Recommended gate | Trade-off | Owner |
|---|---|---|---|
| Internal summarisation with no automated action | Frozen smoke suite, schema check, approval for material changes | Faster iteration; limited incident evidence | Product owner + platform |
| Customer-facing assistant | Segment evals, canary, trace ID, rollback drill | More test maintenance and release lead time | Product + AI platform |
| Regulated or action-capable workflow | Immutable bundle, independent approval, strict floors, change ticket and rollback proof | Highest operating cost; fewer emergency changes | Accountable business owner + risk/security |
Failure modes that the bundle exposes
Floating model aliases: “latest” passes staging and changes overnight. Mitigation: resolve a provider revision into the manifest and block aliases in production configuration.
Evaluation drift: a team updates the test set to match a new workflow while comparing it to last month’s score. Mitigation: retain frozen suites and make a new suite a separately approved version.
Partial rollback: weights roll back but the prompt, retrieval index or tool schema remains new. Mitigation: rollback the complete bundle and test the restore path, not only the model server.
Missing trace correlation: an incident sample cannot be joined to its release. Mitigation: inject release_id into request logs, evaluation records, tool traces and user-feedback events.
What this does not solve
A release bundle does not establish that a model is safe, legally compliant or suitable for every task. It does not remove non-determinism, data-quality problems, provider outages or the need for human escalation. It also adds governance and storage overhead; for a low-risk prototype, the full process may be disproportionate. The sensible boundary is to start with a minimal manifest and increase gate strength as impact and autonomy increase.
Practical starting point
Run a 90-minute release-boundary review: map every runtime input that can alter an answer, select a stable release ID format, freeze one evaluation suite, define two non-negotiable segment thresholds, and perform one rollback drill. Teams that already operate SLOs for inference can attach release IDs to latency and error traces, turning a vague “model change” into an investigable production event.
Sources: NIST AI 600-1, Generative AI Profile; Hugging Face Model Cards documentation; OpenAI model optimisation guidance.
