BLUF — a canary with no eligible traffic has not passed. Keep missing evidence, measurement errors and demonstrated regressions separate, and allow promotion only after a release-specific, workload-matched observation passes an explicit policy. In Argo Rollouts, define both success and failure conditions: otherwise a false success condition can become a failure rather than the intended inconclusive result.
The practical decision is whether to increase traffic to a new AI service, not whether an empty dashboard looks quiet. The implementation below combines tested Prometheus recording-rule fixtures with the actual Argo Prometheus provider and expression evaluator. It is a local synthetic test, not a production rollout, Kubernetes integration test or claim about model accuracy.
Where this gate belongs
An immutable LLM release bundle identifies the candidate and rollback target. Segmented offline release gates check answer quality before exposure. This article adds a different control: what the live rollout controller is allowed to conclude when its operational evidence is absent, invalid or too small.
The Argo analysis documentation distinguishes successful, failed and inconclusive analyses. An inline analysis blocks the rollout while the analysis runs; an inconclusive result can pause progression for intervention. Prefer an explicit inline promotion boundary for this design. A background analysis that is simply stopped when the rollout finishes is not equivalent to a mandatory evidence gate.
Start with one service and one governed request class. Split short interactive requests from long-context work, streaming from non-streaming requests, and materially different routes. Apply the gate to every required class. Canary and control must use different immutable release selectors but the same eligibility definition, observation window and outcome taxonomy. A missing protected class must remain missing; another busy class cannot supply its denominator.
Separate evidence production from release authority
The request boundary emits bounded service, release, class and terminal-outcome labels. Prometheus derives denominators and health comparisons. An eligibility filter removes results that lack the required volume, age or freshness. The resulting gate series contains 1 for a policy pass, 0 for a policy breach, or no sample for insufficient evidence. Argo interprets that contract; the promotion owner retains the explanation and any override decision.
Do not give this evaluator credentials to execute business actions. It needs metric-read access and a tightly scoped release-control integration, not authority to issue purchase orders or change plant controls. Rolling back traffic restores routing to a prior release. It does not reverse an email, payment or physical action already performed by a canary request.
Define the measurement contract before choosing thresholds
For the teaching fixture, ai_requests_total is a terminal-request counter with pre-initialized ok and error outcomes. Its selectors are service="inference", class="short" and release="candidate-a" or "baseline-a". Count errors as outcomes, not as missing requests. Define how client cancellations, timeouts and retries map to the denominator before adopting this metric in a real gateway. A terminal counter alone cannot reveal a growing population of unfinished requests; add admission, outstanding-age and timeout controls.
Let N(r) be sum(increase(ai_requests_total{service="inference",class="short",release="r"}[5m])). Let E(r) be the same increase restricted to outcome="error", divided by N(r). These expressions describe template substitution for the release label, not a PromQL function definition. Pre-initializing both outcomes prevents an absent error series from being confused with a measured zero. Do not add a blanket default-zero fallback to hide a missing exporter.
The example admits a comparison only when both N values reach 100, the newest samples of all currently selected request-counter series are less than 30 seconds old, and both observation-start gauges indicate at least 300 seconds of observation. Those are deliberately explicit operational policies, not statistical confidence guarantees. Prometheus increase extrapolates counter changes; its output is an estimate that can be fractional, not an exact event ledger.
The custom ai_observation_start_seconds gauge records when observation for that release and class began. It must be owned by instrumentation, reset when continuity is lost, and must not be an arbitrary deployment timestamp presented as proof of continuous telemetry. The fixture supplies this gauge synthetically. In production, reconcile expected instances, scrape health and missing intervals separately: checking freshness of existing series does not detect an instance whose series disappeared entirely.
A tested PromQL contract that does not default missing data to healthy
Construct the health value H as the product of three Boolean comparisons: (E(candidate) <= bool 0.02), (E(baseline) <= bool 0.02), and (E(candidate) - E(baseline) <= bool 0.005). This yields 1 only if both absolute error budgets and the relative regression budget pass. A degraded baseline cannot make an equally degraded candidate look acceptable. The two-percent absolute ceiling and half-percentage-point regression allowance are teaching values; choose real budgets from your service contract.
Construct eligibility G with filtering comparisons, without bool: N(candidate) >= 100 and N(baseline) >= 100, plus freshness and observation-age predicates for both releases. Then record H and G as ai:canary_gate, with explicit release="candidate-a" and class="short" labels. In PromQL, and is set intersection: it keeps H only when G has a matching element. It does not perform Boolean arithmetic on the numeric value of G. The synthetic expressions aggregate to one unlabeled element before assigning recording-rule labels, so their matching is intentional.
For each release, the freshness predicate is time() - min(timestamp(ai_requests_total{service="inference",class="short",release="candidate-a"})) < 30, with the baseline selector substituted for its check. Observation age is time() - max(ai_observation_start_seconds{service="inference",class="short",release="candidate-a"}) >= 300. Do not sum different releases before calculating their denominators or compare different class mixtures after aggregation.
Generate the rule for the exact candidate, baseline and class from the release manifest; do not overwrite one global rule that concurrent rollouts share. Evaluate it every 15 seconds in this example. Argo queries the selected ai:canary_gate series with an additional time() - timestamp(ai:canary_gate{same selectors}) < 30 filter. That second guard addresses a recording-rule result that stopped updating even though its underlying counter checks once passed.
Make inconclusive an explicit Argo outcome
The executed AnalysisTemplate uses apiVersion argoproj.io/v1alpha1, kind AnalysisTemplate, and one metric named eligible-health. Its arguments are prometheus-address, canary-release and workload-class. The instant query selects the gate by those release/class arguments and applies the recording-result freshness filter. Do not copy the fixture's synthetic host into a deployment; inject a verified Prometheus endpoint and least-privilege authentication.
| Template field | Tested value | Why it is explicit |
|---|---|---|
| count | 1 | One inline decision after the observation window |
| failureLimit | 0 | A measured policy breach cannot be ignored |
| inconclusiveLimit | 0 | Insufficient evidence requires intervention |
| consecutiveErrorLimit | 0 | Provider failure is not a healthy measurement |
| provider.prometheus.timeout | 10 | Bound the query attempt in seconds |
| successCondition | len(result) == 1 && !isNaN(result[0]) && !isInf(result[0]) && result[0] == 1 | Require exactly one finite passing value |
| failureCondition | len(result) == 1 && !isNaN(result[0]) && !isInf(result[0]) && result[0] == 0 | Fail only on the declared breach value |
The short-circuit length guard prevents indexing an empty vector. Both conditions are false for empty, non-finite or unexpected-cardinality results, which the pinned evaluator classifies as Inconclusive. With only successCondition, the evaluator instead treats a false success result as failure. That distinction is documented behavior, not a cosmetic label: the operator should repair missing evidence rather than claim a measured regression.
Put the inline analysis after a defined exposure and observation step; do not use a zero count for an inline analysis. If traffic remains below the floor, keep the rollout paused, extend observation within a predeclared maximum, or abort exposure and investigate. Repeatedly restarting until a lucky pass appears is not an evidence policy. The controller-limit fields were parsed into the pinned Argo type locally; their full rollout lifecycle still requires a staging-cluster rehearsal.
What the local tests actually returned
The fixture used Go 1.24.6, Argo Rollouts v1.8.3 and promtool 3.5.0. These are reproduction pins, not a claim that these are the newest or preferred production releases. The version-pinned Argo provider and condition evaluator were executed, not reimplemented in a lookalike policy script.
Promtool checked the recording rule and passed ten synthetic time-series cases: healthy, low traffic, no traffic, canary breach, baseline degradation, insufficient observation age, wrong class, missing counter, stale counter and a workload shift with abundant long-class traffic but insufficient short-class requests. The Argo provider then consumed controlled loopback HTTP responses in fourteen cases, exercising instant and range results as well as an HTTP 503. These are two complementary test layers; no real Prometheus server, Kubernetes controller, mesh or language model was involved.
| Observed fixture | Query or provider result | Release interpretation |
|---|---|---|
| Eligible healthy comparison | Gate 1; Successful | Eligible for this operational gate only |
| Eligible canary breach | Gate 0; Failed | Stop progression and investigate or abort |
| Baseline degraded | Gate 0 | Absolute floor blocks misleading relative success |
| Low or absent traffic; wrong class | Empty query result | Inconclusive, not healthy |
| Missing or stale counter; short observation | Empty query result | Restore evidence before deciding |
| NaN; either infinity; two instant samples | Inconclusive | Reject invalid or ambiguous measurement |
| Metrics backend HTTP 503 | Error | Repair provider access; do not invent a result |
| Six range values with a later zero | Failed | Checking only the first value would miss the breach |
| Five range values; NaN; empty range | Inconclusive | Coverage or validity is insufficient |
Retain the template, rules, input-series fixture, go.mod/go.sum, tool versions and observed output together. Reproduce the query layer with promtool check rules rules.yaml and promtool test rules rules-test.yaml. Reproduce the provider layer with go run . against the retained fixture. The provider test reads the actual template, then asserts expected phases; HTTP bodies are explicitly synthetic test inputs, not captured production telemetry.
Range queries need more than result[0]
The Argo Prometheus documentation recommends evaluating all returned values for a range query. The local range test requires six finite values in the declared 0/1 domain; all six must be 1 for success, while any 0 fails after the validity check. A later zero was correctly classified as Failed. An empty range cannot pass merely because an all predicate has no counterexample.
A more subtle boundary remains: the pinned provider flattens matrix sample values before evaluating conditions. A six-value length check does not prove one intended series, six correct timestamps or complete class coverage. Validate the raw response's labels, timestamp grid, expected series population and warnings before relying on a range-based production gate. Two partial series could otherwise masquerade as one complete window. The published instant template avoids that range-shape claim; the separate range tests demonstrate value evaluation only.
Add latency without manufacturing a percentile
The inference SLO framework supplies the user-facing latency boundary. For a classic histogram, aggregate bucket rates across replicas while preserving le and the required release/class dimensions, then calculate histogram_quantile. Never average instance p95 values. The Prometheus histogram guidance explains why precomputed summary quantiles cannot be aggregated into a service percentile.
For a fixed latency budget, a fraction of observations within a matching bucket can be easier to audit than an interpolated percentile. Verify that every relevant instance exposes the bucket and that numerator and denominator share the same eligible population. An empty histogram is still missing evidence. A fast response is not necessarily a correct answer, and a low error rate does not validate tool behavior or task quality.
Before allowing this gate to promote
1. Freeze candidate, baseline, request classes, counters, terminal outcomes and the rollback target. Require every protected class rather than a convenient aggregate.
2. Test no traffic, wrong selectors, missing instances, stale series, backend errors, counter resets and changing workload mixtures. Check raw denominators, query warnings and unfinished-request age alongside the final gate value.
3. Set volume, duration, freshness and absolute/relative budgets before exposure. Document maximum waiting time, override authority and the evidence needed to resume an inconclusive run.
4. Rehearse the actual controller, routing layer and installed CRDs in staging. Verify failure, error and inconclusive behavior; prove that the next traffic step cannot bypass the inline boundary.
5. Retain the release manifest, resolved query, time window, selector set, denominator evidence and final decision. Bound label cardinality and evidence retention; do not place raw prompts or user identities in metric labels.
For a canary-gate review, bring one release manifest, the current promotion template and the PromQL behind its green status. The useful outcome is a tested pass, fail and inconclusive contract with an accountable recovery path—not another dashboard that quietly treats missing work as success.
