BLUF: Use tail sampling to retain diagnostic evidence, not to calculate your service-level error rate. An error-or-latency policy deliberately overrepresents unusual requests. Keep independently instrumented, unsampled request-outcome counters for SLOs; route every span of a trace to the same sampler; and test what happens when evidence arrives after the decision, the sampler restarts or backend membership changes. A retained error span does not prove that the complete trace survived.
In a local synthetic test for this article, OpenTelemetry Collector Contrib retained 28 of 100 request traces, including all ten deliberately failed requests. The request ledger therefore showed a 10% failure fraction, while the retained-trace population showed about 35.7%. Neither number is a production benchmark. The disagreement is the intended consequence of selective retention—not a deterioration in the service.
Two measurements, two different contracts
The OpenTelemetry sampling overview distinguishes early head decisions from tail decisions based on accumulated spans. Tail sampling can preserve errors that were not apparent when a request began. It also adds state, compute and maintenance costs. At low volume, storing all approved telemetry may be simpler than operating a stateful sampling tier.
Define reliability at the user-operation boundary: eligible operations, acceptable outcomes and bad outcomes. Count a recovered model retry according to the eventual user outcome, not automatically as another failed user request. Do not construct the SLO denominator from spans exported after the sampler. The existing burn-rate alerting implementation explains the outcome-counter and alert contract; this article addresses the separate diagnostic-retention decision.
A trace-level ERROR status and a bad user outcome are not interchangeable. An internal attempt may fail before a fallback succeeds; a response may be operationally successful but semantically wrong. Our fixture intentionally makes ten ERROR children correspond to ten bad outcomes so the selection effect is visible. A real implementation must establish that mapping separately.
Architecture: affinity before retention
Applications emit approved spans to first-tier routing Collectors. Those Collectors use a trace-ID-aware load-balancing exporter to select a second-tier sampler. Each sampler buffers its own traces, evaluates retention policies and exports selected evidence. Request-outcome metrics follow a separate, non-sampled path from the application to the metrics backend. “Independent” describes the instrumentation and denominator; shared infrastructure still needs failure analysis.
The OpenTelemetry gateway deployment guidance documents this two-tier arrangement. Use traceID rather than service-name routing for this purpose. All routing replicas must agree on backend identities and membership. A generic round-robin load balancer in front of the sampling tier can distribute different batches from one trace to different samplers and destroy the evidence the policy needs.
Affinity holds for a stable routing view, not forever. The version-pinned load-balancing exporter documentation states that changing backend membership reroutes some trace IDs. Consistent hashing reduces movement; it does not migrate buffered spans or decision caches. Treat scale-down, rolling replacement and DNS convergence as trace-integrity events, not only availability events.
A bounded, version-pinned policy
The executable example uses Collector Contrib 0.135.0, an intentionally fixed reproduction version, not a recommendation to deploy an old release or a claim that it is current. Its official Linux release archive was downloaded, checked against the published SHA-256 checksum and executed locally. Revalidate the policy and dependencies on your approved, patched distribution before deployment. The matching tail-sampling documentation defines the settings used below.
The processor fragment is JSON, which can also be used as YAML syntax. Put it under the processors key in the Collector configuration. The three positive policies are OR-like here: an observed ERROR, a sufficiently long trace or the deterministic baseline can retain a trace. No drop, inverted or composite policies are included.
{"tail_sampling": {"decision_wait": "2s", "num_traces": 1000, "expected_new_traces_per_sec": 100, "decision_cache": {"sampled_cache_size": 10000, "non_sampled_cache_size": 10000}, "policies": [{"name": "errors", "type": "status_code", "status_code": {"status_codes": ["ERROR"]}}, {"name": "slow", "type": "latency", "latency": {"threshold_ms": 1000}}, {"name": "baseline", "type": "probabilistic", "probabilistic": {"sampling_percentage": 10, "hash_salt": "p49-fixture"}}]}}
Wire the sampler pipeline as receivers: [otlp], processors: [tail_sampling], exporters: [file] for a local-only experiment. Configure the file exporter with a dedicated output path and flush_interval: 100ms. Bind the OTLP receiver to loopback. The routing tier uses exporter loadbalancing, routing_key: traceID, a static resolver listing the two sampler gRPC endpoints, and protocol.otlp.tls.insecure: true only for this isolated loopback test. The fixture disables router exporter queues and retries to avoid concealing routing outcomes; that is not a production resilience recommendation. Validate every complete configuration with otelcol-contrib validate --config <file> before starting it.
The two-second wait is deliberately short for fault injection. It starts when the sampler first observes a span, not when the user operation started, and is not an inactivity timeout extended by every arriving child. The latency policy uses earliest observed start and latest observed end; it does not directly measure every application's user-facing latency contract. Incomplete spans and inconsistent clocks can therefore distort both retention and interpretation. Time-accurate observability remains a prerequisite.
num_traces bounds the trace buffer, not its total bytes. expected_new_traces_per_sec assists allocation; it is not an admission limit. Both decision caches are bounded and much larger than the fixture buffer. They remember keep/drop decisions after span data is released, but this memory is not a durable audit record. A cached drop can correctly discard a later ERROR child under the already-made decision.
Deterministic fixture and reconciliation
The fixture ran three local Collector processes: one router and two samplers. It sent synthetic OTLP/HTTP JSON into the router, which forwarded OTLP/gRPC to the samplers. No production prompts, customer identifiers or production telemetry backends were involved. The 100 outcomes were counted in a separate fixture ledger; a production metrics backend and counter exporter were not part of this experiment.
Reproduction manifest: index requests from 0 through 99; derive each trace ID from the first 32 hexadecimal characters of SHA-256 of the UTF-8 string cohort-<index>. Each trace has parent span ID 0000000000000001 and child span ID 0000000000000002. Requests 0–9 have an ERROR child; requests 10–14 have a 1,500 ms parent and successful outcome. Other parents and all children last 10 ms. Parents and children are sent in separate OTLP requests, preserving the same trace IDs. The parent and child share a synthetic start timestamp; this is a selection fixture, not a realistic service timing model.
Expected retention was computed before comparing exported IDs: retain if the fixture is an error, is slow, or passes the pinned probabilistic evaluator. In that version, the baseline hashes the salt p49-fixture followed by the 16 trace-ID bytes using FNV-1a-64 and compares with the 10% threshold. This implementation detail is version-specific, not a portable sampling specification. A finite deterministic cohort need not retain exactly 10% under the baseline policy, and the combined policy deliberately retains more.
The exact expected and actual retained-ID sets matched: 28 traces, each with both spans and each confined to one sampler. Ten were bad user outcomes in the fixture ledger. Compare 10 / 100 = 10% against 10 / 28 ≈ 35.7%. Even though every deliberately failed request in this completed cohort was retained, the sampled fraction cannot be substituted for the request SLI. Sampling happens after application tracing and network transport, so this result does not establish that every production failure would reach the sampler.
Decision table: what survived the fault tests
| Case | Expected retention boundary | Observed local result | Operating decision |
|---|---|---|---|
| Complete cohort, stable membership | Errors, slow traces or baseline match | 28 of 100 traces; exact ID match; two spans per retained trace | Use for diagnosis; retain the independent 100-request denominator |
| ERROR child inside the wait | Parent and error child can influence one decision | Two spans retained after a 0.3-second child delay | Measure arrival spread before selecting the production wait |
| ERROR child after a cached drop | New evidence does not reopen the drop decision | Zero spans retained when the child arrived after the decision | Track lateness; do not promise all errors survive |
| Sampler killed before decision | In-memory parent state is outside exporter durability | Only the later ERROR child survived after restart | Reconcile span completeness, not just trace-ID presence |
| Router membership changed from A to B | Consistent routing does not transfer existing state | Parent dropped on A; ERROR child retained on B | Test membership changes and draining with in-flight traces |
The delayed-child cases used trace IDs selected to fail the baseline policy, avoiding an accidental probabilistic keep. The early child arrived after 0.3 seconds; the late child was sent after a five-second wait, safely beyond the two-second decision window in this run. The restart test killed sampler A before its decision and submitted the ERROR child after restarting A. The topology test changed the router's static backend list from A-only to B-only between parent and child. It demonstrates state fragmentation under deliberate membership replacement, not Kubernetes discovery timing or a production rolling-update benchmark.
Capacity, restarts and long-running AI work
Size pending state from per-shard arrival rate and the decision window, then test burstiness, trace size and shard skew. A trace with many tool calls can consume far more memory than a short health request. Raising decision_wait buys time for late evidence but increases retention latency and pending state. Raising num_traces without a measured memory budget merely moves the failure boundary.
An exporter WAL protects work that reached that exporter queue. It does not automatically persist spans still awaiting a tail decision. The telemetry-backpressure article covers the downstream queue boundary. The current upstream tail-sampling documentation also describes experimental tail-storage support behind a feature gate; this fixture did not configure or test it. Do not silently infer restart durability from either a newer README or a persistent exporter checkbox.
Long-running AI tasks deserve a separate trace design decision. A minutes-long workflow can outlive a practical sampling wait, particularly when spans arrive incrementally. Consider bounded operation traces connected through span links, or an explicitly governed evidence stream for important lifecycle events. That changes cross-trace policy semantics and searchability. It is not a free way to preserve a complete distributed trace or satisfy audit obligations.
Head sampling is another upstream boundary. If the SDK never records or exports a span, a downstream tail sampler cannot reconstruct it. A hybrid policy can protect telemetry capacity, but “keep all errors” then means only errors visible within the surviving input and decision window. Redact or exclude sensitive prompt and response content before it leaves the trusted collection boundary, not merely after a tail decision.
Release checklist and limits
Before production approval, require: an owned request-outcome contract; a separate metrics path with reset and missing-series checks; trace-ID affinity tests across routing replicas; a measured span-arrival distribution; bounded trace, cache and byte budgets; deliberate restart and membership-change tests; retention/completeness reconciliation against synthetic IDs; redaction verification; and a rollbackable Collector configuration.
Monitor sampler buffer pressure, early evictions, late-span behavior, policy errors, accepted/refused telemetry and exporter failures using metric names verified for your deployed version. Preserve sampler health through a monitoring path that remains useful when the trace backend is unavailable. An absence of retained errors can mean healthy service, lost telemetry, an upstream head drop or an already-cached tail drop; do not collapse these possibilities into a green SLO.
This local exercise did not test throughput limits, memory exhaustion, exporter persistence, experimental tail storage, SDK head sampling or notification delivery. It proved the specified selection and fragmentation behavior for synthetic inputs on one pinned binary. The operational recommendation is narrower than “tail sampling is reliable”: separate the reliability denominator from diagnostic retention, and make evidence loss observable.
If you are choosing a sampling budget for an AI platform, bring one request-outcome definition, a redacted trace-shape inventory and your Collector topology to an architecture review. I can help turn them into a versioned retention policy, executable failure fixtures and a capacity decision—without treating a more interesting trace sample as proof of better or worse service reliability.
