BLUF: A sending queue buys time; it does not guarantee delivery. For an OpenTelemetry Collector exporting AIOps evidence, resilience comes from aligning four separate controls: a bounded queue for short backend interruptions, retry deadlines for transient failures, persistent storage for Collector restarts, and explicit overload behaviour when the buffer is exhausted. Size those controls from measured ingress and a declared outage/data-loss budget, monitor accepted, queued, sent and refused records as one flow, and prove recovery with fault injection. “Export failed” is not automatically “data lost”, because a retry may still succeed; “persistent” is not “exactly once”, because disks, hosts, retry windows and downstream semantics still fail.
This is the downstream continuation of building passive, time-accurate OT evidence. That design establishes whether an observation is trustworthy at acquisition. The problem here is whether the same evidence survives the transport path when the observability backend becomes slow, unavailable or overloaded. Without this boundary, an AIOps system loses the very records needed to reconstruct an incident.
Backpressure is a failure-domain decision, not a queue-size setting
A typical Collector pipeline accepts spans, metric points or log records through receivers, processes them and hands batches to an exporter. When the destination cannot accept a request, the exporter may retry and hold work in a sending queue. That queue decouples ingress from temporary egress failure. Once it reaches capacity, however, the Collector must refuse or drop additional work rather than consume memory or disk without limit.
The current OpenTelemetry resiliency documentation distinguishes three mechanisms. An in-memory sending queue protects against a downstream endpoint interruption but loses buffered data if the Collector process terminates. A sending queue backed by the file_storage extension writes queue state to a local write-ahead log and can recover after a process restart. A dedicated message queue between Collector tiers can isolate longer or cross-boundary outages, but it creates another production system with its own retention, availability, security and operating cost.
Treat each mechanism as protection against a named failure. Memory protects against a brief remote pause. A local WAL also protects against process restart, but not necessarily host or disk loss. A replicated message queue can protect a larger boundary, but only while its producer path, retention and cluster remain healthy. None of these statements establishes end-to-end exactly-once delivery.
Architecture diagram: make the protected boundary visible
APPLICATION / OT EVIDENCE SOURCE → local or agent Collector → receiver accepted/refused counters → processors and batching → exporter sending queue → retry policy → telemetry backend.
PERSISTENCE PATH — exporter queue → file_storage WAL on a durable mounted volume → queue replay after Collector restart. The volume, permissions, capacity, cleanup and recovery behaviour are part of the service, not deployment trivia.
CONTROL PATH — Collector internal metrics → independent monitoring destination or local scrape → queue saturation alert → admission/load-shedding policy → incident runbook. Do not depend only on the same backend whose outage you are trying to observe; retain a local or separate health path that can still reveal queue pressure.
OPTIONAL DURABILITY TIER — agent Collector → locally protected exporter → replicated message queue → gateway Collector → backend. Add this only when the required outage window or failure boundary justifies its operational cost.
Separate the four controls
Sending queue capacity answers: how much unsettled export work can be buffered? It must be bounded. Queue capacity is commonly expressed in batches or requests, so batch-size variance matters; a nominal count is not a byte guarantee.
Retry policy answers: for how long and at what cadence should a retryable failure be attempted? The OpenTelemetry resiliency page describes exponential backoff with jitter and notes a five-minute default maximum elapsed time in its example context. Pin and validate the exact Collector distribution and version you deploy rather than copying a default from an article. Permanent errors and non-retryable responses must not churn indefinitely.
Persistent storage answers: does unsettled work survive a Collector process restart? The file_storage extension can back the sending queue with a WAL. It still depends on a writable, sufficiently provisioned directory and the durability of the mounted storage.
Overload policy answers: what happens after capacity is consumed? A bounded system must reject, shed, sample or degrade according to signal criticality. Google SRE’s overload guidance is relevant here: protect the task so it continues serving the load it can handle rather than allowing excess work to trigger memory pressure and cascading failure.
Size from a measured outage budget
Start with four measured inputs per signal and exporter: peak accepted items per second over a defensible window; observed items per export request after batching; measured serialized bytes per request including realistic attribute cardinality; and the outage duration the business expects the Collector layer to bridge. Then add a documented safety factor for burst and batch variability.
The basic capacity model is: required buffered items = peak accepted rate × protected outage seconds × safety factor. Convert items into queue requests using the observed batch distribution, not only an average. Estimate disk from measured serialized request bytes plus WAL overhead. Finally, prove the estimate under load: compression, logs with large bodies, span attributes and changing batch sizes can invalidate spreadsheet sizing.
Capacity and retry time must agree. A large queue paired with a short maximum retry window can expire old work before the queue fills. A long retry window paired with a small queue can fill quickly and reject new evidence. Disk capacity adds a third clock: at current ingress, when will the persistent volume reach the reserve threshold? Alert on the earliest exhausted budget.
Define the data-loss budget explicitly: maximum records or seconds of evidence that may be lost by signal class, the failure boundaries covered, and the conditions that force degradation. Security audit logs, incident traces and high-volume debug telemetry should not inherit the same retention or priority by accident.
Decision table: choose the durability tier
Condition: backend interruptions are brief; replay after Collector termination is not required. Control: bounded in-memory sending queue plus retry. Benefit: simplest and lowest disk overhead. Residual risk: process restart loses buffered work; queue overflow still drops or refuses data.
Condition: Collector restarts must not erase unsettled evidence and the host volume is inside the accepted failure boundary. Control: sending queue backed by file_storage WAL. Benefit: process-level recovery. Residual risk: disk exhaustion, volume corruption, permissions, host loss and retry expiry.
Condition: a gateway or vendor boundary must tolerate a longer outage or survive one Collector host. Control: durable replicated message queue between tiers, with local protection on the producer hop. Benefit: broader decoupling and independently managed retention. Residual risk: substantially more operational complexity; the queue itself becomes critical infrastructure.
Condition: ingress can exceed safe drain capacity even after recovery. Control: admission control, priority classes, intentional sampling or load shedding. Benefit: preserves critical signals and Collector health. Residual risk: deliberate evidence gaps that must be measurable and documented.
Condition: the data-loss budget is effectively zero. Decision: do not claim that a Collector setting solves it. Perform an end-to-end integrity design covering source buffering, acknowledgements, storage replication, downstream deduplication, retention and reconciliation.
Monitor the flow, not one red metric
OpenTelemetry documents queue capacity and current queue size as otelcol_exporter_queue_capacity and otelcol_exporter_queue_size. Track their ratio and time-to-full under current net inflow, but also watch the absolute capacity because configuration changes alter the denominator. Queue saturation is a leading indicator; it does not by itself prove loss.
For each signal, compare receiver accepted and refused counts with exporter sent counts over a window that includes queue change. Enqueue-failed metrics indicate work that failed to enter the sending queue, often because it was full. Exporter send-failed metrics indicate delivery attempts that failed; OpenTelemetry explicitly cautions that these do not inherently imply data loss because retries may follow. The evidence of loss is a terminal outcome: refused/enqueue-failed work, retry expiry/drop logs, unrecoverable WAL failure, or an ingress-to-egress reconciliation gap after the queue has drained.
Keep dimensions bounded: Collector instance, pipeline, exporter and signal are usually actionable; arbitrary endpoint or tenant labels can create a cardinality incident inside the monitoring system. Record configuration version, Collector version and restart count alongside incidents so operators can distinguish load growth from a rollout regression.
Once evidence reaches the analytics layer, preserve this transport state in causal AIOps evidence packets. Missing telemetry during an outage is a competing explanation for an apparent root cause, not a detail to hide.
A production configuration pattern
The minimum pattern is conceptually: define file_storage with a dedicated persistent directory; enable that extension in service.extensions; reference file_storage from the remote exporter’s sending_queue; set an explicit bounded queue_size; configure retry_on_failure with initial interval, maximum interval and maximum elapsed time; and mount storage with a capacity alert and tested permissions. Syntax and supported fields can change, so validate the configuration against the exact Collector build in CI and inspect the exporterhelper documentation for that release.
Run the Collector’s own internal metrics on a path operators can reach during a backend incident. The official documentation says internal metrics are exposed through a Prometheus interface on port 8888 by default, while configuration schema details remain under development and may introduce breaking changes. Treat defaults as discovery aids, not production contracts. Pin the listening interface, network policy, scrape path and authentication appropriate to the deployment.
Avoid circular monitoring. If Collector health metrics are exported only through the blocked exporter, the queue can be full while the dashboard remains green and stale. A local scrape retained by the platform, an out-of-band operations backend or a minimal independent alert path makes the failure visible.
Outage test matrix: verify every failure boundary
Test: backend returns retryable failure or becomes unreachable. Inject: block the egress route or use a controlled fault endpoint. Expected evidence: send failures rise, queue grows, accepted ingress continues within budget, retries back off, then the queue drains after recovery. Pass condition: no terminal-loss indicator within the protected window and recovery does not overload the backend.
Test: queue reaches capacity. Inject: hold the backend down beyond the calculated buffer. Expected evidence: capacity ratio reaches the alert threshold, enqueue failures or receiver refusals become visible, memory/disk remains bounded. Pass condition: Collector stays healthy, loss is quantified by signal, and the runbook applies priority or shedding rather than crashing.
Test: Collector process terminates with buffered work. Inject: terminate and restart the process while the backend remains unavailable. Expected evidence: the WAL reopens and unsettled work reappears. Pass condition: replay proceeds after recovery without silent disappearance; duplicates, if possible, are accepted or reconciled downstream.
Test: persistent disk is full, read-only or unavailable. Inject: use a bounded test volume and controlled permission/failure scenarios. Expected evidence: storage and enqueue errors surface before the host is destabilized. Pass condition: Collector fails in the documented mode, critical signals follow the agreed degradation policy, and no false delivery claim is emitted.
Test: backend recovers at reduced capacity. Inject: throttle the receiver during drain. Expected evidence: backlog decreases without a retry storm or saturation of the recovering service. Pass condition: drain rate, live ingress and backend capacity remain controlled; recovery time meets the objective.
Failure modes that survive a “persistent queue” checkbox
1. Queue units are mistaken for records or bytes, while batch sizes vary. 2. The WAL sits on ephemeral container storage. 3. A restart test proves process recovery but not node replacement or volume loss. 4. Retry duration expires before the advertised outage window. 5. Disk alerts use percentage only and ignore time-to-full. 6. Health metrics travel through the failed backend.
7. Operators equate send_failed with dropped and trigger the wrong incident response. 8. Conversely, they assume every failed send will retry forever. 9. A recovered Collector drains at full speed and overloads the backend again. 10. Multiple replicas share or contend for storage not designed for that access pattern. 11. Deployment changes queue semantics without a versioned recovery test. 12. Sensitive telemetry persists on disk without retention, encryption and access controls appropriate to its content.
Persistent buffering also changes deletion and incident-response procedures. A record removed from the backend may still exist in a WAL or message queue until acknowledged and cleaned up. Security, privacy and contractual retention requirements must be translated into storage configuration and operating procedures; qualified counsel should assess legal obligations.
Implementation checklist
— Classify signals by criticality and data-loss budget. — Measure peak accepted rate, batch distribution and serialized request size. — Name the protected outage duration and failure boundary. — Align queue capacity, retry expiry and disk reserve. — Put WAL data on intentional persistent storage with access controls. — Monitor accepted, refused, queued, enqueue-failed, send-failed and sent flow. — Keep an independent health path. — Alert on saturation and time-to-full, not only failed exports. — Test backend outage, full queue, process restart, disk failure and throttled recovery. — Quantify terminal loss and possible duplicates. — Pin Collector/config versions and rerun tests after changes. — Document the runbook owner and shedding order.
Start with one exporter and one forced outage
Choose the exporter carrying your highest-value incident evidence. Measure its real ingress and batches, state a protected outage window, and configure the smallest durability tier that covers that boundary. Then block the backend, restart the Collector, constrain the disk in a test environment and restore the backend slowly. The deliverable is not a queue that appears enabled; it is a recovery record showing what was accepted, what survived, what was sent, what was lost and how the system behaved at capacity. I help operations teams turn that record into a deployable Collector architecture, capacity model and fault-injection gate before AIOps depends on the data.
Primary and authoritative sources
OpenTelemetry Collector resiliency — sending queues, retry, file-backed WAL, message queues and documented loss circumstances.
OpenTelemetry Collector internal telemetry — queue, enqueue-failed, send-failed, receiver and data-flow metrics, including interpretation limits.
Google SRE: Handling Overload — bounded service behaviour, graceful rejection, throttling and protection against cascading failure.
OpenTelemetry Collector exporterhelper — implementation and version-specific queue/retry configuration reference.
