BLUF — RabbitMQ message TTL limits residence in a queue, not the lifetime of work already delivered to a consumer. Carry an immutable business deadline in the job, check it again immediately before model dispatch, and record expired-not-started separately from failed execution. Lower prefetch reduces local waiting but does not replace this admission gate. Dead-letter delivery is not permission to restart the clock.
Consider a read-only incident-summary job that is useful only while an operator is investigating the current alarm. A consumer receives it promptly but waits for an inference slot. By the time that slot opens, the incident context is obsolete. The broker successfully delivered the message; the application would still be wrong to start it. This article isolates that delivery-to-dispatch boundary. The broader retry-budget design governs how many attempts are allowed, not whether a prefetched job remains useful.
Four clocks that must not be confused
RabbitMQ's TTL documentation defines message TTL as retention in a queue. It documents a delivery race: expiry can occur after bytes have entered the socket and before a consumer receives them. More importantly, a delivered job may sit in your application's own buffer. Neither broker queue depth nor a successful delivery says that inference has started.
The expiration property is a string containing a duration in milliseconds. When per-message and queue-level message TTL both apply, the smaller value wins. By contrast, x-expires concerns deletion of an unused queue. It is not a job deadline, and deleting a queue through expiry does not dead-letter its contents. Execution timeout is a fourth mechanism: it concerns work that has already started and requires a tested cancellation contract.
| Control | Boundary | Does not prove |
|---|---|---|
| Message TTL | Residence in a broker queue | Timely start after delivery |
| Queue expiry: x-expires | Unused queue lifetime | Per-job cancellation |
| Business deadline | Useful work across hops | Backend has stopped |
| Execution timeout | Already-started operation | Remote cancellation succeeded |
Do not use a process-local monotonic timestamp as a portable business deadline. Persist an absolute expiry in the job envelope, with an explicit clock-synchronisation and uncertainty policy. Use monotonic time for local elapsed durations. The deadline is part of the business contract: preserve it through redelivery, retries, dead-letter handling and process restarts.
What the executed broker fixture demonstrated
The local fixture ran RabbitMQ 4.0.5 from Ubuntu package 4.0.5-10ubuntu5.2, Erlang/OTP 27.3.4.6 and Pika 1.3.2. These are reproduction pins, not a recommendation to deploy this release. It used a real isolated loopback broker, classic queues, publisher confirms and manual acknowledgements. The jobs were synthetic and read-only; no model provider was called. It did not simulate broker behavior with a mock.
Each job had a 600 ms message TTL and an immutable expires_at_ms value in its JSON body. The queued case waited one second before retrieval. The prefetch cases published three jobs each, received up to the configured limit, waited 250 ms in the event loop and then another 800 ms before dispatch checks. These deliberately short delays expose the boundary; they are not production timeout recommendations or throughput measurements.
| Fixture case | Observed outcome | Application decision |
|---|---|---|
| Queued beyond 600 ms TTL | Dead-lettered; expiration absent | Reject original stale deadline |
| Prefetch 1; three jobs | One delivered; two expired in broker | Reject one local job |
| Prefetch 3; three jobs | Three delivered before expiry | Reject all three after waiting |
| Requeue after expiry | Original queue empty; dead letter received | Reject unchanged deadline |
With prefetch one, one job was delivered and held unacknowledged; the other two remained broker-side and were absent when the queue was checked after expiry. With prefetch three, all three were delivered before expiry and remained in the local waiting list until the application rejected them. All four delivered jobs were classified expired_not_started, with no model dispatch. This is evidence that prefetch changes where the waiting occurs, not a measured estimate of cost savings.
The separate queued case reached its dead-letter destination with reason expired. Its expiration property was absent, while the absolute deadline in the JSON body remained unchanged. A separate job was retrieved, held beyond expiry and negatively acknowledged with requeue=true. It was no longer retrievable from its original queue and arrived at the dead-letter destination with the original business deadline intact. Both dead-letter consumers rejected stale work. Across these application-observed cases, the saved terminal ledger contains six expired_not_started records. The two broker-expired jobs in the prefetch-one case were not observed by that consumer ledger; do not silently count it as complete end-to-end accounting.
Timeline: delivery is not dispatch
QUEUED → DELIVERED, UNACKNOWLEDGED → LOCAL WAIT → DEADLINE CHECK → MODEL DISPATCH. The broker TTL applies to queue residence. The dispatch check applies after local waiting, including waiting for a concurrency permit. If the deadline has passed, take the terminal-expiry path instead of entering model dispatch. Keep receipt time, dispatch-check time and original expiry as separate trace fields.
For the prefetch-three fixture, all three callbacks ran before the 600 ms deadline. Dispatch checks occurred only after the deliberately delayed local wait. The broker did not recall those delivered bodies. A consumer that checked only in the callback would have missed this later expiry. If an adapter or SDK adds another queue, that queue creates another boundary to inspect.
Implement the last responsible admission check
At publication, assign a stable job ID, immutable expires_at_ms and schema version. Reject missing, malformed or implausibly distant deadlines according to a declared policy. Compute broker TTL from the remaining allowance rather than granting a new full duration on each republish. A passed deadline should be rejected before publication, not turned into a fresh TTL. Validate the envelope at a trusted ingress so an untrusted producer cannot extend its own authority.
On receipt, validate the envelope and register the job as delivered but not started. Use bounded prefetch and a bounded local buffer. Wait for the execution permit without occupying more capacity than the policy allows. Immediately before dispatch, calculate remaining = expires_at_ms − worker_wall_time_ms − clock_uncertainty_ms. Reject when remaining is not positive. If the service contract requires a minimum useful execution window, reserve that window and cleanup allowance too; the fixture tests expiry, not completion-time prediction.
The 100 ms uncertainty margin in this test is an illustrative policy input, not a measured clock bound. A controlled arithmetic case used worker time 10,000 ms and deadline 10,080 ms: without a margin it would admit; with a 100 ms margin it rejects. Production must establish and monitor the actual producer-to-worker uncertainty. If clock health is unknown, a supposedly conservative constant is not proof; stop admission or route to an explicit degraded policy.
After admission, derive a local timeout from the remaining allowance. A successful check cannot guarantee completion before the deadline, and time may pass between the check and the wire call. Put the check as close to the true execution boundary as possible, propagate the deadline downstream, and test the backend's cancellation semantics. Graceful shutdown explains why stopping a waiter does not establish that backend work stopped.
Acknowledgement and terminal-state decisions
| State at boundary | Action | Residual risk |
|---|---|---|
| Expired before dispatch | Persist terminal expiry; acknowledge | Store/ack crash gap |
| Expired; quarantine required | Reject without requeue to configured DLX | Default DLX can lose messages |
| Still useful; capacity available | Recheck then dispatch with remaining timeout | Expiry or cancellation during execution |
| Clock uncertainty unbounded | Stop or use explicit degraded policy | Lost availability |
| Already started; deadline passes | Request cancellation; track actual outcome | Remote work can continue |
For stale work, either persist a terminal expired_not_started outcome and acknowledge it, or reject without requeue to a deliberately configured quarantine path. Neither choice automatically supplies reliable audit evidence. Persisting before acknowledgement can produce redelivery after a crash; acknowledge first and a crash can erase the only outcome record. Use an idempotent terminal store keyed by job identity, replay-safe consumers and reconciliation for missing outcomes. The fixture ledger is a saved local test artifact, not a transactional production outbox or crash-durability proof.
Avoid an infinite requeue loop for already-expired work. The dead-letter documentation states that the original TTL is removed during dead-lettering. It also warns that default dead-letter republishing does not use internal publisher confirms and can lose messages when the destination cannot accept them. Quorum queues support a separately configured at-least-once dead-lettering mode; this classic-queue test does not validate it. Prefer operational policies over hardcoded queue arguments in production; the disposable fixture uses arguments only to isolate its topology.
Prefetch is a capacity trade-off
RabbitMQ's consumer-prefetch documentation explains that a non-global prefetch limit is applied per consumer; zero means unlimited. More consumers therefore change the aggregate delivered-but-unacknowledged population. A small prefetch can limit local residence but may reduce throughput when delivery latency leaves workers idle. A large value can hide a backlog inside consumers while the broker's ready count looks reassuring.
Measure ready, unacknowledged and locally waiting work separately. Record queue residence and consumer residence, then count expired before receipt, expired before dispatch, execution failed, completed and outcome unknown without collapsing them into one error counter. Keep job IDs out of metric labels. Retain them in controlled traces or the outcome store. Telemetry backpressure remains a separate evidence-transport problem: a dropped monitoring record is not proof that a business job was never dispatched.
Release checklist and next decision
1. Reproduce queued expiry, delivered local waiting, requeue and dead-letter handling on the pinned broker and client used by your deployment.
2. Run prefetch and worker-concurrency combinations under realistic queueing. Verify the dispatch spy sees zero calls for expired jobs; do not infer this from a missing success response.
3. Inject clock skew, invalid deadlines, terminal-store failure and worker restart. Preserve original expiry, avoid infinite requeue and identify outcomes that remain unknown.
4. Test real model timeouts separately. Include remote work that continues after cancellation, and do not report that work as expired-not-started.
5. Define the owner and retention of terminal records. Broker expiry without a dead-letter observer may leave no application receipt; reconcile against the producer's accepted-job inventory if complete accounting is required.
Bring one queue topology, its prefetch/concurrency settings and one stale-job example to an AI operations review. The useful deliverable is an executable deadline gate with an outcome ledger and explicit loss boundaries—not another TTL setting that appears to cover the entire workflow.
