BLUF — stop admitting new work, withdraw readiness, drain admitted requests within an explicit budget, and record what remains unfinished. Cancelling an asyncio waiter does not prove that its thread, inference engine or remote operation stopped. A shutdown passes only when request outcomes and backend-work outcomes are accounted for separately—not when the process disappears from a dashboard.
This article includes an executed local ASGI shutdown fixture with short work, a partial stream and a blocking thread. It demonstrates one process boundary, not a Kubernetes, GPU or load-balancer acceptance test. The operating objective complements inference SLOs: a request interrupted by deployment must remain in the reliability denominator.
Define the two ledgers before the timeout
Track each admitted request through completed, failed, cancelled or still outstanding at the observation boundary. Keep rejected-before-admission work separate. A completed response must satisfy the application contract; an HTTP 200 header at the start of a stream is not enough. Use a terminal stream marker or an equivalent validated protocol boundary, not socket closure alone.
Track backend execution independently: not started, running, confirmed stopped, completed or unknown. A cancelled request can still have running backend work. Do not add these two ledgers together as mutually exclusive counts. Record a bounded request class and release identity in metrics; keep correlation identifiers in access-controlled traces rather than high-cardinality metric labels. Never log prompts just to count unfinished requests.
The Uvicorn server-behavior documentation explains that status and headers are sent when the ASGI application starts the response. A later exception cannot retrospectively replace that status with a failure. This is why a green HTTP-status chart can hide truncated generation.
A shutdown state machine with explicit ownership
RUNNING → QUIESCING → DRAINING → RECORDING → EXIT. RUNNING admits work under normal limits. QUIESCING closes the application admission gate and marks readiness false. DRAINING lets the already-admitted cohort finish while enforcing its remaining deadlines. RECORDING writes terminal request outcomes and unresolved backend state within a bounded cleanup budget. EXIT releases resources or hands unresolved durable work to its declared owner.
This is a proposed application contract, not a Kubernetes enum. Make the admission decision and registration of in-flight work atomic within the owning event loop or protected registry. Otherwise a request can pass the gate just before draining begins and never enter the drain count. A multi-worker deployment needs a gate for every worker; changing one process-local Boolean is not a fleet-wide fence.
Liveness and readiness answer different questions. A draining process can still be alive and capable of cleanup while no longer eligible for new traffic. Do not deliberately fail liveness to implement graceful draining. Conversely, readiness withdrawal alone is not instantaneous admission control: existing connections and routing propagation can still deliver requests. Return the documented temporary rejection before starting new inference; apply the existing retry budget instead of encouraging an uncontrolled retry wave.
Timing: the outer clock includes the hook
Kubernetes Pod lifecycle documentation describes termination hooks, signals and the eventual forced-stop boundary. The termination grace period includes preStop execution; it is not a fresh budget that starts after the hook. Endpoint updates happen alongside local shutdown, not as a guaranteed synchronous barrier before the next request.
Timing sketch — Pod termination begins → preStop closes admission and initiates readiness withdrawal → TERM reaches the server → bounded request drain → bounded cleanup and outcome recording → process exit. In parallel, EndpointSlice and proxy routing state propagate. The outer grace deadline can cut across this entire sequence. This sketch is an ordering aid, not a measured cluster trace or a promise that every proxy sees the endpoint change before TERM.
Do not copy a fixed sleep into preStop and call it propagation proof. Measure the last request arriving through your actual ingress and connection-reuse path after withdrawal. If a bounded propagation allowance is necessary, include it in the same outer budget and retain rejection at the application boundary. Protect any drain-control endpoint with a local or authenticated management boundary; the fixture's localhost-only control route is not a public administration API.
| Clock | Owner and boundary | What it does not guarantee |
|---|---|---|
| Pod termination grace | Kubelet/runtime; includes preStop | Successful completion before forced stop |
| preStop duration | Management hook; spends outer budget | Instant propagation to every proxy |
| Server graceful timeout | ASGI server; connection/task wait | Bounded lifespan cleanup or stopped thread |
| Request deadline | Caller/application; remaining usefulness | Backend cancellation after timeout |
| Worker lifetime | Worker supervisor or remote service | Coupling to the waiting coroutine |
Choose the outer grace budget to cover the measured hook duration, server drain, bounded application cleanup and an explicit safety margin. This is a planning constraint, not an exact equality: runtimes and scheduling add overhead, and concurrent phases must not be counted twice. The request deadline still applies during shutdown; extending it merely to finish a rollout can violate the caller's service contract.
In the pinned Uvicorn implementation, the graceful timeout bounds waiting for connections and tasks. Lifespan shutdown follows that wait. Therefore that timeout is not, by itself, a hard limit on every cleanup operation or total process lifetime. Bound application cleanup explicitly and retain the container's outer termination boundary.
What the executed local fixture returned
The run used Python 3.14.9, Uvicorn 0.41.0, asyncio and HTTP/1.1 through h11. These are reproduction pins, not recommended latest versions. The server bound an ephemeral localhost port. It admitted three requests: a 0.2-second coroutine, a stream emitting a chunk every 0.1 seconds for up to 100 iterations, and synthetic blocking work designed to run for four seconds through asyncio.to_thread.
The harness waited for the short request to complete and for the thread to start, then called the local drain route. It verified a late request was rejected, readiness returned an HTTP error and liveness still returned 200. It then sent real SIGTERM to the owned child process with a one-second graceful timeout. Lifespan cleanup deliberately waited 0.2 seconds so thread activity after request cancellation could be observed. These durations are test stimuli, not production settings.
| Observed item | Count | Interpretation |
|---|---|---|
| Admitted requests | 3 | Short, stream and thread-backed work |
| Completed requests | 1 | Short response completed before drain |
| Cancelled request handlers | 2 | Stream and thread waiter |
| Rejected before admission | 1 | Late request did not start work |
| Finally blocks executed | 3 | Cleanup entry, not completion proof |
| Backend work unfinished at lifespan shutdown | 1 | Thread heartbeat continued after cancellation |
The short client received a complete 200 response. The streaming client raised IncompleteRead; the thread-backed request returned an HTTP error. All three admitted handlers executed their finally blocks. That proves entry into the instrumented cleanup, not successful durable persistence or backend cancellation.
The thread emitted a heartbeat 0.196 seconds after its waiting coroutine was cancelled. No worker-finished event was recorded. At lifespan shutdown, one backend operation was still unfinished. The child subsequently exited with return code -15: the pinned Uvicorn signal handler restores and re-raises SIGTERM after shutdown. Do not rewrite that result as a successful thread completion, or generalize it to another process manager.
The JSONL ledger records monotonic event time, event kind and the synthetic route. Its accounting assertion is three admitted requests, one completed and two cancelled, with one separately rejected late request. The backend ledger remains distinct. This fixture has no external writes, production traffic or native GPU workload; a remote provider could continue processing after this local process is gone.
Cancellation is a request to cooperate
Python's asyncio documentation describes CancelledError delivery at an opportunity for the task to resume. Use try/finally for cleanup and normally re-raise cancellation after handling it. Swallowing cancellation can interfere with timeout and structured-concurrency behavior. Moving blocking work to a thread keeps the event loop responsive, but does not create a thread-kill protocol.
For an owned worker, implement a cooperative stop token checked between bounded units of work and record its acknowledgement. For a native library, test its actual cancellation API and safe interruption points. For a remote call, distinguish transport disconnect, cancellation requested and cancellation confirmed by the receiving system. If hard isolation is necessary, a supervised worker process offers a termination boundary, but adds IPC, startup overhead and recovery complexity; killing it still cannot roll back a remote side effect.
| Observed boundary | Decision | Required evidence |
|---|---|---|
| New request after admission stop | Reject before starting work | Gate outcome and no backend dispatch |
| Admitted short work within deadline | Allow bounded completion | Complete validated response |
| Stream truncated after 200 headers | Record incomplete, not successful | Client protocol outcome and terminal marker |
| Waiter cancelled; worker still active | Keep backend unresolved | Stop acknowledgement or completion record |
| Remote write outcome unknown | Reconcile; do not blindly replay | Authoritative destination result |
Acceptance checklist for the real deployment
1. Replay the pinned local test with the actual server entrypoint and process manager. Check normal completion, deadline expiry, client disconnect and repeated termination signals. Retain both client-observed results and server-side outcome records; a zero exit status alone is insufficient.
2. Repeat through staging ingress and Kubernetes termination. Include persistent connections, long streams, concurrent worker processes and routing updates. Verify every newly arriving request is either admitted before the boundary or rejected without starting backend work. Measure endpoint propagation rather than assuming a universal delay.
3. Test a worker that ignores cooperative cancellation. Keep its state unresolved until an owner confirms termination or completion. Reserve capacity for the replacement plus draining work; draining longer reduces interrupted requests but occupies resources longer and can slow rollouts.
4. Inject outcome-store failure. Bound persistence retries and report loss of accounting evidence explicitly. If the process may be killed before it can record a terminal event, use an external admission ledger and lease/reconciliation mechanism. Restore drills address recovery after state loss; they do not replace this live termination test.
5. Freeze the rollout when incomplete streams, oldest outstanding age or unaccounted operations exceed the predeclared release policy. Do not erase those requests by shortening the observation window. Choose thresholds from service commitments and business risk, not the tiny synthetic sample here.
The next useful deliverable
For an AI service shutdown review, bring the actual entrypoint, termination settings, one interrupted stream and one operation whose cancellation outcome is uncertain. The deliverable should be a timeout-ownership matrix, a reproducible drain test and an outcome ledger that distinguishes stopped waiting from stopped work. That is a stronger release criterion than a promise of zero downtime.
