Changing the DNS answer for an AI endpoint does not move an already-open connection. In the isolated experiment below, the cache returned origin B at 1.139 seconds, but the reused HTTP connection still returned origin A at 1.140 seconds. Only retiring the idle connection and opening another moved the request. Test discovery freshness, connection reuse and request outcome separately; a low DNS TTL is not a failover service-level objective.
This is an implementation diagnostic for teams operating inference gateways, retrieval services or external model clients. It is not a measurement of a commercial model API, a production incident report or an Envoy benchmark. The useful buyer decision is whether your acceptance test proves that new work reaches the intended backend—or merely proves that a DNS query now returns a different address.
Three clocks, not one failover timer
The first clock is DNS cache validity. The TTL on a resource record limits how long that record may be cached; it does not impose a lifetime on a TCP connection. RFC 1035 defines the wire format and resource-record TTL. Check the authoritative response and the answer actually returned to the application. A laptop query against a different resolver is not evidence about the application's resolver path.
The second clock is application discovery refresh. A client may resolve only when opening a connection, maintain its own cache, or use a proxy that refreshes asynchronously. Expiry makes a cached result unusable under the cache policy; it does not necessarily trigger a query at that instant. Record both the configured policy and the observed query times. Include container DNS, local stubs and sidecars in the inventory rather than assuming one shared cache.
The third clock is connection reuse. A healthy HTTP connection can continue carrying requests to the old peer after discovery changes. HTTP/2 multiplexing and long-running inference streams complicate retirement further. A successful lookup and a successful request can therefore describe different backends without either observation being false.
Executed local fixture: fresh DNS, old peer
The experiment ran on Python 3.12.13 with dnslib 0.9.26. A UDP DNS authority bound to loopback answered model.test with either 127.0.0.1 or 127.0.0.2. Two local HTTP/1.1 servers used the same port on those different addresses and returned origin-A or origin-B. An explicit single-name application cache consumed the wire response; a small client reused one http.client.HTTPConnection until it was deliberately retired.
Both positive and negative cache lifetimes were one second. There was no background refresh thread: the cache queried the authority on demand after expiry. A diagnostic lookup explicitly refreshed the cache while the original HTTP connection remained open. This design makes the two mechanisms independently observable. The cache is intentionally narrow, not a production recursive resolver: it does not implement a delegation chain, DNSSEC, CNAME handling or a complete cache-key policy.
The sequence was: open a connection to A; change the authoritative answer to B; query before cache expiry; query again after waiting 1.1 seconds; send another request on the existing connection; retire that idle connection only after its response body was consumed; open a new connection. Every observed request returned HTTP 200. The returned origin identity, not the status code alone, exposed the stale route.
| Elapsed / event | Observed state | Request outcome |
|---|---|---|
| 0.038 s: authority changes | Authority B; cache A | Existing connection to A |
| 1.139 s: cache refresh | Authority B; cache B | Connection still open to A |
| 1.140 s: reuse | No new connection | 200 from origin-A |
| 1.141 s: after idle retirement | New connection to B | 200 from origin-B |
| 1.141 s: name repaired | Authority B; cached NXDOMAIN | Lookup still fails |
| 2.243 s: after negative expiry | Fresh answer; new connection | 200 from origin-B |
Times are elapsed monotonic seconds from this single local run, rounded to milliseconds. They are not latency percentiles or a production recovery bound. The two 1.1-second waits were explicit test inputs, not measured DNS propagation times. The test demonstrated connection reuse and negative-cache persistence with real loopback HTTP and DNS traffic; it did not exercise TLS, Envoy, HTTP/2, a TCP timeout or interruption of an inference stream.
To reproduce the decision, keep origin identity in the response or server log, record the socket peer on every new connection, and correlate those observations with DNS results and cache hits. Pin the runtime and resolver library, use an isolated test name and high local ports, and close both listeners afterwards. Do not modify public DNS or send test traffic to a customer endpoint. A useful assertion is: after cache refresh, a reused connection still reports A; after safe idle retirement, a new connection reports B. If the client under test behaves differently, retain its trace rather than forcing this fixture's result onto it.
Negative answers need their own recovery test
For the second phase, the authority returned NXDOMAIN with an SOA record. The fixture used the smaller of the SOA TTL and SOA MINIMUM, both one second, as its negative-cache lifetime. This follows the negative-response lifetime rule described in RFC 2308. The test then restored the authoritative address immediately. The cache continued to return the cached negative result until a later lookup after expiry; at 2.243 seconds the new request reached B.
The phase boundary deliberately invalidated the earlier positive entry so the next lookup observed NXDOMAIN. That is test setup, not a recommendation to flush caches during every incident. NXDOMAIN means a name does not exist. NODATA means the requested record type has no answer for an existing name. Neither is the same as a transport timeout. The fixture tested NXDOMAIN only. Production DNS can also involve serve-stale policies, multiple cache layers and resolver-specific failure handling; this one-second example is not a universal timer.
A failed TCP connection happens after an address has been selected. Diagnose SYN/connect timeout, connection refusal, TLS verification failure and HTTP errors as separate events. This run did not inject a TCP timeout, so it makes no claim about timeout duration or retry behavior. Combine a later timeout experiment with an explicit retry budget, not an unbounded reconnect loop.
A decision table for the incident bridge
| Observation | Next discriminating check | Avoid |
|---|---|---|
| Authority B; application A | Actual resolver path and TTL | Blaming the HTTP pool first |
| Cache B; reused peer A | Connection age and reuse | Assuming TTL closes sockets |
| New peer B; wrong backend | Gateway routing and origin identity | Trusting peer IP alone |
| Authority repaired; NXDOMAIN | Negative TTL and cache layer | Calling it a TCP timeout |
| Address resolved; connect fails | Transport, TLS and deadline trace | Unbounded retries |
Capture a request correlation ID, lookup time, answer and remaining TTL, cache layer, connection creation time, peer address, backend identity and result. Avoid logging prompts, credentials or raw customer payloads merely to diagnose routing. A gateway address may remain unchanged while its own upstream changes; in that case socket peer identity alone cannot establish which model worker handled the request.
Keep TLS identity and active work intact
The fixture used cleartext HTTP on loopback and did not alter a TLS verification setting. That boundary matters: production HTTPS tests must preserve the original hostname for certificate verification, SNI and HTTP authority while controlling only address selection. Replacing the URL hostname with an IP or disabling verification can produce a passing connectivity test that is no longer testing the service's security contract.
Use this acceptance checklist before applying connection retirement in production:
1. Verify the actual client and proxy versions, resolver path, DNS refresh policy and negative-cache settings. Capture effective configuration, not only a deployment template.
2. Confirm that the replacement endpoint presents the expected certificate and supports the same authentication, model route and protocol. Test an incorrect certificate as a rejection case in a separate controlled TLS fixture.
3. Distinguish idle from active connections. Stop assigning new work to a retiring route; allow admitted streams to complete under a documented deadline. Do not tear down every pooled socket to make the DNS test look successful.
4. Record unfinished requests and define whether replay is safe. A partially delivered generation or a tool-using workflow may not be safely repeatable. Coordinate with graceful service shutdown rather than hiding unfinished work behind a reconnect.
5. Measure client-visible completion and error rates alongside backend identity. Connect the acceptance criteria to inference SLOs, not to GPU utilisation or DNS success alone.
These are production acceptance requirements, not claims that this small fixture verified stream preservation or TLS behavior. Aggressive connection cycling increases handshakes and churn; long-lived pools reduce that cost but can delay traffic movement. Choose the policy using your request deadlines and backend retirement window, then test both healthy migration and hard failure.
What to verify if Envoy is in the path
Envoy's service-discovery documentation is the primary reference for its discovery types and refresh controls. This experiment did not run Envoy, so it deliberately does not present a measured strict-DNS versus logical-DNS comparison. Do not relabel the toy client as either mode.
For an Envoy deployment, pin the exact release and configuration and run the same trace through that binary. Inspect the configured discovery type, TTL-respect setting, refresh interval, failure-refresh policy, health checks and connection-drain behavior. Record update attempts, successes and failures together with actual upstream identity. A generic documentation default is not proof of the effective deployment policy.
Define the evidence before the migration
A credible failover acceptance test answers three questions: what did discovery return, which connection carried the request, and which backend completed it? It also states what remained untested. A single dig result or HTTP 200 cannot answer all three.
If your team is changing an inference endpoint, start with a short review of the client/pool inventory and one isolated trace. Bring the resolver path, effective proxy configuration, stream deadlines and rollback criteria to an implementation review with Ade Christanto. The deliverable should be a reproducible acceptance test and an explicit retirement policy—not a promise that every request moves within one TTL.
