A successful request over an existing TLS connection does not prove that a new connection will accept a rotated gateway certificate. In the isolated test below, the established connection kept exchanging data after the listener switched certificates, while a new connection using the old trust store failed. Replacing the CA bundle file did not update the already-created Python SSL context. Rebuilding the context with the intended trust bundle restored new-connection acceptance without disabling verification.
For an inference gateway, that distinction separates a reassuring health check from a useful rotation acceptance test. Test the certificate chain, service identity and effective client trust separately. Then prove that the client can create a fresh connection with those settings. This article reports a synthetic loopback experiment, not a customer incident or a benchmark of an AI provider.
Define the rotation boundary before changing it
A certificate renewal under an already-trusted issuer is not the same change as moving to a different root CA. A new intermediate, different subject alternative name (SAN), incomplete served chain or changed validity window can independently break acceptance. Inventory the leaf, intermediates, intended roots, DNS reference identity and effective trust configuration for each client population. The test deliberately rotates from root A to root B to make the trust boundary visible.
Python's ssl documentation distinguishes SSL contexts from the sockets created with them. A context holds verification settings and loaded trust material. An on-disk bundle, a live context and an established transport are different objects. A mounted secret changing on disk is therefore not evidence that an SDK, process or connection pool has reloaded it.
RFC 9525 separates service-identity verification from certification-path validation. A chain issued by a trusted CA can still identify the wrong service. For this test the reference identity remained model.test, even though the TCP destination was 127.0.0.1. The client passed model.test as server_hostname and disabled Common Name fallback; the intended identity was in the DNS SAN. Do not replace the reference hostname with the destination IP merely to make a diagnostic connect.
Executed fixture and sanitized chain manifest
The run used Python 3.14.9, its linked OpenSSL 3.5.9 and cryptography 50.0.1. The separate openssl command was version 3.5.5. Record both OpenSSL versions: the command-line executable and Python's linked library need not match. Certificate validity checks below used the command-line implementation; network handshakes used Python's library.
The fixture generated temporary RSA-2048 keys and SHA-256-signed test certificates. The old leaf, with DNS SAN model.test, was signed directly by test root A. The new leaf, also model.test, was signed by test intermediate B, which chained to test root B. A separate wrong-identity leaf used DNS SAN other.test under the same B chain. CA certificates carried CA/key-signing extensions; leaves carried server-auth usage. The server sent the leaf and required intermediate, not a replacement for the client's trust anchor.
A sanitized manifest retained issuer names, SANs, validity windows and SHA-256 fingerprints. Temporary private keys were deleted when the fixture exited. No production certificates, DNS records or trust stores were changed. The loopback listener used an operating-system-assigned port and a small echo protocol, not HTTP or an inference API. This deliberately isolates TLS behavior from application retries and model execution.
Both sides were restricted to TLS 1.3. The server issued no session tickets; successful new connections reported session_reused=false. The client used create_default_context(cafile=...) with certificate and hostname verification enabled, Common Name fallback disabled and recorded verify_flags=557088. These are measured settings of this runtime, not a promise about every Python release or SDK default.
Keep the old connection visible while testing the new one
The server selected its current SSL context when accepting each TCP connection. Switching that selection from the old certificate to the new chain affected subsequent accepts, not the already-wrapped socket. The held socket exchanged another message after the switch and retained the same peer-certificate fingerprint. That result does not mean it authenticated the new certificate: it performed no new handshake.
Connection lifecycle — before rotation: client context A → full handshake with old leaf → established socket. After listener rotation: established socket → another successful exchange, still associated with the old peer certificate. Separate path: fresh socket + context A → new B chain → rejection. After explicit context rebuild: fresh socket + context B → new B chain → acceptance. Treat these as separate observations rather than one global “gateway healthy” signal.
The DNS and connection-pool failover experiment already showed why a new DNS answer does not move an existing connection. This experiment adds the cryptographic boundary that the earlier cleartext test explicitly did not exercise. It does not repeat DNS timing, pool implementation or backend-discovery claims.
Handshake acceptance matrix: observed results
| Connection / fault | Observed result | Interpretation |
|---|---|---|
| Fresh old server; trust A | Accepted; not resumed | Baseline |
| Held socket after rotation | Exchange succeeds; same certificate | No new handshake |
| Fresh new server; trust A | Rejected: code 20 | B chain not trusted |
| Bundle file B; existing context A | Rejected: code 20 | File change is not context reload |
| Fresh new server; rebuilt context B | Accepted; not resumed | Intended B trust loaded |
| Missing intermediate; trust B | Rejected: code 20 | Served chain incomplete |
| Wrong SAN; trust B | Rejected: code 62 | Trusted chain, wrong identity |
| Old server; rebuilt context B | Rejected: code 20 | A no longer accepted |
| Old and new servers; trust A+B | Both accepted; not resumed | Temporary overlap demonstrated |
The issuer-related failures returned verification code 20, “unable to get local issuer certificate”; the wrong SAN returned code 62, hostname mismatch. Those codes describe this run. The same code can arise from different chain-building problems, so do not infer “missing root” from code 20 alone. Inspect the served intermediates and the intended trust anchors together.
For the bundle-file test, the fixture overwrote active.pem from A-only to B-only, then attempted a new connection using the existing context. That attempt still failed. A newly constructed context reading the same file accepted the B chain and rejected the A server. This is why the test uses context replacement rather than claiming that another load call removes old trust. A temporary A+B context accepted both servers; that demonstrated an overlap option, not an instruction to retain retired authorities indefinitely.
Verify hostname and validity independently
The offline control used openssl verify with an explicit root file, an untrusted intermediate file, sslserver purpose and strict X.509 checking. It disabled default CA directory/store lookup. The good B chain passed with -verify_hostname model.test. The other.test leaf passed chain verification without a hostname option, then failed when the model.test hostname check was added. Chain success alone was therefore insufficient evidence of service identity in this fixture.
The OpenSSL verify reference documents chain inputs, hostname verification and -attime. The test evaluated the same new leaf at two controlled verification times: two days before and two days after the fixture reference time. Its leaf validity extended only one day on either side; CA validity extended thirty days. The first check failed at leaf depth with code 9, “certificate is not yet valid”; the second failed with code 10, “certificate has expired”. The host clock was not changed.
These were offline path-validation tests, not network handshakes at simulated times. They isolate validity without changing the machine's time or falsely claiming that a live SDK accepted an injected clock. Likewise, the fixture did not fetch or validate OCSP responses or CRLs. Trusted issuer, matching SAN and valid dates do not establish a tested revocation policy.
Operational decision table
| Observation | Next discriminating check | Release decision |
|---|---|---|
| Old stream works; fresh handshake fails | New peer chain and loaded trust | Do not approve from stream success |
| New root file; failures persist | Context generation and reload boundary | Rebuild and retest, not bypass |
| Issuer error with expected root | Served intermediate and chain construction | Repair chain before rollout |
| Trusted chain; hostname mismatch | Reference identity versus DNS SAN | Reject wrong service identity |
| Offline validity failure | Leaf window and intended verification time | Repair issuance or timing |
| Need to remove old CA | Fresh context rejects old-only chain | Bound overlap and retire trust |
A production acceptance record should identify the client runtime and SDK, proxy or service mesh, bundle digest, context generation, connection creation time, reference identity, peer-certificate fingerprint, verification outcome and whether a session was resumed. Keep prompts, API credentials and private keys out of this diagnostic record. Use bounded failure classes in metrics; detailed certificate evidence belongs in controlled logs or release artifacts.
Tie the test to user-visible inference SLOs: successful old streams cannot be the denominator for new-connection health. Track both cohorts during the change. Certificate verification errors are not a reason for unlimited reconnects; the retry-budget operating model provides the separate containment boundary. Retrying an unchanged invalid identity does not repair it.
A rollout checklist with explicit limits
1. Inventory the actual client trust source and reload mechanism. A system bundle, embedded SDK bundle and explicit application bundle may differ. Reproduce the deployment's effective path rather than testing an administrator's shell alone.
2. Stage the intended chain and reference identity in an isolated environment. Require positive controls plus untrusted issuer, missing intermediate and wrong-SAN rejection. Keep certificate verification enabled throughout.
3. If policy permits a temporary A+B trust overlap, define its owner and removal condition. Adding an authority broadens accepted issuers. It is not a harmless availability toggle, and it is inappropriate when policy requires immediate distrust of A.
4. Build and verify a new client context or use the documented reload mechanism. Prove new connections use it before sending new work. Do not infer reload from a changed file hash alone.
5. Stop assigning work to retired pools and drain active streams under explicit deadlines. Forced connection cycling costs handshakes and can interrupt partial generations. Emergency distrust may require terminating existing sessions; that is a separate risk decision, not something this availability-focused fixture proves safe.
6. Test resumption, mTLS, revocation, proxy hops and the actual SDK separately where used. A full handshake test with tickets disabled does not establish how resumed sessions behave across rotation. The fixture also did not measure HTTP/2 multiplexing, cross-signed chain selection or production reload races.
7. Retire old trust through a replacement configuration and prove the old-only chain is rejected on a fresh connection. Preserve an approved rollback boundary without silently restoring an authority that security policy has withdrawn.
The trade-off is explicit: long-lived connections reduce setup cost but can conceal new-handshake failures; aggressive retirement exposes the new configuration sooner but creates churn and can disrupt useful work. The experiment establishes acceptance decisions, not a latency improvement or a universal drain timeout.
Make the next certificate change an evidence-producing release
Before rotating an inference gateway, bring one client/pool configuration, the proposed certificate-chain manifest and the intended trust-removal policy to an implementation review with Ade Christanto. The practical deliverable is a reproducible positive/negative handshake matrix, a documented context-reload boundary and a stream-safe retirement plan. A green request over yesterday's connection is not the release gate.
