An approval authorizes a proposed change to an observed state—not an unlimited right to overwrite whatever exists later. For Kubernetes-oriented AIOps remediation, bind the decision to the object's UID and observed resourceVersion, then make the destination enforce that precondition when it writes. If another actor has changed the object, stop and obtain a new decision. Refreshing the version and retrying under the old approval defeats the control.
This is a concurrency problem during normal operation. It is distinct from deciding which actions need human review, and from reconciling actions after a disaster-recovery restore. The practical deliverable is a conditional-write adapter, a conflict disposition and a test that makes a paused worker lose its authority safely.
The race happens after the reviewer clicks approve
Consider an illustrative controller that proposes changing a workload from two replicas to three. Worker A and worker B both read the same object. A reviewer approves A's proposal. Before A executes, B applies a separately authorized change to four replicas because the operational situation has changed. A then resumes with the old approval.
Both workers can hold valid credentials. Both proposals can have been reasonable at review time. An unconditional write from A still overwrites newer intent. A duplicate-action key does not solve this: A and B are different logical actions, not retries of one request.
The human-in-the-loop architecture guide establishes review gates and evidence packets. Add a state binding to that packet. The industrial AIOps operating boundary determines which remediation is permitted at all; optimistic concurrency does not expand that authority or make safety-critical OT actuation acceptable.
Bind the decision, not just the ticket
A production approval record should identify approval_id, reviewer identity and role, approved_at, expires_at, destination cluster identity, API group/resource/subresource, namespace, name, UID, observed resourceVersion, exact action digest, policy revision and evidence reference. Where your destination supports fencing, record the execution generation separately. A namespace and name identify a location; UID distinguishes the particular object from a replacement created at that location.
The digest must cover a canonical, typed representation of the operation: destination, method, affected fields and values. Specify serialization rules and reject ambiguous values before hashing. A digest detects a difference only if the approved digest is itself trusted. Keep approvals immutable in a protected store or verify a signed envelope; never let the action worker rewrite the version or digest inside an approval record. Authenticate the reviewer and verify that their authority has not been revoked.
Use a trusted clock for expiry. The exact boundary is a policy choice: our local fixture rejects at now greater than or equal to expires_at. A backend check immediately before sending a request cannot guarantee that expiry, policy or leadership still holds when a remote service eventually commits. Stronger guarantees require enforcement in the same destination-side consistency boundary as the write, or an explicitly accepted weaker contract.
resourceVersion is an opaque value for this workflow. Preserve and compare it; do not parse it as a sequence number, increment it or use arithmetic to decide whether an approval is nearly fresh. A watch cache is useful for diagnosis, not a substitute for a write precondition.
Make Kubernetes reject the stale state
The Kubernetes API concepts documentation describes resourceVersion-based lost-update detection for updates and conditional patches. For an existing object, a PUT carries the observed metadata.resourceVersion. A stale update can be rejected as a conflict. Do not catch that response, fetch the latest object and automatically resend the old human decision.
For a narrow change, JSON Patch can express ordered tests and the mutation in one request. For example, PATCH an existing Deployment with Content-Type application/json-patch+json and three operations: test /metadata/uid against the approved UID; test /metadata/resourceVersion against the approved opaque value; replace /spec/replicas with the approved integer. The path must already exist for replace. Construct this request from the protected approval, not from a fresh observation silently substituted by the worker.
Treat a failed JSON Patch test as a failed precondition, not as a guaranteed HTTP 409. Kubernetes Status details and response codes depend on the operation and failure. Test the actual API server and client combination; distinguish a state conflict from malformed input, authentication failure and transport uncertainty. Server-side apply field ownership is also not a replacement for binding approval to observed state.
Stock Kubernetes validates the object preconditions in that request; it does not automatically understand your reviewer, action digest, approval expiry, policy revision or leadership generation. Your authorization service or enforcement adapter must validate those additional constraints. If approval revocation or policy changes must be atomic with a resource mutation, design that boundary deliberately; separate API calls do not create a cross-object transaction.
A successful API write records desired state. It does not prove that the workload converged, that new replicas became ready or that the incident was resolved. Observe the resulting resource and define a separate operational acceptance condition. Do not repeatedly write the same desired state merely because readiness is slow.
The competing-writer timeline
The diagram is a synthetic interleaving, not an incident measurement. Both workers read version A. B commits version B while A is paused. A's original precondition fails. A read used to explain that conflict belongs to diagnosis; a new write requires a newly authorized proposal. Keeping those paths separate prevents a generic retry helper from converting a safe rejection into an overwrite.
Conflict-handling decision table
| Observation | Meaning | Required disposition | Evidence to retain |
|---|---|---|---|
| UID and version match; authorization valid | Candidate is eligible for a conditional attempt | Send the bounded mutation; verify result | Approval ID, request digest, response and resulting version |
| Version differs on the same UID | Observed state is stale | Hold; review a fresh proposal | Approved and current version plus state diff |
| Same name but different UID | Object was deleted and recreated | Reject old approval; re-identify the target | Old and new UID with destination identity |
| Approval expired or policy changed | Permission is no longer current | Stop before dispatch; obtain a new decision | Expiry or policy reason and decision owner |
| Worker resumes after leadership change | Old coordination state is insufficient | Reject at the enforcing boundary where supported | Execution generation and fence rejection |
| Timeout after request transmission | Commit outcome is unknown | Reconcile; do not create a fresh attempt blindly | Stable operation identity and destination evidence |
| Write accepted but workload not ready | Desired-state update is not operational recovery | Observe convergence or escalate | Accepted version, readiness and bounded observation window |
Exercise the interleavings locally
For this article, a standard-library Python fixture was executed with eight passing tests. It models a destination with a lock protecting checks and mutation, an opaque version, an object UID, a policy revision, an execution epoch, an action digest and an expiry. This is a deterministic local model, not a test against a Kubernetes cluster and not a benchmark. Simulated time and explicitly ordered worker steps make the failure reproducible without scheduler luck.
The two-worker case has B commit four replicas from the shared observation, then asserts that A receives CONFLICT and leaves four replicas intact. A separate simulated reviewer decision against the new observation allows a subsequent change. The fixture also checks a paused worker after leader change, deletion/recreation, exact expiry, a changed action, a changed policy and a committed write whose acknowledgement is lost.
The eighth test is a negative control: deliberately replace the approved version with the current version in an unprotected approval dictionary. The stale action then succeeds. That passing test demonstrates the vulnerability, not an acceptable execution path. It is why a hash field and a conditional API call are insufficient if the worker can modify the approval envelope.
The timeout case commits once, raises a simulated TimeoutError, and shows that resending the original precondition conflicts without another effect. This proves only the model's behavior. A matching current replica count in a real cluster does not prove which worker caused it; another controller may have produced the same value. Use authoritative operation/audit evidence where available and retain an unresolved state where causation cannot be established.
To reproduce the experiment in your own harness, use these assertions as the contract: every rejected case has zero effects; the stale worker cannot change B's result; a fresh authorized decision can proceed; the lost-ack case has one effect; and the negative control succeeds only when approval integrity is intentionally broken. Do not promote the model's combined policy-and-write lock into a claim about stock Kubernetes.
A lease is coordination, not a universal fence
Kubernetes Leases support coordination, including heartbeats and leader election. A process can pause, lose its lease and later resume with an already prepared request. Checking leadership and then calling a different system leaves a race between those two operations. A resourceVersion condition protects against a changed object; it does not, by itself, reject an old leader when the object has not changed.
A fencing design needs the receiving boundary to validate the execution generation and mutation together. Depending on the receiver, that might mean a monotonic fencing token or a serially enforced current-epoch check. The fixture uses the latter inside its lock. A token recorded only in the coordinator has no power over a destination that ignores it.
The etcd API documentation describes atomic compare-and-swap transactions over etcd keys. That transaction can protect coordinator state in etcd. It cannot atomically include an unrelated Kubernetes request, ticketing API or industrial controller. Do not manipulate Kubernetes' backing etcd directly to bypass its API; the example concerns the transaction boundary, not an alternative remediation route.
Idempotency and fencing answer different questions. Idempotency asks whether this logical action already ran under the receiver's deduplication contract. Fencing asks whether this actor or generation is still permitted to act. A state precondition asks whether the approved observation still holds. You may need all three, and none establishes universal exactly-once business effects. The workflow restore-drill guide covers reconciliation when local history has been rewound; this article addresses competing live writers.
Acceptance checklist before enabling remediation
1. Use an isolated cluster and a harmless test workload. Verify the exact UID and resourceVersion conditions against the deployed API, including the error shape for a failed JSON Patch test. No live-cluster integration test is claimed here.
2. Pause a worker after approval, change the object through a second identity, then resume the first. Confirm the destination rejects the stale mutation and independently observe that the newer state remains.
3. Delete and recreate the object under the same name. Reuse the old approval and confirm rejection. Test leadership expiry without an object change as well; it exposes systems that confuse version checking with fencing.
4. Inject a lost acknowledgement after commit. Ensure the workflow enters reconciliation rather than generating a new approval or idempotency key automatically. Test absent or unavailable audit evidence as an unresolved outcome.
5. Record attempted writes, conditional rejections, expired approvals, reauthorization outcomes and unresolved operations. Use bounded metric labels; keep detailed resource identities in access-controlled logs. Compare conflict counts with total conditional attempts, not with all alerts.
6. Review conflict cost. Whole-object version binding can reject a proposal after an unrelated status update. Narrower field predicates may improve availability but protect less of the reviewed context. Make that a reviewed policy change with tests, not an automatic retry optimization.
For an architecture review, bring one remediation action, its approval record and the destination's documented write contract. I can help turn those into a conditional-write adapter and a race-test acceptance gate. Where the receiver cannot enforce the necessary preconditions, keep automated execution disabled for that path or explicitly narrow the guarantee. A deliberate hold is operationally expensive, but it is more defensible than letting yesterday's approval overwrite today's decision.
