A restored workflow database is not a restored business process. Recover checkpoints into an isolated environment, keep outbound actions disabled, and reconcile every potentially executed operation against the destination before resuming. PostgreSQL recovery can rewind local records; it cannot rewind an order, approval or ticket already committed in another system. A restore drill passes only when it demonstrates both recoverable state and controlled business effects.
The failure: your checkpoint is older than the world
Consider an illustrative maintenance workflow. At T0, the backup contains an approved request but no completed ticket. At T1, the workflow creates the ticket in the external service. At T2, the checkpoint recording that success is lost in an incident. Restoring T0 makes the request look actionable again. Starting the worker without reconciliation can create another ticket, even when the backup is perfectly consistent.
This is a disaster-recovery boundary, not simply another retry policy. Retry budgets limit attempts during execution. Historical restoration can also erase the local record of attempts, approvals, cancellations and deduplication decisions. An operation ledger restored to the same old point is useful evidence of intent, but cannot prove that nothing happened afterward.
The design question is: which system can authoritatively tell you whether the intended business action already happened? If that answer is unavailable, the safe state is unresolved, not ready to replay.
Define the recovery set before choosing a backup
Inventory the workflow database, application records, operation ledger, durable queue, long-term agent store, artifact manifests and external destinations. For each, record ownership, backup method, recovery point, version compatibility and the authority for reconciliation. Separate data needed to interpret history from credentials that could accidentally execute it.
The LangGraph persistence documentation distinguishes thread-scoped graph checkpoints from stores holding application-defined data across threads. Recovering the checkpointer alone does not establish that the corresponding store, retrieval data or tool schema is compatible. An in-memory checkpointer is not a backup mechanism; use persistence appropriate to the failure boundary you actually need to cover.
Keep the graph code, checkpoint serialization dependencies, database schema and model/configuration manifest identifiable. The LLM release bundle addresses artifact reproducibility; here it becomes an input to state recovery. Restoring old state into new code without compatibility tests is a migration, not a transparent restart. A hosted model revision may no longer be available, so exact regeneration of an earlier answer cannot be assumed.
Database consistency also has a scope. Independent databases, object storage and remote APIs do not acquire a shared recovery point merely because each has a green backup dashboard. Either coordinate their boundary or design explicit reconciliation for the mismatch. The orchestration architecture guide helps locate which layer owns state and which backend must enforce action controls.
SQL dump or base backup plus WAL?
PostgreSQL SQL dump documentation states that pg_dump produces an internally consistent snapshot as of the start of the dump. It covers one database; roles and tablespaces require separate handling. Even pg_dumpall does not synchronize snapshots across all databases. A dump is useful when its recovery point and restore duration fit your objectives, but it is not a base for WAL replay.
Continuous archiving and point-in-time recovery combine a physical base backup with the required sequence of archived WAL. They permit recovery to a selected point on a recoverable timeline. Missing required segments, an unusable base backup or incompatible server versions can invalidate that path. Configuration files such as postgresql.conf and pg_hba.conf are not recovered through WAL changes; preserve the relevant configuration separately.
| Recovery route | What it restores | Key dependency | What it does not prove |
|---|---|---|---|
| Consistent SQL dump | One database snapshot | Complete dump, roles and tested restore procedure | Cross-system consistency or remote-action rollback |
| Base backup plus archived WAL | Database cluster at a recoverable target | Compatible physical backup and required WAL chain | That every acknowledged remote effect is known locally |
| Workflow checkpoints plus store | Thread state and cross-thread application data | Compatible graph, serializers and store boundary | Safe replay of business actions |
| Reconciled operation evidence | Decision about each ambiguous action | Authoritative destination status and stable identity | Universal exactly-once execution |
Physical recovery generally demands closer server-version compatibility than logical dumps. Pin the procedure to the actual PostgreSQL major version and backup tooling. The source documentation supports the backup mechanisms; the workflow release gates in this article are engineering recommendations, not database guarantees.
Timeline: local recovery does not undo external effects
The diagram is an illustrative sequence, not a measured incident. T0 is the recovered local point, T1 is the external commit, and T2 is the loss of newer local evidence. The restored environment stays isolated while a read-only reconciliation path checks the external system. Only a reviewed subset can enter a separately authorized resume path.
Run an isolated restore drill
1. Declare the scenario and clocks. Identify the simulated failure time, target recovery point, affected workflows and business acceptance criteria. State whether the recovery-time objective includes reconciliation and release. Stop the clock at safe business service, not merely at a successful database connection. Record the database recovery point separately from missing business outcomes.
2. Fence before restoring. Provision a separate network and identity boundary with default-deny outbound access. Stop workers, schedulers, queue consumers and inbound triggers. Do not mount production write credentials. Check alternate routes through proxies, SDK endpoints, email, webhooks and message brokers. A setting inside the restored database is not a sufficient fence: the restore can replace it.
3. Prove isolation with a harmless test. Attempt a controlled outbound call toward a dedicated test sink and confirm denial from the restored worker identity. Retain policy and network evidence. A sandbox API endpoint can still send real notifications or incur cost if account boundaries are wrong; verify the receiving tenant as well as its URL.
4. Restore and verify integrity. Validate the backup identifier, database target and tooling version before execution. For a logical restore, make errors fatal and inspect the exit status rather than accepting a partially restored schema. For PITR, prove that recovery reached the intended target and stopped on the intended timeline. Check tables, application invariants and checkpoint decoding using the pinned runtime without starting action-capable workers.
5. Enumerate the uncertainty set. Include operations pending at the restored point, known business events after it, broker redeliveries and work that may have been created entirely after the backup. A scan of restored pending rows alone misses the last category. Preserve available newer evidence before destructive recovery; obtain destination audit history or a controlled change export where needed.
6. Reconcile through a separate read-only adapter. Match tenant, business operation ID, external object ID and input fingerprint. Capture the authoritative state, observation time and evidence reference. Do not treat a stale replica or an eventually consistent search miss as proof of absence. If the destination cannot establish the outcome, quarantine the operation for accountable review.
7. Release selectively, then observe. Record the approved operation cohort and current recovery generation outside the stale restored state. Disable the old worker fleet and stale queue consumers before enabling the new fleet. Enforce generation or fencing checks at the action boundary so an old lease cannot regain write authority. Re-enable only the necessary destination, credentials and workload rate; stop if unexpected effects appear.
Reconciliation decision table
| Restored local state | Destination evidence | Resume decision | Required control |
|---|---|---|---|
| Pending action | Exact matching effect is committed | Record reconciled completion; skip creation | Stable identity and verified result mapping |
| Pending action | Authoritative absence is established | Consider a newly authorized attempt | Current approval, valid input and destination safety contract |
| Pending action | Timeout, stale search or uncertain status | Hold for reconciliation | No automatic replay while outcome is unknown |
| Completed action | Remote object changed or was cancelled | Review current business intent | Never restore an old approval over a later cancellation |
| No local operation | Newer destination history shows an effect | Reconstruct evidence before scheduling | Account for post-backup work, not only restored rows |
| Any action | Idempotency retention or payload identity differs | Stop automatic reuse | Verify receiver contract; escalate unresolved cases |
Do not patch opaque framework checkpoint tables casually to mark work complete. Use supported state APIs or a reviewed migration that preserves graph invariants, and test the resulting next transition. Completion is a business assertion supported by destination evidence, not a convenient value written into a status column.
Idempotency can reduce duplicate effects only within the receiver's documented semantics. Preserve the original logical action identity; do not generate a new key merely because the recovery generation changed. Keep generation identity separate for fencing. A receiver that has expired its deduplication record may treat an old key as new. If the restore horizon exceeds that retention, reconciliation must carry the safety argument.
Measure the drill and inject the awkward cases
Record the actual recovered database point and the interval of missing local evidence. Compare that with the agreed recovery-point objective. Separately count known external effects, reconciled operations, unresolved outcomes and missing business events. A small database-loss window can still contain a costly duplicate action; one number cannot describe both risks.
Break recovery time into provisioning, data restoration, application validation, reconciliation and controlled release. Measure the full path against the recovery-time objective, retaining stage durations to identify the constraint. Reconciliation can dominate even when database recovery is fast. No universal recovery-time target follows from choosing PostgreSQL or LangGraph.
Exercise a missing required WAL segment, a checkpoint/runtime mismatch, a committed remote action with a lost acknowledgement, an expired deduplication record, a stale consumer that restarts, and an operation created after the backup. Expected behavior is explicit blocking or controlled reconciliation, not silent forward progress. Use destination stubs or isolated tenants that independently count attempts and effects; client logs alone cannot prove absence of duplicates.
The release checklist is concrete: isolation tested; backup and target identified; schema and checkpoint decoding verified; uncertainty set enumerated; every resumed action has a disposition; unresolved actions remain fenced; old writers cannot return; current authorization is checked; monitoring and a stop control are ready. Retain the runbook, environment manifest, restore logs, reconciliation decisions and signed release record together.
Limits, privacy and the next implementation step
An isolated drill costs infrastructure, operator time and destination integration work. Read-only reconciliation can hit API limits, lack historical coverage or depend on a vendor that is also unavailable. A conservative hold protects against duplicates but delays legitimate work. An independent ledger helps only if its retention, integrity and failure domain survive the incident; it is not inherently authoritative about a remote commit.
Restores may reintroduce data previously deleted or access previously revoked. Apply current retention, deletion and access decisions before releasing restored data or reconnecting workflows. Involve privacy, security and qualified legal reviewers where obligations are implicated. This is operational design guidance, not legal advice.
For your next architecture review, bring one stateful workflow, its backup inventory and the contract of one side-effecting API. I can help turn those inputs into a recovery-boundary map, a reconciliation adapter and a restore-drill acceptance gate. The useful deliverable is evidence that your system can recover without blindly repeating yesterday's business actions—not another green backup notification.
