Disaster Recovery Runbook Template and Worked Example

A disaster recovery runbook is an executable recovery procedure. It tells an incident team what failed, who may choose the recovery path, what evidence must be captured, which commands are safe, when to stop, and how to prove the customer-facing service is working again.

A backup dashboard is not recovery evidence. Recovery becomes credible only after a clean restore or controlled failover produces a measured recovery point, elapsed time, technical validation, business validation, and an owned list of gaps.

Download the complete DR package

• DR runbook template — owners, RPO/RTO, triggers, commands, stop conditions, validation, communication, and evidence.
• Recovery validation checklist — infrastructure, application, data, and business checks.
• Incident timeline template — detection, decisions, execution, recovery, and follow-up ownership.
• Public DR runbook repository — templates, drill report, readiness scoring utility, tests, CI, and release packaging.

Copy the package into the same version-controlled environment as the service configuration. Replace all placeholders and execute a drill before treating the runbook as approved.

Runbook, DR plan, and incident playbook

These documents have different jobs:

A single large document usually fails under pressure. Keep the incident coordination model short, and maintain executable runbooks per critical service or recovery path.

Minimum DR runbook contract

Every production runbook should contain:

• service and business impact;
• recovery tier, RPO, and RTO;
• incident commander, recovery operator, validator, and communication owner;
• triggers for failover, restore, or rebuild;
• explicit stop conditions;
• dependency map and recovery order;
• approved recovery sources;
• exact commands with expected output;
• routing and application reconnect behavior;
• technical, data, and business validation;
• rollback or fallback path if recovery fails;
• evidence location and next drill date.

Use a decision gate before any mutating action:

Failure confirmed: yes / no
Customer impact: ...
Current writable system: ...
Candidate recovery source: ...
Candidate recovery point: ...
Meets RPO: yes / no / unknown
Estimated recovery time: ...
Meets RTO: yes / no / unknown
Unsafe writes stopped: yes / no / not required
Decision owner: ...
Decision: failover / restore / rebuild / continue diagnosis
Stop conditions: ...

Worked example: PostgreSQL primary failure in Kubernetes

This example assumes:

• PostgreSQL is managed by Patroni;
• etcd or another distributed configuration store is healthy;
• applications connect through PgBouncer and HAProxy or an equivalent routing layer;
• Kubernetes hosts application workloads, not necessarily the database;
• WAL archiving and base backups exist;
• the agreed RPO is five minutes and RTO is thirty minutes.

Adapt every command and threshold to the real topology.

1. Declare roles and preserve evidence

Incident commander: controls scope, risk, and communication
Database recovery operator: executes PostgreSQL actions
Platform operator: controls routing and application workloads
Business validator: validates the customer transaction
Scribe: records timestamps, commands, outputs, and decisions

Record:

• detection time;
• current customer impact;
• recent releases and infrastructure changes;
• Patroni cluster view;
• last known primary;
• replica lag;
• backup and WAL status;
• application error and connection metrics.

Do not restart all nodes or delete pods before preserving state. Destructive troubleshooting can remove the evidence needed to choose the safest recovery path.

2. Run read-only PostgreSQL and Patroni checks

patronictl list
curl -fsS http://127.0.0.1:8008/cluster
systemctl status patroni --no-pager
journalctl -u patroni --since '-30 minutes' --no-pager

On each reachable PostgreSQL node:

select pg_is_in_recovery() as is_replica;
select now() as checked_at,
       pg_last_xact_replay_timestamp() as last_replayed_at,
       now() - pg_last_xact_replay_timestamp() as replay_delay;
select application_name, client_addr, state,
       sent_lsn, write_lsn, flush_lsn, replay_lsn,
       write_lag, flush_lag, replay_lag
from pg_stat_replication;

Stop and escalate when:

• more than one node appears writable;
• consensus health is unknown;
• no replica meets RPO;
• timeline history is inconsistent;
• the recovery target is not understood;
• current writes could destroy forensic or recovery evidence.

3. Choose failover or restore

A replica is not a backup. Logical errors, deletes, and corruption may replicate successfully.

4. Execute a controlled failover

Only after the decision gate passes:

patronictl failover CLUSTER_NAME \
  --candidate CANDIDATE_NODE \
  --force

Prefer a planned switchover when the old primary is still healthy and the goal is maintenance or rehearsal:

patronictl switchover CLUSTER_NAME \
  --leader CURRENT_PRIMARY \
  --candidate CANDIDATE_NODE

Immediately verify cluster role:

patronictl list
curl -fsS http://CANDIDATE_NODE:8008/primary
curl -fsS http://OLD_PRIMARY:8008/replica

Do not continue if the old primary still accepts writes without an understood fencing mechanism.

5. Validate HAProxy and PgBouncer routing

Example checks:

curl -fsS http://HAPROXY_ADMIN_OR_HEALTH_ENDPOINT
psql 'host=PGBOUNCER_HOST port=6432 dbname=APP_DB user=CHECK_USER' \
  -X -v ON_ERROR_STOP=1 \
  -c "select inet_server_addr(), pg_is_in_recovery(), now();"

Validate:

• write endpoint reaches the new primary;
• read endpoint does not send unsafe writes to replicas;
• PgBouncer clears or recycles unusable server connections;
• applications reconnect within the expected retry window;
• connection storms do not exhaust PostgreSQL;
• monitoring, backups, and replication follow the new topology.

Patroni, PgBouncer, and HAProxy solve different problems. Patroni manages leadership, PgBouncer manages connection pressure, and HAProxy or another router directs clients to the correct role.

6. Recover Kubernetes workloads in dependency order

Do not restart all deployments simultaneously. Use a defined sequence:

1. Confirm identity, secrets, DNS, network, and storage access.
2. Confirm the writable PostgreSQL endpoint.
3. Confirm queues, object storage, and external dependencies.
4. Restart or roll application APIs in controlled batches only if needed.
5. Start workers after the database and queues are safe.
6. Run technical and business validation.
7. Re-enable normal traffic and scheduled processing.

Useful checks:

kubectl get nodes -o wide
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get events -A --sort-by=.metadata.creationTimestamp | tail -100
kubectl rollout status deployment/app -n production
kubectl get endpoints,endpointslices -n production

7. Validate recovery at four levels

Infrastructure

• intended primary is the only writable node;
• replicas are following the correct timeline;
• storage, network, certificates, DNS, and secrets are available;
• backup and monitoring jobs resumed.

Application

• readiness and health endpoints succeed;
• authentication works;
• dependency calls succeed;
• error rate and p95/p99 latency return toward baseline;
• queues and workers do not accumulate repeated failures.

Data

• recovered timestamp or LSN is recorded;
• critical record counts and constraints are valid;
• required recent transactions are present or accepted as lost within RPO;
• no unexpected duplicate processing occurs.

Business

• a user can log in;
• the critical customer transaction completes;
• product or service owner confirms expected behavior;
• support confirms customer impact is falling.

A successful psql connection is not full recovery evidence.

Worked drill report

Record the outcome in a compact, reviewable form:

Service: checkout-platform
Scenario: PostgreSQL primary host loss
Started: 2026-08-02T09:00:00Z
Detection confirmed: 09:01
Failover approved: 09:05
New primary writable: 09:09
Application routing restored: 09:12
Business transaction passed: 09:16
Exercise completed: 09:20

Target RPO: 5 minutes
Measured RPO: 42 seconds
Target RTO: 30 minutes
Measured RTO: 16 minutes

Evidence:
- Patroni cluster snapshots before and after
- replication and timeline queries
- HAProxy and PgBouncer routing checks
- application latency/error dashboard
- synthetic checkout transaction

Gaps:
- worker reconnect exceeded target by 70 seconds
- backup monitor followed old primary for four minutes

Owners and deadlines:
- Platform team / worker retry change / 2026-08-16
- DBA / backup monitor role discovery / 2026-08-09

The values above are an example format, not a claim about a real SteadyOps customer or production system.

Recovery communication timeline

Use one source of truth and update it on a fixed cadence:

Avoid declaring recovery solely because infrastructure is green. State whether customer validation and data checks are complete.

Restore path requirements

A restore runbook must specify:

• immutable or protected backup source;
• base backup identifier and integrity result;
• required WAL range or snapshot chain;
• clean target environment;
• point-in-time target;
• secret and encryption-key recovery;
• routing isolation during restore;
• validation before writes are enabled;
• fallback when the selected source is incomplete.

Measure restore time from decision to validated service, not only database process startup.

DR drill cadence

Use risk to set cadence:

• critical data and revenue paths: quarterly or after material architecture change;
• important internal systems: at least twice per year;
• backup-only systems: periodic clean restore plus application validation;
• every major incident: rerun the affected recovery path after fixes.

A tabletop exercise checks understanding. A clean restore or controlled failover checks reality. Mature programs use both.

Related SteadyOps reading

• PostgreSQL at Scale — Patroni, PgBouncer, connection control, failover, backup, and restore.
• Kubernetes Production Readiness Checklist — failure tolerance, releases, observability, and recovery controls before launch.
• Kubernetes Rollback Checklist — when application rollback is safer than disaster recovery.
• Security Evidence Operations Model — incident timelines and recovery evidence as operational records.

Key takeaways

• A DR runbook is an executable, versioned, and tested procedure.
• Failover, restore, and infrastructure rebuild are different decision paths.
• RPO and RTO must be measured during a drill, not inferred from backup status.
• Patroni leadership, client routing, connection pooling, and application retry behavior must be validated together.
• Recovery is complete only after data and customer-facing business checks pass.
• Every drill should produce evidence, gaps, owners, and a next test date.

# Disaster Recovery Runbook

## Service and impact
- Service: <name>
- Business impact: <customer journey>
- Severity: <SEV-1/SEV-2>
- RPO: <minutes>
- RTO: <minutes>

## Ownership
- Incident commander: <role>
- Recovery operator: <role>
- Business validator: <role>

## Trigger and stop conditions
- Trigger: <measurable condition>
- Do not continue when: <data corruption / unknown primary / missing backup>

## Recovery steps
1. Freeze risky writes or traffic.
2. Capture current state and timestamps.
3. Validate backup and recovery data availability.
4. Restore or fail over using the approved procedure.
5. Reconnect dependencies in documented order.
6. Run technical and business smoke tests.

## Validation
- Health endpoint: <URL>
- Database consistency query: <query>
- Critical transaction: <test>
- Monitoring returned to baseline: <dashboard>

## Communication timeline
- Detected:
- Mitigation started:
- Recovery completed:
- Business validation completed:

## Follow-up
- Evidence location: <link>
- Action items: <tickets>

patronictl list
psql -X -v ON_ERROR_STOP=1 -c "select pg_is_in_recovery();"
psql -X -v ON_ERROR_STOP=1 -c "select now(), current_database();"
psql -X -v ON_ERROR_STOP=1 -c "select count(*) from pg_stat_activity;"
curl -fsS https://service.example.com/health