Disaster Recovery Runbook Template for Production Systems

A disaster recovery runbook is an executable recovery procedure, not a long policy document. It tells an engineer what failure has occurred, who owns the decision, which checks come first, when to stop, how to restore or fail over, and how to prove that the customer-facing service is healthy again.

This guide is designed for CTOs, SREs, database owners, and platform engineers who need a usable DR runbook example for PostgreSQL, Kubernetes, web applications, queues, object storage, and supporting infrastructure. It can be copied into Git, Notion, Confluence, or an incident-management repository, but the commands and thresholds must be adapted and tested against the real environment.

The most important rule is simple: a backup report is not recovery evidence. Recovery becomes credible only after a clean restore or failover drill produces a measured recovered point, elapsed time, technical validation, and business validation.

How to use this disaster recovery runbook template

Start with one critical service, not the entire company infrastructure. Choose the system whose outage, data loss, or prolonged degradation would create the largest customer or business impact. Then complete the template in this order:

1. Define service impact and severity.
2. Agree RPO and RTO with the business owner.
3. Name the incident commander, recovery operator, and business validator.
4. Document dependencies and recovery order.
5. Write exact checks, commands, expected results, and stop conditions.
6. Run a drill in a clean environment.
7. Record evidence and convert every gap into an owned action.

A reusable Markdown skeleton and configuration examples are included in the practical implementation sections below. The same template is also available in the SteadyOps disaster recovery runbook repository.

What every DR runbook must contain

A production runbook should be short enough to use under pressure and precise enough to prevent dangerous improvisation. At minimum, it needs:

• Service name, business impact, and incident severity.
• Recovery trigger and explicit stop conditions.
• RPO and RTO.
• Incident commander and recovery operator.
• Dependency inventory and recovery order.
• Backup, snapshot, WAL, or replica source.
• Exact commands with placeholders and expected output.
• Technical health checks.
• Critical business transaction validation.
• Communication timeline and stakeholder owner.
• Evidence location, follow-up actions, and next drill date.

The most common failure is writing a runbook as prose. During an outage nobody wants to interpret a ten-page wiki article. The procedure must make the next safe action obvious: capture state, verify the recovery source, stop unsafe writes, restore or promote, reconnect dependencies, validate, and communicate.

Define RPO and RTO before choosing tools

Many teams begin with products: Patroni, etcd, Kubernetes operators, snapshots, object storage, cloud backup services, or database replicas. Those tools matter, but the recovery contract begins with business tolerances.

RPO defines the maximum acceptable data loss. RTO defines the maximum acceptable recovery time. A service with a 30-minute RPO can use a different architecture from a payment or authentication system that requires near-zero data loss.

Without these answers, every incident becomes a debate. With them, the team can compare architectures and make recovery decisions against agreed business risk.

Separate high availability failover from disaster recovery

High availability and disaster recovery solve different failure modes.

• HA failover usually promotes an already-running healthy replica after primary or infrastructure failure.
• DR restore rebuilds service from backup, WAL, snapshots, object storage, infrastructure code, and secrets after corruption, deletion, regional loss, or broader compromise.

Treating a replica as the only backup is dangerous because corruption or accidental deletes can replicate. Treating restore from backup as a normal application rollback is also dangerous because it is slower, affects data state, and may require coordinated downtime.

The runbook should state which path applies to each trigger:

PostgreSQL disaster recovery procedure

PostgreSQL recovery needs more than pg_dump or a snapshot. The runbook must cover primary failure, unsafe replica lag, corrupted data, accidental deletes, missing WAL, credential recovery, routing, and application validation.

First capture current state with read-only checks:

patronictl list
systemctl status patroni
curl -fsS http://127.0.0.1:8008/cluster
psql -X -v ON_ERROR_STOP=1 -c "select pg_is_in_recovery();"
psql -X -v ON_ERROR_STOP=1 -c "select now() - pg_last_xact_replay_timestamp() as replica_lag;"

Before promotion or restore, confirm:

• Which node is currently writable.
• Whether any replica satisfies the RPO.
• Whether the latest base backup completed.
• Whether required WAL files exist.
• Which client route will point to the recovered primary.
• How application retries behave during the switch.
• Whether backup and monitoring jobs will continue afterward.

After recovery, validate database state and a real application operation. A successful psql connection is necessary but not sufficient.

Kubernetes and service recovery order

Kubernetes can restart workloads, but it does not automatically restore databases, queues, secrets, object storage, external DNS, or application consistency. A service-level DR runbook must document dependency order.

A common order is:

1. Recover network, identity, secrets, and required storage access.
2. Recover the database or confirm the healthy primary.
3. Recover queues and object storage.
4. Start APIs in a controlled mode.
5. Start workers only after their dependencies are safe.
6. Run technical health checks.
7. Run a critical business transaction.
8. Re-enable normal traffic and background processing.

Useful initial checks include:

kubectl get nodes -o wide
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get events -A --sort-by=.metadata.creationTimestamp | tail -50
kubectl rollout status deployment/app -n production
kubectl get pvc -A

Do not restart every component simultaneously. Random restart order can create a second incident through queue duplication, connection storms, stale cache, or workers processing data before the database is consistent.

Recovery validation must include the business path

A runbook is incomplete if it ends with “pods are Running” or “database accepts connections.” Recovery must be proved at three levels:

Infrastructure validation

• Required nodes, storage, network, and certificates are healthy.
• Only the intended primary is writable.
• Replication, backups, and monitoring resumed.
• Queues and workers are not accumulating errors.

Application validation

• Health and readiness endpoints succeed.
• Error rate and p95/p99 latency return to baseline.
• Authentication and dependency calls work.
• Logs contain no repeated recovery-related failures.

Business validation

• A user can log in.
• A customer can complete the critical transaction.
• The recovered data point is acceptable.
• Support or the business owner confirms that customer impact is falling.

The business validator should be named before the incident. SREs can prove infrastructure health, but only the product or service owner can confirm that the recovered behavior is correct for customers.

Incident communication and recovery evidence

The runbook should define one source of truth and a simple timeline:

• Detection time.
• Impact confirmation.
• Mitigation start.
• Failover or restore decision.
• Recovery completion.
• Technical validation.
• Business validation.
• Follow-up actions.

Store commands, timestamps, selected outputs, recovered point, restore duration, screenshots or dashboard links, and approvals in the incident record. This evidence supports postmortems, customer communication, audits, and future runbook improvement.

DR runbook decision matrix

Related SteadyOps reading

• PostgreSQL at Scale — replication, Patroni, connection control, backup, and restore behavior.
• Kubernetes Production Readiness Checklist — controls that should exist before a production launch.
• Kubernetes Rollback Checklist — when the correct response is application rollback rather than disaster recovery.
• Security Evidence Operations Model — recovery records, access events, and incident timelines as operational evidence.

Key takeaways

• A disaster recovery runbook is an executable and tested procedure.
• RPO, RTO, owners, triggers, and stop conditions must exist before an incident.
• HA failover and DR restore are different recovery paths.
• Database connectivity is not enough; validate the real customer transaction.
• A clean restore drill is the evidence that turns backups into a recovery strategy.
• Every drill should produce measured results and owned improvements.

# Disaster Recovery Runbook

## Service and impact
- Service: <name>
- Business impact: <customer journey>
- Severity: <SEV-1/SEV-2>
- RPO: <minutes>
- RTO: <minutes>

## Ownership
- Incident commander: <role>
- Recovery operator: <role>
- Business validator: <role>

## Trigger and stop conditions
- Trigger: <measurable condition>
- Do not continue when: <data corruption / unknown primary / missing backup>

## Recovery steps
1. Freeze risky writes or traffic.
2. Capture current state and timestamps.
3. Validate backup and recovery data availability.
4. Restore or fail over using the approved procedure.
5. Reconnect dependencies in documented order.
6. Run technical and business smoke tests.

## Validation
- Health endpoint: <URL>
- Database consistency query: <query>
- Critical transaction: <test>
- Monitoring returned to baseline: <dashboard>

## Communication timeline
- Detected:
- Mitigation started:
- Recovery completed:
- Business validation completed:

## Follow-up
- Evidence location: <link>
- Action items: <tickets>

patronictl list
psql -X -v ON_ERROR_STOP=1 -c "select pg_is_in_recovery();"
psql -X -v ON_ERROR_STOP=1 -c "select now(), current_database();"
psql -X -v ON_ERROR_STOP=1 -c "select count(*) from pg_stat_activity;"
curl -fsS https://service.example.com/health