Kubernetes Rollback Checklist for Production Deployments

A Kubernetes rollback should not be invented during an incident. The team should know what signal triggers rollback, who owns the decision, which command is safe to run, and how recovery will be validated after traffic returns to the previous version.

This checklist is for production deployments where customer impact matters: web APIs, SaaS services, background workers, payment flows, authentication paths, and PostgreSQL-backed applications. Use it before a release, during a failed rollout, and after the incident review.

Rollback readiness before deployment

A rollback is only safe when the previous version is known, available, and compatible with current data and traffic.

Before a production deployment, confirm:

• the previous container image or artifact still exists in the registry
• Kubernetes rollout history is available for the workload
• database migrations are backward-compatible
• feature flags can disable risky behavior without a full redeploy
• readiness and liveness probes are meaningful
• the deployment owner and rollback owner are known
• dashboards show error rate, latency, restarts, saturation, queue depth, and database connections
• the rollback command was tested in stage or a production-like environment

For a standard Kubernetes Deployment, check the current revision before the release:

kubectl rollout history deployment/my-service -n production
kubectl rollout status deployment/my-service -n production

If your release process cannot identify the previous revision quickly, rollback is not ready enough for a high-risk deployment.

Rollback decision matrix

Use objective criteria. The team should not debate basic rollback rules while customers are already affected.

A useful rule: if the release changed database structure, message format, authentication behavior, or payment logic, rollback needs a human decision with evidence, not only an automatic command.

Standard Kubernetes rollback commands

For a regular Deployment, the basic rollback path is:

kubectl rollout status deployment/my-service -n production
kubectl rollout history deployment/my-service -n production
kubectl rollout undo deployment/my-service -n production
kubectl rollout status deployment/my-service -n production

Rollback to a specific revision:

kubectl rollout undo deployment/my-service -n production --to-revision=12

Pause and resume a rollout:

kubectl rollout pause deployment/my-service -n production
kubectl rollout resume deployment/my-service -n production

Inspect pods and recent events while the rollback is happening:

kubectl get pods -n production -l app=my-service -o wide
kubectl describe deployment/my-service -n production
kubectl get events -n production --sort-by='.metadata.creationTimestamp' | tail -50

Do not declare success when the command exits. Declare success only after the service is healthy from the user path.

Helm rollback commands

If the service is deployed with Helm, first inspect release history:

helm history my-release -n production

Rollback to a known good revision:

helm rollback my-release 12 -n production

Then validate Kubernetes state:

kubectl rollout status deployment/my-service -n production
kubectl get pods -n production -l app=my-service

Helm rollback can reapply manifests, but it does not automatically solve incompatible database migrations, message schema changes, or external dependency changes. Keep those risks in the rollback runbook.

Argo Rollouts commands

If you use Argo Rollouts for canary or blue/green delivery, rollback decisions often happen before 100% traffic is shifted.

Check rollout state:

kubectl argo rollouts get rollout my-service -n production

Abort a bad rollout:

kubectl argo rollouts abort my-service -n production

Promote only when analysis and smoke checks are clean:

kubectl argo rollouts promote my-service -n production

For progressive delivery, connect analysis templates to meaningful signals: 5xx rate, latency, queue depth, failed login rate, checkout success, database saturation, and any business-critical endpoint.

Post-rollback validation checklist

After rollback, validate both technical and customer-facing behavior.

Technical checks:

• service returns HTTP 200 on health and readiness endpoints
• pods are Ready and not restarting
• 5xx rate returned to baseline
• p95/p99 latency returned to expected range
• database connections are stable
• queue depth is not growing unexpectedly
• logs no longer show the release-specific error
• background workers are processing jobs normally

Example checks:

curl -fsS https://api.example.com/health
kubectl get pods -n production -l app=my-service
kubectl logs -n production deploy/my-service --tail=100

Business checks:

• login or authentication flow works
• the main customer action works
• payment, checkout, booking, or lead form works if relevant
• internal users can complete the critical workflow
• support or monitoring confirms customer impact is falling

A rollback that only fixes pod status but leaves the customer path broken is not complete.

Database and migration safety

Kubernetes rollback is simple when the release is stateless. It is harder when code and data changed together.

Safer deployment pattern:

1. Add backward-compatible database changes first.
2. Deploy code that can work with old and new data shapes.
3. Backfill data separately.
4. Switch reads after validation.
5. Remove old columns or paths in a later release.

Avoid:

• dropping columns used by the previous version
• renaming fields without compatibility layer
• irreversible migrations in the same release as application changes
• changing event/message format without worker compatibility
• assuming restore from backup is a normal rollback path

If rollback requires database restore, treat it as disaster recovery, not a normal deployment rollback.

Common rollback anti-patterns

• Using latest tags in production, making the previous version unclear
• Missing kubectl rollout history because deployments are patched inconsistently
• Probes that say healthy before the app can serve real traffic
• Rollback command exists but nobody tested it
• Alerts fire on noisy CPU metrics instead of user impact
• Database migration breaks the previous application version
• Background workers process messages from a newer incompatible version
• The incident channel has no clear decision owner

The fix is boring operational discipline: versioned artifacts, tested rollback commands, objective rollback criteria, and a runbook that engineers can follow under pressure.

Related SteadyOps reading

• Zero-Downtime Blue/Green Deployments — deployment safety, traffic switching, and rollback criteria.
• Kubernetes Production Readiness — production controls that make rollback safer.
• HA & DR Runbooks — when rollback becomes a recovery process instead of a normal release action.
• PostgreSQL at Scale — database behavior that often decides whether rollback is safe.

Key takeaways

• A rollback plan must exist before deployment starts.
• kubectl rollout undo, helm rollback, and argo rollouts abort should be documented and tested.
• Rollback decisions need objective thresholds, not panic.
• Database migrations are the most common reason application rollback becomes unsafe.
• Recovery is complete only after the customer-facing path is validated.

kubectl rollout history deployment/api -n production
kubectl rollout undo deployment/api -n production --to-revision=12
kubectl rollout status deployment/api -n production --timeout=5m
kubectl get pods -n production -l app=api -o wide
kubectl get events -n production --sort-by=.metadata.creationTimestamp | tail -50