Kubernetes Production Readiness Checklist for SaaS Teams

A Kubernetes production readiness checklist is a go-live decision system, not a generic collection of best practices. It should tell a team whether a workload is safe to expose to customers, which missing controls block launch, which risks can be accepted temporarily, and what evidence proves the platform survives realistic failure.

The practical standard is higher than a successful kubectl apply. A production-ready workload must keep serving correctly during node loss, rolling deployment, dependency slowdown, certificate rotation, database failover, traffic spikes, and operator error. It must also have a tested rollback path and named owners for the decisions that cannot be automated safely.

Download the Kubernetes readiness assets

Use the files below as a review package rather than copying isolated YAML from the article:

• Production evidence checklist — controls, owners, validation, and evidence.
• PodDisruptionBudget example — a conservative starting point for replicated APIs.
• Node-drain validation plan — disruption steps and customer-path verification.
• Readiness validation script — read-only status, rollout, endpoint, and event checks.
• Public Kubernetes rollback repository — tested evidence collector, checklist, CI, and release packaging.

The files are reusable starting points. Replace namespaces, labels, thresholds, dependencies, and validation paths before using them in a real environment.

Executive go-live gate

A production review should classify every control as pass, accepted risk, or blocker. Avoid an unweighted checklist where a missing icon or label looks equivalent to an unsafe database migration.

A missing control may be accepted only with an owner, mitigation, deadline, and explicit business approval. Hidden risk is not an acceptable launch state.

Copyable workload checklist

A critical workload should be able to answer yes to the following:

• At least two replicas are scheduled across appropriate failure domains.
• CPU and memory requests reflect measured steady-state and peak behavior.
• Available node capacity can absorb one expected failure plus deployment surge.
• A PodDisruptionBudget protects voluntary disruption without blocking all maintenance.
• Readiness proves the pod can serve its real request path.
• Liveness detects a stuck process without restarting healthy pods during dependency slowdown.
• Startup probes protect slow initialization.
• Graceful shutdown removes traffic before termination and safely drains background work.
• Deployment strategy, maxSurge, and maxUnavailable fit actual capacity.
• Database and message changes remain compatible with the previous application version.
• Rollback criteria and the exact rollback command are agreed before release.
• p95/p99 latency, error rate, saturation, queue depth, dependency health, and version are visible.
• Every critical alert has an owner, severity, dashboard, runbook, and first safe action.
• Production access is named, revocable, least-privileged, and separate from staging.
• Backup, restore, secret recovery, and dependency recovery order have been tested.
• A real business transaction is validated after deployment and recovery.

YAML baseline for a critical API

The following example combines lifecycle behavior, rollout control, resources, and topology. Values are illustrative; the purpose is to show the controls that must be reviewed together.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
  namespace: production
spec:
  replicas: 3
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  selector:
    matchLabels:
      app: payment-api
  template:
    metadata:
      labels:
        app: payment-api
    spec:
      terminationGracePeriodSeconds: 45
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector:
            matchLabels:
              app: payment-api
      containers:
        - name: app
          image: registry.example.com/payment-api@sha256:REPLACE_ME
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 512Mi
          startupProbe:
            httpGet:
              path: /health/startup
              port: 8080
            failureThreshold: 30
            periodSeconds: 5
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            failureThreshold: 3
            periodSeconds: 5
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            failureThreshold: 3
            periodSeconds: 10
          lifecycle:
            preStop:
              exec:
                command: ["/bin/sh", "-c", "sleep 10"]

A sleep hook is not universal advice. It is useful only when the ingress or service-routing path needs time to stop sending new traffic. Validate termination behavior with real requests and long-running work.

A matching disruption budget:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-api
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: payment-api

Run a node-drain test before launch

A static manifest review cannot prove disruption behavior. Execute the test first in a controlled environment that accurately represents production routing and capacity.

Preconditions

1. Confirm the target node does not host an unprotected single-instance dependency.
2. Record deployment revision, replica distribution, PDB status, error rate, latency, and available capacity.
3. Start a continuous synthetic request against the critical customer path.
4. Confirm the rollback and abort commands.
5. Name the test owner and the person authorized to stop the exercise.

Read-only evidence

kubectl get nodes -o wide
kubectl get pods -n production -o wide
kubectl get pdb -n production
kubectl get deployment payment-api -n production -o yaml
kubectl get events -n production --sort-by=.metadata.creationTimestamp

Controlled disruption

kubectl drain NODE_NAME \
  --ignore-daemonsets \
  --delete-emptydir-data=false \
  --grace-period=45 \
  --timeout=10m

Stop immediately when customer errors exceed the agreed threshold, replacement pods cannot schedule, PDB behavior is unexpected, or a stateful dependency is at risk. Uncordon the node only after the failure state and evidence are captured.

Evidence to retain

• start and completion timestamps;
• node, workload, revision, and image digest;
• synthetic transaction result during disruption;
• p95/p99 latency and error-rate graph;
• pod termination and replacement timing;
• capacity before and after rescheduling;
• unexpected events and blocked evictions;
• owner, decision, outcome, and remediation issue.

Probes must model application behavior

Readiness, liveness, and startup probes solve different problems:

• Readiness removes a pod from traffic when it cannot serve safely.
• Liveness restarts a process that cannot recover without restart.
• Startup protects initialization from premature liveness failure.

Do not make liveness depend on PostgreSQL, Redis, DNS, or another remote service. A dependency incident can otherwise restart every healthy application pod and turn degradation into a full outage.

Readiness may include critical dependencies, but it must be designed carefully. If all pods become unready during a shared dependency slowdown, upstream routing and autoscaling behavior must remain understood and observable.

Capacity must cover failure and deployment together

Average utilization is not a safe capacity target. The cluster needs headroom for:

• one expected node or fault-domain failure;
• rolling-update surge;
• autoscaling delay;
• queue backlog drain;
• log and telemetry bursts;
• dependency recovery;
• abnormal but plausible customer demand.

Review both requested capacity and actual behavior:

kubectl top nodes
kubectl top pods -A
kubectl get hpa -A
kubectl describe deployment payment-api -n production
kubectl describe nodes

A deployment with three replicas is not highly available if all three fit only on one node, share one zone, or cannot be rescheduled after a node loss.

Release and database compatibility gate

Kubernetes rollback only helps when the previous application version remains compatible with shared state.

Before release, answer:

• Can the old version read data written by the new version?
• Can the new version run before a backfill completes?
• Are columns or message fields removed only after all consumers stop using them?
• Is the migration lock duration measured?
• Is rollout pause possible before all replicas update?
• Is the previous image digest retained?
• Does rollback include traffic, workers, scheduled jobs, and schema behavior?

Use expand/contract changes for destructive database evolution:

1. Add backward-compatible structures.
2. Deploy code that supports old and new forms.
3. Backfill with bounded rate and observability.
4. Switch reads or writes behind an explicit gate.
5. Remove old structures only after the rollback window expires.

Observability must support a decision

A dashboard is useful only when it answers what changed, who is affected, and whether to continue or roll back.

Minimum release view:

• request rate, error rate, p50/p95/p99 latency;
• saturation for CPU, memory, connections, queues, and storage;
• old and new application versions;
• readiness failures and restart reasons;
• ingress and dependency errors;
• deployment start, traffic change, rollback, and recovery markers;
• one synthetic business transaction.

Every critical alert should link to a runbook that states the first safe action and stop condition. “CPU high” without affected service, customer signal, owner, or response path is not a production-ready alert.

Security and access evidence

Review Kubernetes access as an operational control:

kubectl auth can-i --list --as system:serviceaccount:production:payment-api
kubectl get role,rolebinding,clusterrole,clusterrolebinding -A
kubectl get serviceaccount -A

Validate that:

• shared kubeconfigs and long-lived personal credentials are removed;
• service accounts have the smallest explicit permissions required;
• staging credentials cannot modify production;
• secret values are not stored in Git or plain manifests;
• privileged workloads, host access, and dangerous capabilities are controlled;
• break-glass access is logged, revocable, and tested;
• image provenance and vulnerability policy are enforced consistently.

Recovery is part of readiness

A workload is not production-ready when Kubernetes can recreate pods but the team cannot restore the service’s data and dependencies.

The readiness package must document:

• backup and recovery source;
• last clean restore result;
• RPO and RTO;
• dependency recovery order;
• secret, certificate, DNS, storage, queue, and database recovery;
• technical validation;
• customer-facing business validation;
• evidence location and next drill date.

Use the Disaster Recovery Runbook Template for service-level recovery and the Kubernetes Rollback Checklist for failed releases.

Acceptance record

Conclude the review with a versioned decision:

Decision: GO / CONDITIONAL GO / NO-GO
Workload: payment-api
Artifact: registry.example.com/payment-api@sha256:...
Reviewer: ...
Date: ...
Blockers: ...
Accepted risks: owner + deadline + mitigation
Evidence bundle: URL or repository path
Next validation date: ...

A conditional go-live is legitimate when the residual risk is explicit and owned. An undocumented assumption is not.

Related SteadyOps reading

• Kubernetes Rollback Checklist — release pause, migration compatibility, rollback, and evidence.
• Zero-Downtime Deployments — rolling, Blue/Green, canary, traffic switching, and rollback windows.
• Disaster Recovery Runbook Template — recovery ownership, RPO/RTO, restore, and business validation.
• PostgreSQL at Scale — connection pressure, Patroni, PgBouncer, failover, and restore.

Key takeaways

• Production readiness is a decision backed by tests and evidence, not a successful manifest apply.
• The highest-value controls are failure tolerance, lifecycle behavior, capacity, rollback compatibility, observability, recovery, and ownership.
• Node-drain and customer-path tests reveal gaps that static YAML review cannot.
• A previous application version is useful only while shared data and messages remain backward compatible.
• Accepted risks require an owner, mitigation, and deadline.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
  strategy:
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      terminationGracePeriodSeconds: 45
      containers:
        - name: api
          image: registry.example.com/api:1.4.2
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              memory: 512Mi
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /live
              port: 8080
            periodSeconds: 10
            failureThreshold: 3

kubectl get nodes -o wide
kubectl get deploy,pod,pdb,hpa -n production
kubectl auth can-i --list --as system:serviceaccount:production:api
kubectl rollout status deployment/api -n production
kubectl get events -n production --sort-by=.metadata.creationTimestamp | tail -50