Kubernetes Production Readiness

A Kubernetes production readiness checklist is not a generic best-practices document. It is a practical way to prove that your cluster can survive real production pressure: node loss, bad deploys, traffic spikes, DNS issues, certificate rotation, database failover, security events, and cost growth.

For a SaaS team, production readiness means the platform has clear SLOs, observable failure modes, tested rollback, controlled access, backup evidence, and enough capacity headroom to keep p95 and p99 latency stable. If these controls are missing, Kubernetes becomes a complex way to hide risk until the next incident.

This checklist is written from a DevOps/SRE point of view. Use it before a production launch, during a quarterly platform review, or before asking a Kubernetes consultant to audit your infrastructure.

1. Cluster architecture and high availability

A production Kubernetes cluster should not depend on one node, one zone, one ingress path, or one person who remembers how failover works. Start with the failure model. What happens if a worker node disappears? What happens if an availability zone is degraded? What happens if the ingress controller is restarted during peak traffic?

Minimum checks:

Control plane is managed or deployed with quorum-aware redundancy.
Worker nodes are spread across zones or fault domains.
Critical workloads use topology spread constraints or anti-affinity.
PodDisruptionBudgets protect important services during maintenance.
Cluster autoscaler has enough headroom for traffic bursts and node replacement.
Ingress/controller components have at least two healthy replicas.

Example PodDisruptionBudget for a critical API:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payment-api-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: payment-api

The important part is not the YAML itself. The important part is proving that maintenance, drain, and rollout events do not break availability.

2. Observability and alerting

You cannot operate what you cannot see. Kubernetes observability must cover the cluster, workloads, ingress, dependencies, and business-facing behavior. Node CPU is useful, but it does not tell you whether users are experiencing high latency or whether a database connection pool is saturated.

A production-ready setup should include:

Metrics for nodes, pods, deployments, HPA, ingress, and persistent volumes.
p95/p99 latency and error rate for critical APIs.
Centralized logs with request IDs or correlation IDs.
Alert routing with severity levels and clear owners.
Dashboards for SLOs, deployment health, and dependency saturation.
Runbook links from alerts.

Prometheus alerts should point to action. An alert like “CPU high” is weak. An alert like “checkout API p99 latency above SLO for 10 minutes after deploy” is operationally useful.

3. Security and access control

Production Kubernetes security starts with least privilege. The cluster should not have shared kubeconfigs, unmanaged admin tokens, or service accounts with broad permissions. Access should be named, auditable, and easy to revoke.

Security checks:

No default cluster-admin access for service accounts.
RBAC roles are reviewed and documented.
Production access uses SSO/MFA where possible.
Secrets are not stored in plain Kubernetes manifests.
NetworkPolicy uses default-deny for sensitive namespaces.
Images are scanned before deployment.
Audit logs are shipped to durable storage.

Useful command during review:

kubectl auth can-i --list --as system:serviceaccount:production:app

For SOC 2-ready infrastructure, security controls must produce evidence: access reviews, deployment history, audit logs, and incident records.

4. Release safety and rollback

Kubernetes makes deployment easy, but it does not automatically make deployment safe. A production-ready release process has rollback criteria, health checks, and a clear path to stop a bad rollout before users are heavily affected.

Release checklist:

Readiness probes verify real service readiness.
Liveness probes do not restart healthy-but-slow services unnecessarily.
Rollout status is checked in CI/CD.
Helm or GitOps deploys have rollback commands documented.
Canary or Blue/Green strategy is used for risky services.
Deployment dashboards show version, error rate, latency, and saturation.

Example safe Helm command:

helm upgrade myservice ./chart --install --atomic --timeout 5m

Rollback should be boring. If rollback requires manual database surgery, the release is not safe yet.

5. Backups, disaster recovery, and stateful workloads

Kubernetes does not remove the need for disaster recovery. Stateful workloads still need backup, restore testing, RPO/RTO targets, and clear recovery steps. Persistent volumes are not a backup strategy by themselves.

For stateful systems, verify:

Backup jobs are monitored.
Restore tests are performed in a clean environment.
RPO and RTO are documented.
PostgreSQL, Redis, object storage, and queues have separate recovery plans.
Secrets and manifests can be restored with the application.
Runbooks describe recovery order and validation checks.

If PostgreSQL runs inside or near the cluster, include checks for replication lag, failover procedure, WAL availability, and application smoke tests after recovery.

6. Cost and capacity control

Kubernetes cost optimization is part of readiness. A cluster can be technically available and still financially unhealthy. Production teams need visibility into node pools, namespace cost, over-requested CPU/memory, idle workloads, storage growth, and log volume.

Cost checks:

Every workload has resource requests.
Limits are used carefully to avoid CPU throttling and OOM loops.
Namespaces have owners and cost labels.
Development or preview environments have cleanup rules.
Storage classes match workload requirements.
Logs have retention policies by environment and severity.

The goal is not maximum utilization. The goal is enough capacity headroom for failover and deploy spikes without paying for unused infrastructure forever.