Kubernetes Observability for SRE Teams: SLOs, Alerts, and Traces

Kubernetes observability is the ability to explain customer impact and system behavior from telemetry. It is not the number of dashboards, exporters, or log lines a cluster produces. A useful observability system lets an on-call engineer answer five questions quickly:

1. Which users or transactions are affected?
2. Which service, dependency, cluster, namespace, and version are involved?
3. Is the problem caused by load, a deployment, a dependency, capacity, or configuration?
4. What is the first safe action?
5. How will the team prove that recovery is complete?

This guide is for SREs and platform engineers operating production Kubernetes. It covers the practical path from service-level indicators to metrics, logs, traces, Kubernetes events, alerting, runbooks, cost controls, and game-day validation.

Start with customer-facing SLIs, not with tools

Prometheus, Grafana, Loki, OpenTelemetry, and commercial platforms are implementation choices. The observability design should begin with the customer journey and its service-level indicators.

For a customer-facing API, useful signals usually include:

• Request rate.
• Successful and failed request rate.
• p50, p95, and p99 latency.
• Availability of critical endpoints.
• Queue depth and processing delay.
• Database connection pressure and query latency.
• Dependency errors and timeouts.
• A real business transaction such as login, checkout, upload, or booking.

Infrastructure signals remain important, but they need context. CPU at 85% may be acceptable if latency and errors remain stable. CPU at 55% can still be part of an outage when every request waits on a database lock.

Define an SLO only after deciding what the user experiences. For example:

99.9% of successful checkout API requests complete in under 800 ms over 30 days.

That statement provides a measurable target, a user-facing event, a time window, and a basis for alerting.

Build a telemetry identity model

Metrics, logs, and traces become useful when they can be correlated. Standardize a small set of fields across the stack:

• service.name
• deployment.environment
• k8s.cluster.name
• k8s.namespace.name
• k8s.pod.name
• application version or image digest
• commit SHA
• request ID
• trace ID and span ID
• customer or tenant identifier only when privacy rules permit it

The version and commit SHA matter because many incidents begin with “what changed?” Add deployment annotations or release markers to dashboards so error and latency changes can be compared with the exact release window.

For logs, prefer structured output over free-form text:

{
  "timestamp": "2026-07-10T12:00:00Z",
  "level": "error",
  "service": "checkout-api",
  "environment": "production",
  "version": "1.4.2",
  "request_id": "req-7d2c",
  "trace_id": "5f8b...",
  "dependency": "postgresql",
  "error": "statement timeout"
}

Do not log secrets, authentication tokens, full payment details, or unrestricted personal data merely because structured logs make collection easier.

Prometheus metrics that support an SRE decision

A production Kubernetes metrics stack should cover the workload, the platform, and the critical dependencies.

Workload signals

• Request rate, errors, and duration.
• Concurrency and queue depth.
• Worker throughput and failed jobs.
• Process restarts and OOM events.
• Runtime-specific saturation such as thread pools or event-loop delay.

Kubernetes signals

• Deployment availability and rollout status.
• Pod readiness and restart reasons.
• HPA desired versus available replicas.
• PodDisruptionBudget state.
• Node pressure and scheduling failures.
• Persistent volume latency, errors, and capacity.
• CoreDNS errors and latency.
• Ingress response codes and duration.

Dependency signals

For PostgreSQL, include connections, locks, slow queries, disk latency, replication lag, and transaction age. For queues, include depth, age, consumer lag, retries, and dead letters. For external services, include timeout and error rates by dependency.

Recording rules are useful for expensive or repeated expressions, but they must preserve the labels needed for investigation. Excessive label cardinality can overload Prometheus, so avoid unbounded values such as raw request IDs, user IDs, or full URLs in metric labels.

ServiceMonitor example and scrape validation

A Prometheus Operator ServiceMonitor can discover an application metrics endpoint:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: checkout-api
  namespace: monitoring
spec:
  namespaceSelector:
    matchNames:
      - production
  selector:
    matchLabels:
      app: checkout-api
  endpoints:
    - port: metrics
      path: /metrics
      interval: 30s
      scrapeTimeout: 10s

After applying it, validate the complete path:

kubectl get servicemonitor -n monitoring checkout-api -o yaml
kubectl get service -n production checkout-api --show-labels
kubectl get endpoints -n production checkout-api
kubectl port-forward -n production service/checkout-api 9090:9090
curl -fsS http://127.0.0.1:9090/metrics | head

A valid YAML object does not prove that Prometheus selected it, reached the target, accepted the metric format, or retained the labels you need.

Alert on impact and burn, not every abnormal number

An alert should request human action. A dashboard can display conditions that are interesting but do not require paging.

A critical alert should contain:

• Customer or service impact.
• Severity.
• Current value and threshold.
• A dashboard link.
• A runbook link.
• Owner or escalation route.
• The first safe action.
• Release or version context where possible.

Multi-window burn-rate alerts are stronger than a single raw threshold because they detect both fast severe incidents and slower sustained SLO erosion. The exact formula must match the real SLO and traffic pattern.

Avoid paging directly on:

• CPU or memory percentage without impact or saturation context.
• A single pod restart.
• A transient replica mismatch during a normal rollout.
• Every Kubernetes Warning event.
• Log text matched without rate, grouping, or service context.

Those signals may belong on dashboards or lower-severity notifications, but they should not automatically wake an engineer.

Logs, traces, and Kubernetes events solve different questions

Metrics tell you that behavior changed. Logs explain application events. Traces show request flow and dependency latency. Kubernetes events show scheduling, image, probe, volume, and controller context.

Use them together:

1. An SLO alert identifies checkout error-rate burn.
2. The dashboard shows the increase began after version 1.4.2.
3. Traces show timeouts in the PostgreSQL span.
4. Logs show statement timeout for one query family.
5. Kubernetes events show no scheduling or restart problem.
6. The runbook directs the engineer to pause rollout and inspect database saturation.

Useful event checks:

kubectl get events -A --sort-by=.metadata.creationTimestamp | tail -100
kubectl describe deployment/checkout-api -n production
kubectl logs deployment/checkout-api -n production --since=15m

Events are short-lived in many clusters. Export the events required for incident analysis if the default retention is insufficient.

Control observability cost and cardinality

Observability can become one of the largest infrastructure costs when every log is retained, traces are sampled without policy, and metric labels contain unbounded values.

Create an owned telemetry budget:

• Retention by data type and environment.
• Log volume by service and severity.
• Trace sampling policy for normal traffic, errors, and slow requests.
• Metric series count and top cardinality contributors.
• Storage growth and query cost.
• Data required for incident, security, or compliance evidence.

A sensible policy may retain high-value production logs longer than development debug logs, sample normal traces aggressively, preserve error traces, and reject unbounded metric labels in CI or review.

Cost optimization must not remove the signals required to detect customer impact or investigate a serious incident. First remove duplicate and low-value telemetry, then tune retention and sampling.

Common Kubernetes observability anti-patterns

Dashboards without owners or decisions

A dashboard that nobody uses during an incident is decoration. Each critical dashboard should have an owner and answer a defined operational question.

Liveness probes used as dependency monitors

Restarting healthy application processes because PostgreSQL is slow can multiply connection pressure and deepen the outage. Keep process health and traffic readiness separate.

High-cardinality metric labels

User IDs, request IDs, random paths, and arbitrary exception text can create an uncontrolled number of series. Put correlation IDs in logs and traces instead.

Alerts without runbooks

An alert with no first action forces the on-call engineer to rediscover the system during stress.

No release context

If dashboards cannot distinguish versions or show deploy markers, the team loses one of the fastest incident hypotheses.

Monitoring the monitoring stack only from itself

Use an external or independent check for the observability pipeline and alert delivery path. A failed monitoring system may otherwise report itself as healthy until data stops arriving.

Operational validation and game-day exercise

A production observability system should be tested with a controlled failure, not only reviewed on diagrams.

Example exercise:

1. Deploy a version that deliberately increases latency on a test endpoint or controlled canary.
2. Confirm that SLI metrics change.
3. Confirm that the SLO alert reaches the correct owner.
4. Open the linked dashboard and identify version and affected service.
5. Follow one request from metric to log and trace.
6. Use the runbook to pause or roll back.
7. Confirm that signals return to baseline.
8. Record detection time, acknowledgement time, diagnosis time, and recovery time.

The goal is not to produce a perfect demo. The goal is to discover missing labels, broken links, noisy alerts, unclear ownership, excessive query time, and runbook gaps before a real incident.

Kubernetes observability decision matrix

Related SteadyOps reading

• Kubernetes Production Readiness Checklist — probes, capacity, security, rollback, and recovery controls.
• Kubernetes Rollback Checklist — objective release signals and post-rollback validation.
• Infrastructure Cost Optimization — telemetry retention, resource sizing, and cost ownership.
• Security Evidence Operations Model — logs, access events, incidents, and operational evidence.

Key takeaways

• Start with customer-facing SLIs and SLOs before choosing tools.
• Standardize service, environment, cluster, version, request, and trace identity.
• Use metrics, logs, traces, and Kubernetes events for different questions.
• Page on impact, sustained burn, or actionable saturation—not every abnormal number.
• Control label cardinality, retention, and sampling with an owned telemetry budget.
• Validate the complete incident path through a game-day exercise.

groups:
  - name: api-slo
    rules:
      - alert: ApiHighErrorBudgetBurn
        expr: |
          (sum(rate(http_requests_total{service="api",status=~"5.."}[5m]))
          / sum(rate(http_requests_total{service="api"}[5m]))) > 0.02
        for: 10m
        labels:
          severity: critical
          owner: platform
        annotations:
          summary: API error budget is burning quickly
          runbook_url: https://runbooks.example.com/api-errors