# Kubernetes Node-Drain Validation

## Preconditions

- [ ] The test is approved and scheduled.
- [ ] Customer-impact signals and a synthetic transaction are visible.
- [ ] The target node and workloads are known.
- [ ] PDB, replicas, topology, requests, and available capacity are reviewed.
- [ ] A stop condition and recovery owner are named.

## Capture baseline

```bash
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get pdb -A
kubectl top nodes
kubectl top pods -A
```

Record error rate, p95/p99 latency, queue depth, dependency health, and the critical business transaction.

## Test

```bash
kubectl cordon NODE_NAME
kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-data --grace-period=60 --timeout=10m
```

## Stop conditions

Stop and recover when:

- The critical transaction fails beyond the agreed threshold.
- PDB blocks a workload whose maintenance path is unknown.
- Replacement pods cannot schedule.
- Storage cannot attach in the required fault domain.
- Error rate, queue age, or latency exceeds the agreed limit.
- A hidden single-replica dependency is discovered.

## Validation

- [ ] Critical transaction remains successful.
- [ ] Required replicas become Ready.
- [ ] No uncontrolled restart or rescheduling loop occurs.
- [ ] Ingress, DNS, certificates, logging, and metrics remain available.
- [ ] Capacity headroom remains sufficient.
- [ ] The node can be returned safely.

```bash
kubectl uncordon NODE_NAME
kubectl get nodes
kubectl get pods -A --field-selector=status.phase!=Running
```

Record timestamps, commands, affected workloads, user-facing metrics, stop conditions, recovery time, and follow-up owners.