HAMZA KAREEM / CONCEPTS hamzawi.dev

Lesson 1031 min read

PodDisruptionBudgets — Voluntary vs Involuntary Disruption

The Permission Slip Nobody Has To Ask For

Cloud infrastructurecloud infrastructure / Kubernetes

The mental model

A PDB is a permission desk standing between your pods and anything that wants to politely remove one.

kubectl drain walks up: "I'd like to evict payments-7f9." The desk checks the budget and answers 200 or 429. Node upgrades, cluster-autoscaler scale-down, Karpenter consolidation — all queue at that desk.

apiVersion: policy/v1
kind: PodDisruptionBudget
spec:
  maxUnavailable: 1
  selector:
    matchLabels: {app: payments}

3 replicas, maxUnavailable 1 → exactly one pod leaves at a time. That model is correct, for everything that asks.

Where the analogy breaks

Nothing is obliged to ask.

  • Node kernel panics → pods just vanish. No eviction call, no desk.
  • Spot/preemptible reclaim → 2-minute notice, then gone.
  • OOMKill or a failing liveness probe → a restart, not an eviction.
  • kubectl delete pod → also not an eviction. PDB bypassed entirely.

PDB governs voluntary disruption only. People read minAvailable: 2 as an availability guarantee. It isn't — it's a rate limiter on graceful removals. Surviving involuntary loss is a different toolbox: replica count, topologySpreadConstraints, anti-affinity.

Two failures that actually bite

  1. The PDB that wedges your fleet
replicas: 1 with minAvailable: 1 → budget is permanently zero. `kubectl drain` blocks forever. Your upgrade stalls at node 3 of 40; the autoscaler stops consolidating; your bill drifts up for weeks and nothing pages.

kubectl get pdb -A
NAME       MIN AVAILABLE   ALLOWED DISRUPTIONS
payments   2               1
billing    1               0     <-- wedged

ALLOWED DISRUPTIONS = 0 on a healthy cluster is a bug, not safety.
  1. The budget counts Ready, not Running
PDB math uses pods passing readiness. If 1 of 3 pods is CrashLooping, you already have only 2 Ready → budget 0. You now cannot drain the node holding the broken pod, which is usually the exact node you wanted to recycle.

Rules of thumb

  • Prefer maxUnavailable over minAvailable. maxUnavailable: 1 scales with the deployment; minAvailable: 2 silently becomes "block everything" the moment someone scales to 2.
  • Never put a PDB on a single-replica workload. Scale to 2, or accept the restart.
  • A PDB with no matching pods is dead weight — a label typo gives you zero protection and zero errors.

Your exercise

Run kubectl get pdb -A and read the ALLOWED DISRUPTIONS column. Any zero rows — intentional, or a wedge? Then pick one and try:

kubectl drain <node> --dry-run=server --ignore-daemonsets

Does the desk say yes, or does it refuse?

❓And one to answer: of these three, which does a PDB actually protect you from — a rolling cluster upgrade, a spot reclaim, or an OOMKill? Only one of them asks permission.