Karpenter Spot Interruption PDB: Why One Replica Died Without SIGTERM

A single-replica agent that one client's production alerting depends on was replaced four times in two days, each time without a graceful shutdown, and a gap in alerts was reported. Every replacement was a spot reclaim of a node in a spot ARM Karpenter NodePool. The agent's PodDisruptionBudget, minAvailable: 1 over its one replica, was there to protect it. In practice it blocked the drain, and the pod died without SIGTERM. We moved the agent to an on-demand Graviton node group, turned its PDB off in the production values, and capped it at one replica. Over the following seven days it had no pod replacements and 0 restarts.

The checklist in our Kubernetes production readiness post says production services run at least two replicas and that PDBs keep node drains from taking too many of them at once. Our Kubernetes cost framework adds that Karpenter consolidation is a reason to set PDBs. This post covers the case those two leave out: a workload with one replica, on spot capacity, behind a budget that allows no evictions.

What Karpenter does when the warning arrives

EC2 sends a Spot Instance interruption notice two minutes before it stops or terminates the instance, and Karpenter acts on that warning. Its disruption docs say that once it has the warning, it "will begin draining the node while in parallel provisioning a new node", and that it drains by evicting pods "with the Kubernetes Eviction API to respect PDBs".

That second phrase is where the budget comes in. The same page lists interruption under the forceful methods, the ones that begin draining as soon as the condition is met and that NodePool disruption budgets cannot rate-limit. The pods themselves still go through eviction, and the docs say plainly that "Pods with blocking PDBs will not be evicted by the Termination Controller."

So Karpenter does what it promises. It starts a replacement node and asks the API server to evict the pod. The two-minute clock belongs to EC2, and it runs whether the eviction succeeds or not.

Why a one-replica budget blocks the clean exit

The Eviction API answers 429 Too Many Requests when "the eviction is not currently allowed because of the configured PodDisruptionBudget". With minAvailable: 1 and one pod, every eviction would leave zero available pods, so every eviction gets that answer. The PDB reference describes the field the same way: an eviction is allowed only if at least minAvailable pods "will still be available after the eviction".

An allowed eviction is the graceful path. The API server marks the pod with its grace period, and the kubelet shuts it down cleanly before force-killing it. A refused eviction leaves the pod running until EC2 takes the instance back. The Kubernetes docs on disruptions are explicit that involuntary disruptions "cannot be prevented by PDBs", and a VM that disappears under its pod is on that list.

The budget therefore protected the pod from the one exit that would have sent it SIGTERM, and did nothing against the exit that killed it. AWS's guidance for spot node groups on EKS points the same way: on the managed node groups page it warns that pods on spot nodes "may be forcibly terminated without receiving termination signals", and it recommends On-Demand for "cluster management tools such as monitoring and operational tools".

A query for budgets that allow no evictions

The pattern shows up in the PDB status. disruptionsAllowed is the number of disruptions currently allowed, and expectedPods is how many pods the budget counts. A budget with one expected pod and zero allowed disruptions is the case above.

kubectl get pdb -A -o json | jq -r '
  .items[]
  | select(.status.disruptionsAllowed == 0)
  | [.metadata.namespace, .metadata.name,
     (.status.expectedPods|tostring), (.status.currentHealthy|tostring)]
  | @tsv'

A multi-replica service that is degraded at that moment also shows zero allowed disruptions, so read the expectedPods column before acting on a row. At this client the same one-replica PDB, blocking Karpenter drains, turned up in four more services.

On-demand capacity and no budget for this workload

Our engineer proposed an on-demand Graviton node group plus a relaxed PDB, limited to the production values. What shipped pins the agent to the on-demand Graviton node group and disables its PDB there.

# Illustrative production values for a single-replica agent; acme-agent is an example name.
acme-agent:
  replicaCount: 1
  nodeSelector:
    eks.amazonaws.com/capacityType: ON_DEMAND
  podDisruptionBudget:
    enabled: false

The eks.amazonaws.com/capacityType: ON_DEMAND label is the one Amazon EKS adds to every node in an On-Demand managed node group, per the same AWS page.

While checking the fix, our engineer saw more than one agent pod at times, and they were KEDA scale-outs. We asked the owners whether two instances of the agent are safe to run, and capped it at one replica. On production the KEDA maximum was set to 1 by hand until the release tag carries it, and the lower environments got the cap in their chart values.

# Illustrative fragment of a KEDA ScaledObject; acme-agent is an example name.
spec:
  scaleTargetRef:
    name: acme-agent
  maxReplicaCount: 1

The ScaledObject spec gives maxReplicaCount a default of 100 and passes it to the HPA that KEDA creates. A single-instance workload that inherits that default can grow past one whenever its trigger fires, and a hand-set value on production lasts only until the next release overwrites it. Write it into the release, and check after every deploy that it held.

Two replicas or none

For the four other services the choice depends on whether the workload can run twice. One of them can, and the team agreed on two replicas in every environment with its PDB kept. With two pods, minAvailable: 1 lets Karpenter evict one at a time, which is the drain the budget was written for. We raised the other three with their owners.

For a workload that must stay single, the Kubernetes guide to configuring a PDB lists not using a PDB at all, and tolerating occasional downtime, as the first option for a single-instance stateful application, and the same trade fits a single-instance agent. We would pair that with on-demand capacity, which is what this agent got. A single replica on spot with a blocking budget gets neither the graceful shutdown nor protection from the reclaim.

For a cluster that runs single-replica operational tools on spot nodes, we would start with the PDB status query above and the capacity type under each of those pods. Send us what you find and we will tell you which ones need a second replica and which need on-demand nodes.