Kubernetes Control Plane Scaling Limits: Nodes Are the Wrong Question

Kubernetes control plane scaling limits usually get quoted as one number. Upstream supports 5,000 nodes, so 5,000 nodes is the answer. That number is real and it is published, and it tells you nothing about why your API server is at 90% memory on a 400-node cluster. Three separate pressures push a control plane over. They are independent, you can max any one of them with the other two on the floor, and the fix for one does nothing for the others. Diagnose it wrong and you spend a quarter adding API servers to a cluster whose actual problem was too much state in etcd.

The three walls, and what each one feels like

The upstream large-cluster guidance is specific about what it covers: Kubernetes v1.37 supports configurations meeting all of no more than 110 pods per node, no more than 5,000 nodes, no more than 150,000 total pods, and no more than 300,000 total containers. Four dimensions, one of which everyone quotes.

In October 2026, Joe Betz and Walter Fender published a three-way split of the problem that matches what we see in client clusters better than any node count does.

| Wall | Who hits it | What dominates | The reflex that fails | |---|---|---|---| | Node-bound | Training and RL fleets, large and homogeneous | Every kubelet reporting in, forever | Vertical control plane growth, until lease writes swamp etcd | | Churn-bound | Agent platforms, modest size and a very intense day | Write rate on short-lived custom resources | Anything indexed by node count, because node count is small | | State-bound | Platform clusters with 200 operators installed | Resident bytes multiplied by every API server holding them | Adding an API server, which makes it worse |

The third row is the one that costs people the most money, and we will come back to why.

A state-bound cluster is neither big nor fast. It has tens of thousands of objects that mostly sit there, and every API server holds a copy of all of them. A churn-bound cluster can sit at a few thousand nodes and have a far worse day than a fifteen-thousand-node training fleet, because its objects are alive: sessions created, suspended, resumed and destroyed thousands of times a minute, and every transition is a write.

Finding your wall with the metrics you already have

Before any architecture discussion, read four things. On AKS these are platform metrics in Azure Monitor, and Microsoft names the trigger explicitly: if etcd_database_usage_percentage sits consistently above 50%, shrink the database before you do anything else.

# How much of your total request concurrency each priority level gets
kubectl get --raw /metrics | grep apiserver_flowcontrol_nominal_limit_seats

# Whether API Priority and Fairness is already dropping requests
kubectl get --raw /metrics | grep apiserver_flowcontrol_rejected_requests_total

# Which FlowSchema is catching which traffic
kubectl get flowschemas

# AKS only: confirm the managed control plane has actually scaled up
kubectl describe configmap large-cluster-control-plane-scaling-status -n kube-system

API Priority and Fairness sums --max-requests-inflight and --max-mutating-requests-inflight into a total concurrency limit and divides it among priority levels. The upstream defaults are 400 and 200, and AKS runs those defaults for 600 concurrent requests. If apiserver_flowcontrol_rejected_requests_total is flat at zero while memory climbs, you are not request-bound, and buying request capacity will not help you.

The read that separates node-bound from the other two is the share of write traffic that is only liveness. In the measurements behind that October piece, lease renewals were roughly 70% of write QPS on a node-bound cluster. If liveness is a rounding error in your write mix, your wall is churn or state.

What node liveness costs at scale

At 100,000 nodes with a 10-second heartbeat, liveness alone generates about 10,000 writes per second. Most of what that control plane does is machines politely saying "still here".

Two fixes exist for that load and they compose. A node-facing tier in front of the control plane can absorb renewals and propagate only real state transitions. Separately, routing renewals through etcd's in-memory lease refresh instead of a full Raft commit lands in etcd itself, which means it helps every cluster down to the three-node one.

What you can do today is move the cheap, chatty resources off your main store. Upstream recommends a dedicated etcd instance for Events, and the flag that does it is documented with its exact format:

--etcd-servers-overrides "/events#http://etcd6:2379"

Per the kube-apiserver reference, the override format is group/resource#servers with semicolon-separated URLs, and it "applies only to resources compiled into this server binary". Keep that sentence in mind for the next section, because it is the whole problem with the churn wall.

The churn wall has nothing to do with cluster size

Platforms that pack many agent sessions onto a smaller pool of worker pods tend to express each session's lifecycle as a custom resource. The busiest and shortest-lived objects in the cluster are therefore CRDs, and the write rate has no relationship to how many machines you own. We see this on clusters running sandboxed AI agent workloads, where the node count looks unremarkable and the write graph does not.

The storage-interface work that would let you route those writes elsewhere is KEP-4743, the Kubernetes-etcd interface, owned by SIG etcd and SIG API Machinery. It was targeted at alpha for v1.32, the tracking status reads Deferred, and the issue is labelled lifecycle/frozen. So the routing flag covers built-in resources and your write storm is in custom resources. There is no configuration answer here yet.

That leaves cutting the write rate, which is unglamorous and works. Stop writing status on every intermediate transition. Set ttlSecondsAfterFinished on Jobs so finished objects leave on their own. Set successfulJobsHistoryLimit and failedJobsHistoryLimit on CronJobs. Deployment revision history defaults to 10 and old ReplicaSets are API objects too. Helm keeps revision history by default, and Microsoft's own guidance for large clusters is to run helm upgrade --history-max below 5.

If your platform generates custom-resource write rates you can measure, SIG API Machinery is asking for exactly those numbers, because no scale-tested recipe for a churn-bound cluster has been published by anyone.

Why adding an API server makes the state wall worse

Adding API servers is a documented option in kOps and the standard answer most people will give you. On a state-bound cluster it is the wrong move, because every API server you add replicates the entire watch cache. You buy request capacity and pay for it in memory you did not have.

There is a floor under this that you cannot configure away. With every disableable cache switched off, an API server still carries about 0.9 KB per RBAC object, because the authorizer's informers hold them regardless. A cluster with 20,000 Roles, ClusterRoles and bindings is therefore carrying roughly 18 MB of authorizer cache in every API server replica before a single workload object is counted.

Cache residency, meaning an API server that only keeps what it actually serves, does not exist upstream yet. KEP-5866, Server-side Sharded List and Watch, went alpha in v1.36 with beta targeted for v1.38, and it filters what goes out, so the cache still stores everything. A two-shard prototype of residency cuts heap by roughly half per shard, and it is a prototype branch under review, not something you run next quarter.

The lever you do have is --watch-cache-sizes, and the docs are blunt that "the only meaningful size setting to supply here is zero, which means to disable watch caching for the associated resource". That pushes those reads to etcd, so it is a trade, not a win. Reducing the object set is the better lever: move environment variables out of pod specs into ConfigMaps, split large Secrets and ConfigMaps, keep secrets in an external store rather than etcd where you can, and make sure your controllers set ownerReferences so garbage collection actually removes dependents.

The ladder above 5,000 nodes, and where your provider stops

Betz and Fender publish a ladder of scale-tested configurations, including the rung nobody has built:

| Recipe | Scale | Composition | Evidence | |---|---|---|---| | Classic HA | up to 5k nodes | 2 to 3 identical control planes | measured | | Role-Split | 5k to 20k | separate machine pools for API serving, scheduler, KCM, etcd | shipped, benefit not yet measured | | Node Plane | 20k to 65k | Role-Split plus a node-facing tier that absorbs the leases | measured, n=1 | | Partitioned Store | ~100k | Node Plane plus resource types split across independent stores | externally demonstrated | | Sharded Slice | beyond | K shard units, each a controller plus API server pinned to a hash range | doesn't exist yet |

Read the index on that ladder carefully. You pick a rung by node count, which is the one axis of three with measurements behind it. If your problem is churn or state, the ladder cannot tell you where to stand.

Managed Kubernetes ceilings are not one number either:

| Platform | Nodes per cluster | Pods per cluster | Node pool cap | |---|---|---|---| | Upstream v1.37 | 5,000 supported | 150,000 | not applicable | | AKS | 5,000 | 200,000 with Azure CNI Overlay | 1,000 per node pool, so five user pools to reach 5,000 | | GKE Standard | 65,000 | 200,000 | 1,000 nodes per zone | | GKE Autopilot | 5,000 | 200,000 | not applicable | | EKS | no published quota | no published quota | 450 nodes per managed node group, 30 groups per cluster |

EKS is the interesting row. There is no documented nodes-per-cluster service quota, so your ceiling is upstream's plus arithmetic: at the default 450 nodes per managed node group you need 12 groups to reach 5,000 nodes, and the default 30 groups per cluster puts 13,500 within reach of quota increases you never requested.

The operational traps cost more than the ceilings. On AKS, cluster upgrades are blocked at 5,000 nodes because there is no capacity left for rolling updates inside max surge, and Microsoft tells you to scale below 3,000 nodes first. A managed control plane scale-up can mean elevated API server latency or timeouts for up to 15 minutes. Azure Network Policy Manager supports only 250 nodes. Standard Load Balancer allows 10,000 backend IP configurations per frontend IP, which one service exposing five ports exhausts at 2,000 nodes.

Our call: below 20,000 nodes, do not build a sharded control plane. Role-Split is shipped and uses stock upstream binaries. Above that, a second cluster is usually cheaper than a research project, though capacity and resilience are different reasons to add one and they want different designs, which we worked through in multi-cluster Kubernetes failover.

What we do before anyone re-architects a control plane

Almost every control plane problem we are called into sits well under 1,000 nodes, and the cause is usually one badly behaved client.

Start with etcd. The default storage quota is 2 GiB, 8 GiB is the suggested maximum for normal environments and etcd warns at startup above it, and the maximum request size is 1.5 MiB. AKS provisions 8 GB and still tells you to start reducing object sizes past 2 GB, because a larger database means longer defragmentation and worse read and write performance.

Then audit clients, in this order:

1. Replace polling LISTs with informers from client-go, so controllers watch events instead of re-listing.
2. Add field selectors. /api/v1/pods with no selector on a large cluster is the single most expensive habit we find.
3. Paginate anything that might return thousands of objects, with limit and continue, or kubectl get pods --chunk-size=100.
4. Use resourceVersion=0 for reads that can come from the API server cache instead of etcd, accepting that it does not support pagination.
5. Give DaemonSets startup jitter and exponential backoff on 429 and 5xx, because a DaemonSet that LISTs on startup becomes a thundering herd the moment you add nodes.
6. Tune leader election so lease_duration > renew_deadline > retry_period holds.

Nine times out of ten that work ends the incident, and the cluster never needed a new control plane topology. For the pre-launch version of this list, see our Kubernetes production readiness checklist. If the pressure is GPU fleets rather than operators, the cost side of that story is in GPU cost allocation on Kubernetes.

When to Get Help

Call us when the diagnosis is the hard part: API server memory climbing with no obvious owner, etcd past 2 GB and nobody sure which controller filled it, or a plan to shard a control plane that nobody has stress-tested. We do a read of your control plane metrics against the three axes above, name which wall you are at, and say which of the upstream rungs you need, including when the answer is that you do not need any of them yet. Talk to us.