HPA Target Above 100 Percent: The Node.js Service That Could Not Scale Out

One team ran a production Node.js service under KEDA with a CPU target of 150% of the request, and under a burst of requests it did not scale out. The process runs its JavaScript on one thread and could not get there. A second Node.js service at the same client had the memory version of the problem: OOMKilled 15 times in one day, with a 512M limit, a V8 heap ceiling of 259 MB, 220 to 235 MB in use at rest, and a 150% autoscaling target that never fired. Our engineer moved the first service to targets it can reach, the shared chart got two fixes for defects that fought the autoscaler, and we traced the burst to a cache invalidation that took the service from under 0.1 to about 60 requests per second.

The choice between native HPA and KEDA is in our HPA scale to zero post. Here the autoscaler already exists and points at a number the process cannot produce.

What a utilization target is measured against

A CPU or memory target written as utilization is a percentage of the container's request. The Kubernetes HPA documentation says the controller "calculates the utilization value as a percentage of the equivalent resource request on the containers in each Pod", takes the mean across the targeted pods, and computes:

desiredReplicas = ceil[currentReplicas × (currentMetricValue / desiredMetricValue)]

KEDA's CPU scaler uses the same definition. With metricType: Utilization, the value is "represented as a percentage of the requested value of the resource for the pods."

So a target above 100% asks each pod to use more than it requested before the replica count goes up. That only works when two things hold: the container is allowed to go past its request, and the process inside is able to.

One event loop and a 150% CPU target

The Node.js guide on the event loop puts the constraint plainly: "Node.js runs JavaScript code in the Event Loop (initialization and callbacks), and offers a Worker Pool to handle expensive tasks like file I/O." The Worker Pool takes "I/O for which an operating system does not provide a non-blocking version, as well as particularly CPU-intensive tasks." Request handling written in JavaScript stays on the one event-loop thread, and one thread uses at most one core.

The arithmetic follows from the request. Take a container that requests 1000m. A 150% target asks for 1500m of sustained use per pod, half a core more than one event loop can burn. Under load the loop saturates and requests queue, while the measured utilization sits at or below 100%. Feed 100 against a target of 150 into the formula and the ratio is 0.67, so the controller's answer points down toward the minimum.

A smaller request leaves more room. At 500m the same 150% target is 750m, which one thread can reach. The target has to be read together with the request, per service, and a chart default of 150% applied to every service will be unreachable for some of them.

A memory target past the heap ceiling

The second service shows the same trap on memory. Its pods idled at 220 to 235 MB against a V8 heap ceiling of 259 MB, about half of the 512M container limit. That left very little heap to grow into, and the pods were OOMKilled 15 times in one day while the 150% target never fired.

For a memory target above 100% to fire, the pod has to reach target times request before it reaches its limit. For a Node process there is a nearer edge: the size V8 allows its old space. The Node.js CLI reference describes --max-old-space-size this way: "As memory consumption approaches the limit, V8 will spend more time on garbage collection in an effort to free unused memory." Its example sets 1536 on a machine with 2 GiB, and --max-old-space-size-percentage sets the same ceiling "as a percentage of available system memory."

We would set the heap size on purpose and size the memory target against it, because this service's ceiling sat at about half its limit. Older Node lines also misread the container limit; the Node.js row in our cgroup v2 migration post lists which versions. For this service the plan was split: DevOps raises the production memory limit and the autoscaling target, and the application team investigates the heap.

Where the burst came from

The root-cause analysis found the trigger outside the autoscaler. An edit to one shared definition updated the 61 records that use it and wiped a computed-results cache for 35 sites, about 14.3k Redis keys in 11 seconds. A downstream processor then called the service for every miss, with no dedup and no backoff. The request rate went from under 0.1 to about 60 per second and landed on a service whose CPU target one event loop could not reach.

The code fix went to the application team in four parts: coalesce, dedup, backoff and narrower invalidation. In general terms, coalescing lets one caller rebuild a missing entry while the others wait for it, dedup drops repeated requests for the same work, backoff slows the caller down when the service slows down, and narrower invalidation clears only what an edit changed.

On backoff, the AWS Builders' Library article on timeouts, retries and backoff is blunt: "If calls to a system are timing out, and that system is overloaded, retries can make the overload worse instead of better." It calls backoff "the preferred solution that we use in Amazon." The application-side fix was handed over and we have not seen it ship, so the infrastructure changes below are what stands between the next bulk edit and the service.

The targets we set instead

The mitigation went live on production first: a ScaledObject minimum of 3 replicas, a CPU target of 60% and a memory target of 100%. Staging got the same values so the team could load-test them. A minimum of three means a burst lands on at least three event loops, and 60% of the request stays within reach of one thread for any request up to about 1667m.

# Illustrative ScaledObject; "acme-api" is a placeholder, not a client name.
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: acme-api
spec:
  scaleTargetRef:
    name: acme-api
  minReplicaCount: 3
  maxReplicaCount: 10
  triggers:
    - type: cpu
      metricType: Utilization
      metadata:
        value: "60"
        containerName: "acme-api"
    - type: memory
      metricType: Utilization
      metadata:
        value: "100"
        containerName: "acme-api"

The ScaledObject reference gives maxReplicaCount a default of 100, so set a ceiling you have capacity for.

A live patch to a ScaledObject lasts until the next release renders the chart again. That is why the fix also went out as a hotfix for that one service, built as a release-candidate tag and merged, and why a follow-up now covers the KEDA and HPA targets that were unreachable or flapping across the other backend services on staging and production.

Two chart defects that fight the autoscaler

The shared Helm chart had two problems that make any target less reliable, and both were fixed in the same weeks and tested on staging with 4 services.

The first was replicas. The chart rendered it even when autoscaling was on. The HPA documentation recommends removing spec.replicas from the manifest when an HPA is enabled, because every apply "will instruct Kubernetes to scale the current number of Pods to the value of the spec.replicas key", which can cause "thrashing or flapping behavior." It also warns that removing the field "may incur a one-time degradation of Pod counts as the default value of this key is 1." The chart now leaves replicas out when autoscaling is on and keeps the live count through Helm's lookup:

# Illustrative template for a chart called "acme"; not the client's chart.
spec:
  {{- if .Values.autoscaling.enabled }}
  {{- $live := lookup "apps/v1" "Deployment" .Release.Namespace (include "acme.fullname" .) }}
  {{- if $live }}
  replicas: {{ $live.spec.replicas }}
  {{- end }}
  {{- else }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}

Test this with a real cluster. The Helm docs on lookup are explicit: "When no object is found, an empty value is returned." Helm does not contact the API server during a plain --dry-run, and the docs point to --dry-run=server for testing against a running cluster. The function list adds that under helm template it "always returns an empty response", which matters if a GitOps tool renders your charts that way.

The second was sidecars. Pod-level utilization counts every container in the pod, so a sidecar's usage and request move the number the autoscaler reads. The Kubernetes docs give the example of scaling "based on the resource use of the web application, ignoring the sidecar container" with a ContainerResource metric, and KEDA exposes the same thing as containerName, "the specific container to scale based on its CPU, rather than the entire pod." The chart now sets containerName for pods with sidecars and defaults it to the app's container.

Auditing every autoscaler for the same mistake

Start with every utilization target above 100. These two commands list the HPAs and the ScaledObjects that have one:

kubectl get hpa -A -o json | jq -r '
  .items[] as $h
  | $h.spec.metrics[]?
  | select(.type == "Resource" or .type == "ContainerResource")
  | (.resource // .containerResource) as $m
  | select($m.target.type == "Utilization" and $m.target.averageUtilization > 100)
  | "\($h.metadata.namespace)/\($h.metadata.name) \($m.name) \($m.target.averageUtilization)%"'

kubectl get scaledobjects.keda.sh -A -o json | jq -r '
  .items[] as $so
  | $so.spec.triggers[]
  | select(.type == "cpu" or .type == "memory")
  | select((.metricType // "Utilization") == "Utilization")
  | select((.metadata.value | tonumber) > 100)
  | "\($so.metadata.namespace)/\($so.metadata.name) \(.type) \(.metadata.value)%"'

For each hit, read the target next to the container's request and limit. On memory, target times request has to land below the limit, with room for the runtime's own ceiling. On CPU for a Node.js service, target times request has to land below one core. Then grep the chart for a replicas line that renders next to an autoscaler, and for pods with sidecars that scale on pod-level utilization.

Teams that want a second pair of eyes on their autoscaling can reach us here. We would begin with the list those two commands print and the requests behind each line, before touching any application code.