NLB Hairpin Connection Timeout: Why One Connect in Ten Hangs

About 10% of connections from pods to Postgres and Redis hung at one client. In its lower environments both databases run inside the EKS cluster, and pods reach them through internal Network Load Balancers. The team noticed the hangs on one environment. Our engineer diagnosed an NLB hairpin and moved the target groups from instance targets to ip targets with client IP preservation. The change went into Terraform and was merged and applied on three lower environments.

Why some connects hang and the rest go through

An instance target group keeps the client's source address by default. AWS lists client IP preservation as enabled for instance-type target groups and disabled for IP-type TCP and TLS target groups (NLB target group attributes). The same page says NAT loopback, also called hairpinning, is not supported while preservation is on: when a target behind an internal NLB opens a connection to that NLB, the connection can be routed back to the target that opened it.

The troubleshooting guide spells out the result. A connection from a registered instance to its own load balancer "succeeds only if the request is routed to a different instance". Routed to the same instance, it times out, because the source and destination addresses are the same. AWS adds that this applies to EKS pods on the same worker node even though the pods have different IP addresses (connections time out for requests from a target to its load balancer).

Which connections fail is decided per connection. An NLB picks a target with a flow hash over the protocol, the source address and port, the destination address and port, and the TCP sequence number, and keeps each TCP connection on that one target for its life (routing algorithm). With instance targets, a client pod can sit on a node that is itself registered. The share of connects that hang follows the odds that the hash lands on the client's own node, so it moves as nodes join or leave the target group.

Why it reads as a database problem

The failure is a hang. The server sends no refusal and no error, and the next attempt usually works, because separate TCP connections from one client carry different source ports and sequence numbers and can go to different targets (the same routing page says so). Connection pools retry by design. What surfaces is a slow request now and then, or a job that times out once and passes on a rerun, and the database becomes the first suspect.

To confirm it, count failed connects from inside a pod:

# Illustrative: run from any pod with bash; db.example.internal is a placeholder
HOST=db.example.internal; PORT=5432; failed=0
for i in $(seq 1 100); do
  timeout 3 bash -c "</dev/tcp/$HOST/$PORT" 2>/dev/null || failed=$((failed+1))
done
echo "failed: $failed of 100"

Then check whether that pod's node is one of the registered targets:

# Values in angle brackets are placeholders
kubectl get pod <client-pod> -o wide
aws elbv2 describe-target-health --target-group-arn <target-group-arn>

A steady fraction of failures from a pod whose node appears in the target list is the pattern AWS describes.

Three fixes, and the one we prefer

The troubleshooting page offers two remedies: disable client IP preservation and read the client address from Proxy Protocol v2, or keep containers that must talk to each other on different instances. The third option is to change what the target group registers.

Disabling preservation is one attribute, preserve_client_ip.enabled, and it has a cost that AWS documents on the attributes page. With preservation off, an NLB supports 55,000 simultaneous connections, or about 55,000 connections per minute, to each unique target (IP address and port), and port allocation errors get more likely past that. Proxy Protocol v2 also needs a target that parses the header. AWS warns that applications which do not expect it can fail, and the header rides on health check connections too. For a database behind the NLB, check whether the server accepts the header at all before you count on it.

Keeping clients off the database's nodes means scheduling rules for every client of that database. We would not run a cluster that way.

With ip targets, the target group registers the database pod's address instead of nodes. The client pods stop being targets, so the case AWS describes (a target connecting to its own load balancer) no longer covers them. That is the fix the client got, with preservation kept on. One default needs attention. AWS sets preservation to disabled for IP-type TCP target groups, so switching the target type without setting the attribute turns preservation off as a side effect.

AWS's note about pods on one worker node is written for instance registration. It does not say what happens when a client pod shares a node with the database pod it reaches through the NLB. We would test that pairing on purpose after the switch: schedule a client next to the database pod and run the loop above.

The Terraform change is a replacement

In the AWS provider, target_type on aws_lb_target_group forces a new resource, and its default is instance (aws_lb_target_group). For a forced replacement Terraform destroys the old object first by default (lifecycle meta-argument), and AWS deletes a target group only when no action references it (DeleteTargetGroup). While a listener still points at the old group, that delete is refused with ResourceInUse.

create_before_destroy reverses the order. The lifecycle page warns that the old and new objects then exist at the same time, so resources with unique names need handling, and name_prefix (at most 6 characters) gives each replacement its own name.

# Illustrative: names, port and variables are placeholders
resource "aws_lb_target_group" "db" {
  name_prefix        = "db-"
  port               = 5432
  protocol           = "TCP"
  target_type        = "ip"
  vpc_id             = var.vpc_id
  preserve_client_ip = true

  lifecycle {
    create_before_destroy = true
  }
}

AWS also says preservation changes take effect only for new TCP connections, so recycle connection pools after the change before you measure again.

An ip target group created in Terraform still needs something that registers pod addresses as pods move. The AWS Load Balancer Controller's TargetGroupBinding attaches a Service's pods to an existing target group provisioned outside Kubernetes (TargetGroupBinding). Clusters that let the controller build the NLB from a Service make the same choice with two annotations, service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip and service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: preserve_client_ip.enabled=true. The controller's ip mode needs native VPC pod networking such as the Amazon VPC CNI (Service annotations).

Where the same pattern hides

Look at any internal NLB with instance targets whose clients run on the registered nodes: in-cluster databases and caches, and internal APIs that pods reach by a load balancer hostname instead of the cluster DNS name. Lost client addresses are a related trap on the ingress side, covered in our ingress-nginx retirement guide, and the networking items in the Kubernetes production readiness checklist are the wider list to run through.

When a share of in-cluster connections time out and retries keep hiding it, send us the details. We would start with the target type and the preservation setting on every internal NLB whose clients run on registered nodes, then run the connect loop from one pod per node.