Restrict access to an internal NLB without locking yourself out

An internal Network Load Balancer created without a security group can never be given one afterwards. AWS states the rule directly: "If you create a Network Load Balancer without associating any security groups, you can't associate them with the Network Load Balancer later on." A load balancer in that state has no inbound filter of its own, and the same page says what that means: "all client traffic can reach the Network Load Balancer listeners and all traffic can leave the Network Load Balancer."

We recently locked down an HAProxy box that sat behind an internal NLB, carrying private traffic from API Gateway, Lambda and Kubernetes pods to internal services. It accepted traffic VPC-wide, through a Client VPN rule, and on all TCP ports from a peer network. To restrict access to an internal NLB you need two settings, a clear idea of what a source address can prove, and the right order. The order matters more than the rules do.

The security group has to exist when the load balancer is created

Whether the rest of this work is an afternoon of rule editing or a migration is decided by a choice someone made when the load balancer was first built. The same AWS page: "You can associate security groups with a Network Load Balancer when you create it." Once there is at least one group, "you can change the security groups associated with the Network Load Balancer at any time."

So an empty list is permanent. The Terraform AWS provider is explicit about what that means for your plan: for aws_lb resources of type network, "security groups cannot be added if none are currently present, and cannot all be removed once added. If either of these conditions are met, this will force a recreation of the resource." A new NLB means a new DNS name, new target group registrations and a change window.

Our practical rule after this engagement: every internal NLB gets a security group at creation, even when the first rule set is as wide as the VPC CIDR. Starting wide and narrowing later is a day of editing rules. Starting with nothing is a replacement. The team was about to stand up the next environment's NLB while we were still working on the dev box, and the one thing we asked them to hold was that creation, because the ordering could not be undone. Two environments that had already been built without groups were recreated later, which is exactly the cost we were trying to avoid repeating.

# Illustrative. Fake names; adjust ports, CIDRs and tags for your account.
resource "aws_lb" "internal_proxy" {
  name               = "acme-internal-proxy"
  internal           = true
  load_balancer_type = "network"
  subnets            = var.private_subnet_ids

  # Never leave this empty on a new NLB.
  security_groups = [aws_security_group.internal_proxy_nlb.id]

  enforce_security_group_inbound_rules_on_private_link_traffic = "on"
}

Client IP preservation acts on the target side

The load balancer and its targets evaluate source addresses separately, and treating them as one evaluation produces rules that do nothing.

The security group on the NLB is evaluated against the real client address as traffic arrives at the load balancer. That happens whether or not client IP preservation is on. What preservation controls is the source address your targets receive: with it off, "the source IP address is the private IP address of the Network Load Balancer," so the proxy's access log and the proxy's own security group see the load balancer rather than the caller.

The defaults are easy to get wrong because they depend on target type and protocol. AWS lists them as: instance type target groups enabled, IP type target groups on UDP, TCP_UDP, QUIC and TCP_QUIC enabled and not disableable, and IP type target groups on TCP and TLS disabled. A TCP target group registered by IP, which is the common shape for a proxy fleet, starts with preservation off.

# Illustrative. A TCP/ip target group starts with preservation disabled.
resource "aws_lb_target_group" "internal_proxy" {
  name        = "acme-internal-proxy-443"
  port        = 443
  protocol    = "TCP"
  target_type = "ip"
  vpc_id      = var.vpc_id

  preserve_client_ip = "true"
}

Turning it on has two consequences worth stating before you do. Changes "take effect only for new TCP connections," so existing sessions keep the old source address and a verification run that only looks at warm connections tells you nothing. And NAT loopback stops working: AWS documents that hairpinning "is not supported when client IP preservation is enabled," which matters when a target behind the NLB dials back into the same NLB and the connection lands on itself.

We would turn preservation on for a proxy tier whose targets never call back through their own load balancer, because having the caller's address in the access log and in the box's security group is worth more than the loopback path. Where targets do call back, leave it off and read the client address from the proxy protocol v2 header instead.

What goes in the allowlist, and what the defaults already let through

With real source addresses arriving, the rules become specific. On the dev box we allowed :80 and :443 from the Lambda subnets, the EKS pod subnets, a peer cluster's pod range masqueraded through ip-masq-agent, and the NLB's own address. Then we dropped the VPC-wide rule, the all-TCP rule from the peer network, and the Client VPN security group and route.

Three details change the shape of that rule set.

Health checks need no inbound rule. AWS: "Health checks are subject to outbound rules, but not inbound rules. You must ensure that outbound rules don't block health check traffic. Otherwise, the Network Load Balancer considers the targets unhealthy." People add an inbound rule for the health check port, see healthy targets, and conclude the rule was needed.

On the target side, reference the load balancer's security group rather than copying its CIDRs. The same page suggests adding "a rule to the security groups associated with your targets that references the security group associated with your Network Load Balancer," which "allows clients to send traffic to your targets through your Network Load Balancer, but prevents them from sending traffic directly to your targets." That second clause is the one worth having. A CIDR-based rule on the box leaves the box reachable around the load balancer, and the target registration guide adds that the health check port needs its own rule from the same source.

Blocked traffic is free. Teams ask whether a long allowlist costs load balancer capacity units, and the documentation answers it in one line: "You are not charged for traffic that is blocked by inbound security group rules." There is no cost argument for a wide rule.

Two sources an allowlist cannot tell apart

This is where a CIDR list stops being an access control, and it is the part we documented for the team rather than fixed.

PrivateLink traffic arrives as the load balancer. "Client IP preservation has no effect on inbound traffic from AWS PrivateLink." " The source IP address of the AWS PrivateLink traffic is always the private IP address of the Network Load Balancer." So the API Gateway private integrations reaching this box cannot be distinguished at the target from anything else coming through the load balancer. You can choose whether the NLB's own inbound rules apply to that traffic, with enforce_security_group_inbound_rules_on_private_link_traffic set to on or off, and with enforcement on "the source of the traffic is the private IP address of the client, not the endpoint interface." The target still sees one address for all of it. Anything that needs to tell those callers apart belongs above the network layer.

Pod subnets are node subnets. By default the Amazon VPC CNI "creates them in the same subnet as the node's primary network interface. It also associates the same security groups to the secondary network interface that are associated to the primary network interface." Allowlisting the pod subnets therefore admits every pod scheduled on those nodes, whichever workload it belongs to, unless you run CNI custom networking and give pods their own subnets.

Lambda is the same problem with a different name. Lambda creates a Hyperplane ENI per unique subnet and security group combination, and "Other functions in your account that use the same subnet and security group combination can also use this ENI." An allowlist entry for a Lambda subnet admits every function sharing that combination.

Where the choice exists, we reference a security group instead of a CIDR. A group is assigned to a specific workload; a CIDR covers whatever happens to be addressed inside it. For the two cases above, that means security groups for pods on EKS and a dedicated security group per Lambda function group, and accepting a coarse rule where neither is in place.

Verifying the lockdown while it is live

The order we recommend, starting with the one step we took on the dev box before touching a single rule:

1. Enable preserve_client_ip on the target groups and open a fresh connection. Confirm that the proxy access log now shows real client addresses rather than the load balancer's.
2. Add the narrow rules alongside the wide ones. Nothing changes yet, which is the point.
3. Remove the wide rules, one at a time, watching SecurityGroupBlockedFlowCount_Inbound on the load balancer. AWS notes that "Blocked traffic is not reflected in other metrics," so this metric and VPC flow logs are the only places a legitimate caller you forgot will show up.
4. Ask the application teams to run their own probes against the paths you cannot see from the network side.

Make the access log do the verifying. We moved logs and metrics off the box through an OpenTelemetry Collector installed by an SSM State Manager association, with a JSON access log carrying the real client IP and the NLB health checks archived, both off by default behind flags. Having the client address in a structured field is what turns "did we break anything" into a query.

The guardrails review added, and the one the reboot added

Review put three things into the change that we had not.

Strict CIDR validation with a /8 floor, so no rule can be widened back to a whole private range by a one-character edit. This is the kind of check that belongs in the pipeline rather than in a reviewer's head, which is the argument we make in integrating security into your CI/CD pipeline.

ip-masq-agent as system-node-critical, because the peer cluster's pod range only appears as the allowlisted address while the masquerade is running. A DaemonSet that can be evicted on a busy node is part of the allowlist, so it gets the priority class that says so.

IMDSv2 and an encrypted root volume on the box, which is ordinary hardening we folded in while the instance was already being changed.

The fourth came later and not from review: a security-patch reboot took one box down, so every box got restart: always, and production was flagged as needing two instances behind the load balancer. The broader set of AWS network and IAM controls we standardise on is in Terraform security best practices for AWS.

If you are looking at an internal NLB that accepts the whole VPC, the first thing we check is whether it has a security group at all, because that answers whether this is an afternoon or a migration, and the second is whether your proxy's access log already carries the caller's address, because without that you are narrowing rules blind. Tell us what the load balancer fronts and we will tell you which of the two you are in.