getaddrinfo EAI_AGAIN in Kubernetes: Count the Errors, Then Check Your StatsD Client

A report of getaddrinfo EAI_AGAIN errors from a Node.js scheduler service reached us filed as a proxy issue. The scheduler calls another service by its in-cluster name, so each error was a failed DNS lookup. Before anyone changed a setting, our engineer counted: 14 days of logs held exactly one production event for that service. The count moved the work off the service in the report and onto a shared StatsD module that builds its client with no errorHandler and no cacheDns, and every app that imports the shared Redis stream module carries that client with it.

For the node side of intermittent DNS failures, such as connection-tracking limits and CoreDNS capacity, see the networking and observability items in our Kubernetes production readiness checklist. This case sits one layer up, in application code: Node.js in an nx monorepo, where one shared library ships inside every app that imports it.

What the error code says

EAI_AGAIN is a return code of getaddrinfo(3). The Linux man page defines it as "The name server returned a temporary failure indication. Try again later." Its sibling EAI_FAIL is the permanent version. Node.js reaches getaddrinfo through dns.lookup(), which the Node.js DNS documentation describes as "a synchronous call to getaddrinfo(3) that runs on libuv's threadpool". The same page says various networking APIs call dns.lookup() internally to resolve host names. An HTTP request to a service name depends on that lookup. So does a UDP metric sent to a host name: the dgram documentation says that "If the value of address is a host name, DNS will be used to resolve the address of the host."

Inside a pod the lookup has more steps than on a laptop. The Kubernetes DNS documentation shows the kind of /etc/resolv.conf that kubelet writes for each pod:

nameserver 10.32.0.10
search <namespace>.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The resolv.conf man page defines ndots as the number of dots a name needs "before an initial absolute query will be made", and the default there is 1. At 5, a short service name is tried with the search domains appended before it is tried as written, so one lookup in code can mean several queries to the cluster DNS server. A temporary failure code is a reason to measure how often it happens, and whether anything retried, before anyone changes the proxy or CoreDNS.

Two weeks of logs, split by source

Split by environment and by the process that logged the error, the two weeks of events looked like this:

| Where | Events in 14 days | What the logs showed | |---|---|---| | Production, the scheduler service | 1 | The only production event from the service in the report | | Production, the OpenTelemetry collector | 3 | The collector resolving the Kinesis endpoint | | Staging | 14 | All inside one window on a single day, each followed by a successful axios retry |

The staging row is one DNS blip: fourteen events in one window on one day, each recovered by an axios retry. In production the scheduler logged the error once in two weeks. The collector's three events came from a different process resolving a different name, an AWS endpoint, so they tell you nothing about the call path in the report. (If the collector is on your own list, our post on OpenTelemetry Collector cost control covers its pipeline config.)

That evidence closed the report. A single cluster-wide count of EAI_AGAIN would have mixed the collector's lookups in with the scheduler's and made the reported service look worse than it was. If your logs are exported as JSON lines, a grouping like this one produces the same split:

# Illustrative: field names depend on your log shipper; acme-logs.jsonl is a placeholder export.
grep -h 'EAI_AGAIN' acme-logs.jsonl \
  | jq -r '[.env, .container, .timestamp[0:10],
            (.message | capture("EAI_AGAIN (?<host>[^ ]+)").host)] | @tsv' \
  | sort | uniq -c | sort -rn

The date column shows whether the events cluster on one day. The host column separates an in-cluster service name from an endpoint outside the cluster.

Why the retry held and the metrics client might not

In staging, a successful axios retry followed every failed lookup, so a lookup that failed once worked on the next try. A StatsD client sends UDP packets, and the dgram documentation says where a failed send goes: "If a callback is not given, the error is emitted as an 'error' event on the socket object." The events documentation says what happens next when an emitter has no listener for 'error': "the error is thrown, a stack trace is printed, and the Node.js process exits." A metrics client that emits its send errors with nothing listening can turn one failed lookup into a dead process.

That is where the real defect sat. The shared StatsD module builds its client with no errorHandler and no cacheDns. Since the module is shared, the exposure reaches past the scheduler: any transient DNS failure can kill any app that imports the shared Redis stream module. The defect was already on the team's list, so it needed no new report, and the fix stays with the existing work on the StatsD module. Our record of the work does not show that fix shipping, and what follows is the change the defect calls for rather than a result we measured on that codebase.

The examples below use hot-shots, a Node.js StatsD client whose options carry these two names. Its README lists errorHandler as "A function with one argument. It is called to handle various errors. default: none, errors are thrown/logger to console", and cacheDns with default: false. The version you run matters. The hot-shots changelog entry for 15.0.0, dated 28 May 2026, adds a "Default error listener on every transport socket so that in the cases we didn't have one, an error doesn't crash the host". While writing this post we ran the probe from the last section against 14.3.1 and 17.1.1 on Node.js v22.22.0. With no errorHandler, 14.3.1 exited on the first failed lookup with Unhandled 'error' event; 17.1.1 printed the error and kept running.

Two settings with different jobs

errorHandler keeps the process alive. The Errors section of the hot-shots README says: "For broad error coverage, specify an errorHandler in your root client. It catches errors in socket setup, sending of messages, and closing of the socket." Set it once in the shared module and every app that imports the module gets it.

cacheDns cuts how often the lookup runs. It is off by default and "only used when protocol is udp". When it is on, the client caches the lookup for cacheDnsTtl, which defaults to 60000 milliseconds. With it off, each metric sent to a host name goes through DNS on its own, so a service that sends many metrics a second makes that many lookups, and each one can come back as EAI_AGAIN.

Turning on cacheDns does not make errorHandler optional. In the hot-shots UDP transport, a lookup that fails while refreshing the cache goes to the send callback and nothing is cached, so the next send tries again. In our probe, 14.3.1 with cacheDns: true and no errorHandler still exited. The cache lowers the number of lookups; the handler is what stops a failed one from ending the process. Caching has one cost to plan for: if the address behind the StatsD host name changes, metrics keep going to the old one until cacheDnsTtl runs out.

// Illustrative shared client: the host name and prefix are placeholders.
const StatsD = require('hot-shots');

const statsd = new StatsD({
  host: 'statsd.acme.example.internal',
  port: 8125,
  prefix: 'acme.',
  cacheDns: true,
  cacheDnsTtl: 60000,
  errorHandler: (error) => {
    // A failed metric send is logged and dropped; the process keeps running.
    console.warn(`statsd send failed: ${error.message}`);
  },
});

module.exports = { statsd };

During a long DNS problem a warning per packet can flood the logs. We would count failures in memory and log the count once a minute instead.

Finding every app that ships the shared module

In an nx monorepo the project graph tells you who is exposed. The Nx command reference describes nx graph --focus as showing "the project graph for a particular project and every node that is either an ancestor or a descendant", and --file writes the graph to a file:

# Illustrative: acme-statsd is a placeholder for the shared metrics project.
npx nx graph --focus=acme-statsd --file=acme-statsd-graph.html

Every app in that view ships the client, including apps that only import another shared library built on it, as with the Redis stream module in this case.

Then check the behaviour on the version in the lockfile, because the changelog only tells you what the newest release does. A name under .invalid fails every time: RFC 6761 says users "MAY assume that queries for "invalid" names will always return NXDOMAIN responses." Run this in a scratch pod built from the app's lockfile:

// Illustrative probe: statsd.invalid never resolves; acme.dns_probe is a placeholder metric.
const StatsD = require('hot-shots');

const probe = new StatsD({ host: 'statsd.invalid', port: 8125 });
probe.increment('acme.dns_probe');
setTimeout(() => console.log('still running after a failed lookup'), 2000);

The code in the error will be ENOTFOUND rather than EAI_AGAIN, since the dns.lookup() documentation says err.code is set to 'ENOTFOUND' when the host name does not exist. It still travels the same send path. On 14.3.1 the probe printed this and exited with code 1:

Error: Error sending hot-shots message: Error: getaddrinfo ENOTFOUND statsd.invalid
node:events:497
      throw er; // Unhandled 'error' event
      ^

If "still running after a failed lookup" never prints, every app in the nx graph has the same exposure, and the shared module needs an errorHandler before any DNS or proxy setting changes. Run the probe again once the handler is in.

When a DNS error report reaches your team, send us the two weeks of logs around it. We would group the events by the name being resolved before touching any network setting, then go through every shared library in the repo that opens a socket, starting with the metrics client, and check which ones have no error handler.