Redis Stream Consumer OOMKilled by Its Own Messages

Redis memory on a client's QA environment had been raised three times in a few months, and a stream consumer was crash-looping on two QA environments with more than 100 restarts each. Our engineer traced both to one audit-log stream that held 4.5 GB of the 5.12 GB in use (88%).

Nothing in the mechanism misbehaves. A consumer group, a batch read, a container memory limit and an entry size with no cap add up to a loop that more memory only postpones. The place to break it is the service that publishes the entries. For setting and testing memory limits in general, see our Kubernetes production readiness checklist. The case here is narrower: a consumer whose memory use was set by the entries it read.

What a 3.45 MB entry does inside a 256 MiB limit

Each entry carried the full old and new object, two whole tree structures included, so single entries reached 3.45 MB. An audit record has to say what changed. Copying the whole object before and after says it, at a size set by the object rather than by the change.

The consumer read 10 entries per batch. COUNT on XREADGROUP is the maximum number of entries to return per stream and has no notion of bytes, so ten entries of that size meant tens of MB parsed inside a 256 MiB container limit. Parsing adds to that. The raw reply and the objects decoded from it sit in memory together, and in our experience nested structures take up far more memory decoded than they do on the wire.

A container that keeps allocating past its memory limit is terminated, its status records reason OOMKilled with exit code 137, and the kubelet restarts it if it can be restarted. Here the pod was OOMKilled before it could XACK. For how OOM kills behave on cgroup v2 nodes, see our cgroup v2 migration post.

How the pending entries list keeps the loop going

XREADGROUP records each entry it delivers in the group's pending entries list (PEL), and the entry stays there until the consumer acknowledges it with XACK. A consumer killed in the middle of a batch acknowledges nothing, so every entry in that batch stays pending.

Claiming exists for stale pending entries. XAUTOCLAIM transfers entries that have been pending longer than a minimum idle time to another consumer, and each claim increments the entry's delivery count. That is the right design when a consumer died of something unrelated to the message. When the entry itself is what kills consumers, the claim hands the same oversized entries to the next pod, which dies the same way before it can acknowledge. In this case the next consumer re-claimed the entries, and the count reached more than 4,000 reclaim attempts.

A reclaim loop built on XAUTOCLAIM has a batch size of its own, and its COUNT defaults to 100 unless the caller sets it. Set it as deliberately as the read batch.

Restarting under the same consumer name lands in the same place. The Redis docs give reading with an ID of 0 as the recovery step after a crash, because it returns the consumer's own pending entries, and here those are the entries that caused the crash.

The Redis documentation names the signal to watch: messages that consumers systematically crash on "will exhibit high attempted delivery counts that can be detected by monitoring". Restart counts on their own look like any other crash loop. The delivery count on the oldest pending entries is what separates a poison payload from a bug in the consumer.

Three reads that point at the payload

When a group's consumers keep restarting, we look at three things before anyone changes consumer code:

# Illustrative: the key, group, pod and namespace names are made up.
# Bytes one key and its value take in RAM
redis-cli MEMORY USAGE acme:audit-events

# Pending entries: ID, owner, ms since last delivery, times delivered
redis-cli XPENDING acme:audit-events acme-workers - + 10

# Why the container last exited: look for reason OOMKilled and exitCode 137
kubectl get pod acme-consumer-0 --output=yaml --namespace=acme-qa

When one key holds most of the memory, the pending entries have been delivered hundreds or thousands of times, and the last state is OOMKilled, you are looking at this failure. A leak in the consumer usually takes time to build up. A poison payload kills the pod on the same entries every time it reads them.

Fix the entry size in the publisher

Our engineer placed the fix in the application: publish a field-level diff or strip the two trees, and cap payload size in the publisher. The publisher is the one component that knows what the event means, so it is the only place that can make the entry smaller and keep the record useful. Every consumer group downstream inherits the size it picks, and so does Redis memory, 88% of which sat in this one stream.

Trimming bounds the entry count and leaves the bytes open. MAXLEN sets the maximum number of entries that may remain in the stream, so a stream trimmed to N entries can still hold N entries as large as its largest. A byte cap at publish time bounds the consumer's batch and the Redis footprint together.

A sketch of the publisher side, with the trees recorded as changed or unchanged instead of copied:

# Illustrative: field names, the event shape and the 64 KiB cap are made up.
import json

MAX_ENTRY_BYTES = 64 * 1024
TREE_FIELDS = {"acme_tree_a", "acme_tree_b"}


def audit_payload(old: dict, new: dict) -> bytes:
    changed = {}
    for field in sorted(old.keys() | new.keys()):
        if old.get(field) == new.get(field):
            continue
        if field in TREE_FIELDS:
            changed[field] = "changed"
        else:
            changed[field] = {"old": old.get(field), "new": new.get(field)}
    payload = json.dumps({"id": new["id"], "changed": changed}).encode()
    if len(payload) > MAX_ENTRY_BYTES:
        raise ValueError(f"audit entry is {len(payload)} bytes, cap is {MAX_ENTRY_BYTES}")
    return payload

If the audit trail has to show what changed inside a tree, diff inside the tree instead of copying it. Set the cap from the entry sizes you see today plus headroom. We would rather see a publish fail in the service that owns the event than have every consumer find out downstream. We handed the root cause and the fix to the application team.

Stopgaps on the consumer and the stream

Until a publisher fix ships, the consumer and the stream still have to survive the entries already written. Our engineer offered consumer-side mitigations and XTRIM as a stopgap, and approved the team's memory-limit PR at 768 MiB instead of the 512 MiB they proposed. These are the options we would weigh on any consumer in this state:

• Lower COUNT, and on Redis 8.10 or later add MAXSIZE, a byte cap on the whole reply that arrived in the 8.10.0 GA release in July 2026. At least one entry is always returned, so MAXSIZE bounds the batch and cannot shrink a single entry.
• Cap the retries. XPENDING shows how many times each pending entry was delivered, and past a threshold you pick, the consumer can copy the entry to a separate stream for inspection and XACK it, so one entry stops taking the group down.
• Trim the stream as a stopgap only. By default XTRIM leaves the IDs of trimmed entries in every group's PEL (KEEPREF), and DELREF, available since Redis 8.2, removes those references too. With the default, a consumer reading its pending history gets a null in place of each trimmed entry, and XAUTOCLAIM drops the missing IDs from the PEL and returns them in a separate list. The consumer has to handle that null. On an audit stream, trimming also deletes audit records, so check what else reads the stream before you trim it.
• A higher memory limit keeps today's entries inside the container and puts no limit on the size of the next one.

# Illustrative: names and numbers are placeholders. MAXSIZE needs Redis 8.10+, DELREF needs 8.2+.
XREADGROUP GROUP acme-workers consumer-1 COUNT 10 MAXSIZE 4194304 BLOCK 2000 STREAMS acme:audit-events >
XAUTOCLAIM acme:audit-events acme-workers consumer-1 60000 0-0 COUNT 1
XTRIM acme:audit-events MAXLEN ~ 10000 DELREF

For a consumer that keeps restarting, we would look first at the delivery count on its oldest pending entries, then at the byte size of those entries next to the container's memory limit, before anyone raises the limit again. If those two numbers look like the ones here, send us the details and we will start at the publisher.