Drain SQS Backlog With Lambda: Batch Size, Concurrency and No Purge

One team's production SQS queue held about 39 million messages waiting for its Lambda consumer, and we drained it from the infrastructure side without purging any of them. The event source mapping's batch size went from 10 to 30 to 100, the function's reserved concurrency went from 25 to 40 to 70, and the queue ran with a 180-second visibility timeout. The function's serverless.yml also set reserved concurrency to 10, while production ran 25. The permanent fix for the growth sits with the application team. The drain came down to three settings that limit each other, plus one number in the repository that did not match the account.

A purge is the one step you cannot undo

PurgeQueue deletes the available messages in a queue, in-flight messages included, and AWS states that you can't retrieve any message it deleted. The deletion takes up to 60 seconds. On a backlog of tens of millions that is the quickest way to an empty queue, and each of those messages is work some part of the system still expects to happen.

We leave a purge to whoever owns the data and treat the drain as the default: make the consumer outrun the producer for long enough that the visible count falls. Everything below assumes the consumer can safely process old messages late. Ask the application team that question before you change a single setting.

Batch size multiplies the work per invocation

For a standard queue, the Lambda event source mapping accepts a batch size of up to 10,000 records, and a FIFO queue stops at 10. Any batch size over 10 also needs MaximumBatchingWindowInSeconds set to at least 1 second. Two limits sit on top of the number you configure. The function timeout has to cover a whole batch, and Lambda sends fewer records than the batch size when the records and their metadata would pass the 6 MB payload quota for synchronous invocation.

Batch size is the cheapest lever because it leaves the number of running copies of the function alone. At the same concurrency, each invocation of a batch-100 mapping carries ten times the messages of a batch-10 one. Whether that turns into ten times the throughput depends on time per message. A handler that calls a downstream service once per record gets slower as the batch grows, and the gain shrinks with it.

The drain moved in steps, 10 then 30 then 100. Stepping lets you read the function's duration and error count at each size before you go further, and it keeps the change easy to reverse. The commands are short:

# Illustrative: placeholder UUID, function name and queue URL
aws lambda update-event-source-mapping \
  --uuid a1b2c3d4-5678-90ab-cdef-11111EXAMPLE \
  --batch-size 100 \
  --maximum-batching-window-in-seconds 1

aws lambda put-function-concurrency \
  --function-name example-worker \
  --reserved-concurrent-executions 70

aws sqs set-queue-attributes \
  --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/example-queue \
  --attributes VisibilityTimeout=180

Reserved concurrency sets the ceiling

Reserved concurrency is both the minimum and the maximum number of concurrent instances a function gets, and no other function in the account can use it. AWS suggests it as a way to keep a function from overwhelming downstream resources such as database connections, so check what the consumer calls before you raise it.

The SQS poller on its own would go much higher. In the default mode, Lambda starts with five concurrent invocations once messages are available, adds up to 300 more per minute while messages remain, and stops at 1,250 concurrent invokes for one event source mapping. A function reserved at 25 never runs more than 25 copies, however deep the queue gets. Taking it to 40 and then 70 raised the ceiling the drain could reach, and the batch size decided how much each of those copies took per call.

At the ceiling, invocations are throttled. AWS documents that on a throttled invocation Lambda backs off by reducing the concurrency it gives the event source mapping, and keeps retrying the message until its timestamp passes the queue's visibility timeout. That retry window is the reason the next setting matters.

AWS also offers a cap on the event source itself. Maximum concurrency on an SQS event source mapping takes a value from 2 to 1,000 and limits how many instances that one queue can invoke. It is independent of reserved concurrency, and the docs say to keep the function's reserved concurrency at or above the total maximum concurrency of its SQS event sources, otherwise Lambda might throttle your messages. For a function fed by one queue we would set both: maximum concurrency on the mapping as the working limit, and reserved concurrency at or above it.

For a backlog beyond what 1,250 concurrent invokes can clear, provisioned mode gives the mapping a configured minimum and maximum of dedicated event pollers, each handling up to 10 concurrent invokes. It costs extra, and it cannot be combined with maximum concurrency.

The visibility timeout has to cover a batch and its retries

AWS's guidance is to set the source queue's visibility timeout to at least six times the function's timeout, and, once a batching window is in use, six times the function timeout plus MaximumBatchingWindowInSeconds. The margin gives Lambda room to retry when the function is throttled while it processes a previous batch. Lambda also refuses to create or update a mapping whose function timeout is longer than the queue's visibility timeout. On the SQS side, VisibilityTimeout accepts 0 to 43,200 seconds and defaults to 30.

Run the rule backwards from the drain's 180 seconds and it bounds the function. Without a batching window, six times the timeout fits in 180 seconds for a function timeout of 30 seconds or less. With the 1-second window that any batch above 10 requires, the bound is (180 - 1) / 6, about 29.8 seconds, and a longer window lowers it again. A bigger batch pushes duration up, so the batch size, the function timeout and the visibility timeout belong in the same change.

Partial batch responses before bigger batches

By default, an error anywhere in a batch returns every message in it to the queue, including the ones the function already processed. At a batch size of 10, one bad record sends nine good ones back; at 100 it sends 99. Partial batch responses change that. Set FunctionResponseTypes to ReportBatchItemFailures on the mapping, have the handler return the failed message IDs as itemIdentifier values in a batchItemFailures list, and only those messages become visible again. AWS adds that with this setting Lambda does not scale down polling when invocations fail. A thrown exception still fails the whole batch.

Earlier in the same engagement, in a development environment, our engineer had analysed failures in a different function of the same consumer service. He found downstream latency, a 3-second client timeout retried 3 times, no ReportBatchItemFailures, and a dead-letter queue at 16.5k messages, and suggested fixes. Nothing in our notes ties that function to the production backlog. We mention it because those are the checks we would run on any consumer before raising its batch size: how long one message can take once retries are counted, whether one failure sends the whole batch back, and where a message goes after repeated failures. For the last one, AWS recommends a redrive policy with maxReceiveCount of at least 5 on the source queue, so Lambda gets a few retries before a message moves to the dead-letter queue.

Make the repository agree with production

The function's serverless.yml set reserved concurrency to 10. Production ran 25 before the drain, and the drain took it to 40 and then 70. In the Serverless Framework that value is one optional line per function, reservedConcurrency, and the SQS settings sit under the function's sqs event:

# Illustrative serverless.yml: placeholder names, values are not a recommendation
functions:
  example-worker:
    handler: src/worker.handler
    timeout: 25
    reservedConcurrency: 40
    events:
      - sqs:
          arn: arn:aws:sqs:us-east-1:123456789012:example-queue
          batchSize: 50
          maximumBatchingWindow: 1
          functionResponseType: ReportBatchItemFailures
          maximumConcurrency: 40

What the next deploy does with a gap like 10 against 25 depends on how production got to 25, and the repository does not record that. Either way, the number an engineer reads in the file is not the number the function runs with, and a drain widens the gap. The plan we handed to the application team lists that drift as something to fix. After a drain, write the end state back to the file, or set the function back on purpose, and keep the event source settings next to it. The scheduled comparison we describe for Terraform drift detection works the same way here: compare the deployed function configuration with the file on a schedule, and the mismatch shows up before the next backlog does.

What to watch while it drains

The SQS CloudWatch metrics tell you whether the drain is working:

• ApproximateNumberOfMessagesVisible is the backlog itself, the messages available for retrieval.
• ApproximateNumberOfMessagesNotVisible counts in-flight messages, received but not yet deleted or expired. AWS points to it for spotting processing delays and stuck consumers.
• ApproximateAgeOfOldestMessage shows how long the oldest unprocessed message has waited. With partial batch responses on, AWS reads a sharp rise in it, or NumberOfMessagesDeleted falling to 0, as a sign the function is not returning failed messages correctly.
• Lambda's ConcurrentExecutions shows whether the function actually reached its new reserved concurrency.

A fast drain also changes what everything downstream of the consumer sees. Before this one, the service owners were warned and the support team got a notice that statuses and alerts could flap while the queue drained. If the consumer runs on instances or containers instead of Lambda, the same backlog is a scaling signal: queue backlog per instance is the metric we use for SQS workers in our Auto Scaling guide.

When a queue grows faster than its consumer, we start by putting the event source mapping, the function's concurrency and timeout, and the queue's visibility timeout next to what the repository says they should be. If you want us to look at yours, get in touch.