Sync production data to staging: the rules we wrote after it failed quietly

A Step Functions workflow that restored production databases into staging and QA reported ExecutionSucceeded on runs where the sync had failed. Our engineer worked through the failures with the client one at a time. Some restores ran past the state machine's 4-hour timeout. A DynamoDB step reported success when the restore had only started. The failure notification read "FAILED: null", the sync Lambdas wrote no logs, and one run wiped the automation and password users in staging and QA. We filed each defect separately and wrote down operating rules that hold whether or not a given fix has shipped. Those rules, and the AWS behavior behind each failure, carry over to any pipeline built from the same parts.

A succeeded execution can still hold a failed sync

Step Functions reports on the states it ran. An execution keeps going until it reaches a terminal state or a runtime error occurs, per the state machine structure reference, and the data underneath plays no part in that. AWS documents several ways for an execution to end green while the work failed, and a sync workflow can contain all of them:

• A Task that calls an AWS API with the default Request Response pattern moves on "immediately after it receives an HTTP response", and the integration patterns page adds that "Step Functions will not wait for a job to complete." For a restore, that response only means the request was accepted.
• A Catch that routes to a notify step and then ends normally turns a failed step into a succeeded execution. The Catch examples in the error handling guide end in a Pass state with "End": true.
• A Lambda function that catches its own exception and returns a payload has reported no error, and Retry and Catch only match on errors.

That is why one of the rules our engineer wrote moves the verdict out of Step Functions. The result-check workflow decides whether a sync worked. The execution status only tells you which states ran.

Polling a restore that Step Functions will not wait for

The DynamoDB step shows the problem most clearly. RestoreTableFromBackup creates a new table from a backup, and the table shows CREATING until the restore is finished and ACTIVE afterwards, according to the restore tutorial. The integration patterns table lists Run a Job (.sync) as not supported for the optimized DynamoDB integration and for AWS SDK integrations alike, so the workflow has to poll the table itself.

{
  "Comment": "Illustrative fragment. State names, error names and input fields such as $.targetTable are placeholders.",
  "StartAt": "RestoreTable",
  "States": {
    "RestoreTable": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:dynamodb:restoreTableFromBackup",
      "Parameters": {
        "TargetTableName.$": "$.targetTable",
        "BackupArn.$": "$.backupArn"
      },
      "ResultPath": "$.restore",
      "Catch": [ {
        "ErrorEquals": [ "DynamoDb.TableAlreadyExistsException", "DynamoDb.TableInUseException" ],
        "ResultPath": "$.error",
        "Next": "TargetNotFree"
      } ],
      "Next": "WaitForRestore"
    },
    "WaitForRestore": {
      "Type": "Wait",
      "Seconds": 60,
      "Next": "DescribeTarget"
    },
    "DescribeTarget": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:dynamodb:describeTable",
      "Parameters": { "TableName.$": "$.targetTable" },
      "ResultPath": "$.describe",
      "Next": "RestoreFinished"
    },
    "RestoreFinished": {
      "Type": "Choice",
      "Choices": [
        {
          "And": [
            { "Variable": "$.describe.Table.TableStatus", "StringEquals": "ACTIVE" },
            { "Variable": "$.describe.Table.RestoreSummary", "IsPresent": true },
            { "Variable": "$.describe.Table.RestoreSummary.RestoreInProgress", "BooleanEquals": false },
            { "Variable": "$.describe.Table.RestoreSummary.SourceBackupArn", "StringEqualsPath": "$.backupArn" }
          ],
          "Next": "RestoreConfirmed"
        },
        { "Variable": "$.describe.Table.TableStatus", "StringEquals": "CREATING", "Next": "WaitForRestore" }
      ],
      "Default": "NotOurRestore"
    },
    "RestoreConfirmed": { "Type": "Succeed" },
    "TargetNotFree": { "Type": "Fail", "Error": "TargetTableNotFree", "CausePath": "$.error.Cause" },
    "NotOurRestore": { "Type": "Fail", "Error": "RestoreNotConfirmed", "Cause": "Target is not an active restore of the requested backup." }
  }
}

The DynamoDb. prefix on the error names comes from the SDK integrations list. The Default branch is there for the race in a later section: a table that the application created on its own will be ACTIVE, but it will not name your backup as its SourceBackupArn.

From a shell, aws dynamodb wait table-exists polls Table.TableStatus every 20 seconds and exits with code 255 after 25 failed checks, per the CLI reference. That is 500 seconds. The restores in this workflow ran for more than four hours, so a script built on the waiter needs a loop around it and has to read exit code 255 as "still unknown".

The 4-hour timeout skips your failure handler

A top-level TimeoutSeconds fails the execution with States.Timeout once it is exceeded, and the execution's status becomes TIMED_OUT in DescribeExecution. The error handling guide spells out the consequence. Catchers exist for Task, Parallel and Map states, "but not for top-level state machine execution failures." Any notify step that a Catch would reach never runs when the whole execution times out.

The guide suggests listening for TIMED_OUT events on EventBridge instead. Standard workflows send an Execution Status Change event to the default event bus on every status change, delivered on a best-effort basis per the EventBridge integration page, and a rule on this pattern picks up the endings a Catch cannot see:

{
  "source": ["aws.states"],
  "detail-type": ["Step Functions Execution Status Change"],
  "detail": {
    "status": ["FAILED", "TIMED_OUT", "ABORTED"],
    "stateMachineArn": ["arn:aws:states:REGION:ACCOUNT_ID:stateMachine:example-sync"]
  }
}

Express workflows do not emit these events, so this only covers standard ones. A timeout also says nothing about how far the restores it started have got. Read the targets before anyone reruns a timed-out sync.

"FAILED: null" and Lambdas that log nothing

A failed run arrived with "FAILED: null" in the notification and nothing in the sync Lambdas' log groups, so there was no reason attached and nowhere to look for one. One way to get exactly that message is a template that prints the execution's output on failure. DescribeExecution documents output as "set only if the execution succeeds. If the execution fails, this field is null." The reason sits in error and cause, which the API and the EventBridge event both carry. Build the message from those two fields, and give every Fail state an Error and a Cause (or ErrorPath and CausePath) as described on the Fail state page, so the text names the step that gave up.

For a Lambda function that writes nothing at all, start with its execution role. Lambda sends logs to CloudWatch Logs only when the role has logs:CreateLogGroup, logs:CreateLogStream and logs:PutLogEvents, which the AWSLambdaBasicExecutionRole managed policy grants, according to the Lambda logging guide. The same page warns that logs can take 5 to 10 minutes to show up after an invocation, so allow for that before calling a function silent.

The DynamoDB delete and recreate race

Because RestoreTableFromBackup creates a new table, the target name has to be free. The call fails with TableAlreadyExistsException when a table by that name exists and with TableInUseException while one is being created or deleted. A sync that keeps the staging table's name therefore deletes the table first, and that opens a window. In this workflow the DynamoDB step deleted the target table and the application recreated it before the restore landed.

The restored table also arrives without part of its configuration. The API reference lists what you must set up by hand afterwards: auto scaling policies, IAM policies, CloudWatch metrics and alarms, tags, stream settings and TTL settings. A delete-and-restore sync has to put those back on every run.

For a manual DynamoDB sync, the rule is an order of operations:

1. Scale the consuming service down first, so nothing recreates the table between the delete and the restore.
2. Restore.
3. Scale the service back up.
4. Clear the cache.

The console can show a working table as not ready, so the rule says to check with the CLI. describe-table returns the status and the restore summary under Table:

aws dynamodb describe-table --table-name "$TABLE" \
  --query 'Table.[TableStatus,RestoreSummary.RestoreInProgress,RestoreSummary.SourceBackupArn]' \
  --output text

You want ACTIVE, no restore in progress, and the ARN of the backup you meant to restore.

The operating rules we wrote down

The target environment must run a build no older than production, or the restore fails on whichever schema lags. Teams that promote one build artifact through every environment, the model in our CI/CD architecture guide, can check this rule by comparing two version strings before the sync starts.

A failed sync does not damage the target, because the rollback restores all 18 databases from a pre-run backup. With that rollback in place, the workflow can afford to fail a run on any doubt; the price of a false alarm is one rollback.

If your lower environments keep users of their own, like the automation and password users that one run wiped, make their presence part of the result check. A production backup has no reason to contain them. A restore that someone ran and checked is also better audit evidence than a runbook that says one would work, a point we made in our SOC 2 guide for SaaS companies.

If a sync like this feeds your staging and QA environments, we would start by comparing the runs your state machine marked as succeeded with what the targets held afterwards, then trace every step that calls a restore API and moves on without polling. Tell us how your sync is built and we will begin with those two checks.