API Gateway WebSocket Stage Rollback With a Stated Blast Radius

Rolling back a WebSocket stage in API Gateway takes one update-stage call that points the stage at the deployment it served before, because a stage is associated with a single deployment ID at a time (AWS: deploy WebSocket APIs). We ran three API Gateway changes for a client in the same weeks as its application releases: routing the dev REST and WebSocket APIs through an HAProxy VPC link, moving a staging WebSocket stage that had not been redeployed for years off a legacy domain, and moving a WebSocket API's connect and disconnect routes to a new microservice. All three landed. The staging route move needed one overnight rollback, and the production switch went through.

Each change was scoped before it started. The VPC link switch was tested first on a service outside the release, the legacy stage move came with a stated blast radius and its rollback command, and the route move came with a warning about start order.

A stage serves one deployment

Edits to routes, integrations and authorizers do not reach a live stage on their own. Every one of them needs a new deployment, and the stage keeps serving the deployment it is associated with until someone changes its deploymentId (same AWS page as above). That turns the rollback into a pointer change. Record the deployment ID the stage serves now, make the change, and if something goes wrong, point the stage back.

# WebSocket API: record what the stage serves before the change
aws apigatewayv2 get-stage --api-id "$API_ID" --stage-name "$STAGE" \
  --query DeploymentId --output text

# Confirm that deployment still exists, and see how old it is
aws apigatewayv2 get-deployments --api-id "$API_ID" \
  --query 'Items[].[DeploymentId,CreatedDate,Description]' --output table

# The rollback: point the stage back at the recorded deployment
aws apigatewayv2 update-stage --api-id "$API_ID" --stage-name "$STAGE" \
  --deployment-id "$PREVIOUS_DEPLOYMENT_ID"

REST APIs take the same change as a patch operation, since replace on /deploymentId is a supported operation for UpdateStage (AWS: patch operations):

# REST API: same pointer, set through a patch operation
aws apigateway get-stage --rest-api-id "$REST_API_ID" --stage-name "$STAGE" \
  --query deploymentId --output text

aws apigateway update-stage --rest-api-id "$REST_API_ID" --stage-name "$STAGE" \
  --patch-operations op=replace,path=/deploymentId,value="$PREVIOUS_DEPLOYMENT_ID"

Redeploying a stage that sat untouched for years

The staging WebSocket stage that had to leave the legacy domain had not been redeployed for years. AWS describes each stage as a snapshot of the API, so a fresh deployment of a stale stage can publish route and integration edits that were saved long ago and never went live, next to the one change you meant to make.

Our engineer took a 30-minute QA window with approval. Before it opened he stated the blast radius as new WebSocket connections only, with open sockets unaffected, and gave the single update-stage rollback to the previous deployment. He made no code change. When the window closed he read the logs: the domain was the only thing that had changed, and no new errors had appeared.

Written as a change note, that fits in a few lines:

# Illustrative change note; the variables are placeholders
Window:       30 minutes for QA, approved
Change:       staging stage moves off the legacy domain, no code change
Blast radius: new WebSocket connections only, open sockets unaffected
Rollback:     aws apigatewayv2 update-stage --api-id "$API_ID" \
                --stage-name "$STAGE" --deployment-id "$PREVIOUS_DEPLOYMENT_ID"
Verify:       logs show the domain as the only change, and no new errors

A line like "open sockets unaffected" has an expiry that belongs in the same note. API Gateway caps a WebSocket connection at 2 hours and closes idle ones after 10 minutes, and neither quota can be raised (AWS: WebSocket quotas). Two hours after the switch, every client that is still connected is on a connection it opened through the new deployment. That number belongs in the note for QA, because it bounds how long the old behavior can still appear in their testing.

Testing a VPC link switch beside a release in flight

The dev REST switch to the HAProxy VPC link was announced for a set time. A release was in flight, so our engineer tested the switch first on one microservice outside the change's scope, checked the logs for errors and confirmed the release was undisturbed. We cut kube-proxy over one node at a time for the same reason in our IPVS to nftables playbook: the first target should be small and unrelated to whatever else is moving that day.

The WebSocket API went through the VPC link behind its own flag. With one flag per API, the REST and WebSocket switches can be turned on and rolled back separately, and a problem with sockets does not pull the REST traffic back with it.

Putting the stage deployment under Terraform without production

We brought the WebSocket API's stage deployment under Terraform with production excluded, to break a cycle between the deployment and the API mapping. Leaving production out also meant that no apply from this work could move the production stage.

Once Terraform owns the deployment, check what happens to the old one. The AWS provider recommends create_before_destroy on aws_apigatewayv2_deployment so redeployments are ordered properly, and a changed triggers map forces a new deployment (provider docs). That lifecycle rule creates the replacement before destroying the current resource (Terraform docs). After an apply that replaces the deployment, the ID in your rollback line may no longer exist. That is why the get-deployments call sits in the block above, and it belongs before the window opens.

Our engineer also added a real WSS Connect request to the team's Bruno collection, with a hostname and token per environment. Authorization on a WebSocket API can be configured on the $connect route only (AWS: $connect and $disconnect), so a REST health check says nothing about whether a client with a real token can open a socket. The saved request answers that question in each environment.

A clean plan for a service that was never there

For the route move, our engineer wrote the Terraform and got a clean staging plan. Then he found that the new microservice had never been provisioned in staging at all: no ECR repository, no secrets, no IAM role and no REST route. A plan describes the resources in its own configuration. It cannot tell you that the service those routes will call does not exist. We still want a plan on every pull request, as in our Terraform anti-patterns post, and we read a clean one as a statement about the gateway alone.

He built the missing infrastructure the same day and warned that the app had to be running before the API Gateway flip, or every connect would fail. The AWS page on $connect describes that mechanism: until the integration on $connect completes, the upgrade request stays pending, and if the integration fails the connection is not made (AWS: $connect and $disconnect). A connect route pointed at a service that is not up turns away every new client.

The disconnect route needs a different kind of care. $disconnect runs after the connection has already closed, and AWS calls it a best-effort event whose delivery it cannot guarantee. A service that takes over $disconnect should have another way to clean up records for connections that are gone.

He rolled the WebSocket stage back to its previous deployment overnight and left a one-command infrastructure deploy for the morning.

Two fixes after the switch

The next day the API Gateway deployments were switched once the pod was up. Two follow-on failures were fixed after that: a missing database schema, and a Redis per-instance endpoint that did not resolve, which was replaced with the cluster primary endpoint. Both sat in the service's own dependencies, behind the integration. A route move needs a test of the service and what it depends on, and a test of the gateway alone would have passed. The production cutover ran later and went through. Staging first and production after is the promotion order we lay out in our CI/CD pipeline architecture framework.

If there is a stage change on your list, send us a note. We would start with the deployment ID each stage serves today and how old it is, then check that every route's integration target exists and answers before anything is switched.