Invalid length for parameter imageManifest in an ECR retag step

A custom build in an nx monorepo failed on its tag-all-images step, and the docker-retag target for one service stopped with Invalid length for parameter imageManifest, value: 0. That 0 is the length of the manifest the retag step passed to PutImage. There was nothing to pass: the service was new, and there was no earlier build tag for the retag script to copy from.

Retagging is how a pipeline promotes an image it has already built, the build-once rule we describe in our CI/CD pipeline architecture framework. For what registries and tags are in the first place, start with container fundamentals. The failure described here happens at the copy step of that promotion.

What the message is telling you

The error never reaches ECR. It comes from botocore, the library under the AWS CLI and boto3, which checks every parameter against the service model before the request goes out. For a string, it compares len(param) with the model's minimum and, when it falls short, formats Invalid length for parameter {name}, value: {param}, valid min length: {min_allowed} (botocore validate.py). The report is raised as a ParamValidationError, whose message opens with Parameter validation failed: (botocore exceptions.py). So value: 0 is a length, and the string was empty.

PutImage marks imageManifest as required, with a minimum length of 1 and a maximum of 4194304 (PutImage API reference). An empty string fails the first bound.

Two conclusions follow. IAM permissions, repository policies and tag immutability can be ruled out, because the request was never sent. The bug sits one step earlier, in whatever produced the value passed as --image-manifest.

Where the empty manifest comes from

AWS documents a retag as two CLI calls: read the manifest of the existing tag with batch-get-image, then write it back under the new tag with put-image --image-tag. That skips the Docker pull and push, which for a large image saves real bandwidth and time (Retagging an image in Amazon ECR). These are the commands from that page:

MANIFEST=$(aws ecr batch-get-image --repository-name amazonlinux --image-ids imageTag=latest --output text --query 'images[].imageManifest')

aws ecr put-image --repository-name amazonlinux --image-tag 2017.03 --image-manifest "$MANIFEST"

The first call is the weak point. A missing tag is absent from BatchGetImage's error list. The service still answers with HTTP 200 and reports problems in a failures array next to the images array (BatchGetImage API reference). When the tag does not exist there is no image object for the query images[].imageManifest to select, so MANIFEST comes back empty and a script written like the example moves on to put-image, which stops with value: 0.

Retag scripts differ in their details, and the case below does not depend on them. Any script that copies a manifest from a source tag fails this way when the source tag has no image behind it.

A new service with nothing to copy

In the case we worked on, the failing docker-retag target belonged to a new microservice. Our engineer diagnosed it as a new service with no earlier build tag for the retag script to copy from.

A retag step rests on one assumption: every service it touches already has an image under an earlier tag. The first build of a new service is the one time that assumption is false for certain. The error text points at a parameter length, so the failed tag-all-images step reports a validation problem with a manifest and says nothing about a missing image. That gap between the message and the cause is why the step is worth a guard.

Building the new service instead of retagging it

Our engineer recommended adding the new service to the custom build's selected apps, next to the cronjobs. A selected app is pushed with the tag directly, and the retag step short-circuits for it. The developer rebuilt with that change and confirmed it worked.

We would make that the rule for any new service in a pipeline that promotes by retagging: its first build pushes an image under the tag, so later retags have a source manifest to read. Writing the rule into the checklist for adding a service costs less than diagnosing the same message again on the next one.

Make the retag step say what is missing

The fix in this case was the build selection. For a team running this pattern, we would also make an empty manifest fail early, with a sentence a developer can act on. The check is a few lines of shell between the two calls, and the failures array already names the image ID that could not be returned:

# Illustrative: acme-api, SOURCE_TAG and RELEASE_TAG are made-up names.
MANIFEST=$(aws ecr batch-get-image --repository-name acme-api \
  --image-ids imageTag="$SOURCE_TAG" \
  --output text --query 'images[].imageManifest')

if [ -z "$MANIFEST" ]; then
  echo "acme-api has no image tagged $SOURCE_TAG to retag." >&2
  aws ecr batch-get-image --repository-name acme-api \
    --image-ids imageTag="$SOURCE_TAG" \
    --output text --query 'failures[].failureReason' >&2
  echo "Build and push acme-api with $RELEASE_TAG instead of retagging it." >&2
  exit 1
fi

aws ecr put-image --repository-name acme-api \
  --image-tag "$RELEASE_TAG" --image-manifest "$MANIFEST"

The flags and the images, failures and failureReason fields come from the retag guide and the BatchGetImage reference linked above. If every project's docker-retag target calls the same script, the guard goes into that script once and covers each service.

The put-image errors that come after this one

Once the manifest is there, PutImage can still refuse the call, and two of its documented errors are worth handling in the same script (PutImage API reference). ImageAlreadyExistsException means the image has already been pushed and neither the manifest nor the tag changed since the last push. A rerun of a retag that already succeeded lands here, so decide whether your script treats it as success. ImageTagAlreadyExistsException means the tag already exists in a repository configured for tag immutability, and in that case moving the tag is the thing the repository is set up to stop.

One more requirement from the same page: if the manifest you push has no mediaType field, the request must include imageManifestMediaType.

When a promotion or retag step in your pipeline fails with a message that does not name the real problem, send us the build log. We would start by listing which services have never had a real build pushed under the source tag, and then check what the step does when batch-get-image returns an empty images list.