Terraform Drift Detection in CI/CD Pipeline: A 2026 Production Playbook

How to run Terraform drift detection in CI/CD: detection cadence, tooling trade-offs, a ready-to-run GitHub Actions workflow, and why most teams should not auto-remediate.

By VVV Ops

The Terraform code says the RDS instance has deletion protection on. The AWS console says an engineer turned it off three months ago while working an incident at 2 a.m., and the change never made it back into code. The next terraform apply will either switch it back on without anyone noticing or stop on a "resource has been modified outside of Terraform" error at a bad moment.

That gap is drift. Any Terraform estate that people can change by hand has some, and the useful question is how long it goes unnoticed. This post covers how we set up drift detection in a CI/CD pipeline: what to run, how often, which tools to use, and why we rarely let the pipeline fix drift on its own.

What drift is

Drift is any disagreement between three things: the Terraform code, the remote state file and the resources that exist in the cloud. When all three agree, terraform plan reports no changes. When two of them disagree, what Terraform does next depends on which two.

There are three kinds worth telling apart:

  • Configuration drift. An attribute changed in the cloud but not in code, the console-edit case. Terraform will try to revert it on the next apply.
  • Unmanaged resources. Something was created outside Terraform, such as a Lambda function a developer added by hand or a security group a runbook created. Terraform does not know it exists.
  • State drift. The state file records something that is no longer true because a resource was deleted, replaced or moved. terraform plan -refresh-only shows this, but only when someone runs it.

All three are normal in a live estate. The aim is to find drift within hours and make the decision about it on purpose.

Where drift comes from

In the estates we have worked on, the same sources come up again and again:

| Source | Can policy prevent it? | |---|---| | Console edits during incidents | Yes, with read-only console access in production | | Side effects of cloud services (service-linked IAM roles, default security groups) | Partly, with lifecycle.ignore_changes | | Another IaC tool (CDK, CloudFormation, Pulumi) in the same account | Yes, by giving each tool its own accounts | | Scripts, runbooks and CLI sessions | Yes, with tighter IAM | | Provider bugs and changed default values | No, handle them during provider upgrades |

Console edits during incidents are the source to fix first, and the fix is clear: in production accounts, people get read-only console access, a small group gets a break-glass role, and every use of that role is logged.

Side effects are harder to spot. AWS creates service-linked roles, EKS creates load balancers, and RDS changes parameter groups. None of that is a mistake, but it still shows up in plan output. Put lifecycle.ignore_changes on the specific attributes the cloud owns. A blanket ignore block hides real drift along with the noise.

How often to run the check

Base the schedule on what a missed change would cost. This is the schedule we start with:

| Environment | Minimum cadence | Reason | |---|---|---| | Production that earns revenue | Every 2 hours | Time to detect matters for security changes | | Other production | Every 6 hours | Enough coverage at lower CI cost | | Staging and pre-production | Daily | Drift here is cheap to reconcile | | Developer sandboxes | On demand | No alerts for someone's experiment |

The CI minutes for a scheduled plan are small next to the cost of one unnoticed security group change. The harder cost is alert volume. Without an agreed way to triage drift alerts, on-call stops reading them within weeks. Start at every 6 hours in production, look at how many alerts you get, and adjust from there.

A schedule finds drift on the next run, and values Terraform does not manage, such as secret values set outside it, never appear in a plan. For one client we built a change tracker for that gap. CloudTrail events go through EventBridge into redaction, rule and notification stages, with least-privilege IAM and a dedicated KMS key, and every config or secret change is posted to Slack with the name of the ECS service that consumes the changed key. Our own security review of the build found two problems before rollout: secret values leaking into the notifications, and Slack mention forgery, where a changed value could inject @mentions. Both were fixed. The tracker also masks credentials inside JDBC and R2DBC connection strings and compares key sets across environments, and it runs in production in two regions next to the scheduled plan.

Three ways to resolve a drift finding

Every drift finding ends in one of three decisions. Naming the decision stops the habit of re-running apply to see what happens.

  1. Revert. The code is right and the cloud is wrong, so run terraform apply and let Terraform put the resource back. Use this when the drift came from a console edit that should not have happened.
  2. Align. The cloud is right and the code is wrong, so change the Terraform code to match and commit it. Use this when an emergency fix was meant to stay.
  3. Accept. The attribute belongs to the cloud provider or does not matter, so add lifecycle.ignore_changes for that attribute only. Use this for service-managed fields, autoscaled replica counts and tags added by organization policy.

Record the decision and the cause with each finding. A ticket closed with "ran apply, fixed" and no cause will come back next week.

Tools

The tools fall into four groups. The choice depends on your CI platform and whether you want to pay for a managed product.

| Group | Examples | Good fit | |---|---|---| | A scheduled job in your CI | GitHub Actions or GitLab CI running terraform plan -detailed-exitcode | Smaller estates and teams with disciplined CI | | Open-source orchestrators | Atlantis, Terramate | Teams that self-host and work GitOps-first | | Managed Terraform platforms | HCP Terraform, Spacelift, env0, Scalr | Larger estates that also want a policy engine, RBAC and a UI | | Cloud-native configuration tracking | AWS Config, Azure Policy | Resource history regardless of which tool made the change |

We usually start with a scheduled job or an open-source orchestrator. Drift detection comes down to running terraform plan on a schedule, and you do not need a platform for that alone. Managed platforms make sense when you also want policy as code, role-based access and a UI for people outside engineering. OpenTofu works the same way for everything in this post. The Terraform state documentation explains the mechanics underneath all of these tools.

Do not rely on AWS Config alone in a Terraform estate. It records every configuration change, including the ones Terraform makes, so by itself it cannot tell an approved apply from a console edit.

A GitHub Actions workflow

This is the minimal workflow we start from. It runs every 6 hours against production, uses plan -detailed-exitcode to tell "no changes" (exit code 0) from "drift found" (exit code 2), and posts to Slack only when there is drift:

name: terraform-drift-detection

on:
  schedule:
    - cron: '0 */6 * * *'  # every 6 hours
  workflow_dispatch:

permissions:
  id-token: write
  contents: read

jobs:
  drift-check:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        workspace: [prod-us-east-1, prod-eu-west-1]
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT_ID }}:role/terraform-drift-reader
          aws-region: us-east-1

      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.9.x

      - name: terraform init
        run: terraform init -backend-config="key=${{ matrix.workspace }}.tfstate"

      - name: terraform plan with drift detection
        id: plan
        run: |
          set +e
          terraform plan -detailed-exitcode -lock=false -out=drift.tfplan
          echo "exitcode=$?" >> "$GITHUB_OUTPUT"

      - name: post drift alert to slack
        if: steps.plan.outputs.exitcode == '2'
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "Drift detected in ${{ matrix.workspace }}",
              "blocks": [
                {"type": "section", "text": {"type": "mrkdwn",
                 "text": "Workflow: <${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}|Run ${{ github.run_id }}>"}}
              ]
            }
        env:
          SLACK_WEBHOOK_URL: ${{ secrets.SLACK_DRIFT_WEBHOOK }}

Three details make it safe to run against production. The terraform-drift-reader role has read-only AWS permissions, because a drift check should never be able to apply anything. -lock=false keeps the scan from blocking a real apply that runs at the same time. The matrix runs each workspace separately, so a broken eu-west-1 run does not hide drift in us-east-1. Add a separate step for exit code 1, which means the plan itself failed, so a broken check does not look like a clean one.

For the IAM side, our post on Terraform security practices for AWS covers the assume-role setup we use.

Why we rarely auto-remediate

Many tools offer to re-apply automatically when they find drift, so the cloud always matches the code and nobody has to file a ticket. In production this tends to turn a small drift finding into an outage. Three cases show why:

  1. During an incident an engineer scales up an RDS instance by hand. Thirty minutes later the drift job scales it back down, and the incident gets worse.
  2. The security team closes a security group during an attack. The drift job opens it again.
  3. The cloud provider patches a managed service. The drift job fights the patch in a loop, uses up API quota and gets the account throttled.

We have seen versions of all three in incident reviews. Drift often records a decision someone made for a reason, and an automatic revert throws that information away.

We allow automatic remediation in two places only: short-lived dev environments, where drift is noise and nothing depends on them, and a few narrow cases where the revert is known to be safe, such as a cost-allocation tag that must always have one value. Everything else gets an alert and a person deciding which of the three outcomes applies.

Reducing how much drift can happen

Detection finds drift after it happens. These four controls reduce how much of it happens:

  • Read-only console access in production. If engineers cannot change an instance in the console, they cannot cause drift there. Keep a break-glass role for emergencies and review every use of it.
  • lifecycle.ignore_changes for attributes the cloud owns, such as service-linked roles, autoscaled replica counts and policy tags, so plan stays readable.
  • Separate accounts per IaC tool. Do not run CDK, Pulumi and Terraform in the same AWS account unless ownership of every resource is written down.
  • Rebuild dev environments from CI. If dev is torn down and recreated every night, drift cannot build up there.

Several of the problems in our post on Terraform anti-patterns also make drift more likely.

A 30-day rollout

Starting from no drift detection, this is the order we follow:

| Week | Work | |---|---| | 1 | Create the read-only IAM role and the workflow. Run it once a day without alerts. | | 2 | Remove false positives with targeted lifecycle.ignore_changes. Count real drift per workspace. | | 3 | Move production to every 6 hours. Send alerts to the channel on-call reads. | | 4 | Write the triage runbook for revert, align and accept. Review the first month of findings with the team. |

After a month you have detection that runs, alerts people trust and a triage process on-call will follow. Cadence and prevention can be tuned from there.

Drift detection tends to wait behind feature work until an incident forces it. If you want the read-only role, the workflow and the triage runbook set up on your existing Terraform and CI and handed over with documentation, tell us about your setup and we will say where we would start.

Tags: terraform drift detection in ci cd pipeline, terraform drift detection, terraform plan detailed exitcode, terraform state drift, infrastructure as code drift