Terraform Is Green. Your Cloud Is Broken. Here's Why.

A security team manually tightened an S3 bucket policy in the AWS console during an incident — then moved on. Three weeks later, a deployment ran terraform apply. Terraform, reading a state file that reflected the world before the manual change, reverted the policy. The result: API 403 errors. Two hours to diagnose. The pipeline reported green throughout.

Infrastructure drift is the gap between what Terraform believes is deployed and what is actually running in the cloud. The state file is a snapshot — it captures the state of infrastructure after the last terraform apply, and nothing that happens after that modifies it unless it passes through Terraform. Every manual console change, every auto-scaling adjustment, every third-party cost optimizer that modifies resource configurations is invisible to the state file. The plan looks clean because Terraform is comparing reality to its last known snapshot, not to current reality.

The failure mode is compounding. Organizations at scale — multiple AWS accounts, multiple regions, post-acquisition integrations — have infrastructure surface areas that inevitably accumulate manual changes. On-call engineers hotfix things in the console under incident pressure. AWS Auto Scaling adjusts instance counts. RDS auto-scales storage. ECS Application Auto Scaling modifies task counts in response to traffic. None of these pass through Terraform, yet all change the infrastructure the state file claims to represent. The New Stack’s analysis notes the failure of the standard answer — “just enforce the pipeline” — at enterprise scale: the surface area is too large and too distributed for pipeline discipline alone to prevent drift.

The solution isn’t better Terraform discipline. It’s querying what’s actually running. CloudQuery syncs real cloud infrastructure into SQL-queryable tables by pulling from cloud provider APIs rather than Terraform state. A query identifying all untagged EC2 instances across accounts looks like: SELECT account_id, region, instance_id, tags FROM aws_ec2_instances WHERE tags->>'owner' IS NULL. This returns the ground truth — what the cloud API reports as existing — independent of what any state file believes. The distinction matters: “The state file tells you what Terraform thinks is running. This tells you what’s actually running.”

The operational implication for platform teams is architectural: Terraform state management and cloud reality verification are separate concerns that require separate tooling. IaC handles provisioning intent; cloud asset inventory handles provisioning truth. Relying on one to validate the other produces the false confidence that leads to the S3 policy revert scenario. Teams building agentic incident response pipelines on AWS face this problem acutely — an agent acting on stale state assumptions during an incident can make it worse.

For cloud platform engineers managing production infrastructure, the immediate practical step is establishing drift detection as a separate pipeline stage — not part of terraform plan, but as an independent query against cloud APIs that runs on a schedule and alerts on divergence. This transforms drift from an unknown unknown (discovered during outages) into a known, monitored gap between intent and reality.

Photo by Myburgh Roux on Pexels

Source: The New Stack

Content created with AI assistance and reviewed for accuracy.