Skip to content
Intermediate Estimated time: 20 min Category: DevOps Published

Terraform — State Backend and Migration

The problem

Terraform does not ask the cloud provider what it manages: it asks the state. That file is the only place where the mapping between the aws_instance.web in your code and the real i-0abc123… lives. If the state disappears, Terraform does not think the instance is broken: it thinks it does not exist and creates it again. An apply without state is an apply against phantom infrastructure.

With a single operator and a local terraform.tfstate that almost never hurts. As soon as there are two people — or one person and a pipeline — the three classic failures show up: two concurrent apply runs stepping on each other and leaving the state describing a reality that no longer exists; a terraform.tfstate committed to Git with the database password in plain text; and the laptop that gets wiped, taking away the only record of what was running in production.

A remote backend solves all three at once: shared, encrypted storage with locking. This page covers how to set it up, how to migrate without recreating anything, and the maintenance operations — state mv, import, -refresh-only — you need when reality and the state diverge. All of it applies equally to Terraform and OpenTofu, with tofu instead of terraform; for recent features, check the documentation for your version before copying a flag from any guide, this one included.

📋 Table of Contents

What the state holds and why it is sensitive

The state is a JSON document mapping resource addresses (module.db.aws_db_instance.main) to real provider identifiers, plus a copy of the attributes read during the last refresh and the dependencies between resources. Three things come out of it: what to create, what to destroy and in which order.

The uncomfortable consequence is that it stores the full attributes, sensitive ones included. An RDS password, a private_key generated by the TLS provider, the contents of a kubernetes_secret: all of it sits in the state in the clear. sensitive = true affects what gets printed on screen, not what gets written to the file. See for yourself with terraform state pull | jq '.resources[].instances[].attributes'.

The state is a secret, treat it as one

  • Never in Git: add *.tfstate, *.tfstate.* and .terraform/ to .gitignore from the first commit.
  • Encrypted at rest and restricted to whoever can run apply: reading the state is equivalent to reading every secret that code touches.
  • If a secret has passed through the state, consider it exposed to anyone who had access to the bucket. Rotate it.
  • Referencing secrets from a manager (Vault, Secrets Manager) reduces the surface, even though the value read still ends up in the state. See secrets management.

Local vs remote

Local (default) Remote
Location terraform.tfstate in the directory Shared bucket or service
Collaboration None: the file is yours Team and CI see the same thing
Locking Does not exist Yes, if the backend supports it
Encryption and backup Whatever your disk does SSE/KMS and bucket versioning
Leak risk High (accidental commit) Controlled by IAM

Local is reasonable for a one-day lab. For anything two people can touch, or that a pipeline will apply, go remote.

S3 backend with locking

The backend block lives inside terraform { } and accepts no variables or interpolation: its values must be literals or come from -backend-config. It is the limitation that most surprises people arriving from the rest of HCL.

terraform {
  required_version = ">= 1.5"

  backend "s3" {
    bucket         = "acme-tfstate-prod"
    key            = "platform/network/terraform.tfstate"
    region         = "eu-west-1"
    encrypt        = true
    kms_key_id     = "arn:aws:kms:eu-west-1:123456789012:key/abcd-1234"
    dynamodb_table = "acme-tfstate-locks"
  }
}

key is the path inside the bucket: a different key per component and per environment, which is what actually separates states. encrypt = true forces encryption at rest and kms_key_id adds access control over the key, auditable in CloudTrail. The lock table is created once, in a separate bootstrap configuration — the bucket and the table cannot live in the state they themselves store:

resource "aws_dynamodb_table" "locks" {
  name         = "acme-tfstate-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

The attribute must be named LockID and be the partition key; with any other name lock acquisition fails. A single table serves every state in the account: the row is identified by the state path.

Native S3 locking: version-dependent — not verified here

Recent versions of the S3 backend support locking without DynamoDB, through a .tflock object next to the state (the use_lockfile argument), leaving dynamodb_table as a legacy option. Do not assume which one applies to your installation: check the S3 backend documentation for the exact version you run — and for OpenTofu, if that is your case — and confirm it with a terraform init. What is non-negotiable is not ending up with no locking mechanism at all.

Minimum IAM policy

Just enough permissions to operate on one state, with no access to the rest of the bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::acme-tfstate-prod",
      "Condition": { "StringLike": { "s3:prefix": ["platform/network/*"] } }
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::acme-tfstate-prod/platform/network/*"
    },
    {
      "Effect": "Allow",
      "Action": ["dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:DeleteItem"],
      "Resource": "arn:aws:dynamodb:eu-west-1:123456789012:table/acme-tfstate-locks"
    }
  ]
}
  • The DynamoDB statement is unnecessary with file-based locking: the lock is then just another object under the same prefix.
  • With kms_key_id you must add kms:Encrypt, kms:Decrypt and kms:GenerateDataKey on the key ARN, or everything fails with an AccessDenied that never mentions KMS.
  • A read-only role (s3:GetObject and s3:ListBucket) for reviewing plans is the best privilege separation on this page: many people need to see the plan, very few need to apply it.

Locking and orphaned locks

Before writing, Terraform acquires a lock; when it finishes, it releases it. If the process dies in between — an aggressive Ctrl+C, a cancelled runner, a network drop — the lock stays and the next apply fails with Error: Error acquiring the state lock, followed by a Lock Info block with the ID, Operation, Who and Created. That block is information, not just an error: it tells you whether someone is genuinely working or it is a leftover from three hours ago.

terraform force-unlock 7c1e2b58-...

Check before you force

force-unlock cancels no operation: it only deletes the lock. If the original apply is still alive on another machine, you end up with two concurrent writes and a corrupted state, which is exactly the damage the lock was preventing.

Protocol: look at Who and Created, ask in the team channel, check whether a job is running. Only then unlock. There is a flag to skip the interactive confirmation; do not put it in an unattended script.

For one-off reads where the lock gets in the way — and only reads — most commands accept -lock=false. With plan, never with apply.

migrate-state vs reconfigure

This is the confusion that breaks the most states. When you change the backend block, terraform init offers two incompatible paths:

Flag What it does with the existing state When
-migrate-state Copies it to the new backend Moving the state elsewhere while keeping the infrastructure
-reconfigure Ignores it: starts clean on the new backend The destination backend already holds the right state

Translated: -migrate-state keeps, -reconfigure forgets. If you run -reconfigure when you meant to migrate, the new backend starts empty, the next plan proposes creating the whole infrastructure from scratch, and the old state is left orphaned where it was. It is recoverable — the original file is not deleted — but only if you recognise the symptom in time.

terraform state pull > backup-$(date +%F).tfstate   # 1. back up, always
# 2. add the backend "s3" block to the configuration
terraform init -migrate-state                       # 3. migrate
terraform state list                                # 4. verify
terraform plan                                      #    must say "No changes"

That clean plan in step 4 is the only valid proof that the migration went well. If it proposes creating or destroying anything, stop: the migrated state does not match the infrastructure.

When the backend is parameterised per environment, the values go in standalone .hcl files (bucket, key, region, one key per line) because the block accepts no variables, and are loaded with -backend-config:

terraform init -reconfigure -backend-config=backends/prod.hcl

Here -reconfigure is the right call: you are not moving a state, you are pointing the working directory at another one that already exists. -migrate-state would try to copy the staging state on top of prod.

Workspaces vs separate directories

Workspaces (terraform workspace new staging, select, list, show) create several states for one and the same configuration. What a workspace does isolate is the state. What it does not isolate:

  • Credentials. The provider is configured identically across every workspace, and whatever AWS_* variables you have loaded apply to any of them. An apply in the wrong workspace runs against whichever account was active, with no barrier at all.
  • The code. A change in the .tf affects all of them at once: there is no gradual promotion from staging to prod.
  • The backend. Same bucket and same permissions: whoever can read one workspace can read them all.
  • Human error. Nothing in the prompt tells you which workspace you are in.

The alternative is a directory per environment (envs/dev, envs/staging, envs/prod), each with its own backend, provider and tfvars, sharing code through modules in modules/. Rule of thumb: workspaces for ephemeral variations of the same environment — a test branch, a temporary deployment — and separate directories for environments with different security boundaries. If dev and prod live in separate accounts — and they should — workspaces are not the tool.

Operating on the state

These commands modify the inventory, not the infrastructure: Terraform touches nothing at the provider, it changes what it believes exists.

terraform state list                      # everything it manages
terraform state show aws_instance.web     # attributes of one resource
terraform state pull > snapshot.tfstate   # download the full state
terraform state mv aws_instance.web module.frontend.aws_instance.this

state mv renames or moves an address: it is what avoids destroy-and-recreate when refactoring. If you rename in the .tf without doing the mv, Terraform sees the old address disappear and a new one appear, and plans a destroy and a create. On a database, that is an incident.

state rm does not destroy, and that is the problem

terraform state rm aws_instance.web forgets the resource: it still exists and still bills at the provider, but Terraform stops managing it. Nobody will destroy it, nobody will update it, and it will not show up in any plan. It is the cleanest way to create orphaned resources nobody remembers six months later.

Legitimate use: pulling a resource out of one state to move it to another. In that case do the import at the destination before the rm at the origin, and write down the ID.

Two precautions that save grief: terraform state pull > before.tfstate ahead of any mv or rm, and a terraform plan afterwards, which must come out with no changes.

Importing pre-existing resources

When the infrastructure exists but the state does not know about it — created by hand, inherited, migrated from another tool — import adopts it. Write the resource block in the .tf first, even if incomplete, then associate the Terraform address with the real provider ID:

terraform import aws_instance.web i-0abc123def456
terraform plan     # what differs between the code and reality?

That plan is the real work. import fills the state with the actual attributes but does not write your configuration: until the .tf describes the resource as it truly is, the plan will keep proposing changes. You iterate — adjust HCL, plan, repeat — until you reach "No changes".

The ID format depends on the provider and the resource: an EC2 instance uses i-…, a security group rule uses a composite string, a Kubernetes resource uses namespace/name. It is documented in the Import section of each resource; do not guess it.

Declarative import blocks — check your version

Recent Terraform versions support import blocks in HCL itself, plannable and reviewable in a PR rather than run by hand, with an option to generate a configuration skeleton. It is far more convenient for bulk imports, but confirm availability and exact syntax in the documentation for your version before putting it in a pipeline.

Drift and refresh-only

Drift is the difference between what the state says and what actually exists at the provider: someone changed something in the console, an autoscaler changed a capacity, an external process added a tag.

terraform plan -refresh-only     # only compares, proposes no configuration changes
terraform apply -refresh-only    # updates the state with reality, touching no resources
terraform plan -refresh=false    # the opposite: does not query the provider

The difference from a normal plan matters: plan mixes two things — what changed outside and what you want to change — into one output that is hard to read. -refresh-only isolates the first question: what changed without my permission? The standalone terraform refresh command is deprecated in favour of apply -refresh-only, precisely because the old one updated the state without first showing what it was about to change.

-refresh=false speeds up plan on large configurations at the cost of working with possibly stale data: useful in a fast development loop, a bad idea before an apply in production. And a periodic job running plan -refresh-only and alerting on differences is one of the cheapest and most useful alerts you can add to an IaC-managed environment; see GitHub Actions for the scaffolding.

State in CI: OIDC instead of keys

Storing a long-lived AWS_ACCESS_KEY_ID in repository secrets means having a permanent credential, with write access to the state and to the infrastructure, in a place many people can read or exfiltrate from a modified workflow. With OIDC, the CI provider issues a short-lived token that the cloud validates against an identity provider: there is no key to rotate and none to leak.

permissions:
  id-token: write        # required to request the OIDC token
  contents: read

jobs:
  plan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/terraform-ci-plan
          aws-region: eu-west-1
      - run: terraform init
      - run: terraform plan -out=tfplan

The critical piece is on the cloud side: the role's trust policy must restrict who can assume it — repository, and ideally a specific branch or environment. A role that trusts any repository in the organisation puts the production apply within reach of anyone with workflow permissions. Use two roles: terraform-ci-plan with read access to the state, assumable from any PR, and terraform-ci-apply with write access, assumable only from the main branch and with approval. That way an external PR produces a reviewable plan without being able to write the state. Details in IaC security and secrets in GitOps.

The plan file is sensitive too

A plan file (-out=tfplan) contains the values about to be written, secrets included. Do not publish it as a downloadable artifact and do not paste its full output into a public PR comment.

Backup and versioning

Bucket versioning turns an unfortunate state rm into a five-minute scare: enable it with aws_s3_bucket_versioning and status = "Enabled" on the state bucket. Recovery then means pulling the previous version of the object and pushing it back with terraform state push. That command overwrites the remote state: download the current one first, check it, and only then push.

  • Snapshot before every manual operation. terraform state pull > before.tfstate costs one second.
  • Block public access on the bucket explicitly, without trusting the default.
  • Object Lock or a retention policy if the requirement is that nobody, not even an administrator, can delete history.
  • Bucket in a different account from the infrastructure it manages: compromising production should not also take out the record of what was in it. Same reasoning as in the 3-2-1 strategy.

Troubleshooting

Symptom Cause Fix
Error acquiring the state lock Orphaned lock from a dead process Check Who/Created, then terraform force-unlock <ID>
Backend configuration changed The backend block changed init -migrate-state (copy) or -reconfigure (point elsewhere)
plan proposes creating everything after migrating -reconfigure was used instead of -migrate-state Restore the old state and redo it with -migrate-state
Resource already exists on apply The resource exists but is not in the state terraform import with the real ID
plan destroys and recreates after a rename The address changed without state mv terraform state mv <old> <new>
Changes nobody asked for Drift: manual modification outside Terraform plan -refresh-only to see it and decide
AccessDenied with no mention of KMS Missing permissions on the encryption key Add kms:Decrypt, kms:Encrypt, kms:GenerateDataKey to the role
Applied to the wrong environment Wrong workspace or -backend-config terraform workspace show before every apply

When the state looks inconsistent and you do not know where to start, terraform state pull | jq '.serial, .lineage' gives the clue: serial increases with every write and lineage identifies the state's lineage. Two states with different lineage are not versions of the same one, and Terraform will refuse to mix them; that is the sign that at some point a new state was created instead of migrating the existing one.

Best practices

  • Remote backend with locking from day one. Migrating later works, but the incident that convinces you to migrate is expensive.
  • One state per component and environment, not a monolithic state: it shrinks the blast radius and the plan time.
  • .gitignore with *.tfstate* and .terraform/ before the first commit, not after the first scare.
  • Snapshot before any state mv, rm or push. Those are the three operations with no undo.
  • A clean plan as the success criterion for every migration or import: "No changes", or you are not finished.
  • OIDC in CI, with separate read and write roles.
  • Periodic drift detection, even if it is only a weekly notification.

References