Skip to content

chore(terraform): refine state lock semantics (skip on plan, wait on apply)#997

Open
olepg wants to merge 2 commits into
equinor:mainfrom
olepg:chore/terraform-lock-semantics
Open

chore(terraform): refine state lock semantics (skip on plan, wait on apply)#997
olepg wants to merge 2 commits into
equinor:mainfrom
olepg:chore/terraform-lock-semantics

Conversation

@olepg
Copy link
Copy Markdown
Contributor

@olepg olepg commented Apr 24, 2026

Refine how the reusable terraform.yml workflow interacts with the state lock.

terraform plan: -lock=false

terraform plan is read-only against state and does not need to acquire a lock. Taking one blocks concurrent plans from other PRs (or other working directories sharing the same backend) for no benefit. Skipping the lock lets plan runs proceed in parallel.

terraform apply: new lock_timeout_minutes input (default 0)

By default terraform apply fails immediately if the state lock is held. With concurrent applies queued via the workflow's concurrency group this is rarely an issue, but transient contention from external tooling (manual terraform apply runs, drift-detection jobs, etc.) can sometimes cause spurious failures.

Added a new lock_timeout_minutes input (number, default 0) that is passed to terraform apply as -lock-timeout=<N>m. When set to 0 the flag is omitted entirely, preserving Terraform's native fail-fast behavior — so the default is a no-op for existing consumers. Consumers that hit lock contention can opt into waiting (e.g. lock_timeout_minutes: 20).

This input change is additive; existing consumers do not need to update anything.

Split out from #993.


Caveat on -lock=false

To be upfront: HashiCorp explicitly warns against -lock=false ("dangerous if others might concurrently run commands against the same workspace"), so this is a real trade-off, not a free win. Worth being explicit about the actual race scenarios:

  1. External actor also bypasses locking. Anything that writes state without holding the lock — terraform apply -lock=false from a laptop, terraform state rm/mv/push -lock=false, custom scripts uploading state directly to the backend, or misconfigured wrappers that pass -lock=false by default. Possible but rare; most tooling locks by default.
  2. External actor holds the lock and is mid-apply. With -lock=true (default), our plan would block waiting for their lock, then read consistent post-apply state. With -lock=false, our plan reads the pre-apply snapshot while their apply is still in flight; they then finish and bump the state serial. Our plan now references a state version that no longer exists.

In both cases terraform apply <planfile> validates the saved plan's state lineage/serial against current state and normally refuses to apply a stale one, so the usual failure mode is a failed apply (re-run required), not a silent wrong apply. The check is lineage/serial only, not a full semantic diff, so narrow edge cases involving data source reads can still slip through. Plans also go stale on their own within hours regardless of locking.

@olepg olepg requested a review from a team as a code owner April 24, 2026 11:57
sefornes pushed a commit that referenced this pull request Apr 29, 2026
Two small, independent hardening changes to the reusable `terraform.yml`
workflow.

## 1. Stop leaking `ENCRYPTION_PASSWORD` to every step

Removed `ENCRYPTION_PASSWORD` from the workflow-level `env:` block and
scoped it to only the two steps that actually need it (`Create tarball`
in the plan job, `Extract tarball` in the apply job).

Secrets passed via `secrets:` to a reusable workflow are auto-masked by
GitHub. Promoting them to a workflow-level `env:` widens exposure to
every step (including any third-party actions added later) with no
benefit. The secret's description was updated to note this.

## 2. Make plan-tarball retention configurable (default 5 days, was 35)

Added a new `artifact_retention_days` input (number, default `5`) and
use it for `retention-days` on the `Upload artifact` step.

The artifact contains the full Terraform configuration and plan,
encrypted with `ENCRYPTION_PASSWORD`. It is only consumed by the Apply
job in the same workflow run. 35 days of retention is unnecessary
exposure window if the password ever leaks. Making it an input lets
consumers tune the value to their deployment cadence (e.g. workflows
that span multiple environments over several days can opt for a higher
value).

This is an additive input change — existing consumers get the new 5-day
default automatically without any code changes.

---

The lock-semantics change originally proposed here has been split out
into #997.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant