Skip to content

Bug: data.archive_file race condition causes spurious Lambda updates when module is used with for_each #92

@ajain-zoox

Description

@ajain-zoox

Describe the bug

When the module is instantiated multiple times via for_each, all instances share the same hardcoded output_path for their data "archive_file" resources in lifecycle-management.tf:

data "archive_file" "az_rebalancing_suspender" {
  type        = "zip"
  output_path = "${path.module}/.terraform/lambda/az-rebalancing-suspender.zip"
  ...
}

data "archive_file" "stop_buildkite_agents" {
  type        = "zip"
  output_path = "${path.module}/.terraform/lambda/stop-buildkite-agents.zip"
  ...
}

Because path.module resolves to the same directory for every instance, all instances write to a single shared zip file. Terraform evaluates data sources concurrently during plan, so instances race to write the file. An instance that reads the file while another is in the middle of writing it sees an empty or truncated zip.

Steps To Reproduce

  1. Instantiate the module with for_each across 5+ stack configurations
  2. Run terraform plan with no changes to any module inputs
  3. Observe one or more aws_lambda_function resources showing a source_code_hash change

Expected behavior

terraform plan should show no changes when no module inputs have been modified.

Actual behaviour

terraform plan shows spurious in-place updates to aws_lambda_function resources:

~ source_code_hash = "6LftzyKxqszw3EZWiaZOlfrPDOPVx5cfLWZD0s14ax0=" -> "47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU="

47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU= is the base64 SHA-256 of an empty byte string, confirming the zip was read as empty due to the race. The issue is nondeterministic — which instances are affected varies between plan runs, and a second run often produces a clean result. Applying the plan deploys a broken empty Lambda, silently breaking AZ rebalancing suspension and agent stop logic.

Stack parameters

  • AWS Region: us-west-2
  • Version: v0.7.0 (confirmed present in all versions through latest)

Additional context

Workaround: run terraform plan -parallelism=1 to serialize data source evaluation and eliminate the race. No code changes required.

Proposed fix: make output_path unique per module instance by incorporating a stack-specific identifier, e.g.:

output_path = "${path.module}/.terraform/lambda/${local.stack_name_full}-az-rebalancing-suspender.zip"

This applies to both az_rebalancing_suspender and stop_buildkite_agents. Tested on Terraform v1.12.2.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions