From f9b64b0e24686704f348d6c6f5e41c8817019591 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 5 Jan 2026 16:28:53 +0000 Subject: [PATCH] feat: add build-artifacts input to transfer interim files between plan and apply (#517) Add new `build-artifacts` input that allows users to specify files and directories to upload alongside the tfplan file and restore during apply. This resolves the issue where interim build artifacts (e.g., zip files created by data.archive_file) are missing during apply because only the tfplan file was transferred between workflow runs. Changes: - Add `build-artifacts` input accepting newline or comma-separated paths - Add staging step to prepare build artifacts with manifest for upload - Modify upload steps to include staged artifacts directory - Add restore step to extract artifacts to original locations during apply - Add cleanup steps to remove staging directories after upload/restore - Add test case demonstrating the build artifact issue scenario - Update README with documentation and remove workaround from To-Do Example usage: - uses: op5dev/tf-via-pr@v13 with: command: plan build-artifacts: | build/bundle.zip dist/ --- README.md | 11 +- action.yml | 143 ++++++++++++++++++- tests/fail_build_artifact_missing/.gitignore | 8 ++ tests/fail_build_artifact_missing/main.tf | 92 ++++++++++++ tests/tf.sh | 12 ++ 5 files changed, 260 insertions(+), 6 deletions(-) create mode 100644 tests/fail_build_artifact_missing/.gitignore create mode 100644 tests/fail_build_artifact_missing/main.tf diff --git a/README.md b/README.md index 7b6045b2..2e9497d7 100644 --- a/README.md +++ b/README.md @@ -181,6 +181,7 @@ All supported CLI argument inputs are [listed below](#arguments) with accompanyi | Security | `preserve-plan` | Preserve plan file "tfplan" in the given working directory after workflow execution.
Default: `false` | | Security | `upload-plan` | Upload plan file as GitHub workflow artifact.
Default: `true` | | Security | `retention-days` | Duration after which plan file artifact will expire in days.
Example: `90` | +| Artifact | `build-artifacts` | Paths to files/directories to upload with plan and restore during apply.7
Example: `build/bundle.zip` | | Security | `token` | Specify a GitHub token.
Default: `${{ github.token }}` | | UI | `expand-diff` | Expand the collapsible diff section.
Default: `false` | | UI | `expand-summary` | Expand the collapsible summary section.
Default: `false` | @@ -199,7 +200,14 @@ All supported CLI argument inputs are [listed below](#arguments) with accompanyi 1. The `on-diff` option is true when the exit code of the last TF command is non-zero (ensure `terraform_wrapper`/`tofu_wrapper` is set to `false`).

1. The default behavior of `comment-method` is to update the existing PR comment with the latest plan/apply output, making it easy to track changes over time through the comment's revision history.

[![PR comment revision history comparing plan and apply outputs.](/.github/assets/revisions.png)](https://raw.githubusercontent.com/op5dev/tf-via-pr/refs/heads/main/.github/assets/revisions.png "View full-size image.")

-1. It can be desirable to hide certain arguments from the last run command input to prevent exposure in the PR comment (e.g., sensitive `arg-var` values). Conversely, it can be desirable to show other arguments even if they are not in last run command input (e.g., `arg-workspace` or `arg-backend-config` selection). +1. It can be desirable to hide certain arguments from the last run command input to prevent exposure in the PR comment (e.g., sensitive `arg-var` values). Conversely, it can be desirable to show other arguments even if they are not in last run command input (e.g., `arg-workspace` or `arg-backend-config` selection).

+1. The `build-artifacts` input accepts newline or comma-separated paths to files or directories that should be uploaded alongside the plan file and restored during apply. This is useful when using `data.archive_file` or other resources that generate interim build files needed during apply (e.g., zip archives for Lambda deployments).
+ Example usage: + ```yaml + build-artifacts: | + build/bundle.zip + dist/ + ```
@@ -322,7 +330,6 @@ View [all notable changes](https://github.com/op5dev/tf-via-pr/releases "Release - Handling of inputs which contain space(s) (e.g., `working-directory: path to/directory`). - Handling of comma-separated inputs which contain comma(s) (e.g., `arg-var: token=1,2,3`); workaround with `TF_CLI_ARGS` [environment variable](https://developer.hashicorp.com/terraform/cli/config/environment-variables#tf_cli_args-and-tf_cli_args_name). -- Handling of interim build artifact(s) between `plan` and `apply` commands (e.g., zip archive); workaround with `arg-auto-approve: true` so that `apply` rebuilds artifact(s) for provisioning ([join discussion](https://github.com/OP5dev/TF-via-PR/issues/517)).
diff --git a/action.yml b/action.yml index 03fbfab4..d10a8c5e 100644 --- a/action.yml +++ b/action.yml @@ -204,8 +204,8 @@ runs: if [[ -z "$artifact_id" ]]; then echo "Unable to locate plan file: ${{ steps.identifier.outputs.name }}." && exit 1; fi gh api /repos/${{ github.repository }}/actions/artifacts/${artifact_id}/zip --header "$GH_API" --method GET > "${{ steps.identifier.outputs.name }}.zip" - # Unzip the plan file to the working directory, then clean up the zip file. - unzip "${{ steps.identifier.outputs.name }}.zip" -d "$INPUTS_ARG_CHDIR" + # Unzip the plan file to the working directory (or current directory if not specified). + unzip "${{ steps.identifier.outputs.name }}.zip" -d "${INPUTS_ARG_CHDIR:-.}" rm --force "${{ steps.identifier.outputs.name }}.zip" - if: ${{ inputs.plan-encrypt != '' && steps.download.outcome == 'success' }} @@ -220,6 +220,64 @@ runs: openssl enc -aes-256-ctr -pbkdf2 -salt -in "$path.encrypted" -out "$path.decrypted" -pass file:"$temp_file" -d mv --force --verbose "$path.decrypted" "$path" + - if: ${{ steps.download.outcome == 'success' }} + id: restore-build-artifacts + env: + INPUTS_ARG_CHDIR: ${{ inputs.arg-chdir || inputs.working-directory }} + shell: bash + run: | + # Restore build artifacts from staging directory. + workdir="${INPUTS_ARG_CHDIR:-.}" + staging_dir="$workdir/.tfviapr-artifacts" + manifest_file="$staging_dir/manifest.json" + + # Skip if no build artifacts were staged. + if [[ ! -f "$manifest_file" ]]; then + echo "No build artifacts manifest found, skipping restore." + exit 0 + fi + + echo "Restoring build artifacts from manifest: $manifest_file" + + # Parse manifest and restore each artifact. + artifacts=$(jq -c '.artifacts[]' "$manifest_file" 2>/dev/null || echo "") + if [[ -z "$artifacts" ]]; then + echo "No artifacts in manifest." + rm -rf "$staging_dir" + exit 0 + fi + + while IFS= read -r artifact; do + rel_path=$(echo "$artifact" | jq -r '.path') + staging_name=$(echo "$artifact" | jq -r '.staging') + artifact_type=$(echo "$artifact" | jq -r '.type') + + # Determine destination path. + if [[ "$rel_path" = /* ]]; then + # Absolute path. + dest_path="$rel_path" + else + # Relative path - restore relative to working directory. + dest_path="$workdir/$rel_path" + fi + + # Create parent directory if needed. + dest_dir=$(dirname "$dest_path") + mkdir -p "$dest_dir" + + # Restore file or directory. + if [[ "$artifact_type" == "directory" ]]; then + cp -r "$staging_dir/$staging_name" "$dest_path" + else + cp "$staging_dir/$staging_name" "$dest_path" + fi + echo "Restored build artifact: $staging_dir/$staging_name -> $dest_path" + done <<< "$artifacts" + + # Clean up staging directory. + rm -rf "$staging_dir" + echo "Build artifacts restored successfully." + - if: ${{ steps.plan.outcome == 'success' || steps.download.outcome == 'success' }} env: INPUTS_TOOL: ${{ inputs.tool }} @@ -254,12 +312,72 @@ runs: rm --force "$path" fi + - if: ${{ inputs.build-artifacts != '' && steps.plan.outcome == 'success' }} + id: stage-build-artifacts + env: + INPUTS_ARG_CHDIR: ${{ inputs.arg-chdir || inputs.working-directory }} + INPUTS_BUILD_ARTIFACTS: ${{ inputs.build-artifacts }} + shell: bash + run: | + # Stage build artifacts for upload. + # Create staging directory and manifest file. + workdir="${INPUTS_ARG_CHDIR:-.}" + staging_dir="$workdir/.tfviapr-artifacts" + manifest_file="$staging_dir/manifest.json" + mkdir -p "$staging_dir" + echo '{"artifacts":[' > "$manifest_file" + + # Parse build-artifacts input (newline or comma-separated). + artifacts=$(echo "$INPUTS_BUILD_ARTIFACTS" | tr ',' '\n' | sed 's/^[[:space:]]*//;s/[[:space:]]*$//' | grep -v '^$') + first=true + + while IFS= read -r artifact_path; do + # Resolve path relative to working directory. + if [[ "$artifact_path" = /* ]]; then + # Absolute path - use as-is. + src_path="$artifact_path" + rel_path="$artifact_path" + else + # Relative path - resolve from working directory. + src_path="$workdir/$artifact_path" + rel_path="$artifact_path" + fi + + if [[ ! -e "$src_path" ]]; then + echo "Warning: Build artifact not found: $src_path" + continue + fi + + # Generate a unique staging name to avoid conflicts. + staging_name=$(echo "$rel_path" | md5sum | cut -d' ' -f1) + + # Copy file or directory to staging area. + if [[ -d "$src_path" ]]; then + cp -r "$src_path" "$staging_dir/$staging_name" + artifact_type="directory" + else + cp "$src_path" "$staging_dir/$staging_name" + artifact_type="file" + fi + + # Add to manifest. + if [[ "$first" != "true" ]]; then echo ',' >> "$manifest_file"; fi + first=false + echo "{\"path\":\"$rel_path\",\"staging\":\"$staging_name\",\"type\":\"$artifact_type\"}" >> "$manifest_file" + echo "Staged build artifact: $src_path -> $staging_dir/$staging_name" + done <<< "$artifacts" + + echo ']}' >> "$manifest_file" + echo "Build artifacts manifest created: $manifest_file" + - if: ${{ inputs.command == 'plan' && inputs.upload-plan == 'true' && (github.server_url == 'https://github.com' || contains(github.server_url, '.ghe.com')) }} id: upload uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6.0.0 with: name: ${{ steps.identifier.outputs.name }} - path: ${{ format('{0}{1}tfplan{2}', inputs.arg-chdir || inputs.working-directory, (inputs.arg-chdir || inputs.working-directory) && '/' || '', inputs.plan-encrypt != '' && '.encrypted' || '') }} + path: | + ${{ format('{0}{1}tfplan{2}', inputs.arg-chdir || inputs.working-directory, (inputs.arg-chdir || inputs.working-directory) && '/' || '', inputs.plan-encrypt != '' && '.encrypted' || '') }} + ${{ inputs.build-artifacts != '' && format('{0}{1}.tfviapr-artifacts', inputs.arg-chdir || inputs.working-directory, (inputs.arg-chdir || inputs.working-directory) && '/' || '') || '' }} retention-days: ${{ inputs.retention-days }} overwrite: true @@ -268,9 +386,22 @@ runs: uses: actions/upload-artifact@c24449f33cd45d4826c6702db7e49f7cdb9b551d # v3.2.1-node20 with: name: ${{ steps.identifier.outputs.name }} - path: ${{ format('{0}{1}tfplan{2}', inputs.arg-chdir || inputs.working-directory, (inputs.arg-chdir || inputs.working-directory) && '/' || '', inputs.plan-encrypt != '' && '.encrypted' || '') }} + path: | + ${{ format('{0}{1}tfplan{2}', inputs.arg-chdir || inputs.working-directory, (inputs.arg-chdir || inputs.working-directory) && '/' || '', inputs.plan-encrypt != '' && '.encrypted' || '') }} + ${{ inputs.build-artifacts != '' && format('{0}{1}.tfviapr-artifacts', inputs.arg-chdir || inputs.working-directory, (inputs.arg-chdir || inputs.working-directory) && '/' || '') || '' }} retention-days: ${{ inputs.retention-days }} + - if: ${{ inputs.build-artifacts != '' && (steps.upload.outcome == 'success' || steps.upload-v3.outcome == 'success') }} + env: + INPUTS_ARG_CHDIR: ${{ inputs.arg-chdir || inputs.working-directory }} + shell: bash + run: | + # Clean up build artifacts staging directory after upload. + workdir="${INPUTS_ARG_CHDIR:-.}" + staging_dir="$workdir/.tfviapr-artifacts" + rm -rf "$staging_dir" + echo "Cleaned up build artifacts staging directory." + - if: ${{ inputs.plan-parity == 'true' && (steps.download.outcome == 'success' || inputs.plan-file != '') }} env: INPUTS_ARG_CHDIR: ${{ inputs.arg-chdir || inputs.working-directory }} @@ -636,6 +767,10 @@ inputs: default: "true" description: "Upload plan file as GitHub workflow artifact (e.g., `true`)." required: false + build-artifacts: + default: "" + description: "Newline or comma-separated paths to files/directories to upload with plan and restore during apply (e.g., `build/bundle.zip`)." + required: false validate: default: "false" description: "Check validation of TF code (e.g., `false`)." diff --git a/tests/fail_build_artifact_missing/.gitignore b/tests/fail_build_artifact_missing/.gitignore new file mode 100644 index 00000000..d647a4a8 --- /dev/null +++ b/tests/fail_build_artifact_missing/.gitignore @@ -0,0 +1,8 @@ +# Terraform +.terraform/ +.terraform.lock.hcl +*.tfplan +tfplan + +# Build artifacts (generated during plan) +build/ diff --git a/tests/fail_build_artifact_missing/main.tf b/tests/fail_build_artifact_missing/main.tf new file mode 100644 index 00000000..9037676d --- /dev/null +++ b/tests/fail_build_artifact_missing/main.tf @@ -0,0 +1,92 @@ +# Test case demonstrating build artifact issue (#517) +# This replicates the scenario where data.archive_file generates a zip +# during plan, but the zip is missing during apply because only the +# tfplan file is uploaded/downloaded between workflow runs. +# +# To test this scenario (simulating separate plan/apply jobs): +# 1. Run: terraform init +# 2. Run: terraform plan -out=tfplan +# (This creates build/bundle.zip via data.archive_file) +# 3. Simulate artifact transfer: rm -rf build/ +# 4. Run: terraform apply tfplan +# (This FAILS because bundle.zip no longer exists) +# +# The new `build-artifacts` input solves this by allowing users to +# specify files/directories to upload alongside the tfplan file. +# +# Example workflow usage with build-artifacts: +# - uses: op5dev/tf-via-pr@v13 +# with: +# command: plan +# build-artifacts: build/bundle.zip +# # Or for multiple files/directories: +# # build-artifacts: | +# # build/bundle.zip +# # dist/ + +terraform { + required_providers { + archive = { + source = "hashicorp/archive" + version = "~> 2.0" + } + null = { + source = "hashicorp/null" + version = "~> 3.0" + } + } +} + +# Archive inline content - creates zip during plan phase +data "archive_file" "bundle" { + type = "zip" + output_path = "${path.module}/build/bundle.zip" + + source { + content = "exports.handler = async (event) => { return { statusCode: 200 }; };" + filename = "index.js" + } + + source { + content = jsonencode({ name = "lambda-function", version = "1.0.0" }) + filename = "package.json" + } +} + +# Simulates deploying the archive (e.g., to S3, Lambda, etc.) +# This will fail during apply if the zip file doesn't exist +resource "null_resource" "deploy" { + triggers = { + archive_hash = data.archive_file.bundle.output_base64sha256 + } + + provisioner "local-exec" { + command = <<-EOT + if [ -f "${data.archive_file.bundle.output_path}" ]; then + echo "SUCCESS: Archive found at ${data.archive_file.bundle.output_path}" + echo "Archive size: $(wc -c < "${data.archive_file.bundle.output_path}") bytes" + echo "Archive hash: ${data.archive_file.bundle.output_base64sha256}" + else + echo "ERROR: Archive missing at ${data.archive_file.bundle.output_path}" + echo "" + echo "This failure occurs when:" + echo " 1. 'terraform plan' runs in one workflow job (creates the zip)" + echo " 2. Only the tfplan file is transferred to another job" + echo " 3. 'terraform apply tfplan' runs but the zip is missing" + echo "" + echo "Solution: Use 'build-artifacts' input to transfer the zip file" + exit 1 + fi + EOT + } +} + +output "archive_path" { + value = data.archive_file.bundle.output_path + description = "Path to the generated archive file" +} + +output "archive_size" { + value = data.archive_file.bundle.output_size + description = "Size of the archive in bytes" +} diff --git a/tests/tf.sh b/tests/tf.sh index 1f8563bd..f531f2ae 100755 --- a/tests/tf.sh +++ b/tests/tf.sh @@ -4,3 +4,15 @@ terraform -chdir=tests/pass_one init -no-color 2> >(tee pass_one.txt) > >(tee pa terraform -chdir=tests/fail_format_diff fmt -check=true -diff=true -no-color 2> >(tee fail_format_diff.txt) > >(tee fail_format_diff.txt) terraform -chdir=tests/fail_data_source_error init -no-color 2> >(tee fail_data_source_error.txt) > >(tee fail_data_source_error.txt) terraform -chdir=tests/fail_invalid_resource_type init -no-color 2> >(tee fail_invalid_resource_type.txt) > >(tee fail_invalid_resource_type.txt) + +# Test case for build artifact issue (#517) +# This demonstrates the scenario where interim build artifacts are missing during apply +echo "=== Testing build artifact scenario (issue #517) ===" +terraform -chdir=tests/fail_build_artifact_missing init -no-color +terraform -chdir=tests/fail_build_artifact_missing plan -out=tfplan -no-color +# Simulate artifact transfer (only tfplan, not the build directory) +rm -rf tests/fail_build_artifact_missing/build/ +# This apply should fail because bundle.zip is missing +terraform -chdir=tests/fail_build_artifact_missing apply tfplan -no-color 2> >(tee fail_build_artifact_missing.txt) > >(tee fail_build_artifact_missing.txt) || echo "Expected failure: build artifact missing" +# Cleanup +rm -f tests/fail_build_artifact_missing/tfplan