Skip to content

Commit 09c3d71

Browse files
justin808justin808
authored
Document cpflow workflow testing (#737)
* Document cpflow workflow testing * Address cpflow testing doc reviews
1 parent f028745 commit 09c3d71

5 files changed

Lines changed: 206 additions & 10 deletions

File tree

‎.controlplane/docs/testing-cpflow-github-actions.md‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
# Testing cpflow GitHub Actions Changes
2+
3+
Use this guide when changing generated `cpflow-*` GitHub Actions, updating the
4+
`cpflow` generator version, or debugging review-app automation.
5+
6+
## What To Test
7+
8+
Test the flow in three layers:
9+
10+
1. Local generated-file checks catch YAML, metadata, and lint problems before a PR.
11+
2. GitHub workflow checks prove GitHub can load the workflow and run CI.
12+
3. A real review-app deploy proves the default-branch trusted actions, GitHub
13+
secrets, Docker build, and Control Plane deploy all work together.
14+
15+
The third layer matters because the review-app workflow intentionally checks out
16+
trusted workflow sources from the repository default branch before passing
17+
Control Plane secrets to local composite actions. A PR branch can contain fixed
18+
`.github/actions/*` files, but the deploy job still loads those local actions
19+
from `master` until the fix is merged there.
20+
21+
## Local Checks
22+
23+
After regenerating the flow, run these checks from the repository root. If
24+
`cpflow` is installed as a gem, use `cpflow` directly:
25+
26+
```sh
27+
bin/conductor-exec cpflow generate-github-actions --staging-branch master
28+
bin/test-cpflow-github-flow
29+
```
30+
31+
When testing an unreleased upstream `control-plane-flow` checkout, replace
32+
`cpflow` with that checkout's `bin/cpflow`:
33+
34+
```sh
35+
bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow generate-github-actions --staging-branch master
36+
bin/test-cpflow-github-flow ruby /path/to/control-plane-flow/bin/cpflow
37+
```
38+
39+
Why the explicit description check exists: GitHub parses expression-like snippets
40+
inside composite action metadata, including `description:` fields. Literal
41+
examples such as `${{ vars.SOME_VALUE }}` can fail action loading before any
42+
shell step starts. The wrapper runs `cpflow github-flow-readiness`, parses the
43+
generated YAML, checks action input descriptions for literal GitHub expressions,
44+
and runs `actionlint -ignore 'SC2129' .github/workflows/cpflow-*.yml`.
45+
46+
## PR Checks
47+
48+
Open a normal PR for the generated-file diff and wait for CI. The workflow PR
49+
itself is useful for syntax and CI validation, but it does not fully prove
50+
review-app deployment changes that live under `.github/actions/`.
51+
52+
For top-level workflow edits, you can manually dispatch the PR branch workflow:
53+
54+
```sh
55+
gh workflow run cpflow-deploy-review-app.yml --ref <branch> -f pr_number=<pr-number>
56+
```
57+
58+
This loads the workflow file from `<branch>`, but the deploy workflow's
59+
`Checkout trusted workflow sources` step still checks out `master` before using
60+
local composite actions with secrets. Treat this as a partial smoke test, not as
61+
proof that PR-branch composite action changes work.
62+
63+
## Post-Merge Review-App Test
64+
65+
After the workflow PR merges to `master`, test a real review-app deployment:
66+
67+
1. Pick a same-repository PR to use as the canary.
68+
2. If the review app does not exist yet, comment exactly `+review-app-deploy` on
69+
that PR.
70+
3. If a previous deploy run failed, rerun the failed deploy run after the
71+
workflow PR is merged.
72+
4. Confirm the deploy run checks out `master` at the merge commit in
73+
`Checkout trusted workflow sources`.
74+
5. Confirm `Setup environment` succeeds and prints the expected `cpflow` version.
75+
6. Confirm `Check if review app exists`, `Build Docker image`, and
76+
`Deploy to Control Plane` all run as expected.
77+
7. Open the review-app URL from the PR comment or deployment status and verify
78+
it returns HTTP 200.
79+
80+
Use the generated app name from the workflow log:
81+
82+
```text
83+
APP_NAME: ${REVIEW_APP_PREFIX}-${PR_NUMBER}
84+
```
85+
86+
This is a template from the workflow output, not a literal command to evaluate
87+
unless those environment variables are already set. For this repo, verify the
88+
actual `REVIEW_APP_PREFIX` repository variable before assuming the final app
89+
name.
90+
91+
## Troubleshooting Signals
92+
93+
### Composite action metadata fails before setup
94+
95+
Error shape:
96+
97+
```text
98+
Unrecognized named-value: 'vars'
99+
Failed to load ./.github/actions/cpflow-setup-environment/action.yml
100+
```
101+
102+
Cause: GitHub parsed a literal expression inside composite action metadata,
103+
usually an input description. Because trusted local actions come from `master`,
104+
fix and merge the generated action metadata on `master`, then rerun the deploy.
105+
106+
### Setup succeeds, then `cpflow exists` reports token format
107+
108+
Error shape:
109+
110+
```text
111+
ERROR: Unknown API token format. Please re-run 'cpln profile login' or set the correct CPLN_TOKEN env variable.
112+
```
113+
114+
Cause: the workflow can read `CPLN_TOKEN_STAGING`, but the value is not a valid
115+
Control Plane service-account token for the installed Control Plane CLI. Rotate
116+
the GitHub secret, then rerun the failed deploy job.
117+
118+
### PR pushes do not create a new review app
119+
120+
This is expected. Pushes redeploy only after the review app already exists.
121+
Create the first review app by commenting exactly:
122+
123+
```text
124+
+review-app-deploy
125+
```
126+
127+
## Ways To Make This Easier
128+
129+
- Add a no-secret GitHub Actions smoke workflow that loads generated local
130+
composite actions from the PR branch and fails fast on action metadata parsing.
131+
- Extend `bin/test-cpflow-github-flow` as more local cpflow GitHub Actions
132+
checks become worth standardizing.
133+
- Add an early token sanity step after `Setup environment` so invalid
134+
`CPLN_TOKEN_STAGING` and `CPLN_TOKEN_PRODUCTION` values fail with a named
135+
"validate Control Plane token" step instead of surfacing later during
136+
`cpflow exists`.
137+
- Keep a tiny canary PR open for review-app workflow testing so post-merge
138+
deploy verification does not depend on whichever feature PR happens to exist.
139+
- Upstream the metadata-description check to `cpflow github-flow-readiness` so
140+
downstream repos get the guard automatically.

‎.controlplane/readme.md‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -424,24 +424,33 @@ repository variables, secrets, and docs aligned with `.controlplane/controlplane
424424
For this app, validate a regenerated flow with:
425425

426426
```bash
427-
bundle exec ruby /path/to/control-plane-flow/bin/cpflow generate-github-actions --staging-branch master
428-
bundle exec ruby /path/to/control-plane-flow/bin/cpflow github-flow-readiness
429-
actionlint .github/workflows/cpflow-*.yml
430-
bundle exec rubocop
427+
bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow generate-github-actions --staging-branch master
428+
bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow github-flow-readiness
429+
actionlint -ignore 'SC2129' .github/workflows/cpflow-*.yml
430+
bin/conductor-exec bundle exec rubocop
431431
```
432432

433433
Then open a normal PR and let GitHub Actions prove the generated review-app,
434434
staging, lint, JS, and RSpec workflows before merging. For review-app workflow
435435
changes, test both the local workflow syntax and a real deployment. GitHub runs
436436
`issue_comment` workflows from the default branch, so a `+review-app-deploy`
437437
comment on the PR does not fully exercise command changes that are only on the
438-
PR branch. Before merge, run the PR branch workflow explicitly:
438+
PR branch. For top-level workflow edits, run the PR branch workflow explicitly:
439439

440440
```bash
441441
gh workflow run cpflow-deploy-review-app.yml --ref <branch> -f pr_number=<pr-number>
442442
```
443443

444+
This loads the workflow file from `<branch>`, but trusted local composite
445+
actions still come from the default branch before secrets are used. Treat it as
446+
a partial smoke test, then verify a real deploy after the workflow changes land
447+
on `master`.
448+
444449
After the workflow reports a review-app URL, verify the URL returns HTTP 200.
445450
If a project needs to track generator changes automatically, use a scheduled
446451
maintenance PR or Renovate-style workflow that bumps the `cpflow` version,
447452
regenerates these files, and runs the same validation commands.
453+
454+
For a fuller checklist, including the gotcha that review-app deploys load local
455+
composite actions from `master` before using Control Plane secrets, see
456+
[Testing cpflow GitHub Actions Changes](docs/testing-cpflow-github-actions.md).

‎.controlplane/shakacode-team.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -66,9 +66,14 @@ flow, regenerate the `cpflow-*` actions/workflows in this repo from the target
6666
`cpflow` version or branch using `--staging-branch master`, review the diff, and
6767
keep the repository variables above aligned with `.controlplane/controlplane.yml`. Validate with
6868
`cpflow github-flow-readiness`, `actionlint .github/workflows/cpflow-*.yml`, and
69-
the normal CI checks before merging.
70-
71-
See [readme.md](readme.md) for more details.
69+
the normal CI checks before merging. For review-app workflow changes, remember
70+
that the deploy workflow checks out trusted local actions from `master` before
71+
passing Control Plane secrets; PR-branch composite action changes are not fully
72+
tested until they land on `master` and a real review-app deploy is rerun.
73+
74+
See [readme.md](readme.md) and
75+
[Testing cpflow GitHub Actions Changes](docs/testing-cpflow-github-actions.md)
76+
for more details.
7277

7378
## Links
7479

‎.github/cpflow-help.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,14 +60,14 @@ You asked for review app help. These commands are generated by [cpflow](https://
6060
<details>
6161
<summary>Advanced: testing changes to generated workflows</summary>
6262

63-
When iterating on the generated workflow YAML on a PR branch, comment-triggered runs (`+review-app-deploy`, `+review-app-delete`, `+review-app-help`) execute the workflow code from the repository's default branch — not your PR branch. To exercise the PR-branch workflow code before merging, dispatch the workflow manually with `gh`:
63+
When iterating on the generated workflow YAML on a PR branch, comment-triggered runs (`+review-app-deploy`, `+review-app-delete`, `+review-app-help`) execute the workflow code from the repository's default branch — not your PR branch. To exercise the top-level PR-branch workflow file before merging, dispatch the workflow manually with `gh`:
6464

6565
```sh
6666
gh workflow run cpflow-deploy-review-app.yml --ref <your-pr-branch> -f pr_number=<pr-number>
6767
gh workflow run cpflow-delete-review-app.yml --ref <your-pr-branch> -f pr_number=<pr-number>
6868
gh workflow run cpflow-help-command.yml --ref <your-pr-branch> -f pr_number=<pr-number>
6969
```
7070

71-
`workflow_dispatch` runs use the workflow file from the `--ref` you pass, so this is the supported way to test PR-branch workflow edits before merge. After merge, comment triggers go back to running the default-branch workflow code as usual.
71+
`workflow_dispatch` runs use the workflow file from the `--ref` you pass, but workflows that intentionally check out trusted local actions from the default branch will still load those local composite actions from the default branch before using secrets. Treat this as a partial smoke test for top-level workflow edits. For changes under `.github/actions/`, merge the generated fix to the default branch and rerun a real review-app deploy.
7272

7373
</details>

‎bin/test-cpflow-github-flow‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
4+
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
5+
cd "$ROOT"
6+
7+
cpflow_cmd=(cpflow)
8+
if [[ $# -gt 0 ]]; then
9+
cpflow_cmd=("$@")
10+
fi
11+
12+
echo "==> cpflow github-flow-readiness"
13+
bin/conductor-exec "${cpflow_cmd[@]}" github-flow-readiness
14+
15+
echo "==> parse generated GitHub Actions YAML"
16+
bin/conductor-exec ruby <<'RUBY'
17+
require "yaml"
18+
19+
Dir[".github/actions/**/action.yml", ".github/workflows/*.yml"].sort.each do |path|
20+
YAML.load_file(path, aliases: true)
21+
puts "parsed #{path}"
22+
end
23+
RUBY
24+
25+
echo "==> check composite action input descriptions"
26+
bin/conductor-exec ruby <<'RUBY'
27+
require "yaml"
28+
29+
bad = []
30+
Dir[".github/actions/**/action.yml"].sort.each do |path|
31+
doc = YAML.load_file(path, aliases: true)
32+
doc.fetch("inputs", {}).each do |name, spec|
33+
bad << "#{path}:#{name}" if spec["description"].to_s.include?("${{")
34+
end
35+
end
36+
37+
abort bad.join("\n") unless bad.empty?
38+
puts "no action metadata descriptions contain GitHub expressions"
39+
RUBY
40+
41+
echo "==> actionlint"
42+
actionlint -ignore "SC2129" .github/workflows/cpflow-*.yml

0 commit comments

Comments
 (0)