Skip to content

Commit 0323be2

Browse files
justin808justin808
committed
Document cpflow workflow testing
1 parent f028745 commit 0323be2

4 files changed

Lines changed: 149 additions & 8 deletions

File tree

‎.controlplane/docs/testing-cpflow-github-actions.md‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
# Testing cpflow GitHub Actions Changes
2+
3+
Use this guide when changing generated `cpflow-*` GitHub Actions, updating the
4+
`cpflow` generator version, or debugging review-app automation.
5+
6+
## What To Test
7+
8+
Test the flow in three layers:
9+
10+
1. Local generated-file checks catch YAML, metadata, and lint problems before a PR.
11+
2. GitHub workflow checks prove GitHub can load the workflow and run CI.
12+
3. A real review-app deploy proves the default-branch trusted actions, GitHub
13+
secrets, Docker build, and Control Plane deploy all work together.
14+
15+
The third layer matters because the review-app workflow intentionally checks out
16+
trusted workflow sources from the repository default branch before passing
17+
Control Plane secrets to local composite actions. A PR branch can contain fixed
18+
`.github/actions/*` files, but the deploy job still loads those local actions
19+
from `master` until the fix is merged there.
20+
21+
## Local Checks
22+
23+
After regenerating the flow, run these checks from the repository root. When
24+
testing an unreleased upstream `control-plane-flow` checkout, invoke that
25+
checkout's `bin/cpflow` directly:
26+
27+
```sh
28+
bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow generate-github-actions --staging-branch master
29+
bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow github-flow-readiness
30+
bin/conductor-exec ruby -e 'require "yaml"; Dir[".github/actions/**/action.yml", ".github/workflows/*.yml"].sort.each { |path| YAML.load_file(path, aliases: true); puts "parsed #{path}" }'
31+
bin/conductor-exec ruby -e 'require "yaml"; bad=[]; Dir[".github/actions/**/action.yml"].sort.each { |path| doc=YAML.load_file(path, aliases: true); doc.fetch("inputs", {}).each { |name, spec| bad << "#{path}:#{name}" if spec["description"].to_s.include?("${{") } }; }; abort bad.join("\n") unless bad.empty?; puts "no action metadata descriptions contain GitHub expressions"'
32+
actionlint -ignore 'SC2129' .github/workflows/cpflow-*.yml
33+
```
34+
35+
Why the explicit description check exists: GitHub parses expression-like snippets
36+
inside composite action metadata, including `description:` fields. Literal
37+
examples such as `${{ vars.SOME_VALUE }}` can fail action loading before any
38+
shell step starts.
39+
40+
## PR Checks
41+
42+
Open a normal PR for the generated-file diff and wait for CI. The workflow PR
43+
itself is useful for syntax and CI validation, but it does not fully prove
44+
review-app deployment changes that live under `.github/actions/`.
45+
46+
For top-level workflow edits, you can manually dispatch the PR branch workflow:
47+
48+
```sh
49+
gh workflow run cpflow-deploy-review-app.yml --ref <branch> -f pr_number=<pr-number>
50+
```
51+
52+
This loads the workflow file from `<branch>`, but the deploy workflow's
53+
`Checkout trusted workflow sources` step still checks out `master` before using
54+
local composite actions with secrets. Treat this as a partial smoke test, not as
55+
proof that PR-branch composite action changes work.
56+
57+
## Post-Merge Review-App Test
58+
59+
After the workflow PR merges to `master`, test a real review-app deployment:
60+
61+
1. Pick a same-repository PR to use as the canary.
62+
2. If the review app does not exist yet, comment exactly `+review-app-deploy` on
63+
that PR.
64+
3. If a previous deploy run failed, rerun the failed deploy run after the
65+
workflow PR is merged.
66+
4. Confirm the deploy run checks out `master` at the merge commit in
67+
`Checkout trusted workflow sources`.
68+
5. Confirm `Setup environment` succeeds and prints the expected `cpflow` version.
69+
6. Confirm `Check if review app exists`, `Build Docker image`, and
70+
`Deploy to Control Plane` all run as expected.
71+
7. Open the review-app URL from the PR comment or deployment status and verify
72+
it returns HTTP 200.
73+
74+
Use the generated app name from the workflow log:
75+
76+
```text
77+
APP_NAME: ${REVIEW_APP_PREFIX}-${PR_NUMBER}
78+
```
79+
80+
For this repo, verify the actual `REVIEW_APP_PREFIX` repository variable before
81+
assuming the final app name.
82+
83+
## Troubleshooting Signals
84+
85+
### Composite action metadata fails before setup
86+
87+
Error shape:
88+
89+
```text
90+
Unrecognized named-value: 'vars'
91+
Failed to load ./.github/actions/cpflow-setup-environment/action.yml
92+
```
93+
94+
Cause: GitHub parsed a literal expression inside composite action metadata,
95+
usually an input description. Because trusted local actions come from `master`,
96+
fix and merge the generated action metadata on `master`, then rerun the deploy.
97+
98+
### Setup succeeds, then `cpflow exists` reports token format
99+
100+
Error shape:
101+
102+
```text
103+
ERROR: Unknown API token format. Please re-run 'cpln profile login' or set the correct CPLN_TOKEN env variable.
104+
```
105+
106+
Cause: the workflow can read `CPLN_TOKEN_STAGING`, but the value is not a valid
107+
Control Plane service-account token for the installed Control Plane CLI. Rotate
108+
the GitHub secret, then rerun the failed deploy job.
109+
110+
### PR pushes do not create a new review app
111+
112+
This is expected. Pushes redeploy only after the review app already exists.
113+
Create the first review app by commenting exactly:
114+
115+
```text
116+
+review-app-deploy
117+
```
118+
119+
## Ways To Make This Easier
120+
121+
- Add a no-secret GitHub Actions smoke workflow that loads generated local
122+
composite actions from the PR branch and fails fast on action metadata parsing.
123+
- Add a small repo script, for example `bin/test-cpflow-github-flow`, that wraps
124+
the local YAML, metadata-description, readiness, and `actionlint` checks.
125+
- Add an early token sanity step after `Setup environment` so invalid
126+
`CPLN_TOKEN_STAGING` and `CPLN_TOKEN_PRODUCTION` values fail with a named
127+
"validate Control Plane token" step instead of surfacing later during
128+
`cpflow exists`.
129+
- Keep a tiny canary PR open for review-app workflow testing so post-merge
130+
deploy verification does not depend on whichever feature PR happens to exist.
131+
- Upstream the metadata-description check to `cpflow github-flow-readiness` so
132+
downstream repos get the guard automatically.

‎.controlplane/readme.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -424,10 +424,10 @@ repository variables, secrets, and docs aligned with `.controlplane/controlplane
424424
For this app, validate a regenerated flow with:
425425

426426
```bash
427-
bundle exec ruby /path/to/control-plane-flow/bin/cpflow generate-github-actions --staging-branch master
428-
bundle exec ruby /path/to/control-plane-flow/bin/cpflow github-flow-readiness
427+
bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow generate-github-actions --staging-branch master
428+
bin/conductor-exec ruby /path/to/control-plane-flow/bin/cpflow github-flow-readiness
429429
actionlint .github/workflows/cpflow-*.yml
430-
bundle exec rubocop
430+
bin/conductor-exec bundle exec rubocop
431431
```
432432

433433
Then open a normal PR and let GitHub Actions prove the generated review-app,
@@ -445,3 +445,7 @@ After the workflow reports a review-app URL, verify the URL returns HTTP 200.
445445
If a project needs to track generator changes automatically, use a scheduled
446446
maintenance PR or Renovate-style workflow that bumps the `cpflow` version,
447447
regenerates these files, and runs the same validation commands.
448+
449+
For a fuller checklist, including the gotcha that review-app deploys load local
450+
composite actions from `master` before using Control Plane secrets, see
451+
[Testing cpflow GitHub Actions Changes](docs/testing-cpflow-github-actions.md).

‎.controlplane/shakacode-team.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -66,9 +66,14 @@ flow, regenerate the `cpflow-*` actions/workflows in this repo from the target
6666
`cpflow` version or branch using `--staging-branch master`, review the diff, and
6767
keep the repository variables above aligned with `.controlplane/controlplane.yml`. Validate with
6868
`cpflow github-flow-readiness`, `actionlint .github/workflows/cpflow-*.yml`, and
69-
the normal CI checks before merging.
70-
71-
See [readme.md](readme.md) for more details.
69+
the normal CI checks before merging. For review-app workflow changes, remember
70+
that the deploy workflow checks out trusted local actions from `master` before
71+
passing Control Plane secrets; PR-branch composite action changes are not fully
72+
tested until they land on `master` and a real review-app deploy is rerun.
73+
74+
See [readme.md](readme.md) and
75+
[Testing cpflow GitHub Actions Changes](docs/testing-cpflow-github-actions.md)
76+
for more details.
7277

7378
## Links
7479

‎.github/cpflow-help.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,14 +60,14 @@ You asked for review app help. These commands are generated by [cpflow](https://
6060
<details>
6161
<summary>Advanced: testing changes to generated workflows</summary>
6262

63-
When iterating on the generated workflow YAML on a PR branch, comment-triggered runs (`+review-app-deploy`, `+review-app-delete`, `+review-app-help`) execute the workflow code from the repository's default branch — not your PR branch. To exercise the PR-branch workflow code before merging, dispatch the workflow manually with `gh`:
63+
When iterating on the generated workflow YAML on a PR branch, comment-triggered runs (`+review-app-deploy`, `+review-app-delete`, `+review-app-help`) execute the workflow code from the repository's default branch — not your PR branch. To exercise the top-level PR-branch workflow file before merging, dispatch the workflow manually with `gh`:
6464

6565
```sh
6666
gh workflow run cpflow-deploy-review-app.yml --ref <your-pr-branch> -f pr_number=<pr-number>
6767
gh workflow run cpflow-delete-review-app.yml --ref <your-pr-branch> -f pr_number=<pr-number>
6868
gh workflow run cpflow-help-command.yml --ref <your-pr-branch> -f pr_number=<pr-number>
6969
```
7070

71-
`workflow_dispatch` runs use the workflow file from the `--ref` you pass, so this is the supported way to test PR-branch workflow edits before merge. After merge, comment triggers go back to running the default-branch workflow code as usual.
71+
`workflow_dispatch` runs use the workflow file from the `--ref` you pass, but workflows that intentionally check out trusted local actions from the default branch will still load those local composite actions from the default branch before using secrets. Treat this as a partial smoke test for top-level workflow edits. For changes under `.github/actions/`, merge the generated fix to the default branch and rerun a real review-app deploy.
7272

7373
</details>

0 commit comments

Comments
 (0)