Skip to content

Commit 968a22e

Browse files
monsagriclaude
monsagri
and
claude
authored
ci: auto-chain stage-2 scaffold into stage-3 verify (autogen pipeline) (#462)
* ci: auto-chain stage-2 scaffold into stage-3 verify (autogen pipeline) Merge the scaffold (stage 2) and verify (stage 3) workflows so a single scaffold run auto-verifies its own draft PR in the same run, instead of requiring a second manual dispatch. Why a merge, not a cross-workflow trigger: GitHub's GITHUB_TOKEN cannot start new workflow runs (anti-recursion), so `gh workflow run` / `workflow_run` chaining would need a non-GITHUB_TOKEN credential (GitHub App or PAT) -- a new shared secret on a public repo. Running verify in-run as a reusable-workflow call sidesteps that entirely: no new credential, no new secret. - verify-reusable.yml (new): the verify job, `on: workflow_call`. Single source of truth; byte-identical job body to the previous verify-scaffold.yml. - verify-scaffold.yml: now a thin workflow_dispatch wrapper that calls the reusable workflow (manual re-verify of an existing scaffold/* PR preserved). - scaffold-resource.yml: the scaffold job exposes the new PR number (agent writes scaffold-result.json; case-insensitive branch-match fallback); a downstream `verify` job (needs: scaffold) calls the reusable workflow when a PR was produced and VERIFY_AGENT_ENABLED is true. Auto-verify is gated by the VERIFY_AGENT_ENABLED kill switch only (no per-family safelist). This intentionally drops the previous human gate between scaffold and verify: a scaffold now triggers a real apply against the verification account automatically. Blast radius stays bounded by the kill switch, the per-PR tf-verify-pr<N> project namespacing, and end-of-run destroy of account singletons (unchanged behavior). Stage-1 Slack one-click was intentionally not built: Slack Workflow Builder has no native authenticated HTTP step, so true one-click would require a Slack app install; the drift message's existing "Scaffold it" link + pre-filled gh command remain the stage-2 entry point. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: source agent prompts, model + kill switch from LaunchDarkly AI Configs Move the scaffold and verify agent prompts out of the workflow YAML into LD AI Configs (agent mode, catfood) so prompt/model changes need no PR. Each stage resolves its prompt from the AI-config-backed flag's PRODUCTION- served variation (fallthrough when targeting is on, else offVariation) via the flags REST API, then composes a "## This run" header (per-run values) on top of the LD instructions. The served variation's value._ldMeta.enabled is the kill switch -- pointing a flag's production fallthrough at its 'disabled' variation stops the agent -- so the SCAFFOLD_AGENT_ENABLED / VERIFY_AGENT_ENABLED repo variables are removed. Fail-closed: if LD is unreachable the run aborts; in verify the gate runs BEFORE AWS-OIDC / SSM / pre-clean so a disabled run never assumes the prod token or deletes the project. Pure-YAML (curl + jq, no SDK / Node / mustache): the Go AI SDK has no AgentConfig method (pre-release) and the Node/Python SDKs would add a foreign runtime to a Go repo. Only instructions + model are taken from LD; --allowedTools and the OIDC/build steps stay in the workflow (security-scoped). In-repo .github/agent-prompts/*.md are the prompt fallback, used only when LD is reachable + enabled but a variation carries no instructions. Reads via a read-only catfood token (LD_AICONFIG_READ_TOKEN secret, scoped to flag read on project default). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * ci: address code-review findings on the autogen pipeline workflows Fixes from the #462 review (15 findings; by-design items excluded): - verify-reusable: Notify Slack now also gates on steps.prompt.outcome == 'success', so a kill-switch-disabled or fail-closed (transient-LD) run no longer posts a misleading "verification did not complete" message. Reads verify-result.json by absolute $GITHUB_WORKSPACE path (cwd-independent). - scaffold: pass `family` to the extract step via env ("$FAMILY") instead of inlining ${{ inputs.family }} into the run script (GHA template injection). - both: resolve the LD model with a literal "Anthropic." prefix strip instead of ${model##*.} (which mangled versioned ids like claude-3.5-x), then reject any model name outside [A-Za-z0-9._:-] so it can't inject extra claude_args. - scaffold: resolve the auto-verify PR number SOLELY from scaffold-result.json; drop the branch-name `gh pr list` fallback that could send stage-3 to verify a STALE same-name PR (real prod apply on the wrong PR). Missing file -> verify skipped + warning to dispatch manually. - both: random heredoc delimiter for the multiline prompt output so prompt content can't prematurely close it (and swallow the model_arg line). - scaffold: add per-family `concurrency` so a double dispatch can't race two runs onto the same scaffold/<slug> branch + PR. - both: scope agent allowedTools — git to needed subcommands (no `git config`), make to `make fmt` (not the token-needing `make generate`). (Branch protection on preview-v3 remains the authoritative guard against a direct push.) - both: treat whitespace-only LD instructions as empty -> in-repo fallback; ::add-mask:: the LD read token; label the "## This run" header as DATA to blunt prompt-injection via PR title / operator notes. - verify-scaffold + scaffold: correct stale/incomplete comments (the dead VERIFY_AGENT_ENABLED kill switch; the inherited-secrets list). Validated: actionlint clean (3 workflows); bash/jq sims for model parsing, whitespace fallback, delimiter uniqueness, and the notify if-guard truth table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 4316582 commit 968a22e

5 files changed

Lines changed: 698 additions & 363 deletions

File tree

‎.github/agent-prompts/scaffold-instructions.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
You are stage 2 (scaffolding) of the LaunchDarkly Terraform provider autogen
2+
pipeline. Scaffold a NEW provider resource for a LaunchDarkly API endpoint
3+
family.
4+
5+
The family / resource / mode for THIS run is in the "## This run" block prepended
6+
above these instructions — read it FIRST. It tells you whether you are in
7+
WHOLE-FAMILY mode (scaffold a resource for an entire API family) or SCOPED mode
8+
(implement exactly ONE net-new resource for a named resource + an explicit list
9+
of operationIds inside a partial family), and carries the family tag, the
10+
resource name (scoped mode), the operationIds (scoped mode), and any operator
11+
notes.
12+
13+
Inputs:
14+
- Endpoint summary for the family: ./family-slice.json (repo root). This is the
15+
WHOLE family for context — read it, but do NOT commit it. In SCOPED mode, model
16+
ONLY the operationIds listed in the run context; ignore the rest of the family
17+
and do NOT modify, regenerate, or touch the family's EXISTING resources.
18+
- Full OpenAPI spec (for schemas): https://app.launchdarkly.com/api/v2/openapi.json
19+
- Read .claude/skills/terraform-provider-add-resource/SKILL.md and its
20+
references/patterns.md (both vendored into this repo), and follow that
21+
playbook's Steps 0-10 exactly. Also obey the conventions in CLAUDE.md:
22+
keys.go constants, framework nested attributes (never blocks), helpers from
23+
framework_helpers.go, patch helpers, handleLdapiErr wrapping, acceptance-test
24+
CI matrix entry, docs templates.
25+
- Use the generated API client (github.com/launchdarkly/api-client-go/v22) for
26+
all API calls. If the client lacks this surface, stop and report that instead
27+
of hand-rolling HTTP.
28+
- If the resource is an account-scoped SINGLETON (the API allows only one per
29+
account — its create returns 409/optimistic_locking_error once one exists),
30+
make the acceptance test self-cleaning. The acceptance token targets a
31+
DEDICATED Terraform test account (not a customer/prod account), so add a
32+
PreCheck that deletes ANY pre-existing instance before create
33+
(delete-existing-first — a leftover from an aborted run or a retained
34+
verification run would otherwise 409 the test). This is the repo's
35+
account-singleton orphan-cleanup PreCheck pattern; see references/patterns.md.
36+
Document the one-per-account limit in the schema Description.
37+
38+
Deliverable:
39+
1. Create a branch named scaffold/<resource name in SCOPED mode, else the family
40+
tag> (lowercased, non-alphanumerics replaced with '-') off preview-v3.
41+
2. Implement the resource (and data source if it has GET-by-key), unit-testable
42+
helpers, acceptance tests, docs templates, and provider registration. In
43+
scripts/driftreport/mapping.yaml: in SCOPED mode, move the run's operationIds
44+
out of the family's new_resource_candidates into a new entry under its
45+
resources (the family stays partial); in WHOLE-FAMILY mode, mark the family
46+
covered.
47+
3. Run go build ./... and make fmt; fix what they surface.
48+
4. Wire the resource into CI and docs so the draft PR clears the same quality
49+
gates a human PR must (stage-2 runs have historically skipped BOTH of these —
50+
do not omit them):
51+
- Acceptance matrix: add the resource's test prefix to the test_case matrix in
52+
.github/workflows/test.yml so its TestAcc* tests actually run in CI. Pick a
53+
`-run` prefix that does NOT overlap an existing entry; use a trailing-
54+
underscore prefix when a sibling resource shares a stem (e.g. TestAccMetric_
55+
vs TestAccMetricGroup_).
56+
- Docs: run `go generate .` (the repo-root directive runs tfplugindocs and
57+
`terraform fmt ./examples`; terraform is installed in this job). Do NOT run
58+
`make generate` or `go generate ./launchdarkly/...` — those also run the
59+
integration-configs codegen, which needs an API token this job intentionally
60+
lacks. Commit the generated docs/ (and any examples/ formatting) so CI's
61+
generate-diff check passes.
62+
5. Commit, push the branch, and open a DRAFT pull request against preview-v3
63+
titled "feat: scaffold <resource name or family> resource (autogen stage 2)".
64+
The PR body must state it is agent-scaffolded and needs human review per stage
65+
3 of the autogen pipeline, and note that stage-3 verification runs
66+
automatically on this PR.
67+
6. Write the PR number you just created to scaffold-result.json at the repo root
68+
(do NOT commit it), exactly:
69+
{"pr_number": <N>, "branch": "<your branch name>"}
70+
The pipeline reads this to auto-trigger stage-3 verification on YOUR PR; if you
71+
omit it, verification falls back to a best-effort branch-name match or must be
72+
dispatched by hand. Get <N> from the `gh pr create` output URL (.../pull/<N>)
73+
or `gh pr view <branch> --json number`.
74+
75+
Never push to preview-v3 directly and never merge anything.

‎.github/agent-prompts/verify-instructions.md‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
You are stage 3 (verification) of the LaunchDarkly Terraform provider autogen
2+
pipeline. A stage-2 agent scaffolded a NEW resource on the currently checked-out
3+
branch and opened a draft PR. Verify it against a REAL LaunchDarkly account
4+
before a human finishes it.
5+
6+
The PR number and title for THIS run are in the "## This run" block prepended
7+
above these instructions — read it FIRST. Wherever these instructions say
8+
"the PR number", use that value (e.g. the verification project is keyed
9+
`tf-verify-pr<PR number>`).
10+
11+
Environment already prepared for you:
12+
- The provider is built from this branch and installed to $GOBIN.
13+
- ~/.terraformrc has a dev_override for "launchdarkly/launchdarkly" pointing
14+
there, so `terraform plan`/`apply` run THIS build with NO `terraform init`.
15+
- Real LD credentials are in the environment: LAUNCHDARKLY_ACCESS_TOKEN and
16+
LAUNCHDARKLY_API_HOST=https://app.launchdarkly.com. NEVER print the token.
17+
- `terraform` is on PATH.
18+
- A previous `tf-verify-pr<PR number>` project (if any) was already deleted by a
19+
workflow step, so the account is clean for a project-scoped apply.
20+
21+
Do this, in order:
22+
23+
1. REVIEW the scaffolded code on this branch. Run `git fetch origin preview-v3`
24+
first, then `git diff origin/preview-v3...HEAD` to see only the scaffold, for
25+
correctness against CLAUDE.md and the vendored playbook at
26+
.claude/skills/terraform-provider-add-resource/ (SKILL.md + references/).
27+
Note real defects (schema/CRUD/helper/test/docs issues), not style nits.
28+
29+
2. EXERCISE the resource. Write a minimal terraform config under a temp dir
30+
(e.g. "$RUNNER_TEMP/verify") with a `provider "launchdarkly" {}` block (it
31+
reads creds from the environment), a prerequisite project keyed
32+
`tf-verify-pr<PR number>` (use this key so the live resources are identifiable
33+
and the workflow can pre-clean it next run), and the new resource (plus its
34+
data source if one was scaffolded). If the new resource is account-scoped
35+
rather than project-scoped, there is no project for the workflow to pre-clean,
36+
so name its key/name per-run as `tf-verify-pr<PR number>-run<the run number>`
37+
to avoid colliding with a resource retained by a previous run. (If the
38+
resource is an account SINGLETON, per-run naming cannot prevent a collision —
39+
only one can exist account-wide — so you will be destroying it in step 4; if a
40+
leftover from an earlier run still blocks `apply`/`create`, delete that one
41+
first via a targeted `terraform import` + `destroy`, or report the blocker.)
42+
Then:
43+
terraform -chdir="$RUNNER_TEMP/verify" plan -input=false
44+
terraform -chdir="$RUNNER_TEMP/verify" apply -input=false -auto-approve
45+
A second `plan` afterwards MUST be empty (no perpetual diff). Do NOT commit the
46+
scratch config or any terraform.tfstate*.
47+
48+
3. FIX as needed. If plan/apply or the empty-diff check fails, fix the resource
49+
implementation/tests/docs on the branch, re-run `go install .` (the dev
50+
override auto-picks up the rebuilt binary), and retry. Where practical, also
51+
run the resource's acceptance test:
52+
TF_ACC=1 TF_ACC_TERRAFORM_PATH="$(command -v terraform)" \
53+
go test -run <TestAccName> ./launchdarkly/... -v -timeout 30m
54+
Run `make fmt` and `go build ./...` before committing.
55+
56+
4. RETAIN or DESTROY by scope:
57+
- PROJECT-scoped resources: RETAIN them (no `terraform destroy`). They sit
58+
under the namespaced `tf-verify-pr<PR number>` project that the next run
59+
pre-cleans, so a human can inspect them.
60+
- ACCOUNT-scoped resources — ESPECIALLY account SINGLETONS (the API allows
61+
only one per account; a second create 409s): run
62+
`terraform -chdir="$RUNNER_TEMP/verify" destroy -input=false -auto-approve`
63+
at the end. A retained singleton has no per-run namespace and permanently
64+
holds the account's only slot, which blocks BOTH the next verify run and the
65+
resource's own acceptance test in shared CI (both then 409). `destroy` only
66+
removes what THIS run created (terraform state) — it never touches unrelated
67+
account data. Say in the report which resources you retained vs destroyed.
68+
69+
5. COMMIT any fixes to this branch (clear conventional-commit message; touch only
70+
files relevant to the fix — never the scratch config or state) and push.
71+
72+
6. REPORT. Post a PR comment on the PR. This repo is PUBLIC, so keep the comment
73+
to: the verification verdict (did plan/apply succeed? was the re-plan clean?
74+
which tests ran?), any code fixes you made, and the retained project key
75+
`tf-verify-pr<PR number>`. Do NOT restate API endpoint paths / operationIds /
76+
roadmap detail beyond what the diff already shows, and note the LD console link
77+
is in the private Slack thread. Use:
78+
gh pr comment <PR number> --body-file <file>
79+
80+
7. Write a machine-readable result to `verify-result.json` at the repo root (do
81+
NOT commit it) with exactly these fields:
82+
{"status":"pass"|"fail","project_key":"tf-verify-pr<PR number>",
83+
"ld_url":"https://app.launchdarkly.com/projects/tf-verify-pr<PR number>",
84+
"summary":"<one or two sentences: what you verified and any fixes>"}
85+
status is "pass" only if apply succeeded AND the re-plan was clean.

0 commit comments

Comments
 (0)