Skip to content

feat(workflows): adopt customize.toml pattern for testarch workflows#91

Merged
muratkeremozcan merged 3 commits intomainfrom
add-workflow-customization
Apr 23, 2026
Merged

feat(workflows): adopt customize.toml pattern for testarch workflows#91
muratkeremozcan merged 3 commits intomainfrom
add-workflow-customization

Conversation

@bmadcode
Copy link
Copy Markdown
Contributor

Summary

Apply the workflow customization surface to all 9 testarch workflows (8 bmad-testarch-* + bmad-teach-me-testing), mirroring the rollout already shipped for the bmad-tea agent and for GDS workflows in bmad-module-game-dev-studio#23.

For each workflow:

  • Add customize.toml with a [workflow] block exposing activation_steps_prepend / activation_steps_append, persistent_facts (defaults to file:{project-root}/**/project-context.md), and on_complete.
  • Merge workflow.md into SKILL.md as the single source of truth and delete workflow.md. The new SKILL.md carries Goal/Role, Conventions, a 6-step On Activation block (resolve customization → prepend → persistent_facts → load _bmad/tea/config.yaml → greet → append), Workflow Architecture, Mode Determination, and routing.
  • Wire the workflow.on_complete resolver into the terminal step file of every mode (steps-c, steps-v, steps-e) so the hook fires regardless of which mode the user runs — same multi-terminal pattern used for gds-sprint-status in the GDS rollout.

workflow.yaml is left intact: it carries installation/registry metadata (required_tools, tags, execution_hints) that is a separate concern from the runtime customization surface.

test/test-installation-components.js is updated to validate the new pattern: customize.toml exists with the expected keys, SKILL.md drives activation through the resolver script, and workflow.md is removed.

Files changed

  • 9 new customize.toml
  • 9 deleted workflow.md
  • 9 modified SKILL.md
  • 27 modified terminal step files (3 per workflow × 9 workflows)
  • 1 modified test file

Test plan

  • npm test — all 739 tests pass (52 + 629 + 58 across three suites)
  • npm run lint — clean
  • npm run lint:md — clean
  • npm run format:check — clean
  • npm run validate:schemas — clean
  • Spot-check installation into a fresh project with one workflow customized via team override
  • Run a full workflow in each mode (create / validate / edit) and confirm on_complete resolver fires at each terminal

Apply the workflow customization surface to all 9 testarch workflows
(8 bmad-testarch-* + bmad-teach-me-testing), mirroring the rollout
shipped for the bmad-tea agent and the GDS workflows in
bmad-code-org/bmad-module-game-dev-studio#23.

For each workflow:
- Add customize.toml with [workflow] block exposing
  activation_steps_prepend / activation_steps_append, persistent_facts
  (defaults to file:{project-root}/**/project-context.md), and
  on_complete.
- Merge workflow.md into SKILL.md as the single source of truth and
  delete workflow.md. The new SKILL.md carries Goal/Role, Conventions,
  a 6-step On Activation block (resolve customization → prepend →
  persistent_facts → load _bmad/tea/config.yaml → greet → append),
  Workflow Architecture, Mode Determination, and routing.
- Wire the workflow.on_complete resolver into the terminal step file
  of every mode (steps-c, steps-v, steps-e) so the hook fires
  regardless of which mode the user runs — same multi-terminal pattern
  used for gds-sprint-status in the GDS rollout.

workflow.yaml is left intact: it carries installation/registry
metadata (required_tools, tags, execution_hints) that is a separate
concern from the runtime customization surface.

Update test/test-installation-components.js to validate the new
pattern: customize.toml exists with the expected keys, SKILL.md drives
activation through the resolver script, and workflow.md is removed.
@augmentcode
Copy link
Copy Markdown

augmentcode Bot commented Apr 23, 2026

🤖 Augment PR Summary

Summary: This PR rolls out the customize.toml-based customization surface across the Test Architect (testarch) workflows, aligning them with the existing TEA agent and prior GDS workflow rollout.

Changes:

  • Added customize.toml to 9 workflows to expose activation_steps_prepend, activation_steps_append, persistent_facts (defaulting to file:{project-root}/**/project-context.md), and on_complete.
  • Folded each workflow’s workflow.md content into SKILL.md as the single activation source of truth, and removed the legacy workflow.md files.
  • Updated each workflow’s terminal step files (create/validate/edit) to resolve and execute workflow.on_complete, ensuring the hook runs regardless of mode.
  • Kept workflow.yaml intact for registry/installation metadata (tools, tags, hints) while moving runtime customization guidance into SKILL.md/customize.toml.
  • Updated installation component tests to validate the new structure (presence of customize.toml, activation resolver usage in SKILL.md, and removal of workflow.md).

Technical Notes: Activation now follows a standardized 6-step sequence (resolve customization → prepend → persistent facts → config load → greet → append), and completion behavior is centralized via the workflow.on_complete resolver hook.

🤖 Was this summary useful? React with 👍 or 👎

Copy link
Copy Markdown

@augmentcode augmentcode Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review completed. 2 suggestions posted.

Fix All in Augment

Comment augment review to trigger a new review at any time.

Comment thread test/test-installation-components.js Outdated
console.log(`${colors.yellow}Test Suite 4: Workflow Structure${colors.reset}\n`);

const teachMeWorkflowPath = path.join(projectRoot, 'src/workflows/testarch/bmad-teach-me-testing/workflow.md');
const teachMeSkillPath = path.join(projectRoot, 'src/workflows/testarch/bmad-teach-me-testing/SKILL.md');
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bmad-teach-me-testing is now validated only via SKILL.md existence; since this PR also introduces customize.toml and removes workflow.md for that workflow, consider asserting those conditions too so regressions are caught.

Severity: low

Generating Fix in Augment link...

🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.

assert(false, `${dirName}/customize.toml exists`, `src/workflows/testarch/${dirName}/customize.toml not found`);
}

// workflow.md was folded into SKILL.md and removed (PR: workflow customization rollout).
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR formalizes that workflow.md is removed (content lives in SKILL.md); it may be worth double-checking repo docs like src/workflows/testarch/README.md that still describe workflow.md as the canonical entrypoint so the documentation stays consistent.

Severity: low

Generating Fix in Augment link...

🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented Apr 23, 2026

📝 Walkthrough

Walkthrough

Nine test architecture workflows are restructured: workflow.md files are deleted, SKILL.md files are expanded with detailed activation protocols, and new customize.toml files introduce workflow configuration. All step files across create/validate/edit modes add "On Complete" hooks for customization resolution. Tests updated to validate new SKILL.md and customize.toml structure.

Changes

Cohort / File(s) Summary
SKILL.md Activation Protocol Expansions
src/workflows/testarch/bmad-teach-me-testing/SKILL.md, src/workflows/testarch/bmad-testarch-atdd/SKILL.md, src/workflows/testarch/bmad-testarch-automate/SKILL.md, src/workflows/testarch/bmad-testarch-ci/SKILL.md, src/workflows/testarch/bmad-testarch-framework/SKILL.md, src/workflows/testarch/bmad-testarch-nfr/SKILL.md, src/workflows/testarch/bmad-testarch-test-design/SKILL.md, src/workflows/testarch/bmad-testarch-test-review/SKILL.md, src/workflows/testarch/bmad-testarch-trace/SKILL.md
Each workflow's SKILL.md now defines a comprehensive activation pipeline: customization resolver with fallback merge strategy, configurable pre/post-greet activation steps, persistent facts loading with file-based glob support, runtime config loading from _bmad/tea/config.yaml, user greeting with language selection, and explicit tri-modal architecture (create/validate/edit) with routing logic. Path resolution rules expanded to cover bare paths, skill-root, project-root, and sibling file resolution.
workflow.md File Deletions
src/workflows/testarch/bmad-teach-me-testing/workflow.md, src/workflows/testarch/bmad-testarch-atdd/workflow.md, src/workflows/testarch/bmad-testarch-automate/workflow.md, src/workflows/testarch/bmad-testarch-ci/workflow.md, src/workflows/testarch/bmad-testarch-framework/workflow.md, src/workflows/testarch/bmad-testarch-nfr/workflow.md, src/workflows/testarch/bmad-testarch-test-design/workflow.md, src/workflows/testarch/bmad-testarch-test-review/workflow.md, src/workflows/testarch/bmad-testarch-trace/workflow.md
All workflow metadata and instructions removed; functionality migrated to SKILL.md and customize.toml.
Workflow Customization Configuration
src/workflows/testarch/bmad-teach-me-testing/customize.toml, src/workflows/testarch/bmad-testarch-atdd/customize.toml, src/workflows/testarch/bmad-testarch-automate/customize.toml, src/workflows/testarch/bmad-testarch-ci/customize.toml, src/workflows/testarch/bmad-testarch-framework/customize.toml, src/workflows/testarch/bmad-testarch-nfr/customize.toml, src/workflows/testarch/bmad-testarch-test-design/customize.toml, src/workflows/testarch/bmad-testarch-test-review/customize.toml, src/workflows/testarch/bmad-testarch-trace/customize.toml
New TOML configuration files under [workflow] section: activation_steps_prepend and activation_steps_append (empty arrays), persistent_facts array with file-based glob reference to {project-root}/**/project-context.md, and on_complete string hook (empty by default).
Step Completion Hooks
src/workflows/testarch/bmad-teach-me-testing/steps-c/step-05-completion.md, src/workflows/testarch/bmad-testarch-atdd/steps-c/step-05-validate-and-complete.md, src/workflows/testarch/bmad-testarch-automate/steps-c/step-04-validate-and-summarize.md, src/workflows/testarch/bmad-testarch-ci/steps-c/step-04-validate-and-summary.md, src/workflows/testarch/bmad-testarch-framework/steps-c/step-05-validate-and-summary.md, src/workflows/testarch/bmad-testarch-nfr/steps-c/step-05-generate-report.md, src/workflows/testarch/bmad-testarch-test-design/steps-c/step-05-generate-output.md, src/workflows/testarch/bmad-testarch-test-review/steps-c/step-04-generate-report.md, src/workflows/testarch/bmad-testarch-trace/steps-c/step-05-gate-decision.md
All create-mode step files add "On Complete" hook that resolves workflow.on_complete via resolve_customization.py and conditionally executes resolved non-empty instruction as final terminal action before exit.
Validate Step Completion Hooks
src/workflows/testarch/bmad-teach-me-testing/steps-v/step-v-01-validate.md, src/workflows/testarch/bmad-testarch-atdd/steps-v/step-01-validate.md, src/workflows/testarch/bmad-testarch-automate/steps-v/step-01-validate.md, src/workflows/testarch/bmad-testarch-ci/steps-v/step-01-validate.md, src/workflows/testarch/bmad-testarch-framework/steps-v/step-01-validate.md, src/workflows/testarch/bmad-testarch-nfr/steps-v/step-01-validate.md, src/workflows/testarch/bmad-testarch-test-design/steps-v/step-01-validate.md, src/workflows/testarch/bmad-testarch-test-review/steps-v/step-01-validate.md, src/workflows/testarch/bmad-testarch-trace/steps-v/step-01-validate.md
All validate-mode step files add post-validation "On Complete" hook that resolves and conditionally executes workflow.on_complete as final terminal instruction.
Edit Step Completion Hooks
src/workflows/testarch/bmad-teach-me-testing/steps-e/step-e-02-apply-edits.md, src/workflows/testarch/bmad-testarch-atdd/steps-e/step-02-apply-edit.md, src/workflows/testarch/bmad-testarch-automate/steps-e/step-02-apply-edit.md, src/workflows/testarch/bmad-testarch-ci/steps-e/step-02-apply-edit.md, src/workflows/testarch/bmad-testarch-framework/steps-e/step-02-apply-edit.md, src/workflows/testarch/bmad-testarch-nfr/steps-e/step-02-apply-edit.md, src/workflows/testarch/bmad-testarch-test-design/steps-e/step-02-apply-edit.md, src/workflows/testarch/bmad-testarch-test-review/steps-e/step-02-apply-edit.md, src/workflows/testarch/bmad-testarch-trace/steps-e/step-02-apply-edit.md
All edit-mode step files add post-completion "On Complete" hook that resolves and conditionally executes workflow.on_complete as final terminal instruction before exit.
Installation Tests
test/test-installation-components.js
Test Suite 4 updated to validate new workflow structure: replaces workflow.md validation with checks for SKILL.md containing resolve_customization.py invocation, adds assertions for customize.toml presence with [workflow] section containing required keys, verifies persistent facts configuration, and explicitly asserts workflow.md deletion across all nine workflows.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~35 minutes

Possibly related PRs

  • feat: knowledge base arrangement #63: Modifies the same workflow files (SKILL.md, workflow.md deletions, step markdown files) and test-installation-components.js with overlapping configuration refactoring.
  • fix: issues 32 through 37 #38: Updates workflow activation/config plumbing in SKILL.md and customize.toml conventions, with matching changes to test-installation-components.js validation logic.
  • feat: pactjs utils support #41: Alters test-installation-components.js test logic and updates workflow/step markdown under src/workflows/testarch with similar configuration structure validation.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately summarizes the main change: adopting a customize.toml pattern across testarch workflows. It is concise, clear, and specifically identifies the primary objective.
Description check ✅ Passed The description provides a comprehensive overview of the changes, including the rationale, specific actions taken (add customize.toml, merge workflow.md into SKILL.md, wire on_complete resolver), files modified, and test plan status. It directly relates to the changeset.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch add-workflow-customization

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 10

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
test/test-installation-components.js (1)

243-265: ⚠️ Potential issue | 🟡 Minor

Coverage gap: bmad-teach-me-testing skipped for customize.toml / workflow.md-removal assertions.

Per the PR summary, all 9 workflows (including bmad-teach-me-testing) received the new customize.toml + SKILL.md pattern and had workflow.md deleted. However, this test only spot-checks SKILL.md existence and BMM-reference absence for teach-me-testing, while the workflowDirs loop (which runs the full suite of [workflow]-key assertions, persistent_facts glob check, and the workflow.md removal negative assertion) covers only the 8 bmad-testarch-* dirs. A regression that reintroduced workflow.md or dropped keys from bmad-teach-me-testing/customize.toml would go undetected.

🧪 Suggested consolidation

Either add 'bmad-teach-me-testing': 'teach-me-testing' to workflowDirs (and drop the separate teach-me block), or duplicate the customize.toml / workflow.md-removal assertions into the teach-me-testing branch.

   const workflowDirs = {
+    'bmad-teach-me-testing': 'teach-me-testing',
     'bmad-testarch-framework': 'framework',
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/test-installation-components.js` around lines 243 - 265, The test
currently only validates SKILL.md for teach-me-testing (teachMeSkillPath) but
omits the customize.toml/workflow.md-removal assertions run in the workflowDirs
loop; update the test so bmad-teach-me-testing is included in that loop by
adding the key 'bmad-teach-me-testing': 'teach-me-testing' to the workflowDirs
object (and remove the separate teachMeSkillPath block) OR copy the same
customize.toml, persistent_facts, and workflow.md negative assertions into the
existing teach-me-testing try/catch branch so the same checks run for
teachMeSkillPath and the workflowDirs iteration.
♻️ Duplicate comments (6)
src/workflows/testarch/bmad-teach-me-testing/steps-c/step-05-completion.md (1)

341-345: ⚠️ Potential issue | 🟠 Major

Missing error handling for script failure.

Same issue as other terminal steps: lacks fallback if resolve_customization.py fails, inconsistent with the SKILL.md activation error handling pattern. The "follow it" instruction is also ambiguous.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-teach-me-testing/steps-c/step-05-completion.md`
around lines 341 - 345, Add robust error handling around the call to
resolve_customization.py and make the "follow it" instruction explicit: when
running python3 _bmad/scripts/resolve_customization.py --skill {skill-root}
--key workflow.on_complete, capture and handle failures (non-zero exit,
exceptions, or empty output) and log the error, then fall back to the same
default/abort behavior used in SKILL.md activation steps; if the resolver
returns a non-empty string, treat that string as the exact terminal instruction
to execute (document which executor/command runs it), otherwise proceed with the
defined fallback path.
src/workflows/testarch/bmad-teach-me-testing/customize.toml (1)

33-35: ⚠️ Potential issue | 🟡 Minor

Glob pattern could match multiple files with undefined behavior.

Same issue as bmad-testarch-nfr/customize.toml: the file:{project-root}/**/project-context.md glob could match multiple files across subdirectories with undefined merge behavior.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-teach-me-testing/customize.toml` around lines 33
- 35, The persistent_facts entry uses a recursive glob
"file:{project-root}/**/project-context.md" that can match multiple files and
cause undefined merges; replace it with a single, deterministic path or a more
specific glob so only one file is matched (e.g., point to the exact
project-context.md location or restrict the pattern to a single expected
directory) and update the persistent_facts entry (the string
"file:{project-root}/**/project-context.md" under persistent_facts) to that
deterministic path.
src/workflows/testarch/bmad-teach-me-testing/steps-v/step-v-01-validate.md (1)

265-269: ⚠️ Potential issue | 🟠 Major

Missing error handling for script failure.

This "On Complete" hook follows the existing pattern from bmad-testarch-trace (shown in context snippets) but lacks fallback error handling if the resolver script fails, inconsistent with the SKILL.md activation pattern that provides explicit fallback instructions. The "follow it as the final terminal instruction" phrase is also ambiguous in an AI instruction context.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-teach-me-testing/steps-v/step-v-01-validate.md`
around lines 265 - 269, The "On Complete" step calls the resolver script
(python3 _bmad/scripts/resolve_customization.py --skill {skill-root} --key
workflow.on_complete) but lacks error handling and a clear fallback and the
phrase "follow it as the final terminal instruction" is ambiguous; update the
step to run the resolver and if the script exits non‑zero or returns an empty
value, log or display a clear fallback instruction (matching SKILL.md's
activation pattern) and proceed with that fallback, and clarify that when a
non‑empty workflow.on_complete is returned you should execute those instructions
as the final step (or state the explicit fallback steps to take if execution
fails) so callers of resolve_customization.py and readers of the doc know both
success and error paths.
src/workflows/testarch/bmad-testarch-test-design/steps-v/step-01-validate.md (1)

69-73: ⚠️ Potential issue | 🟠 Major

Missing error handling for script failure.

Same issue as step-04-generate-report.md: this "On Complete" hook lacks a fallback if the resolver script fails, creating an inconsistency with the SKILL.md activation pattern that provides explicit error handling. Additionally, "follow it as the final terminal instruction" is ambiguous—clarify whether this means execute as a command or interpret as AI instructions.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-testarch-test-design/steps-v/step-01-validate.md`
around lines 69 - 73, The "On Complete" step invoking python3
_bmad/scripts/resolve_customization.py to resolve workflow.on_complete lacks
error handling and is ambiguous about execution semantics; update the step to
run the resolver guardedly (capture script non-zero exit and missing/empty
output) and provide a clear fallback/explicit error message and behavior (e.g.,
log error and continue or abort), and clarify whether the resolved
workflow.on_complete is to be executed as a shell command versus treated as AI
terminal instructions; reference the resolver script (resolve_customization.py)
and the resolved key (workflow.on_complete) and align behavior with the SKILL.md
activation pattern used in step-04-generate-report.md for consistent error
handling.
src/workflows/testarch/bmad-testarch-test-review/customize.toml (1)

33-35: ⚠️ Potential issue | 🟡 Minor

Glob pattern could match multiple files with undefined behavior.

Same issue as other customize.toml files: the file:{project-root}/**/project-context.md glob could match multiple files with undefined merge behavior.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-testarch-test-review/customize.toml` around lines
33 - 35, The persistent_facts entry uses a recursive glob
"file:{project-root}/**/project-context.md" which can match multiple files and
cause undefined merge behavior; update the persistent_facts configuration to
target a single, deterministic file or a stricter path (for example replace the
glob with an explicit path to the intended project-context.md or narrow the glob
to a single known folder) so only one file is matched; locate the
persistent_facts key in customize.toml and change the value referencing
"file:{project-root}/**/project-context.md" accordingly.
src/workflows/testarch/bmad-testarch-test-review/steps-v/step-01-validate.md (1)

69-73: ⚠️ Potential issue | 🟠 Major

Missing error handling for script failure.

Same issue as other terminal steps: lacks fallback if the resolver script fails, and "follow it" instruction is ambiguous.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-testarch-test-review/steps-v/step-01-validate.md`
around lines 69 - 73, The step's invocation of the resolver script (python3
_bmad/scripts/resolve_customization.py --skill {skill-root} --key
workflow.on_complete) lacks error handling and the phrase "follow it" is
ambiguous; update the step to (1) check the resolver process exit status and
capture stderr/stdout, (2) if the script fails or returns empty, fall back to a
clear default behavior (e.g., proceed with no-op or a documented default
terminal action) and log the error output, and (3) clarify that if
workflow.on_complete resolves to non-empty text the agent must treat that text
as the final terminal instruction to execute exactly once before exiting;
reference resolve_customization.py, the --key workflow.on_complete flag, and the
workflow.on_complete result when implementing these checks and the fallback.
🧹 Nitpick comments (3)
src/workflows/testarch/bmad-testarch-atdd/customize.toml (1)

8-10: Nit: merge-rules comment references keys not present in this file.

The header comment mentions "arrays-of-tables with code/id: replace matching items" but this [workflow] schema only contains scalars and flat arrays. Consider trimming to the two rules that apply here to avoid confusing readers of the workflow customization surface (the rule is still accurate, just not exercised). Same applies to the identical copies in bmad-testarch-test-design, bmad-testarch-framework, and bmad-testarch-trace.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-testarch-atdd/customize.toml` around lines 8 -
10, The header comment currently flags three merge rules including
"arrays-of-tables with `code`/`id`: replace matching items," which doesn't apply
to this `[workflow]` file (it only uses scalars and flat arrays); remove or trim
that third rule so the header only documents "scalars: override wins" and the
array-append behavior for persistent_facts/activation_steps_* to avoid
confusion, and apply the same edit to the matching headers in the other variants
(`bmad-testarch-test-design`, `bmad-testarch-framework`, `bmad-testarch-trace`)
to keep them consistent.
src/workflows/testarch/bmad-testarch-nfr/SKILL.md (1)

40-42: Minor: persistent_facts prefix semantics double-embed {project-root}.

Step 3 states file: entries are "paths or globs under {project-root}", but the default in customize.toml is file:{project-root}/**/project-context.md (i.e., the entry already contains {project-root} after the file: prefix). The wording could be read as "the path is relative to {project-root}", which would suggest file:**/project-context.md instead. Consider clarifying that {project-root} may appear inside the value and is just a resolvable placeholder, to prevent override authors from writing relative paths that don't resolve. Same wording is used in the atdd and test-review SKILL.md copies.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-testarch-nfr/SKILL.md` around lines 40 - 42, Step
3's wording is ambiguous about how file: entries in {workflow.persistent_facts}
are resolved; update the SKILL.md text to clearly state that entries prefixed
with file: are interpreted as paths or globs that may be absolute, may include
the {project-root} placeholder (as in the customize.toml default
file:{project-root}/**/project-context.md), or may be given as paths relative to
{project-root} — resolve any relative paths under {project-root} and expand
globs accordingly. Apply the same clarification to the ATDD and test-review
SKILL.md copies so all three documents use the same explicit phrasing about
placeholder resolution and relative vs absolute paths.
test/test-installation-components.js (1)

289-290: Verify negative assertions don't silently pass on empty content.

assert(!skillContent.includes('Read {skill-root}/workflow.md'), ...) and the [workflow.md](workflow.md) check will pass even if the SKILL.md were accidentally truncated or blank. The existing positive assertions (On Activation, resolver invocation, etc.) mitigate this, but consider guarding with a minimum length/size check once near the top of the pathExists(skillMdPath) block to make the negative assertions meaningful.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/test-installation-components.js` around lines 289 - 290, The negative
contains checks on skillContent can falsely pass if the file is empty; update
the pathExists(skillMdPath) block so that after reading skillContent you assert
it has a sensible minimum size (e.g., assert(skillContent &&
skillContent.trim().length > 0, `SKILL.md for ${dirName} should not be empty`)
or a specific length threshold) before running the negative assertions that
reference skillContent.includes(...); place this guard immediately after
skillContent is loaded so functions/variables like pathExists(skillMdPath) and
skillContent are validated first.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/workflows/testarch/bmad-teach-me-testing/SKILL.md`:
- Around line 12-13: The invocation mapping in SKILL.md declares pause/resume
capability but lacks explicit resume keywords; update the mapping that sets mode
to "create" (the line starting "If user invoked with 'create' or 'teach' or
'learn' or 'start'") to include "resume" and "continue" as triggers, and make
the same addition to the duplicated mapping block referenced at lines 104-117 so
the menu/routing matches the documented pause-and-resume behavior.

In `@src/workflows/testarch/bmad-testarch-automate/SKILL.md`:
- Around line 63-67: The doc currently presents a "tri-modal step-file
architecture" but also lists a separate Resume option, causing ambiguity; update
the SKILL.md wording so Resume is described explicitly as a continuation of the
Create flow (not a fourth mode) — for example change the Resume menu entry to "-
[R] Resume — Resume an interrupted Create workflow" and adjust the surrounding
description where "tri-modal step-file architecture" and the menu entries for
Create/Validate/Edit (the Create mode (steps-c/), Validate mode (steps-v/), Edit
mode (steps-e/), and the Resume entry) are documented (also apply the same
change around lines referenced in the second occurrence) so the doc consistently
frames Resume as a Create-flow continuation rather than a standalone mode.

In `@src/workflows/testarch/bmad-testarch-automate/steps-e/step-02-apply-edit.md`:
- Around line 62-66: The terminal hook invocation using "python3
.../resolve_customization.py --skill {skill-root} --key workflow.on_complete"
lacks a failure fallback; update the step documentation for
resolve_customization.py (the call with --key workflow.on_complete) to state
that if the script fails you must manually resolve workflow.on_complete by
checking, in order, {skill-root}/customize.toml,
{project-root}/_bmad/custom/{skill-name}.toml, and
{project-root}/_bmad/custom/{skill-name}.user.toml, and only if the resolved
workflow.on_complete is non-empty follow it as the final terminal instruction
before exiting.

In `@src/workflows/testarch/bmad-testarch-ci/SKILL.md`:
- Around line 63-67: The SKILL.md wording is ambiguous about "Resume" suggesting
both tri-modal and four-mode behavior; update the mode model entries (the
tri-modal list around the "Create mode (steps-c/), Validate mode (steps-v/),
Edit mode (steps-e/)" block and the related bullet list around the Resume entry)
to explicitly state that "[R] Resume — Resume an interrupted workflow
(Create-mode continuation)" so Resume is clearly a Create-flow path, and ensure
any other mentions in the nearby section (previously lines ~75-84) are changed
consistently to reflect Resume = Create continuation.

In `@src/workflows/testarch/bmad-testarch-ci/steps-e/step-02-apply-edit.md`:
- Around line 64-66: The current terminal step assumes running
resolve_customization.py --key workflow.on_complete always succeeds; update the
step to handle resolver failure by checking the script's exit status or output
and treating failures as non-fatal: if the resolver returns a successful result
and workflow.on_complete is non-empty, execute it as the final terminal
instruction; if the resolver fails (non-zero exit, missing output, or parse
error), skip executing workflow.on_complete, log or print a clear warning about
the resolver failure, and exit normally; reference the resolve_customization.py
invocation and the workflow.on_complete key when implementing this fallback.

In `@src/workflows/testarch/bmad-testarch-framework/SKILL.md`:
- Around line 63-67: Update the tri-modal wording in SKILL.md so Resume is
framed as part of Create mode: modify the Create mode bullet (the one mentioning
"Create mode (steps-c/)") to note that the resume path is included (e.g.,
reference step-01b-resume.md) and make the same change in the corresponding
block around lines 75-84 to ensure consistency between the tri-modal description
and the Resume routing; update any text that currently lists Resume as a
standalone choice so it is described as a Create-mode branch under steps-c/ and
step-01b-resume.md.

In `@src/workflows/testarch/bmad-testarch-nfr/customize.toml`:
- Around line 33-35: The persistent_facts glob persistent_facts =
["file:{project-root}/**/project-context.md"] is too broad and can match
multiple project-context.md files leading to undefined behavior; either narrow
it to a single explicit path (e.g., change the value to
file:{project-root}/project-context.md) or, if multiple files must be supported,
update the loading logic that consumes persistent_facts to implement and
document an explicit strategy (e.g., "first match wins" or "merge all") and add
tests to cover multiple-match behavior so the behavior of persistent_facts is
deterministic.

In `@src/workflows/testarch/bmad-testarch-test-design/SKILL.md`:
- Around line 63-67: The documentation currently describes both "tri-modal" and
a separate "Resume" mode, causing a contradiction; update the SKILL.md text
where Create/Validate/Edit modes are listed (the "Create mode (steps-c/)" bullet
and similar mentions around lines ~75-84) to explicitly fold Resume behavior
into Create mode by noting "Create mode (steps-c/): primary execution flow (new
+ resume)" and remove any standalone "Resume" mode references so the taxonomy
reads consistently as three modes: Create (including resume), Validate
(steps-v/), and Edit (steps-e/).

In
`@src/workflows/testarch/bmad-testarch-test-review/steps-c/step-04-generate-report.md`:
- Around line 113-117: The "On Complete" step currently calls
resolve_customization.py for workflow.on_complete without fallback or clear
semantics; update the step to catch script errors and provide a deterministic
fallback: if running python3
{project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --key
workflow.on_complete fails, skip the on_complete hook and proceed to exit, and
if the resolver returns a non-empty workflow.on_complete, explicitly state that
it should be executed as a shell command (not merely "followed") as the final
terminal instruction before exiting.

In `@src/workflows/testarch/bmad-testarch-trace/SKILL.md`:
- Around line 63-67: Update the tri-modal description in SKILL.md to explicitly
state whether "Resume" is a separate mode or an entrypoint of Create mode (e.g.,
change the Create mode bullet under the header so it reads that Create
(steps-c/) includes the resume entrypoint), and make the same clarification
within the later block referenced around lines 75-84; locate and edit the
bullets for "**Create mode (steps-c/)**", "**Validate mode (steps-v/)**", and
"**Edit mode (steps-e/)**" so they consistently indicate that "Resume" is an
entrypoint of Create rather than a fourth mode (or vice versa if you intend
Resume to be a distinct mode), ensuring the document no longer presents
contradictory mode semantics.

---

Outside diff comments:
In `@test/test-installation-components.js`:
- Around line 243-265: The test currently only validates SKILL.md for
teach-me-testing (teachMeSkillPath) but omits the
customize.toml/workflow.md-removal assertions run in the workflowDirs loop;
update the test so bmad-teach-me-testing is included in that loop by adding the
key 'bmad-teach-me-testing': 'teach-me-testing' to the workflowDirs object (and
remove the separate teachMeSkillPath block) OR copy the same customize.toml,
persistent_facts, and workflow.md negative assertions into the existing
teach-me-testing try/catch branch so the same checks run for teachMeSkillPath
and the workflowDirs iteration.

---

Duplicate comments:
In `@src/workflows/testarch/bmad-teach-me-testing/customize.toml`:
- Around line 33-35: The persistent_facts entry uses a recursive glob
"file:{project-root}/**/project-context.md" that can match multiple files and
cause undefined merges; replace it with a single, deterministic path or a more
specific glob so only one file is matched (e.g., point to the exact
project-context.md location or restrict the pattern to a single expected
directory) and update the persistent_facts entry (the string
"file:{project-root}/**/project-context.md" under persistent_facts) to that
deterministic path.

In `@src/workflows/testarch/bmad-teach-me-testing/steps-c/step-05-completion.md`:
- Around line 341-345: Add robust error handling around the call to
resolve_customization.py and make the "follow it" instruction explicit: when
running python3 _bmad/scripts/resolve_customization.py --skill {skill-root}
--key workflow.on_complete, capture and handle failures (non-zero exit,
exceptions, or empty output) and log the error, then fall back to the same
default/abort behavior used in SKILL.md activation steps; if the resolver
returns a non-empty string, treat that string as the exact terminal instruction
to execute (document which executor/command runs it), otherwise proceed with the
defined fallback path.

In `@src/workflows/testarch/bmad-teach-me-testing/steps-v/step-v-01-validate.md`:
- Around line 265-269: The "On Complete" step calls the resolver script (python3
_bmad/scripts/resolve_customization.py --skill {skill-root} --key
workflow.on_complete) but lacks error handling and a clear fallback and the
phrase "follow it as the final terminal instruction" is ambiguous; update the
step to run the resolver and if the script exits non‑zero or returns an empty
value, log or display a clear fallback instruction (matching SKILL.md's
activation pattern) and proceed with that fallback, and clarify that when a
non‑empty workflow.on_complete is returned you should execute those instructions
as the final step (or state the explicit fallback steps to take if execution
fails) so callers of resolve_customization.py and readers of the doc know both
success and error paths.

In
`@src/workflows/testarch/bmad-testarch-test-design/steps-v/step-01-validate.md`:
- Around line 69-73: The "On Complete" step invoking python3
_bmad/scripts/resolve_customization.py to resolve workflow.on_complete lacks
error handling and is ambiguous about execution semantics; update the step to
run the resolver guardedly (capture script non-zero exit and missing/empty
output) and provide a clear fallback/explicit error message and behavior (e.g.,
log error and continue or abort), and clarify whether the resolved
workflow.on_complete is to be executed as a shell command versus treated as AI
terminal instructions; reference the resolver script (resolve_customization.py)
and the resolved key (workflow.on_complete) and align behavior with the SKILL.md
activation pattern used in step-04-generate-report.md for consistent error
handling.

In `@src/workflows/testarch/bmad-testarch-test-review/customize.toml`:
- Around line 33-35: The persistent_facts entry uses a recursive glob
"file:{project-root}/**/project-context.md" which can match multiple files and
cause undefined merge behavior; update the persistent_facts configuration to
target a single, deterministic file or a stricter path (for example replace the
glob with an explicit path to the intended project-context.md or narrow the glob
to a single known folder) so only one file is matched; locate the
persistent_facts key in customize.toml and change the value referencing
"file:{project-root}/**/project-context.md" accordingly.

In
`@src/workflows/testarch/bmad-testarch-test-review/steps-v/step-01-validate.md`:
- Around line 69-73: The step's invocation of the resolver script (python3
_bmad/scripts/resolve_customization.py --skill {skill-root} --key
workflow.on_complete) lacks error handling and the phrase "follow it" is
ambiguous; update the step to (1) check the resolver process exit status and
capture stderr/stdout, (2) if the script fails or returns empty, fall back to a
clear default behavior (e.g., proceed with no-op or a documented default
terminal action) and log the error output, and (3) clarify that if
workflow.on_complete resolves to non-empty text the agent must treat that text
as the final terminal instruction to execute exactly once before exiting;
reference resolve_customization.py, the --key workflow.on_complete flag, and the
workflow.on_complete result when implementing these checks and the fallback.

---

Nitpick comments:
In `@src/workflows/testarch/bmad-testarch-atdd/customize.toml`:
- Around line 8-10: The header comment currently flags three merge rules
including "arrays-of-tables with `code`/`id`: replace matching items," which
doesn't apply to this `[workflow]` file (it only uses scalars and flat arrays);
remove or trim that third rule so the header only documents "scalars: override
wins" and the array-append behavior for persistent_facts/activation_steps_* to
avoid confusion, and apply the same edit to the matching headers in the other
variants (`bmad-testarch-test-design`, `bmad-testarch-framework`,
`bmad-testarch-trace`) to keep them consistent.

In `@src/workflows/testarch/bmad-testarch-nfr/SKILL.md`:
- Around line 40-42: Step 3's wording is ambiguous about how file: entries in
{workflow.persistent_facts} are resolved; update the SKILL.md text to clearly
state that entries prefixed with file: are interpreted as paths or globs that
may be absolute, may include the {project-root} placeholder (as in the
customize.toml default file:{project-root}/**/project-context.md), or may be
given as paths relative to {project-root} — resolve any relative paths under
{project-root} and expand globs accordingly. Apply the same clarification to the
ATDD and test-review SKILL.md copies so all three documents use the same
explicit phrasing about placeholder resolution and relative vs absolute paths.

In `@test/test-installation-components.js`:
- Around line 289-290: The negative contains checks on skillContent can falsely
pass if the file is empty; update the pathExists(skillMdPath) block so that
after reading skillContent you assert it has a sensible minimum size (e.g.,
assert(skillContent && skillContent.trim().length > 0, `SKILL.md for ${dirName}
should not be empty`) or a specific length threshold) before running the
negative assertions that reference skillContent.includes(...); place this guard
immediately after skillContent is loaded so functions/variables like
pathExists(skillMdPath) and skillContent are validated first.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: cbf99bdd-0bd1-4612-af31-ba794f0abdf3

📥 Commits

Reviewing files that changed from the base of the PR and between 8d5b612 and 808bdff.

📒 Files selected for processing (55)
  • src/workflows/testarch/bmad-teach-me-testing/SKILL.md
  • src/workflows/testarch/bmad-teach-me-testing/customize.toml
  • src/workflows/testarch/bmad-teach-me-testing/steps-c/step-05-completion.md
  • src/workflows/testarch/bmad-teach-me-testing/steps-e/step-e-02-apply-edits.md
  • src/workflows/testarch/bmad-teach-me-testing/steps-v/step-v-01-validate.md
  • src/workflows/testarch/bmad-teach-me-testing/workflow.md
  • src/workflows/testarch/bmad-testarch-atdd/SKILL.md
  • src/workflows/testarch/bmad-testarch-atdd/customize.toml
  • src/workflows/testarch/bmad-testarch-atdd/steps-c/step-05-validate-and-complete.md
  • src/workflows/testarch/bmad-testarch-atdd/steps-e/step-02-apply-edit.md
  • src/workflows/testarch/bmad-testarch-atdd/steps-v/step-01-validate.md
  • src/workflows/testarch/bmad-testarch-atdd/workflow.md
  • src/workflows/testarch/bmad-testarch-automate/SKILL.md
  • src/workflows/testarch/bmad-testarch-automate/customize.toml
  • src/workflows/testarch/bmad-testarch-automate/steps-c/step-04-validate-and-summarize.md
  • src/workflows/testarch/bmad-testarch-automate/steps-e/step-02-apply-edit.md
  • src/workflows/testarch/bmad-testarch-automate/steps-v/step-01-validate.md
  • src/workflows/testarch/bmad-testarch-automate/workflow.md
  • src/workflows/testarch/bmad-testarch-ci/SKILL.md
  • src/workflows/testarch/bmad-testarch-ci/customize.toml
  • src/workflows/testarch/bmad-testarch-ci/steps-c/step-04-validate-and-summary.md
  • src/workflows/testarch/bmad-testarch-ci/steps-e/step-02-apply-edit.md
  • src/workflows/testarch/bmad-testarch-ci/steps-v/step-01-validate.md
  • src/workflows/testarch/bmad-testarch-ci/workflow.md
  • src/workflows/testarch/bmad-testarch-framework/SKILL.md
  • src/workflows/testarch/bmad-testarch-framework/customize.toml
  • src/workflows/testarch/bmad-testarch-framework/steps-c/step-05-validate-and-summary.md
  • src/workflows/testarch/bmad-testarch-framework/steps-e/step-02-apply-edit.md
  • src/workflows/testarch/bmad-testarch-framework/steps-v/step-01-validate.md
  • src/workflows/testarch/bmad-testarch-framework/workflow.md
  • src/workflows/testarch/bmad-testarch-nfr/SKILL.md
  • src/workflows/testarch/bmad-testarch-nfr/customize.toml
  • src/workflows/testarch/bmad-testarch-nfr/steps-c/step-05-generate-report.md
  • src/workflows/testarch/bmad-testarch-nfr/steps-e/step-02-apply-edit.md
  • src/workflows/testarch/bmad-testarch-nfr/steps-v/step-01-validate.md
  • src/workflows/testarch/bmad-testarch-nfr/workflow.md
  • src/workflows/testarch/bmad-testarch-test-design/SKILL.md
  • src/workflows/testarch/bmad-testarch-test-design/customize.toml
  • src/workflows/testarch/bmad-testarch-test-design/steps-c/step-05-generate-output.md
  • src/workflows/testarch/bmad-testarch-test-design/steps-e/step-02-apply-edit.md
  • src/workflows/testarch/bmad-testarch-test-design/steps-v/step-01-validate.md
  • src/workflows/testarch/bmad-testarch-test-design/workflow.md
  • src/workflows/testarch/bmad-testarch-test-review/SKILL.md
  • src/workflows/testarch/bmad-testarch-test-review/customize.toml
  • src/workflows/testarch/bmad-testarch-test-review/steps-c/step-04-generate-report.md
  • src/workflows/testarch/bmad-testarch-test-review/steps-e/step-02-apply-edit.md
  • src/workflows/testarch/bmad-testarch-test-review/steps-v/step-01-validate.md
  • src/workflows/testarch/bmad-testarch-test-review/workflow.md
  • src/workflows/testarch/bmad-testarch-trace/SKILL.md
  • src/workflows/testarch/bmad-testarch-trace/customize.toml
  • src/workflows/testarch/bmad-testarch-trace/steps-c/step-05-gate-decision.md
  • src/workflows/testarch/bmad-testarch-trace/steps-e/step-02-apply-edit.md
  • src/workflows/testarch/bmad-testarch-trace/steps-v/step-01-validate.md
  • src/workflows/testarch/bmad-testarch-trace/workflow.md
  • test/test-installation-components.js
💤 Files with no reviewable changes (9)
  • src/workflows/testarch/bmad-testarch-nfr/workflow.md
  • src/workflows/testarch/bmad-testarch-automate/workflow.md
  • src/workflows/testarch/bmad-testarch-ci/workflow.md
  • src/workflows/testarch/bmad-teach-me-testing/workflow.md
  • src/workflows/testarch/bmad-testarch-test-review/workflow.md
  • src/workflows/testarch/bmad-testarch-framework/workflow.md
  • src/workflows/testarch/bmad-testarch-trace/workflow.md
  • src/workflows/testarch/bmad-testarch-test-design/workflow.md
  • src/workflows/testarch/bmad-testarch-atdd/workflow.md

Comment on lines +12 to +13
**Meta-Context:** This workflow uses continuable architecture with state persistence across sessions. Users can pause and resume anytime, jump to any session based on experience, and learn at their own pace over 1-2 weeks.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Resume capability is declared but not explicitly routable

You state users can “pause and resume anytime,” but the invocation mapping/menu doesn’t include a resume keyword/choice. Add an explicit resume trigger to keep behavior aligned with the contract.

Suggested change
**Meta-Context:** This workflow uses continuable architecture with state persistence across sessions. Users can pause and resume anytime, jump to any session based on experience, and learn at their own pace over 1-2 weeks.
- If user invoked with "create" or "teach" or "learn" or "start" or "resume" or "continue" → Set mode to **create**

As per coding guidelines, "Focus on inconsistencies, contradictions, edge cases and serious issues."

Also applies to: 104-117

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-teach-me-testing/SKILL.md` around lines 12 - 13,
The invocation mapping in SKILL.md declares pause/resume capability but lacks
explicit resume keywords; update the mapping that sets mode to "create" (the
line starting "If user invoked with 'create' or 'teach' or 'learn' or 'start'")
to include "resume" and "continue" as triggers, and make the same addition to
the duplicated mapping block referenced at lines 104-117 so the menu/routing
matches the documented pause-and-resume behavior.

Comment on lines +63 to +67
This workflow uses **tri-modal step-file architecture**:

- **Create mode (steps-c/)**: primary execution flow
- **Validate mode (steps-v/)**: validation against checklist
- **Edit mode (steps-e/)**: revise existing outputs
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Tri-modal claim conflicts with the explicit Resume menu option

To avoid ambiguous mode definitions, describe Resume as a Create-flow continuation path rather than a fourth mode.

Suggested change
This workflow uses **tri-modal step-file architecture**:
- **Create mode (steps-c/)**: primary execution flow
- **Validate mode (steps-v/)**: validation against checklist
- **Edit mode (steps-e/)**: revise existing outputs
- **[R] Resume** — Resume an interrupted **Create** workflow

As per coding guidelines, "Focus on inconsistencies, contradictions, edge cases and serious issues."

Also applies to: 75-84

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-testarch-automate/SKILL.md` around lines 63 - 67,
The doc currently presents a "tri-modal step-file architecture" but also lists a
separate Resume option, causing ambiguity; update the SKILL.md wording so Resume
is described explicitly as a continuation of the Create flow (not a fourth mode)
— for example change the Resume menu entry to "- [R] Resume — Resume an
interrupted Create workflow" and adjust the surrounding description where
"tri-modal step-file architecture" and the menu entries for Create/Validate/Edit
(the Create mode (steps-c/), Validate mode (steps-v/), Edit mode (steps-e/), and
the Resume entry) are documented (also apply the same change around lines
referenced in the second occurrence) so the doc consistently frames Resume as a
Create-flow continuation rather than a standalone mode.

Comment on lines +62 to +66
## On Complete

Run: `python3 {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --key workflow.on_complete`

If the resolved `workflow.on_complete` is non-empty, follow it as the final terminal instruction before exiting.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Check whether resolver script exists:"
fd -i 'resolve_customization.py'

echo
echo "Find all new workflow.on_complete hook usages in testarch steps:"
rg -n --type=md -C2 'resolve_customization\.py --skill \{skill-root\} --key workflow\.on_complete|## On Complete' src/workflows/testarch

echo
echo "Find fallback wording usage in workflow docs:"
rg -n --type=md -C2 'If the script fails' src/workflows/testarch src/agents

Repository: bmad-code-org/bmad-method-test-architecture-enterprise

Length of output: 29523


Add explicit failure fallback for workflow.on_complete resolver to match main workflow pattern.

The resolve_customization.py call for the terminal hook lacks a failure branch, unlike the main workflow resolver documented in SKILL.md files. If resolution fails, there is no deterministic path to retrieve workflow.on_complete. Add an explicit fallback:

Suggested replacement for lines 62-66
Suggested change
## On Complete
Run: `python3 {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --key workflow.on_complete`
If the resolved `workflow.on_complete` is non-empty, follow it as the final terminal instruction before exiting.
## On Complete
Run: `python3 {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --key workflow.on_complete`
**If the script fails**, resolve `workflow.on_complete` manually from:
1. `{skill-root}/customize.toml`
2. `{project-root}/_bmad/custom/{skill-name}.toml`
3. `{project-root}/_bmad/custom/{skill-name}.user.toml`
If the resolved `workflow.on_complete` is non-empty, follow it as the final terminal instruction before exiting.

This applies to 40+ step files across all testarch workflows (test-design, test-review, trace, ci, atdd, framework, nfr, automate, and teach-me-testing).

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-testarch-automate/steps-e/step-02-apply-edit.md`
around lines 62 - 66, The terminal hook invocation using "python3
.../resolve_customization.py --skill {skill-root} --key workflow.on_complete"
lacks a failure fallback; update the step documentation for
resolve_customization.py (the call with --key workflow.on_complete) to state
that if the script fails you must manually resolve workflow.on_complete by
checking, in order, {skill-root}/customize.toml,
{project-root}/_bmad/custom/{skill-name}.toml, and
{project-root}/_bmad/custom/{skill-name}.user.toml, and only if the resolved
workflow.on_complete is non-empty follow it as the final terminal instruction
before exiting.

Comment on lines +63 to +67
This workflow uses **tri-modal step-file architecture**:

- **Create mode (steps-c/)**: primary execution flow
- **Validate mode (steps-v/)**: validation against checklist
- **Edit mode (steps-e/)**: revise existing outputs
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Please remove ambiguity around Resume in the mode model

Current wording implies both tri-modal and four-mode behavior. Clarify that Resume is a Create flow path.

Suggested change
This workflow uses **tri-modal step-file architecture**:
- **Create mode (steps-c/)**: primary execution flow
- **Validate mode (steps-v/)**: validation against checklist
- **Edit mode (steps-e/)**: revise existing outputs
- **[R] Resume** — Resume an interrupted workflow (Create-mode continuation)

As per coding guidelines, "Focus on inconsistencies, contradictions, edge cases and serious issues."

Also applies to: 75-84

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-testarch-ci/SKILL.md` around lines 63 - 67, The
SKILL.md wording is ambiguous about "Resume" suggesting both tri-modal and
four-mode behavior; update the mode model entries (the tri-modal list around the
"Create mode (steps-c/), Validate mode (steps-v/), Edit mode (steps-e/)" block
and the related bullet list around the Resume entry) to explicitly state that
"[R] Resume — Resume an interrupted workflow (Create-mode continuation)" so
Resume is clearly a Create-flow path, and ensure any other mentions in the
nearby section (previously lines ~75-84) are changed consistently to reflect
Resume = Create continuation.

Comment on lines +64 to +66
Run: `python3 {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --key workflow.on_complete`

If the resolved `workflow.on_complete` is non-empty, follow it as the final terminal instruction before exiting.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add explicit resolver-failure fallback for terminal completion.

At Line 64, the terminal path assumes resolve_customization.py succeeds. If it fails (missing script, path issue, parse error), completion behavior is undefined at the last step, which can cause brittle exits. Please define non-fatal fallback behavior (or explicit manual resolution path) in this section.

Suggested patch
Suggested change
Run: `python3 {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --key workflow.on_complete`
If the resolved `workflow.on_complete` is non-empty, follow it as the final terminal instruction before exiting.
Run: `python3 {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --key workflow.on_complete`
If resolution succeeds and `workflow.on_complete` is non-empty, follow it as the final terminal instruction before exiting.
If resolution fails, skip `workflow.on_complete` and exit normally.

As per coding guidelines, "Focus on inconsistencies, contradictions, edge cases and serious issues."

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-testarch-ci/steps-e/step-02-apply-edit.md` around
lines 64 - 66, The current terminal step assumes running
resolve_customization.py --key workflow.on_complete always succeeds; update the
step to handle resolver failure by checking the script's exit status or output
and treating failures as non-fatal: if the resolver returns a successful result
and workflow.on_complete is non-empty, execute it as the final terminal
instruction; if the resolver fails (non-zero exit, missing output, or parse
error), skip executing workflow.on_complete, log or print a clear warning about
the resolver failure, and exit normally; reference the resolve_customization.py
invocation and the workflow.on_complete key when implementing this fallback.

Comment on lines +63 to +67
This workflow uses **tri-modal step-file architecture**:

- **Create mode (steps-c/)**: primary execution flow
- **Validate mode (steps-v/)**: validation against checklist
- **Edit mode (steps-e/)**: revise existing outputs
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Please align “tri-modal” wording with Resume routing

Resume is currently listed as a standalone choice while the architecture is declared tri-modal. This should be phrased as a Create-mode branch for consistency.

Suggested change
This workflow uses **tri-modal step-file architecture**:
- **Create mode (steps-c/)**: primary execution flow
- **Validate mode (steps-v/)**: validation against checklist
- **Edit mode (steps-e/)**: revise existing outputs
- **Create mode (steps-c/)**: primary execution flow (includes resume path via `step-01b-resume.md`)

As per coding guidelines, "Focus on inconsistencies, contradictions, edge cases and serious issues."

Also applies to: 75-84

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-testarch-framework/SKILL.md` around lines 63 -
67, Update the tri-modal wording in SKILL.md so Resume is framed as part of
Create mode: modify the Create mode bullet (the one mentioning "Create mode
(steps-c/)") to note that the resume path is included (e.g., reference
step-01b-resume.md) and make the same change in the corresponding block around
lines 75-84 to ensure consistency between the tri-modal description and the
Resume routing; update any text that currently lists Resume as a standalone
choice so it is described as a Create-mode branch under steps-c/ and
step-01b-resume.md.

Comment on lines +33 to +35
persistent_facts = [
"file:{project-root}/**/project-context.md",
]
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Glob pattern could match multiple files with undefined behavior.

The persistent_facts glob file:{project-root}/**/project-context.md will match any project-context.md file in any subdirectory under the project root. If multiple files exist (e.g., docs/project-context.md and tests/project-context.md), the loading and merge behavior is not specified, potentially leading to duplicate or conflicting context being loaded.

Consider either:

  1. Using a specific path without wildcards: file:{project-root}/project-context.md
  2. Documenting the expected behavior when multiple matches occur
  3. Implementing a "first match" or "merge all" strategy explicitly
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-testarch-nfr/customize.toml` around lines 33 -
35, The persistent_facts glob persistent_facts =
["file:{project-root}/**/project-context.md"] is too broad and can match
multiple project-context.md files leading to undefined behavior; either narrow
it to a single explicit path (e.g., change the value to
file:{project-root}/project-context.md) or, if multiple files must be supported,
update the loading logic that consumes persistent_facts to implement and
document an explicit strategy (e.g., "first match wins" or "merge all") and add
tests to cover multiple-match behavior so the behavior of persistent_facts is
deterministic.

Comment on lines +63 to +67
This workflow uses **tri-modal step-file architecture**:

- **Create mode (steps-c/)**: primary execution flow
- **Validate mode (steps-v/)**: validation against checklist
- **Edit mode (steps-e/)**: revise existing outputs
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Mode taxonomy is internally inconsistent (tri-modal + separate Resume)

This reads as three modes and four modes at the same time. Clarify Resume as part of Create-mode routing to keep behavior/documentation deterministic.

Suggested change
This workflow uses **tri-modal step-file architecture**:
- **Create mode (steps-c/)**: primary execution flow
- **Validate mode (steps-v/)**: validation against checklist
- **Edit mode (steps-e/)**: revise existing outputs
This workflow uses **tri-modal step-file architecture**:
- **Create mode (steps-c/)**: primary execution flow (new + resume)
- **Validate mode (steps-v/)**: validation against checklist
- **Edit mode (steps-e/)**: revise existing outputs

As per coding guidelines, "Focus on inconsistencies, contradictions, edge cases and serious issues."

Also applies to: 75-84

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-testarch-test-design/SKILL.md` around lines 63 -
67, The documentation currently describes both "tri-modal" and a separate
"Resume" mode, causing a contradiction; update the SKILL.md text where
Create/Validate/Edit modes are listed (the "Create mode (steps-c/)" bullet and
similar mentions around lines ~75-84) to explicitly fold Resume behavior into
Create mode by noting "Create mode (steps-c/): primary execution flow (new +
resume)" and remove any standalone "Resume" mode references so the taxonomy
reads consistently as three modes: Create (including resume), Validate
(steps-v/), and Edit (steps-e/).

Comment on lines +113 to +117
## On Complete

Run: `python3 {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --key workflow.on_complete`

If the resolved `workflow.on_complete` is non-empty, follow it as the final terminal instruction before exiting.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Missing error handling and ambiguous instruction.

This "On Complete" hook lacks a fallback if resolve_customization.py fails, while the SKILL.md activation sequence (shown in context snippets) provides explicit fallback instructions when the resolver script fails. This inconsistency could leave the workflow stuck if the script errors at completion time.

Additionally, "follow it as the final terminal instruction" is ambiguous in an AI instruction context—it's unclear whether this means execute as a shell command, display to the user, or interpret as AI directives.

Recommended fix:

Align with SKILL.md error handling pattern and clarify instruction
Suggested change
## On Complete
Run: `python3 {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --key workflow.on_complete`
If the resolved `workflow.on_complete` is non-empty, follow it as the final terminal instruction before exiting.
## On Complete
Run: `python3 {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --key workflow.on_complete`
**If the script fails**, skip the on_complete hook and proceed to exit.
If the resolved `workflow.on_complete` is non-empty, execute it as the final instruction before exiting.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In
`@src/workflows/testarch/bmad-testarch-test-review/steps-c/step-04-generate-report.md`
around lines 113 - 117, The "On Complete" step currently calls
resolve_customization.py for workflow.on_complete without fallback or clear
semantics; update the step to catch script errors and provide a deterministic
fallback: if running python3
{project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --key
workflow.on_complete fails, skip the on_complete hook and proceed to exit, and
if the resolver returns a non-empty workflow.on_complete, explicitly state that
it should be executed as a shell command (not merely "followed") as the final
terminal instruction before exiting.

Comment on lines +63 to +67
This workflow uses **tri-modal step-file architecture**:

- **Create mode (steps-c/)**: primary execution flow
- **Validate mode (steps-v/)**: validation against checklist
- **Edit mode (steps-e/)**: revise existing outputs
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Clarify whether Resume is a mode or a create-flow entrypoint

The document calls the architecture tri-modal (Create/Validate/Edit), but then presents Resume as a separate mode option. Please make this explicit to avoid contradictory mode semantics.

Suggested change
This workflow uses **tri-modal step-file architecture**:
- **Create mode (steps-c/)**: primary execution flow
- **Validate mode (steps-v/)**: validation against checklist
- **Edit mode (steps-e/)**: revise existing outputs
This workflow uses **tri-modal step-file architecture**:
- **Create mode (steps-c/)**: primary execution flow (includes resume entrypoint)
- **Validate mode (steps-v/)**: validation against checklist
- **Edit mode (steps-e/)**: revise existing outputs

As per coding guidelines, "Focus on inconsistencies, contradictions, edge cases and serious issues."

Also applies to: 75-84

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/workflows/testarch/bmad-testarch-trace/SKILL.md` around lines 63 - 67,
Update the tri-modal description in SKILL.md to explicitly state whether
"Resume" is a separate mode or an entrypoint of Create mode (e.g., change the
Create mode bullet under the header so it reads that Create (steps-c/) includes
the resume entrypoint), and make the same clarification within the later block
referenced around lines 75-84; locate and edit the bullets for "**Create mode
(steps-c/)**", "**Validate mode (steps-v/)**", and "**Edit mode (steps-e/)**" so
they consistently indicate that "Resume" is an entrypoint of Create rather than
a fourth mode (or vice versa if you intend Resume to be a distinct mode),
ensuring the document no longer presents contradictory mode semantics.

@muratkeremozcan muratkeremozcan merged commit e25a274 into main Apr 23, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants