fix: neft planning in test design#107
Merged
Merged
Conversation
🤖 Augment PR SummarySummary: This PR updates TEA guidance and workflows so Changes:
Technical Notes: The workflow boundary is clarified: 🤖 Was this summary useful? React with 👍 or 👎 |
![augmentcode[bot]](https://avatars.githubusercontent.com/in/1027498?s=60&v=4){kind=small}
| [](https://docs.astral.sh/uv/) | ||
|
|
||
| TEA (Test Engineering Architect) is a standalone BMAD module that delivers risk-based test strategy, test automation guidance, and release gate decisions. It provides a single expert agent (Murat, Master Test Architect and Quality Advisor) and nine workflows spanning Teach Me Testing (TEA Academy), framework setup, test design, ATDD, automation, traceability, NFR assessment, CI guidance, and test review. | ||
| TEA (Test Engineering Architect) is a standalone BMAD module that delivers risk-based test strategy, test automation guidance, and release gate decisions. It provides a single expert agent (Murat, Master Test Architect and Quality Advisor) and nine workflows spanning Teach Me Testing (TEA Academy), test design, framework setup, CI guidance, ATDD, automation, test review, NFR Evidence Audit, and traceability. |
There was a problem hiding this comment.
This PR makes user-facing docs/workflow behavior changes, but it doesn’t look like CHANGELOG.md was updated; the repo guidelines require logging these under ## [Unreleased] (Rule: AGENTS.md).
Severity: medium
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
| name: bmad-testarch-nfr | ||
| # prettier-ignore | ||
| description: 'Assess NFRs like performance security and reliability. Use when the user says "lets assess NFRs" or "I want to evaluate non-functional requirements"' | ||
| description: 'Audit NFR evidence for performance, security, reliability, and maintainability. Use when implementation evidence exists and the user says "lets assess NFRs", "audit NFR evidence", or "evaluate non-functional requirements"' |
There was a problem hiding this comment.
This description lists “maintainability” as a core audited NFR category, but the workflow steps appear to dispatch/aggregate security, performance, reliability, and scalability (and Step 2’s category list is ADR-checklist-based), which could mislead users about what coverage they’re getting.
Severity: medium
Other Locations
src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04-evaluate-and-score.md:165src/workflows/testarch/bmad-testarch-nfr/steps-c/step-04e-aggregate-nfr.md:31src/workflows/testarch/bmad-testarch-nfr/steps-c/step-02-define-thresholds.md:56src/workflows/testarch/bmad-testarch-nfr/nfr-report-template.md:176
🤖 Was this useful? React with 👍 or 👎, or 🚀 if it prevented an incident/outage.
📝 WalkthroughWalkthroughThis PR comprehensively renames "NFR Assessment" to "NFR Evidence Audit" across documentation and workflow files, and restructures the NFR and test-design workflows to clarify separation of concerns: test-design plans NFR thresholds and evidence needs before implementation, while nfr-assess audits implemented evidence afterward. ChangesNFR Evidence Audit Redesign and Test-Design Integration
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 10
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
docs/how-to/workflows/run-trace.md (1)
447-452:⚠️ Potential issue | 🟠 Major | ⚡ Quick winUnify the Phase 2 NFR artifact name in this section.
Line 447 uses “NFR Evidence Audit”, but Line 450 still instructs users to provide
nfr-assessment.md. This creates a conflicting contract for the same evidence input and can cause wrong artifact handoff during gate decisions.As per coding guidelines, "Focus on inconsistencies, contradictions, edge cases and serious issues."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/how-to/workflows/run-trace.md` around lines 447 - 452, The section uses two different names for the same Phase 2 artifact ("NFR Evidence Audit" vs `nfr-assessment.md`); pick one canonical name and make the doc consistent—e.g., keep the heading "NFR Evidence Audit" and update the code block content from `nfr-assessment.md` to `nfr-evidence-audit.md` (or alternatively rename the heading to "NFR Assessment" and keep `nfr-assessment.md`); update every occurrence of the artifact name and filename in this file so the displayed title and the referenced artifact file match (search for "NFR Evidence Audit", "NFR Assessment", and "`nfr-assessment.md`" and replace accordingly).src/workflows/testarch/bmad-testarch-test-design/steps-c/step-02-load-context.md (1)
166-173:⚠️ Potential issue | 🟠 Major | ⚡ Quick winAdd
probability-impact.mdto System-Level required fragments.Line 168–173 currently omits the scoring reference used by the risk step, which can produce inconsistent scoring behavior between modes.
### System-Level Mode (Required) - `adr-quality-readiness-checklist.md` - `nfr-criteria.md` - `probability-impact.md` - `test-levels-framework.md` - `risk-governance.md` - `test-quality.md`🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/workflows/testarch/bmad-testarch-test-design/steps-c/step-02-load-context.md` around lines 166 - 173, Add the missing fragment `probability-impact.md` to the "System-Level Mode (Required)" fragment list so the risk scoring reference is included; update the list under the "System-Level Mode (Required)" heading to insert `probability-impact.md` among the required fragments (ensure it appears alongside `adr-quality-readiness-checklist.md`, `nfr-criteria.md`, `test-levels-framework.md`, `risk-governance.md`, and `test-quality.md`) so the risk step uses consistent scoring across modes.
🧹 Nitpick comments (3)
src/workflows/testarch/bmad-testarch-nfr/checklist.md (1)
230-230: ⚡ Quick winAlign artifact filename with Evidence Audit terminology.
Line 230 still uses
nfr-assessment.mdwhile the deliverable is explicitly renamed to “NFR Evidence Audit Report.” This creates a cross-doc naming mismatch for readers and automation that relies on artifact labels.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/workflows/testarch/bmad-testarch-nfr/checklist.md` at line 230, Update the checklist entry that references the artifact filename by replacing the string `nfr-assessment.md` with the canonical deliverable name used elsewhere, “NFR Evidence Audit Report” (or its corresponding artifact filename used in automation), so the checkbox line in src/workflows/testarch/bmad-testarch-nfr/checklist.md matches the renamed deliverable and avoids cross-doc naming mismatches.src/workflows/testarch/bmad-testarch-nfr/nfr-report-template.md (1)
9-9: ⚡ Quick winAvoid mixed machine-readable naming between “assessment” and “evidence audit.”
The template text is renamed, but canonical identifiers in the same template still use
...assess/nfr_assessment. Please either fully align identifiers or explicitly document these as backward-compatibility aliases to prevent downstream contract confusion.Also applies to: 17-17, 449-450
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/workflows/testarch/bmad-testarch-nfr/nfr-report-template.md` at line 9, The template header was renamed to "evidence audit" but canonical identifiers still use "assess"/"nfr_assessment", causing mixed machine-readable naming; update all internal identifiers to be consistent (either change every instance of nfr_assessment, assess, etc. to use evidence_audit/evidence_audit_id or conversely rename the header back to assessment) or add explicit backward-compatibility aliases and a short comment block documenting that nfr_assessment and assess remain acceptable legacy keys; search and update the occurrences referenced (the header and the other spots around the earlier 17 and later 449-450 locations) and ensure identifiers in variables, metadata keys, and any templates (e.g., nfr_assessment, assess) are uniformly renamed or aliased.src/workflows/testarch/bmad-testarch-test-design/steps-c/step-05-generate-output.md (1)
132-136: ⚡ Quick winScope NFR output requirements to in-scope NFRs only.
This list is unconditional, but checklist criteria are conditional. Make the requirement explicit to avoid generating fabricated/empty NFR sections.
- Risk assessment matrix - NFR planning summary (thresholds, missing thresholds, planned evidence, and NFR-derived risks) when NFRs are in scope - Coverage matrix and priorities - NFR coverage and planned evidence mapping when NFRs are in scope - Execution strategy🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/workflows/testarch/bmad-testarch-test-design/steps-c/step-05-generate-output.md` around lines 132 - 136, Update the unconditional output checklist so NFR-related items are only required when NFRs are in scope: modify the lines containing "NFR planning summary (thresholds, missing thresholds, planned evidence, and NFR-derived risks)" and "NFR coverage and planned evidence mapping" to append a conditional note such as "when NFRs are in scope" (or similar phrasing) so the generated document does not produce fabricated or empty NFR sections.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/explanation/tea-overview.md`:
- Around line 219-226: The Release row in the phase table currently implies
`nfr-assess` always runs at release, which contradicts earlier guidance that
`nfr-assess` is conditional; update the Release table row to mark `nfr-assess`
as conditional (e.g., "nfr-assess (when NFR evidence exists and matters)") and
adjust the Purpose cell to reflect a conditional NFR evidence audit and go/no-go
decision so the `nfr-assess` entry aligns with the earlier conditional guidance
for `nfr-assess`.
In `@docs/how-to/brownfield/use-tea-for-enterprise.md`:
- Line 270: Update the artifact filename referenced in the evidence list:
replace the entry named "nfr-assessment.md" with "nfr-evidence-audit.md" so it
matches the “NFR Evidence Audit” terminology; locate the string
"nfr-assessment.md" in the document (file contains the evidence list) and change
it to "nfr-evidence-audit.md", ensuring any surrounding descriptive text still
reads correctly.
In `@docs/how-to/workflows/run-nfr-assess.md`:
- Line 418: Replace the starred command occurrences so the guide uses the plain
command name; find and change all instances of "*nfr-assess" to "nfr-assess"
(the three occurrences mentioned plus any other stray leading-asterisk usages)
so the document consistently shows the runnable command string and avoids
copy/paste errors.
In `@src/workflows/testarch/bmad-teach-me-testing/steps-c/step-04-session-01.md`:
- Around line 98-99: Update the mismatched fragment count: change the bullet
text that currently reads "**35 Knowledge Fragments:**" to the canonical "**42
Knowledge Fragments:**" so it matches the later script that references 42
fragments; locate and edit the line containing the literal "- **35 Knowledge
Fragments:**" in step-04-session-01.md to replace 35 with 42.
In
`@src/workflows/testarch/bmad-testarch-nfr/steps-c/step-02-define-thresholds.md`:
- Around line 43-46: Specify and implement a deterministic precedence rule for
selecting/merging NFR values when multiple test-design files are present: in the
logic that loads test-design outputs (lookups for test-design-architecture.md,
test-design-qa.md, or any test-design output under {test_artifacts}/ as
described in step-02-define-thresholds.md), enforce a fixed priority list (e.g.,
test-design-architecture.md > test-design-qa.md > other test-design files) and
always take a value from the highest-priority file that defines it; for
conflicting thresholds across files either choose the value from the
higher-priority file or apply a deterministic merge policy (e.g., pick the
strictest/most conservative numeric threshold), and fall back to raw documents
only for remaining UNKNOWNs as existing logic requires.
In `@src/workflows/testarch/bmad-testarch-nfr/workflow.yaml`:
- Line 4: The workflow description string currently lists "maintainability" but
the implemented orchestrated subagents and aggregation use "security",
"performance", "reliability", and "scalability"; update the description in the
workflow.yaml (the description field) to replace "maintainability" with
"scalability" so the description matches the NFR domains actually implemented by
this workflow.
In
`@src/workflows/testarch/bmad-testarch-test-design/test-design-architecture-template.md`:
- Around line 128-134: Update the NFR table block (the pipe-delimited table
starting with the header "NFR Category | Threshold / Requirement | Current
Design Support | Gap / Decision Needed | Planned Evidence") to include
additional rows for "Scalability", "Compliance", and "Custom" (in addition to
the existing Security, Performance, Reliability, Maintainability rows), and fill
the Threshold / Planned Evidence cells with the suggested placeholders (e.g.,
{Scale/concurrency/capacity requirement}, {Capacity test, autoscale evidence},
{Regulatory/privacy/audit requirement}, {Compliance report, audit evidence},
{Project-specific NFR requirement}, {Project-specific evidence artifact}) so
teams are prompted to capture scale, regulatory and project‑specific NFRs.
In `@src/workflows/testarch/bmad-testarch-test-design/test-design-qa-template.md`:
- Around line 130-135: Update the NFR table in test-design-qa-template.md to
include the missing categories (Scalability, Compliance, and Custom) and reorder
rows so Scalability appears before Maintainability as in the suggested diff;
ensure each new row supplies the same columns (Requirement / Threshold, Planned
Validation, Tool / Level, Evidence Artifact, Priority) with the example
placeholders from the suggestion (e.g., Scalability: {Capacity/concurrency/soak
validation}, Tool {k6/APM}, Evidence {Capacity report}; Compliance:
{Policy/privacy/audit validation}, Tool {CI/audit checks}, Evidence
{Audit/compliance evidence}; Custom: {Project-specific validation}, Tool
{Tool/Level}, Evidence {Artifact}) and keep existing Security, Performance,
Reliability, and Maintainability rows unchanged other than reordering.
In `@src/workflows/testarch/bmad-testarch-test-design/test-design-template.md`:
- Around line 83-89: The NFR table currently only lists four categories
(Security, Performance, Reliability, Maintainability) so expand the table rows
in test-design-template.md to include additional supported NFR categories: add
rows for Scalability (with Planned Validation like capacity/concurrency/soak
validation and Evidence Needed like capacity/load evidence), Compliance (with
Policy/privacy/audit validation and compliance/audit evidence), and a Custom row
(with project-specific validation and artifact); update the corresponding
"Planned Validation" and "Evidence Needed" cells for these new rows to match the
suggested entries so the epic NFR planning covers all supported categories.
In `@src/workflows/testarch/bmad-testarch-trace/trace-template.md`:
- Line 681: The template label “NFR Evidence Audit” is inconsistent with
placeholders/keys that use the nfr_assessment naming; pick one canonical term
and apply it consistently—either (A) rename the display label to “NFR
Assessment” or (B) update all placeholders/keys (e.g., {NFR_FILE_PATH}, any
variables or keys named nfr_assessment) to use nfr_evidence_audit (or
nfr_evidence) throughout the trace-template.md and any referenced artifacts;
ensure matching names in functions/methods that read/write those keys (search
for nfr_assessment and NFR_FILE_PATH) so producers/consumers use the same
contract.
---
Outside diff comments:
In `@docs/how-to/workflows/run-trace.md`:
- Around line 447-452: The section uses two different names for the same Phase 2
artifact ("NFR Evidence Audit" vs `nfr-assessment.md`); pick one canonical name
and make the doc consistent—e.g., keep the heading "NFR Evidence Audit" and
update the code block content from `nfr-assessment.md` to
`nfr-evidence-audit.md` (or alternatively rename the heading to "NFR Assessment"
and keep `nfr-assessment.md`); update every occurrence of the artifact name and
filename in this file so the displayed title and the referenced artifact file
match (search for "NFR Evidence Audit", "NFR Assessment", and
"`nfr-assessment.md`" and replace accordingly).
In
`@src/workflows/testarch/bmad-testarch-test-design/steps-c/step-02-load-context.md`:
- Around line 166-173: Add the missing fragment `probability-impact.md` to the
"System-Level Mode (Required)" fragment list so the risk scoring reference is
included; update the list under the "System-Level Mode (Required)" heading to
insert `probability-impact.md` among the required fragments (ensure it appears
alongside `adr-quality-readiness-checklist.md`, `nfr-criteria.md`,
`test-levels-framework.md`, `risk-governance.md`, and `test-quality.md`) so the
risk step uses consistent scoring across modes.
---
Nitpick comments:
In `@src/workflows/testarch/bmad-testarch-nfr/checklist.md`:
- Line 230: Update the checklist entry that references the artifact filename by
replacing the string `nfr-assessment.md` with the canonical deliverable name
used elsewhere, “NFR Evidence Audit Report” (or its corresponding artifact
filename used in automation), so the checkbox line in
src/workflows/testarch/bmad-testarch-nfr/checklist.md matches the renamed
deliverable and avoids cross-doc naming mismatches.
In `@src/workflows/testarch/bmad-testarch-nfr/nfr-report-template.md`:
- Line 9: The template header was renamed to "evidence audit" but canonical
identifiers still use "assess"/"nfr_assessment", causing mixed machine-readable
naming; update all internal identifiers to be consistent (either change every
instance of nfr_assessment, assess, etc. to use evidence_audit/evidence_audit_id
or conversely rename the header back to assessment) or add explicit
backward-compatibility aliases and a short comment block documenting that
nfr_assessment and assess remain acceptable legacy keys; search and update the
occurrences referenced (the header and the other spots around the earlier 17 and
later 449-450 locations) and ensure identifiers in variables, metadata keys, and
any templates (e.g., nfr_assessment, assess) are uniformly renamed or aliased.
In
`@src/workflows/testarch/bmad-testarch-test-design/steps-c/step-05-generate-output.md`:
- Around line 132-136: Update the unconditional output checklist so NFR-related
items are only required when NFRs are in scope: modify the lines containing "NFR
planning summary (thresholds, missing thresholds, planned evidence, and
NFR-derived risks)" and "NFR coverage and planned evidence mapping" to append a
conditional note such as "when NFRs are in scope" (or similar phrasing) so the
generated document does not produce fabricated or empty NFR sections.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 1ec1c787-b407-4612-84a1-c95d3ed3dc75
⛔ Files ignored due to path filters (10)
src/agents/bmad-tea/resources/tea-index.csvis excluded by!**/*.csvsrc/module-help.csvis excluded by!**/*.csvsrc/workflows/testarch/bmad-testarch-atdd/resources/tea-index.csvis excluded by!**/*.csvsrc/workflows/testarch/bmad-testarch-automate/resources/tea-index.csvis excluded by!**/*.csvsrc/workflows/testarch/bmad-testarch-ci/resources/tea-index.csvis excluded by!**/*.csvsrc/workflows/testarch/bmad-testarch-framework/resources/tea-index.csvis excluded by!**/*.csvsrc/workflows/testarch/bmad-testarch-nfr/resources/tea-index.csvis excluded by!**/*.csvsrc/workflows/testarch/bmad-testarch-test-design/resources/tea-index.csvis excluded by!**/*.csvsrc/workflows/testarch/bmad-testarch-test-review/resources/tea-index.csvis excluded by!**/*.csvsrc/workflows/testarch/bmad-testarch-trace/resources/tea-index.csvis excluded by!**/*.csv
📒 Files selected for processing (48)
README.mddocs/explanation/engagement-models.mddocs/explanation/risk-based-testing.mddocs/explanation/tea-overview.mddocs/explanation/testing-as-engineering.mddocs/glossary/index.mddocs/how-to/brownfield/use-tea-for-enterprise.mddocs/how-to/brownfield/use-tea-with-existing-tests.mddocs/how-to/workflows/run-nfr-assess.mddocs/how-to/workflows/run-test-design.mddocs/how-to/workflows/run-trace.mddocs/index.mddocs/reference/commands.mddocs/reference/knowledge-base.mdsrc/agents/bmad-tea/customize.tomlsrc/workflows/testarch/README.mdsrc/workflows/testarch/bmad-teach-me-testing/data/role-paths.yamlsrc/workflows/testarch/bmad-teach-me-testing/data/tea-resources-index.yamlsrc/workflows/testarch/bmad-teach-me-testing/steps-c/step-04-session-01.mdsrc/workflows/testarch/bmad-teach-me-testing/steps-c/step-04-session-07.mdsrc/workflows/testarch/bmad-teach-me-testing/templates/certificate-template.mdsrc/workflows/testarch/bmad-testarch-nfr/SKILL.mdsrc/workflows/testarch/bmad-testarch-nfr/checklist.mdsrc/workflows/testarch/bmad-testarch-nfr/instructions.mdsrc/workflows/testarch/bmad-testarch-nfr/nfr-report-template.mdsrc/workflows/testarch/bmad-testarch-nfr/steps-c/step-01-load-context.mdsrc/workflows/testarch/bmad-testarch-nfr/steps-c/step-01b-resume.mdsrc/workflows/testarch/bmad-testarch-nfr/steps-c/step-02-define-thresholds.mdsrc/workflows/testarch/bmad-testarch-nfr/steps-c/step-04-evaluate-and-score.mdsrc/workflows/testarch/bmad-testarch-nfr/steps-c/step-04a-subagent-security.mdsrc/workflows/testarch/bmad-testarch-nfr/steps-c/step-04b-subagent-performance.mdsrc/workflows/testarch/bmad-testarch-nfr/steps-c/step-04c-subagent-reliability.mdsrc/workflows/testarch/bmad-testarch-nfr/steps-c/step-04d-subagent-scalability.mdsrc/workflows/testarch/bmad-testarch-nfr/steps-c/step-04e-aggregate-nfr.mdsrc/workflows/testarch/bmad-testarch-nfr/steps-c/step-05-generate-report.mdsrc/workflows/testarch/bmad-testarch-nfr/workflow-plan.mdsrc/workflows/testarch/bmad-testarch-nfr/workflow.yamlsrc/workflows/testarch/bmad-testarch-test-design/checklist.mdsrc/workflows/testarch/bmad-testarch-test-design/steps-c/step-02-load-context.mdsrc/workflows/testarch/bmad-testarch-test-design/steps-c/step-03-risk-and-testability.mdsrc/workflows/testarch/bmad-testarch-test-design/steps-c/step-04-coverage-plan.mdsrc/workflows/testarch/bmad-testarch-test-design/steps-c/step-05-generate-output.mdsrc/workflows/testarch/bmad-testarch-test-design/test-design-architecture-template.mdsrc/workflows/testarch/bmad-testarch-test-design/test-design-qa-template.mdsrc/workflows/testarch/bmad-testarch-test-design/test-design-template.mdsrc/workflows/testarch/bmad-testarch-trace/checklist.mdsrc/workflows/testarch/bmad-testarch-trace/trace-template.mdwebsite/astro.config.mjs
Comment on lines
+219
to
226
| | Phase | TEA Workflows | Frequency | Purpose | | ||
| | ------------ | --------------------------------------------------------- | ---------------- | --------------------------------------------------------------- | | ||
| | **Learning** | `teach-me-testing` | Per learner | Progressive testing education | | ||
| | **Phase 2** | (none) | - | Planning phase - PM defines FRs/NFRs | | ||
| | **Phase 3** | `test-design` (system-level), `framework`, `ci` | Once per project | System testability, NFR planning, and test infrastructure setup | | ||
| | **Phase 4** | `test-design`, `atdd`, `automate`, `test-review`, `trace` | Per epic/story | Test planning per epic, then per-story testing | | ||
| | **Release** | `nfr-assess`, `trace` (Phase 2: gate) | Per epic/release | NFR evidence audit and go/no-go decision | | ||
|
|
There was a problem hiding this comment.
Release workflow table conflicts with conditional nfr-assess guidance
This row implies nfr-assess always runs at release, but earlier in this file it’s explicitly conditional on evidence relevance. Please align this row to avoid conflicting operator guidance.
Suggested change
| | Phase | TEA Workflows | Frequency | Purpose | | |
| | ------------ | --------------------------------------------------------- | ---------------- | --------------------------------------------------------------- | | |
| | **Learning** | `teach-me-testing` | Per learner | Progressive testing education | | |
| | **Phase 2** | (none) | - | Planning phase - PM defines FRs/NFRs | | |
| | **Phase 3** | `test-design` (system-level), `framework`, `ci` | Once per project | System testability, NFR planning, and test infrastructure setup | | |
| | **Phase 4** | `test-design`, `atdd`, `automate`, `test-review`, `trace` | Per epic/story | Test planning per epic, then per-story testing | | |
| | **Release** | `nfr-assess`, `trace` (Phase 2: gate) | Per epic/release | NFR evidence audit and go/no-go decision | | |
| | **Release** | `nfr-assess` (when NFR evidence exists and matters), `trace` (Phase 2: gate) | Per epic/release | Conditional NFR evidence audit and go/no-go decision | |
As per coding guidelines, “Focus on inconsistencies, contradictions, edge cases and serious issues.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/explanation/tea-overview.md` around lines 219 - 226, The Release row in
the phase table currently implies `nfr-assess` always runs at release, which
contradicts earlier guidance that `nfr-assess` is conditional; update the
Release table row to mark `nfr-assess` as conditional (e.g., "nfr-assess (when
NFR evidence exists and matters)") and adjust the Purpose cell to reflect a
conditional NFR evidence audit and go/no-go decision so the `nfr-assess` entry
aligns with the earlier conditional guidance for `nfr-assess`.
| - traceability-matrix.md (from Phase 1) | ||
| - test-review.md (from quality audit) | ||
| - nfr-assessment.md (from NFR assessment) | ||
| - nfr-assessment.md (from NFR Evidence Audit) |
There was a problem hiding this comment.
Artifact filename is out of sync with “NFR Evidence Audit” terminology
The evidence list still references nfr-assessment.md, which contradicts the updated naming and can break release checklist consistency.
Suggested change
| - nfr-assessment.md (from NFR Evidence Audit) | |
| - nfr-evidence-audit.md (from NFR Evidence Audit) |
As per coding guidelines, “Focus on inconsistencies, contradictions, edge cases and serious issues.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/how-to/brownfield/use-tea-for-enterprise.md` at line 270, Update the
artifact filename referenced in the evidence list: replace the entry named
"nfr-assessment.md" with "nfr-evidence-audit.md" so it matches the “NFR Evidence
Audit” terminology; locate the string "nfr-assessment.md" in the document (file
contains the evidence list) and change it to "nfr-evidence-audit.md", ensuring
any surrounding descriptive text still reads correctly.
| 3. **Update this assessment** (TEA, 1 hour) | ||
| - Re-run `*nfr-assess` with new results | ||
| 3. **Update this audit** (TEA, 1 hour) | ||
| - Re-run `*nfr-assess` with new evidence |
There was a problem hiding this comment.
Use nfr-assess consistently (remove the leading *).
Line 418, Line 533, and Line 732 use *nfr-assess while the rest of the guide uses nfr-assess. The starred form is copy/paste-hostile and can mislead users into running an invalid command.
Also applies to: 533-533, 732-732
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/how-to/workflows/run-nfr-assess.md` at line 418, Replace the starred
command occurrences so the guide uses the plain command name; find and change
all instances of "*nfr-assess" to "nfr-assess" (the three occurrences mentioned
plus any other stray leading-asterisk usages) so the document consistently shows
the runnable command string and avoids copy/paste errors.
Comment on lines
+98
to
99
| - **9 Workflows:** Teach Me Testing, Test Design, Framework, CI, ATDD, Automate, Test Review, NFR Evidence Audit, Trace | ||
| - **35 Knowledge Fragments:** Distilled expertise on patterns, best practices, Playwright Utils |
There was a problem hiding this comment.
Fix fragment count mismatch in Session 1 content.
Line 99 says “35 Knowledge Fragments,” but this script later teaches “42 fragments,” creating an internal contradiction for learners. Please align this to the canonical count.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/workflows/testarch/bmad-teach-me-testing/steps-c/step-04-session-01.md`
around lines 98 - 99, Update the mismatched fragment count: change the bullet
text that currently reads "**35 Knowledge Fragments:**" to the canonical "**42
Knowledge Fragments:**" so it matches the later script that references 42
fragments; locate and edit the line containing the literal "- **35 Knowledge
Fragments:**" in step-04-session-01.md to replace 35 with 42.
Comment on lines
+43
to
+46
| - Look for `test-design-architecture.md`, `test-design-qa.md`, or any test-design output in `{test_artifacts}/` that contains an NFR section. | ||
| - If found, **load it as the primary source of NFR categories and thresholds**. Use those values directly; do not re-derive them from scratch. | ||
| - Only fall back to raw documents (tech-spec, PRD, story) for categories or thresholds that are still missing or marked **UNKNOWN** in the test-design output. | ||
| - If no test-design NFR plan exists, proceed with step 1 using raw documents. |
There was a problem hiding this comment.
Define deterministic precedence when multiple test-design NFR plans exist.
Lines 43–46 allow several candidate test-design files but don’t specify conflict resolution. If thresholds differ across files, the workflow can produce inconsistent outputs run-to-run.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@src/workflows/testarch/bmad-testarch-nfr/steps-c/step-02-define-thresholds.md`
around lines 43 - 46, Specify and implement a deterministic precedence rule for
selecting/merging NFR values when multiple test-design files are present: in the
logic that loads test-design outputs (lookups for test-design-architecture.md,
test-design-qa.md, or any test-design output under {test_artifacts}/ as
described in step-02-define-thresholds.md), enforce a fixed priority list (e.g.,
test-design-architecture.md > test-design-qa.md > other test-design files) and
always take a value from the highest-priority file that defines it; for
conflicting thresholds across files either choose the value from the
higher-priority file or apply a deterministic merge policy (e.g., pick the
strictest/most conservative numeric threshold), and fall back to raw documents
only for remaining UNKNOWNs as existing logic requires.
| name: bmad-testarch-nfr | ||
| # prettier-ignore | ||
| description: 'Assess NFRs like performance security and reliability. Use when the user says "lets assess NFRs" or "I want to evaluate non-functional requirements"' | ||
| description: 'Audit NFR evidence for performance, security, reliability, and maintainability. Use when implementation evidence exists and the user says "lets assess NFRs", "audit NFR evidence", or "evaluate non-functional requirements"' |
There was a problem hiding this comment.
Domain list in description contradicts the implemented NFR workflow.
Line 4 says the workflow audits maintainability, but the orchestrated subagent set and aggregation in this PR are security, performance, reliability, and scalability. This mismatch can cause incorrect user expectations and wrong category selection.
Suggested change
| description: 'Audit NFR evidence for performance, security, reliability, and maintainability. Use when implementation evidence exists and the user says "lets assess NFRs", "audit NFR evidence", or "evaluate non-functional requirements"' | |
| description: 'Audit NFR evidence for performance, security, reliability, and scalability. Use when implementation evidence exists and the user says "lets assess NFRs", "audit NFR evidence", or "evaluate non-functional requirements"' |
As per coding guidelines, "Focus on inconsistencies, contradictions, edge cases and serious issues."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/workflows/testarch/bmad-testarch-nfr/workflow.yaml` at line 4, The
workflow description string currently lists "maintainability" but the
implemented orchestrated subagents and aggregation use "security",
"performance", "reliability", and "scalability"; update the description in the
workflow.yaml (the description field) to replace "maintainability" with
"scalability" so the description matches the NFR domains actually implemented by
this workflow.
Comment on lines
+128
to
+134
| | NFR Category | Threshold / Requirement | Current Design Support | Gap / Decision Needed | Planned Evidence | | ||
| | --------------- | ------------------------------------------------- | --------------------------- | ------------------------------ | --------------------------------------- | | ||
| | Security | {Auth/authz/data protection requirement} | {Supported/partial/unknown} | {Architecture gap or decision} | {Security tests, scan, audit log} | | ||
| | Performance | {Latency/throughput/resource requirement} | {Supported/partial/unknown} | {Architecture gap or decision} | {k6/APM/load report} | | ||
| | Reliability | {Availability/error-rate/recovery requirement} | {Supported/partial/unknown} | {Architecture gap or decision} | {Burn-in, monitoring, failover report} | | ||
| | Maintainability | {Coverage/code quality/observability requirement} | {Supported/partial/unknown} | {Architecture gap or decision} | {Coverage, static analysis, docs audit} | | ||
|
|
There was a problem hiding this comment.
Expand NFR category rows to match workflow scope.
The template omits scalability, compliance, and project-specific/custom NFR categories required elsewhere, so teams can miss mandatory planning coverage.
Suggested change
| | NFR Category | Threshold / Requirement | Current Design Support | Gap / Decision Needed | Planned Evidence | | |
| | --------------- | ------------------------------------------------- | --------------------------- | ------------------------------ | --------------------------------------- | | |
| | Security | {Auth/authz/data protection requirement} | {Supported/partial/unknown} | {Architecture gap or decision} | {Security tests, scan, audit log} | | |
| | Performance | {Latency/throughput/resource requirement} | {Supported/partial/unknown} | {Architecture gap or decision} | {k6/APM/load report} | | |
| | Reliability | {Availability/error-rate/recovery requirement} | {Supported/partial/unknown} | {Architecture gap or decision} | {Burn-in, monitoring, failover report} | | |
| | Maintainability | {Coverage/code quality/observability requirement} | {Supported/partial/unknown} | {Architecture gap or decision} | {Coverage, static analysis, docs audit} | | |
| | NFR Category | Threshold / Requirement | Current Design Support | Gap / Decision Needed | Planned Evidence | | |
| | --------------- | ------------------------------------------------- | --------------------------- | ------------------------------ | --------------------------------------- | | |
| | Security | {Auth/authz/data protection requirement} | {Supported/partial/unknown} | {Architecture gap or decision} | {Security tests, scan, audit log} | | |
| | Performance | {Latency/throughput/resource requirement} | {Supported/partial/unknown} | {Architecture gap or decision} | {k6/APM/load report} | | |
| | Reliability | {Availability/error-rate/recovery requirement} | {Supported/partial/unknown} | {Architecture gap or decision} | {Burn-in, monitoring, failover report} | | |
| | Scalability | {Scale/concurrency/capacity requirement} | {Supported/partial/unknown} | {Architecture gap or decision} | {Capacity test, autoscale evidence} | | |
| | Maintainability | {Coverage/code quality/observability requirement} | {Supported/partial/unknown} | {Architecture gap or decision} | {Coverage, static analysis, docs audit} | | |
| | Compliance | {Regulatory/privacy/audit requirement} | {Supported/partial/unknown} | {Architecture gap or decision} | {Compliance report, audit evidence} | | |
| | Custom | {Project-specific NFR requirement} | {Supported/partial/unknown} | {Architecture gap or decision} | {Project-specific evidence artifact} | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@src/workflows/testarch/bmad-testarch-test-design/test-design-architecture-template.md`
around lines 128 - 134, Update the NFR table block (the pipe-delimited table
starting with the header "NFR Category | Threshold / Requirement | Current
Design Support | Gap / Decision Needed | Planned Evidence") to include
additional rows for "Scalability", "Compliance", and "Custom" (in addition to
the existing Security, Performance, Reliability, Maintainability rows), and fill
the Threshold / Planned Evidence cells with the suggested placeholders (e.g.,
{Scale/concurrency/capacity requirement}, {Capacity test, autoscale evidence},
{Regulatory/privacy/audit requirement}, {Compliance report, audit evidence},
{Project-specific NFR requirement}, {Project-specific evidence artifact}) so
teams are prompted to capture scale, regulatory and project‑specific NFRs.
Comment on lines
+130
to
+135
| | NFR Category | Requirement / Threshold | Planned Validation | Tool / Level | Evidence Artifact | Priority | | ||
| | --------------- | ----------------------- | ------------------------------------------ | -------------------- | ----------------------------- | -------- | | ||
| | Security | {Requirement} | {Auth/authz/security validation} | {API/E2E/SAST/DAST} | {Report or test result path} | {P0-P3} | | ||
| | Performance | {Requirement} | {Load/stress/baseline validation} | {k6/APM/Lighthouse} | {Report or dashboard} | {P0-P3} | | ||
| | Reliability | {Requirement} | {Error/retry/failover validation} | {API/E2E/monitoring} | {Burn-in/log/metric evidence} | {P0-P3} | | ||
| | Maintainability | {Requirement} | {Coverage/static analysis/docs validation} | {CI/static analysis} | {Coverage or quality report} | {P0-P3} | |
There was a problem hiding this comment.
Align QA NFR coverage table with full in-scope category set.
The current rows don’t cover scalability, compliance, or custom NFR categories, which creates a planning hole vs workflow/checklist expectations.
Suggested change
| | NFR Category | Requirement / Threshold | Planned Validation | Tool / Level | Evidence Artifact | Priority | | |
| | --------------- | ----------------------- | ------------------------------------------ | -------------------- | ----------------------------- | -------- | | |
| | Security | {Requirement} | {Auth/authz/security validation} | {API/E2E/SAST/DAST} | {Report or test result path} | {P0-P3} | | |
| | Performance | {Requirement} | {Load/stress/baseline validation} | {k6/APM/Lighthouse} | {Report or dashboard} | {P0-P3} | | |
| | Reliability | {Requirement} | {Error/retry/failover validation} | {API/E2E/monitoring} | {Burn-in/log/metric evidence} | {P0-P3} | | |
| | Maintainability | {Requirement} | {Coverage/static analysis/docs validation} | {CI/static analysis} | {Coverage or quality report} | {P0-P3} | | |
| | NFR Category | Requirement / Threshold | Planned Validation | Tool / Level | Evidence Artifact | Priority | | |
| | --------------- | ----------------------- | ------------------------------------------ | -------------------- | ----------------------------- | -------- | | |
| | Security | {Requirement} | {Auth/authz/security validation} | {API/E2E/SAST/DAST} | {Report or test result path} | {P0-P3} | | |
| | Performance | {Requirement} | {Load/stress/baseline validation} | {k6/APM/Lighthouse} | {Report or dashboard} | {P0-P3} | | |
| | Reliability | {Requirement} | {Error/retry/failover validation} | {API/E2E/monitoring} | {Burn-in/log/metric evidence} | {P0-P3} | | |
| | Scalability | {Requirement} | {Capacity/concurrency/soak validation} | {k6/APM} | {Capacity report} | {P0-P3} | | |
| | Maintainability | {Requirement} | {Coverage/static analysis/docs validation} | {CI/static analysis} | {Coverage or quality report} | {P0-P3} | | |
| | Compliance | {Requirement} | {Policy/privacy/audit validation} | {CI/audit checks} | {Audit/compliance evidence} | {P0-P3} | | |
| | Custom | {Requirement} | {Project-specific validation} | {Tool/Level} | {Artifact} | {P0-P3} | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/workflows/testarch/bmad-testarch-test-design/test-design-qa-template.md`
around lines 130 - 135, Update the NFR table in test-design-qa-template.md to
include the missing categories (Scalability, Compliance, and Custom) and reorder
rows so Scalability appears before Maintainability as in the suggested diff;
ensure each new row supplies the same columns (Requirement / Threshold, Planned
Validation, Tool / Level, Evidence Artifact, Priority) with the example
placeholders from the suggestion (e.g., Scalability: {Capacity/concurrency/soak
validation}, Tool {k6/APM}, Evidence {Capacity report}; Compliance:
{Policy/privacy/audit validation}, Tool {CI/audit checks}, Evidence
{Audit/compliance evidence}; Custom: {Project-specific validation}, Tool
{Tool/Level}, Evidence {Artifact}) and keep existing Security, Performance,
Reliability, and Maintainability rows unchanged other than reordering.
Comment on lines
+83
to
+89
| | NFR Category | Requirement / Threshold | Risk Link | Planned Validation | Evidence Needed | | ||
| | --------------- | ----------------------- | --------- | ------------------------------------------ | -------------------------------- | | ||
| | Security | {Requirement} | {R-ID} | {API/E2E/SAST/DAST validation} | {Test report, scan, audit log} | | ||
| | Performance | {Requirement} | {R-ID} | {Load/stress/baseline validation} | {k6/APM/Lighthouse report} | | ||
| | Reliability | {Requirement} | {R-ID} | {Error/retry/failover validation} | {Burn-in, logs, monitoring data} | | ||
| | Maintainability | {Requirement} | {R-ID} | {Coverage/static analysis/docs validation} | {Coverage or quality report} | | ||
|
|
There was a problem hiding this comment.
Broaden epic NFR planning rows to cover all supported NFR categories.
This section currently hardcodes four categories and can miss scalability/compliance/custom NFR planning expected by the workflow.
Suggested change
| | NFR Category | Requirement / Threshold | Risk Link | Planned Validation | Evidence Needed | | |
| | --------------- | ----------------------- | --------- | ------------------------------------------ | -------------------------------- | | |
| | Security | {Requirement} | {R-ID} | {API/E2E/SAST/DAST validation} | {Test report, scan, audit log} | | |
| | Performance | {Requirement} | {R-ID} | {Load/stress/baseline validation} | {k6/APM/Lighthouse report} | | |
| | Reliability | {Requirement} | {R-ID} | {Error/retry/failover validation} | {Burn-in, logs, monitoring data} | | |
| | Maintainability | {Requirement} | {R-ID} | {Coverage/static analysis/docs validation} | {Coverage or quality report} | | |
| | NFR Category | Requirement / Threshold | Risk Link | Planned Validation | Evidence Needed | | |
| | --------------- | ----------------------- | --------- | ------------------------------------------ | -------------------------------- | | |
| | Security | {Requirement} | {R-ID} | {API/E2E/SAST/DAST validation} | {Test report, scan, audit log} | | |
| | Performance | {Requirement} | {R-ID} | {Load/stress/baseline validation} | {k6/APM/Lighthouse report} | | |
| | Reliability | {Requirement} | {R-ID} | {Error/retry/failover validation} | {Burn-in, logs, monitoring data} | | |
| | Scalability | {Requirement} | {R-ID} | {Capacity/concurrency/soak validation} | {Capacity/load evidence} | | |
| | Maintainability | {Requirement} | {R-ID} | {Coverage/static analysis/docs validation} | {Coverage or quality report} | | |
| | Compliance | {Requirement} | {R-ID} | {Policy/privacy/audit validation} | {Compliance/audit evidence} | | |
| | Custom | {Requirement} | {R-ID} | {Project-specific validation} | {Project-specific artifact} | |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/workflows/testarch/bmad-testarch-test-design/test-design-template.md`
around lines 83 - 89, The NFR table currently only lists four categories
(Security, Performance, Reliability, Maintainability) so expand the table rows
in test-design-template.md to include additional supported NFR categories: add
rows for Scalability (with Planned Validation like capacity/concurrency/soak
validation and Evidence Needed like capacity/load evidence), Compliance (with
Policy/privacy/audit validation and compliance/audit evidence), and a Custom row
(with project-specific validation and artifact); update the corresponding
"Planned Validation" and "Evidence Needed" cells for these new rows to match the
suggested entries so the epic NFR planning covers all supported categories.
| - **Tech Spec:** {TECH_SPEC_PATH} (if available) | ||
| - **Test Results:** {TEST_RESULTS_PATH} | ||
| - **NFR Assessment:** {NFR_FILE_PATH} (if available) | ||
| - **NFR Evidence Audit:** {NFR_FILE_PATH} (if available) |
There was a problem hiding this comment.
Keep template terminology and placeholder contract consistent for NFR evidence.
The display label now says “NFR Evidence Audit”, but this template still uses nfr_assessment naming in related placeholders/keys. Keeping both terms in one contract increases ambiguity for producers/consumers of trace artifacts.
As per coding guidelines, "Focus on inconsistencies, contradictions, edge cases and serious issues."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/workflows/testarch/bmad-testarch-trace/trace-template.md` at line 681,
The template label “NFR Evidence Audit” is inconsistent with placeholders/keys
that use the nfr_assessment naming; pick one canonical term and apply it
consistently—either (A) rename the display label to “NFR Assessment” or (B)
update all placeholders/keys (e.g., {NFR_FILE_PATH}, any variables or keys named
nfr_assessment) to use nfr_evidence_audit (or nfr_evidence) throughout the
trace-template.md and any referenced artifacts; ensure matching names in
functions/methods that read/write those keys (search for nfr_assessment and
NFR_FILE_PATH) so producers/consumers use the same contract.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
test-designwith thresholds, unknowns, planned evidence, and coverage mapping.nfr-assessas NFR Evidence Audit while keeping the command stable for compatibility.GATEmenu intent and update docs/navigation around Design, Audit, and Trace boundaries.Validation
npm test