Skip to content

audit: harden 10 agents against 9 RCA patterns (generic guardrails)#15

Open
singabhi wants to merge 2 commits into
lst97:mainfrom
singabhi:audit/naamjap-rca-generic-hardening
Open

audit: harden 10 agents against 9 RCA patterns (generic guardrails)#15
singabhi wants to merge 2 commits into
lst97:mainfrom
singabhi:audit/naamjap-rca-generic-hardening

Conversation

@singabhi

Copy link
Copy Markdown

5-Whys analysis of real-world failure patterns observed in evidence from a separate project. All 9 patterns confirmed with file-path or commit evidence; all insertions are generic — no project-specific nouns appear in agent files.

Patterns addressed:
P1 UI control shipped without backing implementation
P2 Status entry marked complete without traceable artifact
P3 User preference written but never read
P4 New entity introduced; downstream surfaces not updated
P5 UI contrast not verified across all visual contexts
P6 Feature ships on one platform; other platforms not recorded
P7 Deployment prerequisite enforced by warn-only CI gate
P8 Mock-only unit suite passes; cross-surface path broken
P9 Code review approved without checking P1–P8

Agent files edited (localized insertions only, marker: audit/naamjap-rca):
agents/quality-testing/code-reviewer.md (P1 P3 P4 P5 P9)
agents/development/full-stack-developer.md (P1 P3 P4)
agents/development/mobile-developer.md (P1 P6)
agents/business/product-manager.md (P1 P2)
agents/specialization/documentation-expert.md (P2)
agents/quality-testing/architect-review.md (P4)
agents/development/ui-designer.md (P5)
agents/quality-testing/qa-expert.md (P6 P8)
agents/infrastructure/deployment-engineer.md (P7)
agents/quality-testing/test-automator.md (P7 P8)

New files:
audits/2026-05-naamjap-rca.md RCA evidence + 5-Whys per pattern
audits/stub-declaration-contract.md UI-logic pairing / stub contract
audits/PR_BODY.md Draft PR body

5-Whys analysis of real-world failure patterns observed in evidence from a
separate project. All 9 patterns confirmed with file-path or commit evidence;
all insertions are generic — no project-specific nouns appear in agent files.

Patterns addressed:
  P1 UI control shipped without backing implementation
  P2 Status entry marked complete without traceable artifact
  P3 User preference written but never read
  P4 New entity introduced; downstream surfaces not updated
  P5 UI contrast not verified across all visual contexts
  P6 Feature ships on one platform; other platforms not recorded
  P7 Deployment prerequisite enforced by warn-only CI gate
  P8 Mock-only unit suite passes; cross-surface path broken
  P9 Code review approved without checking P1–P8

Agent files edited (localized insertions only, marker: audit/naamjap-rca):
  agents/quality-testing/code-reviewer.md       (P1 P3 P4 P5 P9)
  agents/development/full-stack-developer.md    (P1 P3 P4)
  agents/development/mobile-developer.md        (P1 P6)
  agents/business/product-manager.md            (P1 P2)
  agents/specialization/documentation-expert.md (P2)
  agents/quality-testing/architect-review.md    (P4)
  agents/development/ui-designer.md             (P5)
  agents/quality-testing/qa-expert.md           (P6 P8)
  agents/infrastructure/deployment-engineer.md  (P7)
  agents/quality-testing/test-automator.md      (P7 P8)

New files:
  audits/2026-05-naamjap-rca.md          RCA evidence + 5-Whys per pattern
  audits/stub-declaration-contract.md    UI-logic pairing / stub contract
  audits/PR_BODY.md                      Draft PR body

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@qodo-code-review

Copy link
Copy Markdown

Review Summary by Qodo

Harden agent prompts against 9 recurring failure patterns with generic guardrails

✨ Enhancement 📝 Documentation

Grey Divider

Walkthroughs

Description
• Hardened 10 agent prompts with 9 RCA-derived guardrails against recurring failure patterns
• Added explicit checks for UI-implementation pairing, preference read sites, and cross-surface
  propagation
• Introduced stub declaration contract and status accuracy rules to prevent silent defects
• Created comprehensive RCA evidence document with 5-Whys analysis grounded in real commits
Diagram
flowchart LR
  RCA["9 Failure Patterns<br/>P1-P9 Identified"]
  RCA -->|"P1: UI without impl"| CodeReviewer["code-reviewer.md"]
  RCA -->|"P2: Stale status"| DocExpert["documentation-expert.md"]
  RCA -->|"P3: Pref no read"| FullStack["full-stack-developer.md"]
  RCA -->|"P4: Isolated feature"| ArchReview["architect-review.md"]
  RCA -->|"P5: No contrast check"| UIDesigner["ui-designer.md"]
  RCA -->|"P6: Platform gap"| Mobile["mobile-developer.md"]
  RCA -->|"P7: Warn-only gate"| Deploy["deployment-engineer.md"]
  RCA -->|"P8: Mock-only tests"| TestAuto["test-automator.md"]
  RCA -->|"P9: Review gap"| QAExpert["qa-expert.md"]
  CodeReviewer --> Contract["stub-declaration-contract.md"]
  DocExpert --> Contract
  FullStack --> Contract
  ArchReview --> Contract
  UIDesigner --> Contract
  Mobile --> Contract
  Deploy --> Contract
  TestAuto --> Contract
  QAExpert --> Contract
  Contract --> Evidence["2026-05-naamjap-rca.md"]
Loading

Grey Divider

File Changes

1. agents/business/product-manager.md ✨ Enhancement +6/-0

Added status integrity and stub declaration rules

agents/business/product-manager.md


2. agents/development/full-stack-developer.md ✨ Enhancement +3/-0

Added UI-implementation pairing and preference read site checks

agents/development/full-stack-developer.md


3. agents/development/mobile-developer.md ✨ Enhancement +6/-0

Added UI-implementation and cross-platform parity guardrails

agents/development/mobile-developer.md


View more (10)
4. agents/development/ui-designer.md ✨ Enhancement +1/-0

Added visual context completeness verification requirement

agents/development/ui-designer.md


5. agents/infrastructure/deployment-engineer.md ✨ Enhancement +1/-0

Added hard-fail CI gate enforcement principle

agents/infrastructure/deployment-engineer.md


6. agents/quality-testing/architect-review.md ✨ Enhancement +3/-0

Added cross-surface propagation enumeration check

agents/quality-testing/architect-review.md


7. agents/quality-testing/code-reviewer.md ✨ Enhancement +9/-0

Added integration surface correctness checklist with P1-P9 sign-off

agents/quality-testing/code-reviewer.md


8. agents/quality-testing/qa-expert.md ✨ Enhancement +2/-0

Added cross-platform parity and integration test coverage principles

agents/quality-testing/qa-expert.md


9. agents/quality-testing/test-automator.md ✨ Enhancement +2/-0

Added CI gate completeness and cross-surface integration coverage rules

agents/quality-testing/test-automator.md


10. agents/specialization/documentation-expert.md ✨ Enhancement +6/-0

Added traceable completion artifact requirement for status entries

agents/specialization/documentation-expert.md


11. audits/2026-05-naamjap-rca.md 📝 Documentation +400/-0

Comprehensive RCA evidence document with 5-Whys for all 9 patterns

audits/2026-05-naamjap-rca.md


12. audits/stub-declaration-contract.md 📝 Documentation +94/-0

Integration surface and stub declaration conventions reference

audits/stub-declaration-contract.md


13. audits/PR_BODY.md 📝 Documentation +95/-0

PR summary with pattern index and agent file change walkthrough

audits/PR_BODY.md


Grey Divider

Qodo Logo

@qodo-code-review

qodo-code-review Bot commented May 11, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Grey Divider


Remediation recommended

1. Duplicate principle numbering ✓ Resolved 🐞 Bug ⚙ Maintainability
Description
In agents/infrastructure/deployment-engineer.md, the new P7 principle is numbered "2." but the
existing "Infrastructure as Code" principle remains numbered "2.", producing duplicate numbering in
the same list. This creates ambiguity when humans (or any tooling) reference principles by number.
Code

agents/infrastructure/deployment-engineer.md[R67-70]

1. **Automate Everything:** All aspects of the build, test, and deployment process must be automated. There should be no manual intervention required.
+2. **No Advisory-Only Gates (P7):** <!-- added: audit/naamjap-rca --> Every deployment prerequisite — secret, entitlement, service account, configuration key, platform registration — must be enforced by a pipeline step that **fails** the build when the prerequisite is absent. A warning-only gate provides false confidence: warnings accumulate, stop being read, and the gap they guard becomes invisible. If a prerequisite genuinely cannot be automated (e.g., an IDE-only registration step), the pipeline must hard-fail with a clear message and a documented override process that requires an explicit human sign-off — not a permanent downgrade to a warning.
2. **Infrastructure as Code:** All infrastructure, from networks to Kubernetes clusters, must be defined and managed in code.
3. **Build Once, Deploy Anywhere:** Create a single, immutable build artifact that can be promoted across different environments (development, staging, production) using environment-specific configurations.
Evidence
The "Guiding Principles" ordered list contains two consecutive entries numbered "2.": the newly
added "No Advisory-Only Gates (P7)" and the pre-existing "Infrastructure as Code" item, indicating
the insertion was not renumbered along with the rest of the list.

agents/infrastructure/deployment-engineer.md[65-71]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`agents/infrastructure/deployment-engineer.md` has duplicate numbering in the "Guiding Principles" list (two items numbered `2.`), which makes the document confusing to reference.
## Issue Context
A new P7 principle was inserted as item `2.` but the subsequent pre-existing items were not renumbered.
## Fix Focus Areas
- agents/infrastructure/deployment-engineer.md[65-74]
## Expected change
Renumber the ordered list so each principle has a unique, sequential number (e.g., keep P7 as `2.` and shift the following items to `3.`, `4.`, etc.), or switch to markdown auto-numbering style consistently.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Qodo Logo

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a comprehensive set of guardrails and technical standards across various agent prompt files to address nine recurring failure patterns identified in a recent audit. These patterns include UI-only stubs, stale status markers, and incomplete cross-surface data propagation. The feedback identifies a numbering conflict in the deployment engineer's guiding principles and suggests including a missing reference to artifact-backed completion (P2) in the code reviewer's integration surface sign-off checklist.

Comment thread agents/infrastructure/deployment-engineer.md
Comment thread agents/quality-testing/code-reviewer.md Outdated
- deployment-engineer.md: renumber guiding principles (1-8) after P7
  insertion caused duplicate number 2
- code-reviewer.md: add P2 (artifact-backed completion) to the P9
  integration surface sign-off checklist; re-letter items (a)-(h)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant