improve: enhance security-auditor with code-focused methodology and supply chain audit#485
Closed
davila7 wants to merge 1 commit into
Closed
improve: enhance security-auditor with code-focused methodology and supply chain audit#485davila7 wants to merge 1 commit into
davila7 wants to merge 1 commit into
Conversation
…upply chain audit - Remove fabricated JSON communication protocol blocks with hardcoded fake metrics (347 controls, 87% compliance) - Remove hardcoded delivery notification with fake numbers - Add Bash to tools list to enable npm audit, pip-audit, trivy, syft execution - Replace generic noun-phrase checklists with verb-driven 5-phase audit methodology - Add concrete Grep regex patterns for secrets, AWS keys, private keys, weak crypto, injection, XSS, container, CI/CD - Add supply chain security phase: npm audit, pip-audit, trivy, SBOM generation, lock file verification - Add CWE-referenced severity classification table (Critical/High/Medium/Low) - Update compliance frameworks to current versions: PCI DSS v4.0.1, ISO 27001:2022, NIST CSF 2.0, OWASP Top 10 2025, OWASP API Security 2023, SLSA Level 1-4 - Remove reference to non-existent context manager - Remove organizational audit content (physical security, interview notes, vendor assessments) - Trim from 285 to 150 lines while increasing actionable content density Automated review cycle | Co-Authored-By: Claude Code <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
| Metric | Count |
|---|---|
| Total Components | 755 |
| ✅ Passed | 360 |
| ❌ Failed | 395 |
| 999 |
❌ Failed Components (Top 5)
| Component | Errors | Warnings | Score |
|---|---|---|---|
vercel-edge-function |
3 | 4 | 81/100 |
prompt-engineer |
2 | 0 | 90/100 |
neon-expert |
2 | 2 | 88/100 |
agent-overview |
2 | 1 | 89/100 |
unused-code-cleaner |
2 | 1 | 89/100 |
...and 390 more failed component(s)
📊 View Full Report for detailed error messages and all components
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
1 issue found across 1 file
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="cli-tool/components/agents/security/security-auditor.md">
<violation number="1" location="cli-tool/components/agents/security/security-auditor.md:15">
P2: The glob pattern does not actually match standard `Dockerfile` names, so container-related files can be missed during audit scoping.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
|
|
||
| ### Phase 1: Scoping | ||
|
|
||
| - Use Glob to map repository structure: `**/*.{js,ts,py,go,java,tf,yml,yaml,json,toml,Dockerfile}` |
Contributor
There was a problem hiding this comment.
P2: The glob pattern does not actually match standard Dockerfile names, so container-related files can be missed during audit scoping.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At cli-tool/components/agents/security/security-auditor.md, line 15:
<comment>The glob pattern does not actually match standard `Dockerfile` names, so container-related files can be missed during audit scoping.</comment>
<file context>
@@ -1,286 +1,150 @@
+
+### Phase 1: Scoping
+
+- Use Glob to map repository structure: `**/*.{js,ts,py,go,java,tf,yml,yaml,json,toml,Dockerfile}`
+- Read package manifests (`package.json`, `requirements.txt`, `go.mod`, `pom.xml`, `Cargo.toml`) to understand the dependency surface
+- Read existing security policies, threat models, and previous audit findings if present
</file context>
Suggested change
| - Use Glob to map repository structure: `**/*.{js,ts,py,go,java,tf,yml,yaml,json,toml,Dockerfile}` | |
| - Use Glob to map repository structure: `**/*.{js,ts,py,go,java,tf,yml,yaml,json,toml}`, `**/Dockerfile*` |
Owner
Author
|
Closing per maintainer decision. The security-auditor agent will be re-evaluated in a future automated review cycle if needed. Branch deleted. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated Component Improvement
Component:
cli-tool/components/agents/security/security-auditor.mdTrending rank: #7 (4,416 downloads, 313/week)
Changes Applied
Removed all fabricated JSON blocks — Deleted fake communication protocol JSON at lines 133-141 and progress tracking JSON at lines 196-207 (hardcoded: 347 controls, 52 findings, 87% compliance score). Deleted hardcoded delivery notification with same fake metrics.
Added Bash to tools — Changed
tools: Read, Grep, Globtotools: Read, Bash, Grep, Globto enable runningnpm audit,pip-audit,trivy,syft, and other security scanners.Replaced generic invocation steps with 5-phase audit methodology — Removed the "query context manager" step (referencing a non-existent agent). Added Phase 1 Scoping, Phase 2 Automated Pattern Scanning, Phase 3 Dependency/Supply Chain Audit, Phase 4 Control Review, Phase 5 Finding Classification & Reporting.
Added concrete Grep regex patterns — Secrets/credentials, AWS access keys (
AKIA[0-9A-Z]{16}), private keys (PEM headers), weak crypto (md5|sha1|des|rc4), SQL injection, OS command injection, XSS (innerHTML,dangerouslySetInnerHTML), container misconfigs (FROM :latest,USER root), unpinned CI/CD actions.Added supply chain security section —
npm audit --audit-level=high,pip-audit,trivy,syftSBOM generation, lock file verification, unpinned dependency detection, dependency confusion risk assessment.Added CWE-referenced severity classification table — Critical (CWE-798, CWE-89, CWE-78), High (CWE-287, CWE-306, CWE-502), Medium (CWE-311, CWE-330, CWE-601), Low (CWE-200, CWE-693).
Updated compliance frameworks to current versions — PCI DSS v4.0.1 (June 2024), ISO 27001/27002:2022, NIST CSF 2.0 (Feb 2024), NIST SSDF SP 800-218 v1.1, OWASP Top 10 2025, OWASP API Security Top 10 2023, CIS Benchmarks (current), SLSA Level 1–4. Rendered as a table with version column and code-audit scope.
Converted noun-phrase checklists to verb-driven instructions — e.g. "Data classification" → "Verify data classified as sensitive is encrypted at rest (AES-256 or equivalent)".
Removed organizational audit content — Eliminated physical security, interview notes, vendor assessments, SLA validation, and other content irrelevant to a code-level auditor.
Rewrote description examples — Three focused examples with concrete assistant responses mentioning specific tools (npm audit, CWE references, PKCE, NIST SP 800-61).
No
model:field added — Frontmatter remains clean.Size Impact
Research Summary
The original component was a generic security-audit template with heavy organizational audit language (physical security, vendor assessments, interview schedules) that does not apply to Claude Code's code-focused context. It contained fabricated metrics presented as live output, no scanning commands, outdated compliance versions (OWASP 2021 implied), and referenced a non-existent context manager agent. This improvement refocuses the agent on what it can actually do: grep source code, run CLI scanners, read config files, and classify findings against current standards.
Validation
Automated review cycle by Component Improvement Loop
Summary by cubic
Refocused the
security-auditorcomponent on code and supply chain analysis. Replaced vague checklists and fake metrics with a concrete 5-phase audit and runnable tools for real findings.Bash:npm audit,npx better-npm-audit,pip-audit,trivy,syft.Greppatterns for secrets, weak crypto, injection, XSS, and container/CI/CD misconfigs.cli-tool/components/); no new components (nodocs/components.jsonregeneration). No new environment variables or secrets.Written for commit c580985. Summary will update on new commits.