improve: enhance security-auditor agent#427
Closed
davila7 wants to merge 1 commit into
Closed
Conversation
- Replace fictional JSON communication protocol blocks with real tool-use instructions (Glob, Read, Grep) - Add 5-phase audit methodology: Scoping, Pattern Scanning, Control Review, Finding Classification, Reporting - Add concrete Grep regex patterns for secrets, crypto weaknesses, injection, container, and CI/CD risks - Update compliance frameworks to current versions: PCI DSS v4.0, ISO 27001:2022, NIST CSF 2.0, OWASP API Security Top 10 2023, SLSA - Add four new audit domains: Supply Chain Security, Container and Kubernetes Security, CI/CD Pipeline Security, Secrets Management - Replace hardcoded example numbers (347 controls, 52 findings, 87%) with template placeholders - Add CWE-referenced finding classification table (Critical/High/Medium/Low) - Convert noun-phrase checklists to verb-driven instructional content Automated review cycle | Co-Authored-By: Claude Code <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
|
| Metric | Count |
|---|---|
| Total Components | 753 |
| ✅ Passed | 360 |
| ❌ Failed | 393 |
| 620 |
❌ Failed Components (Top 5)
| Component | Errors | Warnings | Score |
|---|---|---|---|
vercel-edge-function |
3 | 4 | 81/100 |
neon-expert |
2 | 1 | 89/100 |
agent-overview |
2 | 1 | 89/100 |
unused-code-cleaner |
2 | 1 | 89/100 |
context7 |
2 | 2 | 88/100 |
...and 388 more failed component(s)
📊 View Full Report for detailed error messages and all components
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
3 issues found across 1 file
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="cli-tool/components/agents/security/security-auditor.md">
<violation number="1" location="cli-tool/components/agents/security/security-auditor.md:20">
P2: Don't ask the agent to identify known vulnerable packages from manifest reads alone. With only Read/Grep/Glob, it can inventory dependencies but cannot verify advisories.</violation>
<violation number="2" location="cli-tool/components/agents/security/security-auditor.md:147">
P2: This task is not executable with the declared tools. Secret detection in git history requires VCS/history access, not just Read/Grep/Glob over the current tree.</violation>
<violation number="3" location="cli-tool/components/agents/security/security-auditor.md:160">
P2: Update the PCI DSS row to the active version. Listing v4.0 as current will benchmark audits against a retired standard.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
| - Secrets rotation policies and enforcement | ||
| - Vault or secrets manager integration vs. environment variable sprawl | ||
| - `.gitignore` and pre-commit hook coverage for secrets | ||
| - Detection of secrets in git history |
Contributor
There was a problem hiding this comment.
P2: This task is not executable with the declared tools. Secret detection in git history requires VCS/history access, not just Read/Grep/Glob over the current tree.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At cli-tool/components/agents/security/security-auditor.md, line 147:
<comment>This task is not executable with the declared tools. Secret detection in git history requires VCS/history access, not just Read/Grep/Glob over the current tree.</comment>
<file context>
@@ -1,287 +1,180 @@
+- Secrets rotation policies and enforcement
+- Vault or secrets manager integration vs. environment variable sprawl
+- `.gitignore` and pre-commit hook coverage for secrets
+- Detection of secrets in git history
+
+### Incident Response
</file context>
Suggested change
| - Detection of secrets in git history | |
| - Detection of secrets in the current repository contents, and explicit note when git-history scanning requires separate VCS access |
| | Framework | Current Version | Key Focus | | ||
| |-----------|----------------|-----------| | ||
| | SOC 2 Type II | 2017 (TSC) | Trust service criteria | | ||
| | PCI DSS | v4.0 (March 2024) | Payment card data protection | |
Contributor
There was a problem hiding this comment.
P2: Update the PCI DSS row to the active version. Listing v4.0 as current will benchmark audits against a retired standard.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At cli-tool/components/agents/security/security-auditor.md, line 160:
<comment>Update the PCI DSS row to the active version. Listing v4.0 as current will benchmark audits against a retired standard.</comment>
<file context>
@@ -1,287 +1,180 @@
+| Framework | Current Version | Key Focus |
+|-----------|----------------|-----------|
+| SOC 2 Type II | 2017 (TSC) | Trust service criteria |
+| PCI DSS | v4.0 (March 2024) | Payment card data protection |
+| ISO 27001 / 27002 | 2022 | ISMS controls |
+| HIPAA | Current | Healthcare data privacy |
</file context>
Suggested change
| | PCI DSS | v4.0 (March 2024) | Payment card data protection | | |
| | PCI DSS | v4.0.1 (June 2024) | Payment card data protection | |
|
|
||
| 1. Ask the user to clarify: audit goals, compliance targets, systems in scope, and timeline constraints. | ||
| 2. Use Glob to map the repository structure: identify languages, frameworks, infrastructure-as-code, CI/CD configs, and Dockerfiles. | ||
| 3. Use Read to examine package manifests (`package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pom.xml`) and identify dependencies and known vulnerable packages. |
Contributor
There was a problem hiding this comment.
P2: Don't ask the agent to identify known vulnerable packages from manifest reads alone. With only Read/Grep/Glob, it can inventory dependencies but cannot verify advisories.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At cli-tool/components/agents/security/security-auditor.md, line 20:
<comment>Don't ask the agent to identify known vulnerable packages from manifest reads alone. With only Read/Grep/Glob, it can inventory dependencies but cannot verify advisories.</comment>
<file context>
@@ -1,287 +1,180 @@
+
+1. Ask the user to clarify: audit goals, compliance targets, systems in scope, and timeline constraints.
+2. Use Glob to map the repository structure: identify languages, frameworks, infrastructure-as-code, CI/CD configs, and Dockerfiles.
+3. Use Read to examine package manifests (`package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pom.xml`) and identify dependencies and known vulnerable packages.
+4. Note what is explicitly out of scope to avoid scope creep.
+
</file context>
Suggested change
| 3. Use Read to examine package manifests (`package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pom.xml`) and identify dependencies and known vulnerable packages. | |
| 3. Use Read to examine package manifests (`package.json`, `requirements.txt`, `go.mod`, `Gemfile`, `pom.xml`) to inventory dependencies and flag packages that should be checked against an advisory source for known vulnerabilities. |
Owner
Author
|
Closing as superseded by later improvements to security-auditor.md (see PR #485, which itself is queued for review). This branch is 141 commits behind main. Branch deleted. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated Component Improvement
Changes
Communication ProtocolandProgress trackingsections that referenced a non-existent "context manager". Replaced startup instructions with real tool-use instructions.Research Summary
The original component had structural problems that made it non-functional: it referenced a context manager API that does not exist, used JSON blocks to simulate inter-agent communication (which agents cannot actually do), and contained hardcoded fictional metrics. The body was organized as flat noun-phrase checklists with no executable guidance. Compliance frameworks referenced outdated versions. The component also lacked modern security domains (supply chain, containers, CI/CD, secrets management) that are now standard in security audits.
Validation
Automated review cycle by Component Improvement Loop
Summary by cubic
Reworked the security-auditor component to be actionable and aligned with real tool usage, replacing fictional protocols with a structured, modern audit workflow. This improves accuracy, updates frameworks, and adds concrete scanning patterns and domains.
Glob,Read, andGrep; removed hardcoded metrics.cli-tool/components/). No new components; nodocs/components.jsonregeneration needed. No new environment variables or secrets.Written for commit 1098924. Summary will update on new commits.