Skip to content

vulnerability-writeup instructs a report filename the findings schema rejects #47

Description

@mariohercules

Summary

The findings schema requires a write-up's filename to equal its directory slug, but the vulnerability-writeup skill — the one whose sub-agent actually creates the file — instructs the opposite, twice, with two different example filenames.

The schema enforces the equality with a backreference:

"reportPath": {
"type": "string",
"pattern": "^findings/([a-z0-9][a-z0-9._-]*)/\\1\\.md$"
}

"reportPath": { "type": "string", "pattern": "^findings/([a-z0-9][a-z0-9._-]*)/\\1\\.md$" }

The orchestrating skills state the convention correctly — deep-security-scan/SKILL.md line 143 and security-diff-scan/SKILL.md line 106 both say findings/<slug>/<slug>.md, as do references/scan-artifacts.md and references/final-report.md.

But the write-up skill itself says:

skills/vulnerability-writeup/SKILL.md line 71:

create one directory containing an appropriately-named report file as markdown (e.g. freebsd-shm-uaf.md) […] Don't just use a nondescript file name like report.md.

skills/vulnerability-writeup/references/report-format.md line 5:

Choose an appropriate file name for the report that makes it clear what the vulnerability is at a glance. For example, freebsd-shm-ftruncate-uaf-lpe.md.

The two examples are different filenames, which reads as "any descriptive name". Neither passes the schema unless the slug happens to match, and the sub-agent prompt template in that skill passes only an output directory — never the filename constraint and never the lowercase/charset constraint.

Affected version and environment

  • Released package: @openai/codex-security@0.1.1
  • Confirmed on current main at f22d4a36f26d16287bcdfd707b369116e02a08c3
  • Python 3.12.13

Steps to reproduce

Validating candidate paths with the plugin's own validator (finalize_scan_contract.validate_against_schema) against the bundled findings schema:

ACCEPT  findings/unsafe-archive-extraction/unsafe-archive-extraction.md
REJECT  findings/unsafe-archive-extraction/freebsd-shm-uaf.md
        -> findings.schema.findings[0].writeup.reportPath: string does not match schema pattern
REJECT  findings/archive/freebsd-shm-ftruncate-uaf-lpe.md
        -> findings.schema.findings[0].writeup.reportPath: string does not match schema pattern

The two rejected names are exactly the examples the write-up skill and its reference give. Uppercase and spaces fail the same way, so a slug like Archive-Extraction is also rejected.

Expected behavior

The write-up skill should state the same convention as the orchestrators and the schema: the report file must be findings/<slug>/<slug>.md, lowercase, matching its directory.

Actual behavior

The skill that creates the file instructs a naming scheme the schema rejects, and explicitly discourages the only shape that would reliably satisfy it if the slug were reused.

Impact

The severity depends on which instruction the model follows, and I want to be accurate about that rather than overstate it. The orchestrating skills are the ones that set writeup.reportPath, and they carry the correct convention — so a run where the orchestrator both names and records the path consistently is fine.

The failure appears when the sub-agent follows its own skill and names the file descriptively. Then either the orchestrator records the real filename and finalization rejects the document, or it records the conventional path and reportPath points at a file that does not exist. The first case fails late: it happens at seal time, after all analysis work is done, so a long deep scan is lost at the final step.

Note that skills/security-scan/SKILL.md — the default standard scan — does not mention write-ups at all, so this is specific to the deep and diff scan paths.

Suggested direction

Replace the "appropriately-named" guidance in both vulnerability-writeup/SKILL.md and references/report-format.md with the findings/<slug>/<slug>.md rule, and include the filename in the sub-agent prompt template rather than only the output directory. Keeping the "not report.md" intent is compatible with this, since the slug is already descriptive.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions