Skip to content

feat: add SARIF 2.1.0 output for code scanning integrations#153

Open
omobolajiadeyan wants to merge 5 commits into
OWASP:mainfrom
omobolajiadeyan:feat/sarif-output
Open

feat: add SARIF 2.1.0 output for code scanning integrations#153
omobolajiadeyan wants to merge 5 commits into
OWASP:mainfrom
omobolajiadeyan:feat/sarif-output

Conversation

@omobolajiadeyan

Copy link
Copy Markdown
Contributor

Closes #82.

Adds --sarif-out <path> to agent-harness run so CI pipelines can surface
security regression failures through GitHub Code Scanning and other SARIF-aware
dashboards.

Changes

  • Add src/agent_harness/sarif.py — converts HarnessResult to SARIF 2.1.0
  • Add --sarif-out flag to the run subcommand in cli.py
  • Add two CLI tests: failing assertions emit SARIF results; passing runs emit empty results
  • Document --sarif-out and GitHub Code Scanning upload in docs/ci-github-actions.md

Design notes

  • Only fail and error assertions produce SARIF results. pass and not_run are omitted because SARIF results represent findings, not confirmations.
  • Rule IDs use the format agent-harness/<assertion-id>. Assertion IDs come from scenario definitions and are stable across runs.
  • fail maps to SARIF level error; error maps to warning.
  • locations is an empty array — there are no source locations because assertions evaluate runtime agent behaviour, not static code.
  • The SARIF document includes scenario_id, mode, and overall_result as run-level properties for downstream consumers.

Validation

python -m pytest tests/test_cli.py -k sarif
python -m ruff check src tests
python -m mypy src

Example CI step

- name: Run security regression
  run: |
    agent-harness run scenarios/goal_hijack/basic.yaml \
      --trace-file traces/basic.json \
      --sarif-out results/basic.sarif

- name: Upload SARIF
  uses: github/codeql-action/upload-sarif@v3
  with:
    sarif_file: results/basic.sarif
  if: always()

@omobolajiadeyan

Copy link
Copy Markdown
Contributor Author

Follow-up after the failed CI run: I fixed the Ruff import-order failure in commit b3fb2ef by sorting the new SARIF CLI import.\n\nCurrent PR status:\n- lint: passing\n- test: passing\n- Run security regression scenarios: passing\n\nLocal checks also passed for:\n- python -m ruff check src tests\n- python -m agent_harness.cli validate scenarios\n\nThe change remains scoped to SARIF 2.1.0 output and GitHub Code Scanning integration support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add SARIF output for code scanning integrations

1 participant