Skip to content

Commit 8af748b

Browse files
sameehjsameehj
committed
advisory: tests, CSAF mandatory-test gate, and CI
Add scripts/test_gen_advisory.py (record->model logic, CSAF semantic invariants, reproducibility, batch/default-tree resolution, fail-loud behaviour) and a CSAF 2.0 conformance gate (scripts/csaf_validate.mjs) running the strict schema plus all mandatory tests via @secvisogram/csaf-validator-lib. advisory.yml runs unit, schema (CycloneDX 1.6 strict + overlay JSON Schema + reproducibility) and CSAF mandatory-test jobs. Both the schema and CSAF jobs also exercise the zero-argument default/batch path against the canonical advisories/ tree, proving `make advisory` and the script are interchangeable. CVE-2026-5999 is a synthetic CVSS v3.1 fixture that exercises the CSAF scores[] path the v4-only records do not. Signed-off-by: Sameeh Jubran <sameeh@wolfssl.com>
1 parent c60c2d0 commit 8af748b

8 files changed

Lines changed: 1336 additions & 0 deletions

File tree

‎.github/workflows/advisory.yml‎

Lines changed: 217 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
name: Advisory Tests
2+
3+
# START OF COMMON SECTION
4+
on:
5+
push:
6+
branches: [ 'master', 'main', 'release/**' ]
7+
pull_request:
8+
branches: [ '*' ]
9+
10+
# Defence-in-depth: this workflow only reads the tree and validates generated
11+
# advisories (no API writes, no git push, no release upload), so pin the token
12+
# to read-only per GitHub's supply-chain hardening guidance.
13+
permissions:
14+
contents: read
15+
16+
concurrency:
17+
group: ${{ github.workflow }}-${{ github.ref }}
18+
cancel-in-progress: true
19+
# END OF COMMON SECTION
20+
21+
jobs:
22+
# Tier 1 - pure-Python unit + semantic tests for scripts/gen-advisory.
23+
# No build, no pip deps. Runs in seconds and is the cheapest gate for the
24+
# record->model logic and the CSAF semantic invariants (every product_id
25+
# defined/used, no contradicting status, flags only on not-affected
26+
# products, no cvss_v4 in CSAF 2.0 scores, canonical CWE names, ...).
27+
unit:
28+
name: gen-advisory unit tests
29+
if: github.repository_owner == 'wolfssl'
30+
runs-on: ubuntu-24.04
31+
timeout-minutes: 5
32+
steps:
33+
- uses: actions/checkout@v4
34+
35+
- name: Syntax check
36+
run: python3 -m py_compile scripts/gen-advisory
37+
38+
- name: Unit tests
39+
run: python3 -W error::ResourceWarning -m unittest scripts/test_gen_advisory.py -v
40+
41+
# Tier 2 - format-level validation: generate per-CVE and bundled advisories
42+
# from the committed CVE fixtures + example overlay, then validate the
43+
# CycloneDX VEX against the 1.6 strict schema (same validator the SBOM
44+
# workflow uses) and the VEX overlay against its JSON Schema. Also pins
45+
# SOURCE_DATE_EPOCH reproducibility for both emitters.
46+
schema:
47+
name: advisory schema validation
48+
if: github.repository_owner == 'wolfssl'
49+
runs-on: ubuntu-24.04
50+
needs: unit
51+
timeout-minutes: 10
52+
steps:
53+
- uses: actions/checkout@v4
54+
55+
- name: Install validators
56+
# cyclonedx-bom provides the CycloneDX 1.6 strict JSON validator (same
57+
# pin as .github/workflows/sbom.yml); jsonschema validates the VEX
58+
# overlay against scripts/advisory-vex-overlay.schema.json. Pinned so
59+
# a validator release cannot silently change what "valid" means.
60+
run: |
61+
python3 -m pip install --user --upgrade pip
62+
python3 -m pip install --user 'cyclonedx-bom==7.*' 'jsonschema==4.*'
63+
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
64+
65+
- name: Overlay validates against its JSON Schema
66+
run: |
67+
python3 - <<'PY'
68+
import json, jsonschema
69+
schema = json.load(open('scripts/advisory-vex-overlay.schema.json'))
70+
overlay = json.load(open('scripts/advisory-vex-overlay.example.json'))
71+
jsonschema.Draft202012Validator.check_schema(schema)
72+
jsonschema.Draft202012Validator(schema).validate(overlay)
73+
print('OK: example overlay matches advisory-vex-overlay.schema.json')
74+
PY
75+
76+
- name: Generate advisories (per-CVE + bundled)
77+
# Mirrors how a release would be cut: one document per CVE, plus a
78+
# bundled per-release advisory carrying both. SOURCE_DATE_EPOCH makes
79+
# the run deterministic for the reproducibility check below.
80+
run: |
81+
mkdir -p /tmp/adv
82+
for id in CVE-2026-5501 CVE-2026-5778 CVE-2026-5999; do
83+
SOURCE_DATE_EPOCH=1700000000 \
84+
python3 scripts/gen-advisory \
85+
--cve-record "scripts/testdata/$id.json" \
86+
--vex-overlay scripts/advisory-vex-overlay.example.json \
87+
--csaf-out "/tmp/adv/$id.csaf.json" \
88+
--cdx-vex-out "/tmp/adv/$id.cdx.json"
89+
done
90+
SOURCE_DATE_EPOCH=1700000000 \
91+
python3 scripts/gen-advisory \
92+
--cve-record scripts/testdata/CVE-2026-5501.json \
93+
--cve-record scripts/testdata/CVE-2026-5778.json \
94+
--vex-overlay scripts/advisory-vex-overlay.example.json \
95+
--advisory-id wolfSSL-SA-5.9.1 \
96+
--csaf-out /tmp/adv/wolfSSL-SA-5.9.1.csaf.json \
97+
--cdx-vex-out /tmp/adv/wolfSSL-SA-5.9.1.cdx.json
98+
99+
- name: CycloneDX VEX validates per CycloneDX 1.6 strict schema
100+
run: |
101+
python3 - <<'PY'
102+
import glob, sys
103+
from cyclonedx.validation.json import JsonStrictValidator
104+
from cyclonedx.schema import SchemaVersion
105+
v = JsonStrictValidator(SchemaVersion.V1_6)
106+
paths = sorted(glob.glob('/tmp/adv/*.cdx.json'))
107+
assert paths, 'no CycloneDX VEX documents were generated'
108+
for p in paths:
109+
errs = v.validate_str(open(p).read())
110+
if errs:
111+
print(f'INVALID: {p}: {errs}', file=sys.stderr)
112+
sys.exit(1)
113+
print(f'OK: {p}')
114+
PY
115+
116+
- name: Reproducibility - two runs are byte-identical
117+
run: |
118+
mkdir -p /tmp/adv-r2
119+
SOURCE_DATE_EPOCH=1700000000 \
120+
python3 scripts/gen-advisory \
121+
--cve-record scripts/testdata/CVE-2026-5501.json \
122+
--cve-record scripts/testdata/CVE-2026-5778.json \
123+
--vex-overlay scripts/advisory-vex-overlay.example.json \
124+
--advisory-id wolfSSL-SA-5.9.1 \
125+
--csaf-out /tmp/adv-r2/wolfSSL-SA-5.9.1.csaf.json \
126+
--cdx-vex-out /tmp/adv-r2/wolfSSL-SA-5.9.1.cdx.json
127+
diff /tmp/adv/wolfSSL-SA-5.9.1.csaf.json \
128+
/tmp/adv-r2/wolfSSL-SA-5.9.1.csaf.json
129+
diff /tmp/adv/wolfSSL-SA-5.9.1.cdx.json \
130+
/tmp/adv-r2/wolfSSL-SA-5.9.1.cdx.json
131+
132+
- name: Default/batch path matches `make advisory`
133+
# No record flags: gen-advisory falls back to the canonical
134+
# advisories/ tree (the exact inputs `make advisory` feeds it via
135+
# --records-dir/--vex-overlay), proving the script and the build target
136+
# are interchangeable and that the committed real records + overlay
137+
# generate and validate.
138+
run: |
139+
python3 scripts/gen-advisory --out-dir /tmp/adv-default
140+
python3 - <<'PY'
141+
import glob, sys
142+
from cyclonedx.validation.json import JsonStrictValidator
143+
from cyclonedx.schema import SchemaVersion
144+
v = JsonStrictValidator(SchemaVersion.V1_6)
145+
paths = sorted(glob.glob('/tmp/adv-default/*.cdx.json'))
146+
assert paths, 'batch mode produced no CycloneDX documents'
147+
for p in paths:
148+
errs = v.validate_str(open(p).read())
149+
if errs:
150+
print(f'INVALID: {p}: {errs}', file=sys.stderr)
151+
sys.exit(1)
152+
print(f'OK: {p}')
153+
PY
154+
155+
- name: Upload generated advisories
156+
if: always()
157+
uses: actions/upload-artifact@v4
158+
with:
159+
name: advisories-${{ github.sha }}
160+
path: /tmp/adv/*.json
161+
if-no-files-found: warn
162+
retention-days: 90
163+
164+
# Tier 2 - CSAF 2.0 conformance: the real gate. JSON-schema validity is
165+
# necessary but not sufficient; CSAF defines mandatory tests (section 6.1.*)
166+
# -- CVSS/vector consistency, contradicting product status, product_id
167+
# defined/used, tracking.version vs revision_history, CWE name match, ... --
168+
# that a bare schema pass accepts. scripts/csaf_validate.mjs runs the strict
169+
# 2.0 schema + all mandatory tests via the Secvisogram reference
170+
# implementation (bundles every schema incl. the first.org CVSS schemas, so
171+
# it is fully offline once installed).
172+
csaf-conformance:
173+
name: CSAF 2.0 mandatory tests
174+
if: github.repository_owner == 'wolfssl'
175+
runs-on: ubuntu-24.04
176+
needs: unit
177+
timeout-minutes: 10
178+
steps:
179+
- uses: actions/checkout@v4
180+
181+
- uses: actions/setup-node@v4
182+
with:
183+
node-version: '20'
184+
185+
- name: Install csaf-validator-lib (pinned)
186+
# Pinned: csaf-validator-lib implements the CSAF mandatory tests, and
187+
# an unpinned upgrade could change pass/fail semantics under us. The
188+
# bare `csaf-validator-lib` name on npm is an unrelated placeholder;
189+
# the reference implementation is the @secvisogram scope.
190+
run: npm install --no-save @secvisogram/csaf-validator-lib@2.0.25
191+
192+
- name: Generate CSAF advisories (per-CVE + bundled)
193+
run: |
194+
mkdir -p /tmp/adv
195+
for id in CVE-2026-5501 CVE-2026-5778 CVE-2026-5999; do
196+
python3 scripts/gen-advisory \
197+
--cve-record "scripts/testdata/$id.json" \
198+
--vex-overlay scripts/advisory-vex-overlay.example.json \
199+
--csaf-out "/tmp/adv/$id.csaf.json"
200+
done
201+
python3 scripts/gen-advisory \
202+
--cve-record scripts/testdata/CVE-2026-5501.json \
203+
--cve-record scripts/testdata/CVE-2026-5778.json \
204+
--vex-overlay scripts/advisory-vex-overlay.example.json \
205+
--advisory-id wolfSSL-SA-5.9.1 \
206+
--csaf-out /tmp/adv/wolfSSL-SA-5.9.1.csaf.json
207+
208+
- name: CSAF strict schema + mandatory tests
209+
run: node scripts/csaf_validate.mjs /tmp/adv/*.csaf.json
210+
211+
- name: CSAF default/batch path (canonical advisories/ tree)
212+
# Same conformance gate, but driven through the zero-argument default
213+
# path `make advisory` uses, against the committed real records +
214+
# advisories/vex-overlay.json.
215+
run: |
216+
python3 scripts/gen-advisory --out-dir /tmp/adv-default
217+
node scripts/csaf_validate.mjs /tmp/adv-default/*.csaf.json

‎scripts/csaf_validate.mjs‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
// CSAF 2.0 conformance gate for documents emitted by scripts/gen-advisory.
2+
//
3+
// JSON-schema validity is necessary but NOT sufficient for CSAF: the standard
4+
// defines a battery of *mandatory tests* (section 6.1.*) -- CVSS/vector
5+
// consistency, contradicting product status, product_id defined/used,
6+
// tracking.version vs revision_history, and so on -- that a bare schema pass
7+
// happily accepts. This runner uses the Secvisogram reference implementation
8+
// (@secvisogram/csaf-validator-lib) which bundles every schema (incl. the
9+
// first.org CVSS schemas) and implements those mandatory tests, so the check
10+
// is fully offline and reproducible once the pinned dependency is installed.
11+
//
12+
// Gate = the strict CSAF 2.0 schema test + all mandatory tests. Optional and
13+
// informative tests are reported as warnings only (they encode house-style
14+
// preferences, not conformance).
15+
//
16+
// Usage: node scripts/csaf_validate.mjs <doc.csaf.json> [<doc2.csaf.json> ...]
17+
// Exit 0 if every document passes the gate, 1 otherwise.
18+
19+
import { readFileSync } from 'node:fs'
20+
import validate from '@secvisogram/csaf-validator-lib/validate.js'
21+
import * as schemaTests from '@secvisogram/csaf-validator-lib/schemaTests.js'
22+
import * as mandatoryTests from '@secvisogram/csaf-validator-lib/mandatoryTests.js'
23+
import * as optionalTests from '@secvisogram/csaf-validator-lib/optionalTests.js'
24+
25+
const files = process.argv.slice(2)
26+
if (files.length === 0) {
27+
console.error('usage: node scripts/csaf_validate.mjs <doc.csaf.json> ...')
28+
process.exit(2)
29+
}
30+
31+
// The gate: strict 2.0 schema + every mandatory test.
32+
const gateTests = [schemaTests.csaf_2_0_strict, ...Object.values(mandatoryTests)]
33+
// Reported for visibility but non-fatal.
34+
const advisoryTests = [...Object.values(optionalTests)]
35+
36+
function summarize(testResults) {
37+
// testResults: [{ name, isValid, errors, warnings, infos }]
38+
const failed = []
39+
for (const t of testResults) {
40+
if (t.isValid === false || (t.errors && t.errors.length > 0)) {
41+
failed.push(t)
42+
}
43+
}
44+
return failed
45+
}
46+
47+
let anyInvalid = false
48+
49+
for (const file of files) {
50+
let doc
51+
try {
52+
doc = JSON.parse(readFileSync(file, 'utf8'))
53+
} catch (e) {
54+
console.error(`ERROR: cannot read/parse ${file}: ${e.message}`)
55+
anyInvalid = true
56+
continue
57+
}
58+
59+
const gate = await validate(gateTests, doc)
60+
const advisory = await validate(advisoryTests, doc)
61+
62+
if (gate.isValid) {
63+
console.log(`OK ${file} (strict schema + ${Object.keys(mandatoryTests).length} mandatory tests)`)
64+
} else {
65+
anyInvalid = true
66+
console.error(`FAIL ${file}`)
67+
for (const t of summarize(gate.tests)) {
68+
for (const err of t.errors || []) {
69+
console.error(` [${t.name}] ${err.instancePath || '/'}: ${err.message}`)
70+
}
71+
}
72+
}
73+
74+
// Surface optional-test warnings without failing the build.
75+
const optWarn = summarize(advisory.tests)
76+
for (const t of optWarn) {
77+
for (const err of t.errors || []) {
78+
console.warn(` warn ${file} [${t.name}] ${err.instancePath || '/'}: ${err.message}`)
79+
}
80+
}
81+
}
82+
83+
process.exit(anyInvalid ? 1 : 0)

‎scripts/include.am‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -193,3 +193,13 @@ EXTRA_DIST += scripts/gen-advisory \
193193
scripts/cwe-names.json \
194194
scripts/advisory-vex-overlay.schema.json \
195195
scripts/advisory-vex-overlay.example.json
196+
197+
# Advisory regression suite + CSAF 2.0 conformance gate, with the frozen CVE
198+
# fixtures they run against. Shipped so a downstream consumer / CRA reviewer
199+
# can re-run the advisory tests from a release tarball.
200+
EXTRA_DIST += scripts/csaf_validate.mjs \
201+
scripts/test_gen_advisory.py \
202+
scripts/testdata/README.md \
203+
scripts/testdata/CVE-2026-5501.json \
204+
scripts/testdata/CVE-2026-5778.json \
205+
scripts/testdata/CVE-2026-5999.json

0 commit comments

Comments
 (0)