Skip to content

Commit 68c8b73

Browse files
jarvis9443jarvis9443
committed
feat(fuzz): mutation fuzzer with schema-conformance oracle
A small mutation-based fuzzer that runs the validator against AST-mutated copies of real-world OpenAPI specs and checks two oracles: 1. No crashes (validate_request must not throw a Lua error). 2. Schema conformance — a request generated to satisfy an operation's schema must be accepted by the validator. This is the productionised form of the harness used during v1.0.3 QA. Locally it reproduces the path-extension Bug 1 against pre-fix v1.0.3 and the utf8_len(table) Bug 3 against unpatched jsonschema, and is clean against the current main + jsonschema main. Wired into CI: - fuzz.yml — runs on every PR / push to main, 120s budget. Fails the job on any crash or candidate false-negative; uploads fuzz/out/ as an artifact. - fuzz-nightly.yml — runs daily at 18:00 UTC, 600s budget. On failure uploads findings, then opens (or comments on) a fuzz-nightly tracking issue assigned to @jarvis9443. See fuzz/README.md for architecture, mutator list, and how to extend.
1 parent ff9db5c commit 68c8b73

8 files changed

Lines changed: 758 additions & 1 deletion

File tree

‎.github/workflows/fuzz-nightly.yml‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
name: fuzz-nightly
2+
3+
on:
4+
schedule:
5+
- cron: '0 18 * * *' # 18:00 UTC = 02:00 China next day
6+
workflow_dispatch:
7+
inputs:
8+
budget:
9+
description: 'Fuzz budget in seconds'
10+
required: false
11+
default: '600'
12+
13+
permissions:
14+
contents: read
15+
issues: write
16+
17+
jobs:
18+
fuzz:
19+
runs-on: ubuntu-22.04
20+
env:
21+
OPENRESTY_PREFIX: "/usr/local/openresty"
22+
FUZZ_BUDGET: ${{ github.event.inputs.budget || '600' }}
23+
24+
steps:
25+
- name: Check out code
26+
uses: actions/checkout@v4
27+
28+
- name: Install system dependencies
29+
run: |
30+
sudo apt-get update
31+
sudo apt-get install -y build-essential libncurses5-dev libreadline-dev libssl-dev perl lua5.1 liblua5.1-0-dev
32+
33+
- name: Install OpenResty
34+
run: |
35+
wget -qO - https://openresty.org/package/pubkey.gpg | sudo gpg --dearmor -o /usr/share/keyrings/openresty.gpg
36+
echo "deb [signed-by=/usr/share/keyrings/openresty.gpg] http://openresty.org/package/ubuntu $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/openresty.list
37+
sudo apt-get update
38+
sudo apt-get install -y openresty
39+
40+
- name: Install LuaRocks
41+
run: |
42+
curl -fsSL https://raw.githubusercontent.com/apache/apisix/master/utils/linux-install-luarocks.sh | sh
43+
44+
- name: Install Lua dependencies
45+
run: |
46+
sudo luarocks install jsonschema
47+
sudo luarocks install lua-resty-radixtree
48+
49+
- name: Run mutation fuzzer
50+
id: fuzz
51+
continue-on-error: true
52+
run: |
53+
export PATH=$OPENRESTY_PREFIX/nginx/sbin:$OPENRESTY_PREFIX/bin:$PATH
54+
make fuzz FUZZ_BUDGET=$FUZZ_BUDGET
55+
56+
- name: Upload findings
57+
if: steps.fuzz.outcome == 'failure'
58+
uses: actions/upload-artifact@v4
59+
with:
60+
name: fuzz-findings-${{ github.run_id }}
61+
path: fuzz/out/
62+
retention-days: 90
63+
64+
- name: Open / update tracking issue
65+
if: steps.fuzz.outcome == 'failure'
66+
env:
67+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
68+
run: |
69+
set -euo pipefail
70+
DATE=$(date -u +%Y-%m-%d)
71+
TITLE="Nightly fuzz failure: $DATE"
72+
RUN_URL="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
73+
SUMMARY=$(cat fuzz/out/summary.json 2>/dev/null || echo '{}')
74+
FIRST5=$(head -n 5 fuzz/out/crashes.jsonl 2>/dev/null || echo '(no crashes.jsonl found)')
75+
BODY=$(cat <<EOF
76+
The nightly fuzz run failed.
77+
78+
- Run: $RUN_URL
79+
- Summary: \`$SUMMARY\`
80+
- Findings artifact: \`fuzz-findings-${{ github.run_id }}\` (attached to the run, retained 90 days)
81+
82+
First 5 findings:
83+
\`\`\`
84+
$FIRST5
85+
\`\`\`
86+
87+
To reproduce locally:
88+
\`\`\`
89+
make fuzz FUZZ_BUDGET=$FUZZ_BUDGET
90+
\`\`\`
91+
EOF
92+
)
93+
94+
# De-dup: reuse any open issue with the fuzz-nightly label.
95+
existing=$(gh issue list --label fuzz-nightly --state open \
96+
--json number --jq '.[0].number' || echo "")
97+
if [ -n "$existing" ]; then
98+
gh issue comment "$existing" --body "$BODY"
99+
gh issue edit "$existing" --add-assignee jarvis9443 || true
100+
else
101+
# Ensure label exists (idempotent).
102+
gh label create fuzz-nightly --color FBCA04 \
103+
--description "Findings from the nightly fuzz job" 2>/dev/null || true
104+
gh issue create --title "$TITLE" \
105+
--label fuzz-nightly,bug \
106+
--assignee jarvis9443 \
107+
--body "$BODY"
108+
fi
109+
110+
- name: Fail the job if fuzz failed
111+
if: steps.fuzz.outcome == 'failure'
112+
run: exit 1

‎.github/workflows/fuzz.yml‎

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
name: fuzz
2+
3+
on:
4+
push:
5+
branches: [ main ]
6+
pull_request:
7+
branches: [ main ]
8+
9+
jobs:
10+
fuzz:
11+
runs-on: ubuntu-22.04
12+
env:
13+
OPENRESTY_PREFIX: "/usr/local/openresty"
14+
FUZZ_BUDGET: "120"
15+
16+
steps:
17+
- name: Check out code
18+
uses: actions/checkout@v4
19+
20+
- name: Install system dependencies
21+
run: |
22+
sudo apt-get update
23+
sudo apt-get install -y build-essential libncurses5-dev libreadline-dev libssl-dev perl lua5.1 liblua5.1-0-dev
24+
25+
- name: Install OpenResty
26+
run: |
27+
wget -qO - https://openresty.org/package/pubkey.gpg | sudo gpg --dearmor -o /usr/share/keyrings/openresty.gpg
28+
echo "deb [signed-by=/usr/share/keyrings/openresty.gpg] http://openresty.org/package/ubuntu $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/openresty.list
29+
sudo apt-get update
30+
sudo apt-get install -y openresty
31+
32+
- name: Install LuaRocks
33+
run: |
34+
curl -fsSL https://raw.githubusercontent.com/apache/apisix/master/utils/linux-install-luarocks.sh | sh
35+
36+
- name: Install Lua dependencies
37+
run: |
38+
sudo luarocks install jsonschema
39+
sudo luarocks install lua-resty-radixtree
40+
41+
- name: Run mutation fuzzer
42+
run: |
43+
export PATH=$OPENRESTY_PREFIX/nginx/sbin:$OPENRESTY_PREFIX/bin:$PATH
44+
make fuzz FUZZ_BUDGET=$FUZZ_BUDGET
45+
46+
- name: Upload findings
47+
if: failure()
48+
uses: actions/upload-artifact@v4
49+
with:
50+
name: fuzz-findings-${{ github.run_id }}
51+
path: fuzz/out/
52+
retention-days: 30

‎Makefile‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,11 @@ test-conformance:
4242
lint:
4343
luacheck -q lib/
4444

45+
### fuzz: Run mutation fuzzer (FUZZ_BUDGET seconds, default 60)
46+
FUZZ_BUDGET ?= 60
47+
fuzz:
48+
python3 fuzz/mutate_fuzz.py --budget $(FUZZ_BUDGET) --out fuzz/out
49+
4550
### clean: Remove build artifacts
4651
clean:
47-
rm -rf *.rock
52+
rm -rf *.rock fuzz/out

‎fuzz/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
out/
2+
out_*/

‎fuzz/README.md‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Mutation Fuzzer for `lua-resty-openapi-validator`
2+
3+
A small mutation fuzzer that runs the validator against AST-mutated copies of
4+
real-world OpenAPI specs and checks two oracles:
5+
6+
1. **No crashes**: `validate_request` must not throw a Lua error
7+
(caught with `pcall`).
8+
2. **Schema conformance**: a request **generated to satisfy** an operation's
9+
schema must be **accepted** by the validator. A rejection is a candidate
10+
false-negative bug.
11+
12+
The fuzzer is the productionised form of the harness used during
13+
[v1.0.3 QA](../../../qa/lua-resty-openapi-validator-v1.0.3.md). It reproduces
14+
the bugs that QA found (path-extension Bug 1 against the unfixed validator,
15+
`utf8_len(table)` Bug 3 against the unfixed jsonschema).
16+
17+
## Architecture
18+
19+
```
20+
mutate_fuzz.py (Python orchestrator)
21+
├─ pick a seed spec from fuzz/seeds/
22+
├─ apply N random mutations (mutators below)
23+
├─ generate schema-conforming positive requests
24+
└─ resty -e RUNNER_LUA (validator subprocess, one per round)
25+
└─ for each case: pcall(v:validate_request, req)
26+
└─ JSONL result on stdout: {phase, accepted, err}
27+
```
28+
29+
Mutators (`fuzz/mutate_fuzz.py`):
30+
31+
| name | what it does | targets |
32+
|---|---|---|
33+
| `path_extension` | append `.json` / `.csv` etc. to a random path | path-routing edge cases (Bug 1) |
34+
| `nullable_enum` | inject `null` into an enum + flip `nullable: true` | nullable-enum handling (Bug 2) |
35+
| `length_on_array` | move `maxLength` onto an `array`/`object` schema | type-inappropriate keywords (Bug 3) |
36+
| `param_style` | flip parameter `style`/`explode` | parameter parsing (Bug 4 family) |
37+
| `required_phantom` | add a non-existent property name to `required` | schema-validation edge cases |
38+
| `swap_scalar_type` | swap `type: integer` ↔ `type: string` | coercion paths |
39+
40+
Generator (`sample_value`): produces JSON values that match a JSON Schema
41+
fragment (string / integer / number / boolean / array / object / enum), with
42+
a depth limit. Path/query/header parameters that are `required: true` are
43+
filled in; the request body is sampled from the operation's
44+
`requestBody` schema if present.
45+
46+
## Run locally
47+
48+
```bash
49+
make fuzz # 60s budget
50+
make fuzz FUZZ_BUDGET=300 # 5 min
51+
python3 fuzz/mutate_fuzz.py --budget 60 --seed 7 # reproducible
52+
```
53+
54+
Output:
55+
56+
- `fuzz/out/crashes.jsonl` — one JSON object per finding
57+
- `fuzz/out/summary.json` — `{rounds, cases_run, elapsed_s, crash_count}`
58+
- exits non-zero on any crash or candidate false-negative (CI-friendly)
59+
60+
## Add a seed
61+
62+
Drop any OpenAPI 3.x spec into `fuzz/seeds/`. Smaller specs (50–100 ops)
63+
give more mutation rounds per second; very large specs (>500 ops) slow
64+
each round. Recommended size: 30 KB – 300 KB.
65+
66+
## Add a mutator
67+
68+
1. Add `def my_mutator(spec, rng): ...` near the other mutators.
69+
2. Append to `MUTATORS` list.
70+
3. Mutator must mutate `spec` **in place** and return a label string
71+
(or just its function name) for logging.
72+
73+
## Noise filter
74+
75+
`gen_cases` does not try to satisfy every JSON Schema construct — `oneOf` /
76+
`allOf` / `discriminator` / complex `pattern` are common in real specs but
77+
hard to satisfy generically. Errors mentioning these are filtered as
78+
generator artefacts, not validator bugs. The list lives near the bottom
79+
of `mutate_fuzz.py`. If the filter masks a real bug you find by other
80+
means, narrow / shrink it; if it lets through too much noise, widen it.
81+
82+
## CI
83+
84+
- **PR**: `.github/workflows/fuzz.yml` — 120s budget, fails the PR on any finding.
85+
- **Nightly**: `.github/workflows/fuzz-nightly.yml` — 600s budget, on
86+
failure uploads `fuzz/out/` as an artifact and opens (or comments on)
87+
a `fuzz-nightly` issue assigned to `@jarvis-api7`.

0 commit comments

Comments
 (0)