Commit 2e9b3f5
authored
Skip invalid payloads in _scan_payload_dir instead of crashing (#1810)
## Summary
`Director._scan_payload_dir` (`garak/payloads.py`) catches
`JSONDecodeError`/`KeyError` for a payload file and logs `"Invalid
payload, skipping"`, but it does **not** `continue`. Execution falls
through to the code that reads `payload_types` and records the file:
```python
except (json.JSONDecodeError, KeyError) as exc:
msg = f"payload scan: Invalid payload, skipping: {payload_path}"
logging.debug(msg, exc_info=exc)
# raise garak.exception.PayloadFailure(msg) from exc
payload_name = payload_path.stem
payloads_found[payload_name] = {
"path": payload_path,
"types": payload_types, # <- read even when the except fired
}
```
Consequences:
- **Crash:** if the first/only scanned file is invalid, `payload_types`
was never bound → `UnboundLocalError`, which aborts the whole payload
inventory (`_refresh_payloads`).
- **Silent data corruption:** if a valid file was processed earlier in
the loop, the stale `payload_types` is reused, registering the invalid
file under the **previous** file's types.
This is reachable in normal use — `PAYLOAD_DIR` includes user /
`XDG_DATA_DIR` custom payload directories, so a single stray or
malformed `.json` there breaks payload loading.
## Fix
Add `continue` in the `except` block so invalid payloads are skipped,
matching the log message ("skipping") and the commented-out `raise`. One
line; no other behavior changes.
## Why this is not a duplicate
Checked open PRs (`_scan_payload_dir`, `payload scan`, `payloads` in
title) and open issues (`invalid payload`) on NVIDIA/garak — none
address this.
## Testing
- Added `tests/test_payloads.py::test_scan_payload_dir_skips_invalid`,
which scans a temp dir containing a malformed JSON, a JSON missing
`payload_types`, and one valid payload; asserts only the valid payload
is registered and no stale types leak. It **fails on `main`**
(`UnboundLocalError`) and **passes** with this change.
- Commands run:
- `python3 -m pytest
tests/test_payloads.py::test_scan_payload_dir_skips_invalid
tests/test_payloads.py::test_non_json_direct_load
tests/test_payloads.py::test_payload_Director` → 3 passed
- `python3 -m black --check garak/payloads.py tests/test_payloads.py` →
clean
## AI assistance
AI assistance (Claude) was used to help locate the bug, implement the
fix, and write the test. I reviewed every changed line and ran the tests
above.2 files changed
Lines changed: 18 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
153 | 153 | | |
154 | 154 | | |
155 | 155 | | |
| 156 | + | |
156 | 157 | | |
157 | 158 | | |
158 | 159 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
| |||
81 | 82 | | |
82 | 83 | | |
83 | 84 | | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
84 | 101 | | |
85 | 102 | | |
86 | 103 | | |
| |||
0 commit comments