Unified run.spec selection grammar

[patriciapampanelli]

Summary

Introduces a single selection grammar — run.spec — for choosing probes and buffs, replacing the fragmented --probes / --probe_tags / --buffs flags and their config keys. One transport-agnostic engine parses both the CLI string (--run_spec) and the config form (run: { spec: { include: [...], exclude: [...] } }) into the same internal Spec, then resolves it to canonical plugin paths.

Resolves #1744

Grammar

• Plugin paths: probes.<module>[.<Class>], buffs.<module>[.<Class>], probes.*
• Empty selection: none (or probes.none) selects no probes — an explicit empty selection, distinct from an unspecified spec (which defaults to probes.*)
• Filters: tier:<N> (inclusive threshold — tiers 1..N), tag:<prefix> (membership)
• Exclusion: any selector prefixed with - (e.g. -probes.dan.DanInTheWild, -tier:3)
• Default (no probe plugin-path selector) → all active probes (probes.*), injected only at resolve time.

What changed

• New module garak/_spec.py — Selector / Spec / Resolution + parsing and resolution. _resolve_plugin_paths is the single shared resolution core. A none selector kind represents an explicit empty selection.

• garak/cli.py — adds --run_spec (underscore, consistent with garak's other flags such as --probe_tags / --list_probes); --probes/--probe_tags/--buffs are deprecated and map onto run.spec (with --run_spec winning on conflict).

• garak/command.py — print_buffs() gains an optional selected_buffs argument so --list_buffs matching --list_probes / --list_detectors. Previously the buff listing ignored any selection and always listed every buff; called without the argument, behaviour is unchanged (lists all).

• garak/_config.py — run.spec default; legacy config keys (plugins.probe_spec, plugins.buff_spec, run.probe_tags) map onto run.spec post-merge (mirroring model_type → target_type).

• Report snapshot now carries run.spec instead of the legacy keys.

• Downstream consumers (report_digest.py, aggregate_reports.py) read run.spec, with a fallback for reports predating this change.

• Bundled configs (bag.yaml, fast.json) migrated to run.spec.

• Docs updated (configurable.rst, usage.rst, faster.rst, new _spec.rst, cliref.rst regenerated from --help).

Selection of "none" vs unspecified (standardized)

A single rule governs empty vs default selection, consistently across CLI and config:

• none (e.g. --probes none or probe_spec: none) → explicit empty selection (probes.none); resolves to no probes. A run with no probes is a deliberate no-op ("No probes, nothing to do").
• vacuous values (omitted, empty, or auto) → unspecified; resolves to all active probes (probes.*).

This restores the legacy meaning of none (which selected nothing) while keeping the unified grammar's implicit default. The only behavior change vs the legacy flags is that --probes "" / --probes auto now default to all (they previously selected nothing); these are degenerate inputs and the new behavior is the more intuitive one.

Deprecation

--probes / --probe_tags / --buffs and plugins.probe_spec / plugins.buff_spec / run.probe_tags are deprecated (emit a deprecation notice). Detectors are out of scope here and keep their own plugins.detector_spec surface, to be migrated at #1789.

Examples

CLI

# all active probes (default — no selector needed)
garak -t test.Blank
# a whole probe family
garak -t test.Blank --run_spec "probes.dan"
# a single probe class
garak -t test.Blank --run_spec "probes.dan.DanInTheWild"
# family minus one class
garak -t test.Blank --run_spec "probes.dan, -probes.dan.DanInTheWild"
# tier threshold (tiers 1..2) filtered by tag, with one exclusion
garak -t test.Blank --run_spec "tier:2, tag:owasp:llm01, -probes.dan.AutoDAN"
# probes + a buff
garak -t test.Blank --run_spec "probes.dan, buffs.encoding"
# explicit empty selection (no probes)
garak -t test.Blank --run_spec "probes.none"

Config (YAML)

run:
  spec:
    include:
      - probes.dan
      - tag:owasp:llm01
      - buffs.encoding
    exclude:
      - probes.dan.DanInTheWild

Manual — selection resolution

• all active probes (no selector): garak --list_probes
• probe family: garak --list_probes --run_spec "probes.dan"
• explicit empty selection: garak --list_probes --run_spec "probes.none" → 0 probes (default still lists all)
• single class: garak --list_probes --run_spec "probes.dan.DanInTheWild"
• family minus one class: garak --list_probes --run_spec "probes.dan, -probes.dan.DanInTheWild"
• tier threshold (1..1): garak --list_probes --run_spec "tier:1"
• negative tier excludes exactly that tier (1..3 then drop tier 2 → {1,3}): garak --list_probes --run_spec "tier:3, -tier:2"
• tag filter: garak --list_probes --run_spec "tag:owasp:llm01"
• buff selection: garak --list_buffs --run_spec "buffs.encoding"

Manual — deprecation, conflict, errors

• --probes none is a deliberate no-op: garak -t test.Blank --probes none → "No probes, nothing to do" (does not run all probes)
• --probes deprecation maps to run.spec: garak --list_probes -p dan
• --probe_tags deprecation maps to tag:: garak --list_probes --probe_tags owasp:llm01
• --buffs deprecation maps to buffs.: garak --list_buffs -b encoding
• config probe_spec: none selects nothing (parity with CLI); auto/empty default to all
• --run_spec wins over legacy flag: garak --list_probes --run_spec "probes.dan" -p atkgen
• friendly error + exit 1 on invalid spec: garak --list_probes --run_spec "detectors.foo"
• aborts with empty-result reason: garak --list_probes --run_spec "probes.dan, -probes.dan"