Skip to content

Commit 8d99783

Browse files
Oxygen56claude
andcommitted
feat: log skip reason when detectors are skipped in scan
Previously, when a detector had skip=True (e.g., AlwaysSkip detector), the harness silently continued with no indication to the user about why the detector was not run. Changes: - Add skip_reason attribute to Detector base class - Log the reason at INFO level when a detector is skipped - Default reason is 'disabled via config' when skip_reason is not set Fixes #1061 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Oxygen <1391083091@qq.com>
1 parent 6dbfae5 commit 8d99783

2 files changed

Lines changed: 13 additions & 2 deletions

File tree

garak/detectors/base.py

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,10 @@ class Detector(Configurable):
4242
"skip": False,
4343
}
4444

45+
skip_reason: str = None
46+
"""Human-readable reason why this detector was skipped. Set when ``skip`` is
47+
True to help users understand why a detector did not run on a probe."""
48+
4549
_run_params = {"seed"}
4650

4751
def _set_description(self):

garak/harnesses/base.py

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -196,9 +196,16 @@ def run(self, model, probes, detectors, evaluator, announce_probe=True) -> None:
196196
attempt_iterator = tqdm.tqdm(attempt_results, leave=False)
197197
detector_probe_name = d.detectorname.replace("garak.detectors.", "")
198198
attempt_iterator.set_description("detectors." + detector_probe_name)
199+
if d.skip:
200+
reason = getattr(d, "skip_reason", None) or "disabled via config"
201+
logging.info(
202+
"skipping detector %s on probe %s: %s",
203+
detector_probe_name,
204+
probe.probename,
205+
reason,
206+
)
207+
continue
199208
for attempt in attempt_iterator:
200-
if d.skip:
201-
continue
202209
attempt.detector_results[detector_probe_name] = list(
203210
d.detect(attempt)
204211
)

0 commit comments

Comments
 (0)