fix(jsonschema_bench): add per-sample validation timeout to prevent eval hangs

[hancheolcho]

Summary

jsonschema_bench can hang an entire evaluation indefinitely during scoring. schema_compliance() validates each prediction with an unbounded call to schema_conform_with_format_checker (jsonschema Draft202012Validator + FormatChecker). A single pathological schema/instance — e.g. a deeply-recursive $ref, or a format regex with catastrophic backtracking — spins the CPU forever, so the scoring phase never returns. Generation completes, but the process then sits at ~100% CPU with no progress and no error/traceback.

Fix

Wrap the per-sample validation in a wall-clock SIGALRM timeout (_run_with_timeout). On timeout the sample is treated as non-compliant (return False) — the same outcome the existing except Exception branch already produces for un-processable schemas — instead of hanging.

• Per-sample budget defaults to 40s, overridable via the JSONSCHEMA_BENCH_TIMEOUT_S environment variable.
• If SIGALRM is unavailable (non-main thread, or an unsupported platform), validation runs unguarded, so behavior never regresses to a crash.
• The change is confined to lm_eval/tasks/jsonschema_bench/metrics.py; samples that already validate within the budget are scored exactly as before.

Notes

• No change to scores for samples that validate within the time budget.
• Only the jsonschema_bench task is affected.

[CLAassistant]

All committers have signed the CLA.

[chuenchen309]

This is a real and well-scoped fix — an unbounded validation call in the scoring phase hanging the whole eval (generation done, then ~100% CPU with no traceback) is a nasty failure mode. The implementation is carefully defensive: guarding signal.signal against the non-main-thread ValueError / missing-SIGALRM AttributeError and running unguarded in that case means it degrades to current behavior rather than crashing, and the timeout result (return False) matches the existing except Exception path, so scores for in-budget samples are untouched.

One subtlety a reviewer might reasonably worry about: does a Python SIGALRM handler actually interrupt a catastrophic-backtracking format regex, given the match runs in the C re/sre engine? I tested it to be sure:

signal.signal(signal.SIGALRM, lambda *a: (_ for _ in ()).throw(TimeoutError()))
signal.setitimer(signal.ITIMER_REAL, 2)
re.match(r'(a+)+$', 'a'*40 + 'b')   # classic catastrophic backtracking
# -> TimeoutError raised at ~2.0s

It does — CPython's SRE engine polls the pending-signal flag during matching, so both motivating cases (deeply-recursive $ref at the Python level and a pathological format regex in C) are actually interrupted. Good.

The one behavioral caveat, which the code comment already acknowledges, is that when metrics run off the main thread the guard silently no-ops and the hang can still occur. That's an acceptable limitation for the common (main-thread) path and is documented, so I'd just make sure that's the intended contract. Nothing blocking from my read — the change is correct and confined to jsonschema_bench.