fix: core evaluator/cache/utils correctness fixes

[Luodian]

Core evaluator / cache / utils correctness fixes

A cluster of standalone correctness fixes for the eval pipeline, response cache, and utils. No new dependencies; all changes are additive or in-place bug fixes.

lmms_eval/caching/cache.py

• save_to_cache accumulated per-item results into serializable_item and appended to serializable_obj. Before, the loop mutated subitem.arguments but never appended anything, so the cache always pickled an empty list.
• Reviewer caveat: the request cache key does not fingerprint dataset contents, so with --cache_requests (opt-in, default off) a dataset changed in place under an unchanged task config will reuse stale requests. That key design predates this PR — flagging it because the cache now actually works and the caveat becomes reachable.

lmms_eval/caching/response_cache.py

• Task fingerprinting now recurses into nested task groups (_collect_fingerprints), so subtasks get real config fingerprints instead of silently falling back to empty strings.

lmms_eval/evaluator.py

• Secret redaction: sensitive keys are redacted from results["config"] before it is written out. Matching is suffix-based (*api_key / *_token / *_secret / *password, plus the exact-name set and key=value assignments inside model_args strings), so openai_api_key= and friends don't slip through.
• Padding aliasing fix: rank-padding built [pad_source] * n, aliasing a single Instance so all padding requests shared one resps/token_counts list. Now each padding slot is a distinct isolated clone (_clone_padding_request inside the loop).
• Multi-rank auto-init (torchrun path only): on distributed_executor_backend="torchrun", when world_size > 1 and torch.distributed is uninitialized, auto-init a gloo process group so backends that never call init_process_group don't crash evaluator collectives — guarded on MASTER_ADDR/MASTER_PORT being present (clear warning + skip otherwise, instead of raising). The default accelerate path is untouched: the evaluator's Accelerator() fallback creates its own state, so GPU collectives are never silently downgraded onto a pre-created gloo group.
• torchrun gather fix: all_gather_into_tensor now uses a 1-D input tensor and a world_size-sized buffer.
• Defensive skip: docs that land in a rank's shard with no requests are skipped instead of crashing at requests[0].doc.

lmms_eval/utils.py

• Robust get_git_commit_hash / get_git_branch_name: anchored at the repo root (works regardless of CWD), with env fallbacks (LMMS_EVAL_GIT_COMMIT / GIT_COMMIT / GITHUB_SHA, LMMS_EVAL_GIT_BRANCH / GIT_BRANCH / BRANCH_NAME / GITHUB_REF_NAME) and detached-HEAD handling.
• Local dataset builders (csv/json/parquet/text/pandas/arrow) now resolve relative dataset_kwargs.data_files paths to absolute against the YAML dir; HF-hub dataset paths are left untouched.
• py3.13 fix: re.split(..., maxsplit=maxsplit) (positional maxsplit is deprecated in 3.13).

Validation

• py_compile clean on all four files; pre-commit black+isort clean; ruff introduces no new findings.
• Ported helpers unit-exercised: git resolution + env fallbacks + detached HEAD, data_files resolution, secret redaction (prefixed keys + string-form assignments), and padding-clone isolation (N distinct objects for N slots).

Review pass (2026-07-06)

Self-review follow-up commit: gated the gloo auto-init to the torchrun path with the MASTER_ADDR/PORT guard (see above), moved the padding clone inside the loop so slots are truly isolated, switched redaction to suffix matching, and dropped an unused build_eval_output_dir helper that had no callers.

[Luodian]

@kcz358 @pufanyi This batch of 12 PRs (#1378–#1389) upstreams fixes and features from our fork — all now ready for review. Every PR went through a full review pass today; findings were fixed on the branches and each PR body has a "Review pass (2026-07-06)" section summarizing what changed. A map to make this reviewable:

Suggested merge order — plain numeric order (#1378 → #1389) avoids all conflicts. The actual constraints:

• fix: core evaluator/cache/utils correctness fixes #1378 before feat: auto-strip thinking blocks from generations before scoring #1380 (both touch evaluator.py)
• feat: add verifiers module - pluggable per-sample verification pipeline #1379 before refactor(tasks): migrate 10 benchmarks from llm_judge to the verifiers pipeline #1388 (stacked: refactor(tasks): migrate 10 benchmarks from llm_judge to the verifiers pipeline #1388 = 10 task migrations on top of the verifiers module; once feat: add verifiers module - pluggable per-sample verification pipeline #1379 lands, refactor(tasks): migrate 10 benchmarks from llm_judge to the verifiers pipeline #1388 gets rebased so the duplicated base commits drop out)
• feat(gemini): migrate to google-genai SDK, keep gemini_api alias #1382 before feat(loggers): add SwanLab logger as a wandb alternative (--swanlab_args) #1385 (both touch pyproject.toml) and before feat(models): add diffusers-based world-model generation backends (Wan 2.1/2.2, Cosmos, LTX-Video, MAGI-1, fal.ai) #1389 (both register models; feat(models): add diffusers-based world-model generation backends (Wan 2.1/2.2, Cosmos, LTX-Video, MAGI-1, fal.ai) #1389 inlines a minimal ParallelPlan that switches to feat(gemini): migrate to google-genai SDK, keep gemini_api alias #1382's model_utils/parallel.py — one-line dedupe at merge)
• fix(tasks): utils improvements across 7 benchmarks #1386 before refactor(tasks): migrate 10 benchmarks from llm_judge to the verifiers pipeline #1388 (both touch tasks/videoevalpro/utils.py; trivial re-merge either way)

What's in the batch:

PR	Area
#1378	core evaluator / cache / utils correctness fixes
#1379	new verifiers module (pluggable per-sample verification)
#1380	opt-in strip-thinking filter (TaskConfig.auto_strip_thinking)
#1381	llm_judge OpenAI providers: base URL, reasoning-model params, PIL images
#1382	gemini → google-genai SDK migration (video/audio upload, concurrency)
#1383	HTTP eval server job-scheduler fixes
#1384	remove Lance-based data paths (public repo = HF/local data only)
#1385	SwanLab logger (--swanlab_args), wandb alternative
#1386	utils fixes across 7 benchmarks
#1387	IntPhys2 + VANTAGE-Bench tasks (PhysBench pulled — see PR for why)
#1388	migrate 10 benchmarks from llm_judge to verifiers (stacked on #1379)
#1389	8 diffusers/API world-model generation backends

CI note — pre-existing, not from this batch: compare-task-input-boundary is currently red on every task-touching PR in the repo (including merged #1374–#1376). Root cause: lmms_eval/tasks/mmmu/utils.py builds the LLM-judge client at import time (module-level server = get_server(...)), and the CI runner has no OPENAI_API_KEY, so the snapshot tool crashes while importing mmmu_val from its spec — before any comparison happens. Same reason --tasks mmmu_val fails for anyone without an OpenAI key. Happy to send a small follow-up making that init lazy, which would turn the check green repo-wide.

Ping me here or on the individual PRs with anything.