[BUG] [eval] --max-samples 10 and then without, you get the cached data from the previous run.

[NathanHB]

Describe the bug

A clear and concise description of what the bug is.

To Reproduce

Please provide all the steps needed to reproduce the behavior, or provide a minimal working example if needed. We will ignore issues missing this section.

Expected behavior

A clear and concise description of what you expected to happen.

Version info

Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.

[shipbehaves]

had this open while tracing the cache wrapper. root cause is that the cache key has no notion of how many samples a run requested.

the path is cache_dir / task_name / task_hash / sampling_method.parquet (get_cache_path), and task_hash (_get_task_hash) is a hash of the sorted task-config strings only. neither carries max_samples. samples are then keyed per doc.id inside that file.

the reason it's deterministic and not flaky: get_docs does rnd.seed(42); rnd.shuffle(eval_docs); eval_docs[:n_samples], and doc.id = str(ix) is the pre-shuffle index. fixed seed means --max-samples 10 and a full run produce the exact same first 10 ids. so on the second run get_samples_to_process_and_cache sees those 10 ids already in existing_indices, skips them, and only processes the rest. nothing signals that the cached set came from a truncated run, so the 10 truncated rows are served back as-is.

two fix directions:

• fold a run-scope marker (sample count, or a per-invocation run id) into the key, so a different requested size lands in a different cache slot. simple but trades away the cross-run reuse the cache is there to give you.
• keep the per-doc reuse but stop trusting a strict subset: in get_samples_to_process_and_cache, only treat a task as fully cached when the cached id set covers the requested docs; otherwise reprocess the missing ones. that keeps reuse and fixes the truncation case, since the full run's extra ids are simply not cached yet.

second option seems closer to the intent of per-doc caching. happy to put up a PR if you want it.

[NathanHB]

Hey @shipbehaves ! Feel free to open a PR for this :)

[shipbehaves]

Thanks @NathanHB! Before opening a PR I re-checked the scenario on current main, and good news: the reported symptom no longer reproduces. Verified at the unit level (a subset run then a full run against the same cache returns the full set, reusing the cached docs and reprocessing only the missing ones) and end to end (lighteval accelerate on arc:challenge with --max-samples 10, then without it, sharing the cache: the first run cached exactly 10 samples, the second returned all 1172).

Since the behavior is already correct, I opened #1291 with a regression test that locks it in so this cannot silently come back.

One thing the end-to-end check surfaced, might deserve its own issue: the two runs landed in different task-hash cache slots. The task hash is built from LightevalTaskConfig.__str__(lite=True), which includes the solver's function repr with a memory address, so the hash changes every process and cache slots are never shared across CLI invocations for tasks with a solver. That quietly disables cross-run cache reuse (and incidentally is the second reason the original symptom cannot occur across two invocations today). Happy to open a small PR to stabilize the hash input if you want it.