Skip to content

Commit ad8cf9a

Browse files
fix lm-eval version conflict in puzzletron requirements (#1257)
## Summary - lm-eval 0.4.10 has API changes that break `lm_eval_hf.py`, which monkey-patches `HFLM.create_from_arg_obj` - Pin `examples/puzzletron/requirements.txt` from `lm-eval==0.4.10` to `lm-eval==0.4.8` to match `examples/llm_eval/requirements.txt` - Add version compatibility warning to `lm_eval_hf.py` for early detection if versions drift again <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated the lm-eval dependency version to 0.4.8 in the Puzzletron example * Added compatibility version checking to the LM Eval example with appropriate notifications <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: jrausch <jrausch@nvidia.com> Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com> Co-authored-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
1 parent 3f41819 commit ad8cf9a

2 files changed

Lines changed: 8 additions & 1 deletion

File tree

examples/llm_eval/lm_eval_hf.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,15 @@
4040
import warnings
4141

4242
import datasets
43+
import lm_eval
4344
from lm_eval import utils
4445
from lm_eval.__main__ import cli_evaluate, parse_eval_args, setup_parser
46+
47+
if not lm_eval.__version__.startswith("0.4.8"):
48+
warnings.warn(
49+
f"lm_eval_hf.py is tested with lm-eval 0.4.8; found {lm_eval.__version__}. "
50+
"Later versions may have incompatible API changes."
51+
)
4552
from lm_eval.api.model import T
4653
from lm_eval.models.huggingface import HFLM
4754
from quantization_utils import quantize_model

examples/puzzletron/requirements.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
lm-eval==0.4.10
1+
lm-eval==0.4.8
22
math-verify
33
ray
44
# Likely works for transformers v5 also, but we need to test it

0 commit comments

Comments
 (0)