What
Add support for VLM evaluation by integrating trusted lmms-eval harness. Direct multimodal analogue to current text-only evaluation using lm-eval. Provides reproducible and comparable numbers across wide variety of benchmarks without building or maintaining our own harness.
There is currently no support for evaluation of VLMs.
Scope
Touches:
pyproject.toml — new [project.entry-points."lmms_eval.models"] entry (metadata only; no
new runtime dependency — lmms-eval is imported lazily).kempnerforge/data/vlm_dataset.py – promote _pil_to_tensor -> public pil_to_tensor and extract small build_tokenizer helper.tests/unit/test_vlm_dataset.py – Change _pil_to_tensor calls to pil_to_tensor
New:
kempnerforge/eval/vlm/ — new subpackage: the KempnerForgeVLM(lmms) chat adapter + a
ModelManifest, with loader/preprocessing/generation as helpers.scripts/vlm_eval_harness.py — new CLI.docs/how-to/run-vlm-evaluation.md (+ toctree wiring).tests/unit/eval/vlm/ (CPU) + a gated 1-GPU integration test.
No new JobConfig fields anticipated. Task selection and runtime options are CLI.
Backward compatibility
Fully backward compatible. New, parallel subsystem.
What
Add support for VLM evaluation by integrating trusted
lmms-evalharness. Direct multimodal analogue to current text-only evaluation usinglm-eval. Provides reproducible and comparable numbers across wide variety of benchmarks without building or maintaining our own harness.There is currently no support for evaluation of VLMs.
Scope
Touches:
pyproject.toml— new[project.entry-points."lmms_eval.models"]entry (metadata only; nonew runtime dependency —
lmms-evalis imported lazily).kempnerforge/data/vlm_dataset.py– promote_pil_to_tensor-> publicpil_to_tensorand extract smallbuild_tokenizerhelper.tests/unit/test_vlm_dataset.py– Change_pil_to_tensorcalls topil_to_tensorNew:
kempnerforge/eval/vlm/— new subpackage: theKempnerForgeVLM(lmms)chat adapter + aModelManifest, with loader/preprocessing/generation as helpers.scripts/vlm_eval_harness.py— new CLI.docs/how-to/run-vlm-evaluation.md(+toctreewiring).tests/unit/eval/vlm/(CPU) + a gated 1-GPU integration test.No new
JobConfigfields anticipated. Task selection and runtime options are CLI.Backward compatibility
Fully backward compatible. New, parallel subsystem.