Skip to content

fix: preserve OpenAI max_new_tokens#1318

Merged
kcz358 merged 1 commit intoEvolvingLMMs-Lab:mainfrom
Genmin:fix/openai-max-new-tokens
May 6, 2026
Merged

fix: preserve OpenAI max_new_tokens#1318
kcz358 merged 1 commit intoEvolvingLMMs-Lab:mainfrom
Genmin:fix/openai-max-new-tokens

Conversation

@Genmin
Copy link
Copy Markdown
Contributor

@Genmin Genmin commented May 2, 2026

Summary

  • remove the silent 4096 cap from OpenAI-compatible simple and chat payload builders
  • use the requested max_new_tokens for reasoning-model max_completion_tokens instead of hardcoding 5000
  • add payload-level regression coverage for simple, chat, and reasoning-model paths

Fixes #1317

Testing

  • uv run --with pytest python -m pytest test/models/test_openai.py -q
  • uv run --with ruff ruff check lmms_eval/models/chat/openai.py lmms_eval/models/simple/openai.py test/models/test_openai.py
  • git diff --check

@kcz358 kcz358 merged commit 55c7eba into EvolvingLMMs-Lab:main May 6, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

lmms-eval’s OpenAI chat adapter does not preserve lm-eval-harness text-task semantics

2 participants