feat(entrypoints): job scheduler improvements for the HTTP eval server

[Luodian]

Summary

• Fix a latent crash in the HTTP eval server's job scheduler: single-GPU jobs went through accelerate launch --num_processes 1, which uses accelerate's simple_launcher and never sets WORLD_SIZE/RANK/MASTER_ADDR — lmms_eval's unconditional Accelerator() construction then crashes with ValueError: ... WORLD_SIZE expected, but not set. Single-GPU jobs now invoke python -m lmms_eval directly; multi-GPU jobs go through accelerate.commands.launch --multi_gpu.
• Fix a second crash class: the scheduler used to drain the eval subprocess's stdout via asyncio.StreamReader.readline() inside the server's event loop, which raises ValueError("Separator is not found, and chunk exceed the limit") as soon as a single chunk (long tqdm line, traceback, JSON dump) exceeds asyncio's 64 KiB separator limit. Output is now redirected straight to a per-job log file at the OS level, and failures raise a RuntimeError with the log tail attached for fast debugging.
• Scrub Slurm/torch.distributed env vars (SLURM_PROCID, RANK, WORLD_SIZE, MASTER_ADDR, etc.) from the eval subprocess's environment, so Accelerate doesn't mistake the child for a Slurm-launched distributed task when the HTTP server itself happens to run under a distributed launcher.
• Split command-building, subprocess execution, and log-tailing into testable static methods (_build_eval_cmd, _run_subprocess_with_log, _tail_log).

Test plan

• python3 -m py_compile on all lmms_eval/entrypoints/*.py
• Import-check lmms_eval.entrypoints.{protocol,client,job_scheduler,server_args,http_server} and the package __init__
• ruff check on changed files (clean)
• black --check / isort --check on changed files (clean)
• pytest test/entrypoints/ -v — 15/15 passed (includes a new test asserting the subprocess starts in its own session)

Review pass (2026-07-06)

Self-review follow-up commit:

• Whole-process-group kill on cancel: the eval subprocess is now spawned with start_new_session=True and cancellation kills the group (os.killpg, ProcessLookupError-guarded, plain kill() fallback). Previously only the direct child was SIGKILLed, so on the multi-GPU path the accelerate launch wrapper's GPU worker grandchildren were orphaned holding GPU memory.
• Dropped a speculative _iter_stream_lines helper (and its tests) that had no production consumer — it was pre-landed for a future SSE streaming endpoint and can return with that feature.