feat: harness agnostic TerminalBench environment

[elisam0]

anyterminal_agent

Runs any Gym agent inside a Terminal Bench task container and evaluates the result by executing the task's tests/test.sh inside the same container. Works with hermes_agent, claude_code_agent, or any other compatible Gym agent.

Unlike anyswe_agent (which runs agent and eval in two concurrent containers), anyterminal_agent runs everything sequentially in one container: the agent completes its work, then test.sh runs and writes a binary reward (0.0 / 1.0) to /logs/verifier/reward.txt. The test directory is mounted read-only so the agent cannot tamper with the tests before evaluation.

Quickstart

1. Prepare the dataset — downloads tasks via Harbor and writes the input JSONL:

# Download tasks + build dataset + build SIFs (default)
python responses_api_agents/anyterminal_agent/prepare.py

# Skip SIF builds — Apptainer will pull docker:// images at runtime
python responses_api_agents/anyterminal_agent/prepare.py --no-build-sif

# Smoke test — first 5 tasks only
python responses_api_agents/anyterminal_agent/prepare.py --limit 5 --no-build-sif

Requires harbor on PATH. Tasks are cached at ~/.cache/harbor/tasks/terminal-bench/.

2. Start the environment:

ng_run "+config_paths=[responses_api_agents/anyterminal_agent/configs/anyterminal_hermes.yaml,responses_api_models/vllm_model/configs/vllm_model.yaml]"

3. Collect rollouts:

ng_collect_rollouts \
  +agent_name=anyterminal_hermes \
  +input_jsonl_fpath=responses_api_agents/anyterminal_agent/data/terminal_bench.jsonl \
  +output_jsonl_fpath=results/anyterminal_rollouts.jsonl

Each rollout row contains reward (0.0 or 1.0), the full agent trajectory, and mask_sample for timeouts or unreliable rewards.

Agent wiring

Swap the agent by changing three fields in the YAML (or overriding on the CLI):

agent_server_module: responses_api_agents.hermes_agent.app
agent_server_class: HermesAgent
agent_config_class: HermesAgentConfig
agent_kwargs:
  max_turns: 90
  terminal_backend: local
  terminal_timeout: 180

Agent dependencies install once at startup into a portable Python prefix mounted read-only inside the task container at /agent_deps_mount. Add setup_scripts/<agent_dir>_deps.sh to support a new agent.

Container images

Each Terminal Bench task specifies a Docker image in its task.toml. You can either:

• Pull at runtime (default, tb_sif_dir: null): Apptainer pulls docker://<image> on first use. Requires internet access on compute nodes.
• Pre-build SIFs (prepare.py --sif-dir PATH): Converts each image to a .sif file. Faster and works on air-gapped clusters.

Key config options

Field	Default	Description
tb_tasks_cache_dir	~/.cache/harbor/tasks	Where Harbor stores downloaded task definitions
tb_sif_dir	null	Pre-built SIF directory; null = pull docker:// at runtime
tb_agent_timeout	1800	Seconds before the agent is killed
tb_eval_timeout	300	Seconds for test.sh to complete
apptainer_memory_limit_mb	32768	Per-container memory cap
concurrency	8	Max concurrent tasks dispatched to Ray

Hermes agent changes

Updated HermesAgentConfig to expose Hermes 2.1 configuration fields (compression_enabled, compression_threshold, delegation_max_iterations, checkpoints_enabled) and generate the agent's YAML config dynamically via _build_config(). Also updated defaults: max_turns 30 → 90, terminal_timeout 60 → 180, temperature now defaults to None (model default).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[cmunley1]

/claude review

[claude]

SHIP WITH CARE

New anyterminal_agent environment that runs any Gym agent inside Terminal Bench task containers. The core design is sound — sequential agent→eval in a single Apptainer container, file-based reward communication, Ray-dispatched parallelism. Async patterns are correct (Ray futures awaited, no httpx, no ray.get() in async).

Three findings worth resolving before or shortly after merge:

1. RISK — body.metadata crash (anyterminal_agent/app.py:576)

dict(body.metadata) raises TypeError when metadata is None. Guard with dict(body.metadata or {}). See inline comment.

2. RISK — SIGTERM handler race (hermes_agent/app.py:288-305)

add_signal_handler / remove_signal_handler on the shared event loop is not concurrency-safe. Concurrent responses() calls overwrite each other's handlers. See inline comment for the race sequence and fix options.

3. RISK — Breaking default changes to HermesAgentConfig

max_turns 30→90, temperature 1.0→None, terminal_timeout 60→180, and compression_enabled flipped from hard-coded False to default True. These silently change behavior for all existing hermes_agent users. The compression change is the most impactful — it alters trajectories and eval scores. Consider keeping hermes_agent's existing defaults and putting the new values only in the anyterminal_hermes.yaml config. See inline comment.

4. NOTE — No tests

CLAUDE.md requires test coverage for new environments. There are no tests for anyterminal_agent. At minimum, unit tests for _instruction_from_input, _read_task_meta, reward-file parsing, and the mask_sample logic would catch regressions in the scoring path.

[cmunley1]

can you add 5 example input + rollouts? https://docs.nvidia.com/nemo/gym/main/contribute/environments/new-environment#4-generate-example-rollouts

[cmunley1]

looks good
i have used the older version with various harnesses with good results. will add results here in a day or two, but dont think its blocking. useful would be to reproduce reported TB scores from some leading models.

[cmunley1]

i wonder if this could cause a memory leak without cleanup

[elisam0]

Added atexit cleanup so the temp dir is removed on process exit. This is a disk resource under /tmp and /tmp is cleared on reboot anyway, so the practical impact is low.

[cmunley1]

Name                                            Stmts   Miss  Cover
-------------------------------------------------------------------
responses_api_agents/anyterminal_agent/app.py     348    226    35%
-------------------------------------------------------------------
TOTAL                                             348    226    35%
FAIL Required test coverage of 96.0% not reached. Total coverage: 35.06%

coverage seems a bit low. i dont think we need 96% here but would be nice to get a bit higher, e.g.

Name                                     Stmts   Miss  Cover
------------------------------------------------------------
responses_api_agents/swe_agents/app.py     955    237    75%
------------------------------------------------------------
TOTAL                                      955    237    75%

[cmunley1]

some tb2 avg@1 scores from testing this

gpt 5.4
hermes    34.8%
openclaw  18.0%
opencode  42.7%
pi        28.1%

opus 4.7
hermes    57.3%
openclaw  68.5%
opencode  42.7%
pi        61.8%