Skip to content

Commit 2da9142

Browse files
wprazuchritaneves
authored andcommitted
docs(config): document the Domain enum (#1205 friction #9 / FEP-1023) (#1633)
## What Adds a docstring to the `Domain` enum in `config_types.py` — a one-line description per value with a recognizable example benchmark, plus guidance to **pick the most specific domain** and use `other` only as a catch-all (addresses the [#600](#600) 'agent is too broad / no guidance' concern **without** redefining the taxonomy). ``` math — mathematical problem solving (e.g. AIME, MATH, GSM8K). coding — code generation, repair, or execution (e.g. SWE-bench, LiveCodeBench). agent — multi-step, tool-using tasks (e.g. tau2, workplace_assistant). Prefer a more specific value when it fits. ... (knowledge, instruction_following, long_context, safety, games, translation, e2e, rlhf, other) ``` ## Why Epic [#1205](#1205) friction 9 (naming confusion). Surfaces on IDE hover and when reading `config_types`; complements the inline `domain` comment added in #1597. ## Tests `test_domain_enum_documents_every_value` asserts every `Domain` value appears in the docstring, so a newly-added domain can't ship undocumented. --------- Signed-off-by: Wojciech Prazuch <wprazuch@nvidia.com> Signed-off-by: Rita Fernandes Neves <rfernandesne@nvidia.com>
1 parent a567232 commit 2da9142

2 files changed

Lines changed: 31 additions & 0 deletions

File tree

nemo_gym/config_types.py

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -406,6 +406,27 @@ class BenchmarkDatasetConfig(BaseModel):
406406

407407

408408
class Domain(str, Enum):
409+
"""The capability a resources server primarily evaluates or trains.
410+
411+
Pick the single domain that best fits the task. If several seem to apply, choose the most
412+
specific one (e.g. prefer `math` or `coding` over `agent`); use `other` only when none
413+
of the specific values fit. The values:
414+
415+
- `math` — mathematical problem solving (e.g. AIME, MATH, GSM8K).
416+
- `coding` — code generation, repair, or execution (e.g. SWE-bench, LiveCodeBench).
417+
- `agent` — multi-step, tool-using / environment-interacting tasks (e.g. tau2,
418+
workplace_assistant). Prefer a more specific value when the task is really math/coding/etc.
419+
- `knowledge` — factual or domain-knowledge question answering (e.g. GPQA, MMLU).
420+
- `instruction_following` — adherence to explicit formatting/constraints (e.g. IFEval).
421+
- `long_context` — reasoning over long inputs (e.g. RULER, long-document QA).
422+
- `safety` — refusing harmful content / resisting jailbreaks & prompt injection.
423+
- `games` — interactive game environments (e.g. blackjack, tetris).
424+
- `translation` — machine translation quality (e.g. WMT).
425+
- `e2e` — end-to-end pipelines spanning multiple capabilities at once.
426+
- `rlhf` — preference / reward-model / LLM-as-judge evaluations.
427+
- `other` — catch-all when no specific domain above applies.
428+
"""
429+
409430
MATH = "math"
410431
CODING = "coding"
411432
AGENT = "agent"

tests/unit_tests/test_config_types_help.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -197,3 +197,13 @@ class TestConfig(BaseNeMoGymCLIConfig):
197197
# Validate parameters section
198198
assert "param" in parsed["parameters"]
199199
assert "[required]" in parsed["parameters"]["param"]
200+
201+
202+
def test_domain_enum_documents_every_value() -> None:
203+
# Each Domain value must be documented in the enum docstring, so newly-added domains can't
204+
# ship without guidance (the docstring is the source of truth surfaced in IDEs / on hover).
205+
from nemo_gym.config_types import Domain
206+
207+
doc = Domain.__doc__ or ""
208+
for member in Domain:
209+
assert f"`{member.value}`" in doc, f"Domain.{member.name} ('{member.value}') is undocumented"

0 commit comments

Comments
 (0)