Skip to content

Commit 5e19493

Browse files
author
Dylan Huang
committed
Enhance CLI documentation generation by updating subparser help extraction. Introduce a method to hide suppressed commands from help output and ensure accurate help text is included for subparsers.
1 parent 3ff6f8c commit 5e19493

3 files changed

Lines changed: 139 additions & 9 deletions

File tree

‎docs/cli-reference.md‎

Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
# CLI Reference
2+
3+
**ep** - eval-protocol: Tools for evaluation and reward modeling
4+
5+
## Global Options
6+
7+
| Option | Type | Default | Required | Description |
8+
|--------|------|---------|----------|-------------|
9+
| `--verbose`, `-v` | | false | No | Enable verbose logging |
10+
| `--profile` | | - | No | Fireworks profile to use (reads ~/.fireworks/profiles/<name>/auth.ini and settings.ini) |
11+
| `--server` | | - | No | Fireworks API server hostname or URL (e.g., dev.api.fireworks.ai or https://dev.api.fireworks.ai) |
12+
13+
## Commands
14+
15+
### `ep logs`
16+
17+
Serve logs with file watching and real-time updates
18+
19+
| Option | Type | Default | Required | Description |
20+
|--------|------|---------|----------|-------------|
21+
| `--port` | int | `8000` | No | Port to bind to (default: 8000) |
22+
| `--debug` | | false | No | Enable debug mode |
23+
| `--disable-elasticsearch-setup` | | false | No | Disable Elasticsearch setup |
24+
| `--use-env-elasticsearch-config` | | false | No | Use env vars for Elasticsearch config (requires ELASTICSEARCH_URL, ELASTICSEARCH_API_KEY, ELASTICSEARCH_INDEX_NAME) |
25+
| `--use-fireworks` | | false | No | Force Fireworks tracing backend for logs UI (overrides env auto-detection) |
26+
| `--use-elasticsearch` | | false | No | Force Elasticsearch backend for logs UI (overrides env auto-detection) |
27+
28+
### `ep upload`
29+
30+
Scan for evaluation tests, select, and upload as Fireworks evaluators
31+
32+
| Option | Type | Default | Required | Description |
33+
|--------|------|---------|----------|-------------|
34+
| `--path` | | `.` | No | Path to search for evaluation tests (default: current directory) |
35+
| `--entry` | | - | No | Entrypoint of evaluation test to upload (module:function or path::function). For multiple, separate by commas. |
36+
| `--id` | | - | No | Evaluator ID to use (if multiple selections, a numeric suffix is appended) |
37+
| `--display-name` | | - | No | Display name for evaluator (defaults to ID) |
38+
| `--description` | | - | No | Description for evaluator |
39+
| `--force` | | false | No | Overwrite existing evaluator with the same ID |
40+
| `--yes`, `-y` | | false | No | Non-interactive: upload all discovered evaluation tests |
41+
| `--env-file` | | - | No | Path to .env file containing secrets to upload (default: .env in current directory) |
42+
43+
### `ep create`
44+
45+
Resource creation commands
46+
47+
#### `ep create rft`
48+
49+
Create a Reinforcement Fine-tuning Job on Fireworks
50+
51+
| Option | Type | Default | Required | Description |
52+
|--------|------|---------|----------|-------------|
53+
| `--evaluator` | | - | No | Evaluator ID or fully-qualified resource (accounts/{acct}/evaluators/{id}); if omitted, derive from local tests |
54+
| `--dataset` | | - | No | Use existing dataset (ID or resource 'accounts/{acct}/datasets/{id}') to skip local materialization |
55+
| `--dataset-jsonl` | | - | No | Path to JSONL to upload as a new Fireworks dataset |
56+
| `--dataset-builder` | | - | No | Explicit dataset builder spec (module::function or path::function) |
57+
| `--dataset-display-name` | | - | No | Display name for dataset on Fireworks (defaults to dataset id) |
58+
| `--base-model` | | - | No | Base model resource id |
59+
| `--warm-start-from` | | - | No | Addon model to warm start from |
60+
| `--output-model` | | - | No | Output model id (defaults from evaluator) |
61+
| `--epochs` | int | `1` | No | - |
62+
| `--batch-size` | int | `128000` | No | - |
63+
| `--learning-rate` | float | `3e-05` | No | - |
64+
| `--max-context-length` | int | `65536` | No | - |
65+
| `--lora-rank` | int | `16` | No | - |
66+
| `--gradient-accumulation-steps` | int | - | No | Number of gradient accumulation steps |
67+
| `--learning-rate-warmup-steps` | int | - | No | Number of LR warmup steps |
68+
| `--accelerator-count` | int | - | No | - |
69+
| `--region` | | - | No | Fireworks region enum value |
70+
| `--display-name` | | - | No | RFT job display name |
71+
| `--evaluation-dataset` | | - | No | Optional separate eval dataset id |
72+
| `--eval-auto-carveout` | | true | No | - |
73+
| `--no-eval-auto-carveout` | | true | No | - |
74+
| `--chunk-size` | int | `100` | No | Data chunk size for rollout batching |
75+
| `--temperature` | float | - | No | - |
76+
| `--top-p` | float | - | No | - |
77+
| `--top-k` | int | - | No | - |
78+
| `--max-output-tokens` | int | `32768` | No | - |
79+
| `--response-candidates-count` | int | `8` | No | - |
80+
| `--extra-body` | | - | No | JSON string for extra inference params |
81+
| `--mcp-server` | | - | No | The MCP server resource name to use for the reinforcement fine-tuning job. |
82+
| `--wandb-enabled` | | false | No | - |
83+
| `--wandb-project` | | - | No | - |
84+
| `--wandb-entity` | | - | No | - |
85+
| `--wandb-run-id` | | - | No | - |
86+
| `--wandb-api-key` | | - | No | - |
87+
| `--job-id` | | - | No | Specify an explicit RFT job id |
88+
| `--yes`, `-y` | | false | No | Non-interactive mode |
89+
| `--dry-run` | | false | No | Print planned REST calls without sending |
90+
| `--force` | | false | No | Overwrite existing evaluator with the same ID |
91+
| `--skip-validation` | | false | No | Skip local dataset and evaluator validation before creating the RFT job |
92+
| `--ignore-docker` | | false | No | Ignore Dockerfile even if present; run pytest on host during evaluator validation |
93+
| `--docker-build-extra` | | `` | No | Extra flags to pass to 'docker build' when validating evaluator (quoted string, e.g. "--no-cache --pull --progress=plain") |
94+
| `--docker-run-extra` | | `` | No | Extra flags to pass to 'docker run' when validating evaluator (quoted string, e.g. "--env-file .env --memory=8g") |
95+
96+
### `ep local-test`
97+
98+
Select an evaluation test and run it locally. If a Dockerfile exists, build and run via Docker; otherwise run on host.
99+
100+
| Option | Type | Default | Required | Description |
101+
|--------|------|---------|----------|-------------|
102+
| `--entry` | | - | No | Entrypoint to run (path::function or path). If not provided, a selector will be shown (unless --yes). |
103+
| `--ignore-docker` | | false | No | Ignore Dockerfile even if present; run pytest on host |
104+
| `--yes`, `-y` | | false | No | Non-interactive: if multiple tests exist and no --entry, fails with guidance |
105+
| `--docker-build-extra` | | `` | No | Extra flags to pass to 'docker build' (quoted string, e.g. "--no-cache --pull --progress=plain") |
106+
| `--docker-run-extra` | | `` | No | Extra flags to pass to 'docker run' (quoted string, e.g. "--env-file .env --memory=8g") |

‎eval_protocol/cli.py‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -500,20 +500,34 @@ def _configure_parser(parser: argparse.ArgumentParser) -> argparse.ArgumentParse
500500
# )
501501

502502
# Hidden command: export-docs (for generating CLI reference documentation)
503-
export_docs_parser = subparsers.add_parser(
504-
"export-docs",
505-
help=argparse.SUPPRESS, # Hidden from help output
506-
)
503+
export_docs_parser = subparsers.add_parser("export-docs", help=argparse.SUPPRESS)
507504
export_docs_parser.add_argument(
508505
"--output",
509506
"-o",
510507
default="./docs/cli-reference.md",
511508
help="Output markdown file path (default: ./docs/cli-reference.md)",
512509
)
513510

511+
# Update metavar to only show visible commands (exclude those with SUPPRESS)
512+
_hide_suppressed_subparsers(parser)
513+
514514
return parser
515515

516516

517+
def _hide_suppressed_subparsers(parser: argparse.ArgumentParser) -> None:
518+
"""Update subparsers to exclude commands with help=SUPPRESS from help output."""
519+
for action in parser._actions:
520+
if isinstance(action, argparse._SubParsersAction):
521+
# Filter _choices_actions to only visible commands
522+
choices_actions = getattr(action, "_choices_actions", [])
523+
visible_actions = [a for a in choices_actions if a.help != argparse.SUPPRESS]
524+
action._choices_actions = visible_actions
525+
# Update metavar to match
526+
visible_names = [a.dest for a in visible_actions]
527+
if visible_names:
528+
action.metavar = "{" + ",".join(visible_names) + "}"
529+
530+
517531
def parse_args(args=None):
518532
"""Parse command line arguments."""
519533
parser = build_parser()

‎eval_protocol/cli_commands/export_docs.py‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,11 +13,12 @@
1313
logger = logging.getLogger(__name__)
1414

1515

16-
def _get_parser_info(parser: argparse.ArgumentParser) -> Dict:
16+
def _get_parser_info(parser: argparse.ArgumentParser, subparser_help: str = "") -> Dict:
1717
"""Extract information from an ArgumentParser."""
1818
info = {
1919
"prog": parser.prog,
2020
"description": parser.description or "",
21+
"help": subparser_help, # The help text from add_parser()
2122
"epilog": parser.epilog or "",
2223
"arguments": [],
2324
"subparsers": {},
@@ -26,9 +27,16 @@ def _get_parser_info(parser: argparse.ArgumentParser) -> Dict:
2627
# Extract arguments
2728
for action in parser._actions:
2829
if isinstance(action, argparse._SubParsersAction):
29-
# Handle subparsers
30+
# Handle subparsers - also extract the help text for each
3031
for name, subparser in action.choices.items():
31-
info["subparsers"][name] = _get_parser_info(subparser)
32+
# Get the help text from the subparser action's _parser_class
33+
subparser_help_text = ""
34+
if hasattr(action, "_choices_actions"):
35+
for choice_action in action._choices_actions:
36+
if choice_action.dest == name:
37+
subparser_help_text = choice_action.help or ""
38+
break
39+
info["subparsers"][name] = _get_parser_info(subparser, subparser_help_text)
3240
elif isinstance(action, argparse._HelpAction):
3341
# Skip help action, it's always present
3442
continue
@@ -100,8 +108,10 @@ def _generate_command_section(
100108
lines.append(f"{heading} `{full_command}`")
101109
lines.append("")
102110

103-
if info["description"]:
104-
lines.append(info["description"])
111+
# Use help text (from add_parser) or description (from ArgumentParser)
112+
description = info.get("help") or info.get("description") or ""
113+
if description and description != argparse.SUPPRESS:
114+
lines.append(description)
105115
lines.append("")
106116

107117
# Arguments table

0 commit comments

Comments
 (0)