feat(matcher): match sigil-prefixed positionals (fixes #361)

dig ns foo.bar @8.8.8.8 explained @8.8.8.8 with the wrong positional's
text: positionals were assigned purely by order, so the @-token landed
on whatever was next in line.

Option gains a prefix field — a literal sigil a token must start with
for that positional to claim it (dig's [@server]). Prefixed positionals
leave ordered consumption entirely: the matcher tries the prefix pool
first, then falls back to today's ordered/variadic logic, so ns and
foo.bar now correctly consume name and type.

The LLM prompt asks for the sigil from the synopsis; both sanitize
sites (response.py, postprocess.py) restrict it to a corpus-grounded
allowlist (@, +, :) so optional sub-components like ssh's
[user@]hostname can't knock a popular operand out of ordered matching.
Eval on the 12-page corpus: zero false-positive prefix emissions,
including ssh.

The e2e fixture db is rebuilt with llm:codex/gpt-5.4/medium (gpt-5.2
reproducibly dropped dig's positionals) in one parallel pass; snapshots
refreshed for the richer help-text markdown and new positional help
boxes, plus a new test pinning the issue-361 command end-to-end.

biome now skips .claude/ (agent worktrees with nested configs) and the
untracked parse-conformance corpus so make lint reflects repo code.

Author: idank

AGENTS.md

@@ -35,13 +35,13 @@ Use the LLM eval (`tests/evals/llm/llm_eval.py`) to compare before/after metrics
 git stash push -- explainshell/extraction/llm/
 
 # 2. Run on the old code
-python tests/evals/llm/llm_eval.py run --label baseline --model openai/gpt-5-mini --jobs 10 -d "baseline before <short summary of change>"
+python tests/evals/llm/llm_eval.py run --label baseline --model codex/gpt-5.4/medium --jobs 10 -d "baseline before <short summary of change>"
 
 # 3. Restore your changes
 git stash pop
 
 # 4. Run on the new code
-python tests/evals/llm/llm_eval.py run --label change --model openai/gpt-5-mini --jobs 10 -d "<short summary of change>"
+python tests/evals/llm/llm_eval.py run --label change --model codex/gpt-5.4/medium --jobs 10 -d "<short summary of change>"
 
 # 5. Compare the two run directories (oldest first)
 python tests/evals/llm/llm_eval.py compare tests/evals/llm/runs/<baseline-run> tests/evals/llm/runs/<change-run>
@@ -51,10 +51,10 @@ python tests/evals/llm/llm_eval.py compare tests/evals/llm/runs/<baseline-run> t
 
 ```bash
 # Run on the default corpus, parallelizing realtime calls
-python tests/evals/llm/llm_eval.py run --label smoke --model openai/gpt-5-mini --jobs 10
+python tests/evals/llm/llm_eval.py run --label smoke --model codex/gpt-5.4/medium --jobs 10
 
 # Run on specific files (overrides --corpus)
-python tests/evals/llm/llm_eval.py run --label probe --model openai/gpt-5-mini --jobs 10 path/to/file.1.gz
+python tests/evals/llm/llm_eval.py run --label probe --model codex/gpt-5.4/medium --jobs 10 path/to/file.1.gz
 
 # Use --batch <size> instead of --jobs to route through the provider's batch API
 # (cheaper, but minutes-to-hours of queue latency; pays off only on much larger corpora).
@@ -121,7 +121,7 @@ make ubuntu-archive UBUNTU_RELEASE=resolute
 make arch-archive
 
 # Process a man page into the database
-python -m explainshell.manager extract --mode llm:openai/gpt-5-mini /path/to/manpage.1.gz
+python -m explainshell.manager extract --mode llm:codex/gpt-5.4/medium /path/to/manpage.1.gz
 ```
 
 ## Project Structure
@@ -194,15 +194,16 @@ SQLite with two tables:
 - **mapping** - command name → manpage id lookup (many-to-one, with score for preference)
 
 Key classes (Pydantic models in models.py):
-- `Option` - text, short/long flag lists, has_argument, positional, nested_cmd
-- `ParsedManpage` - container with options/positionals properties and `find_option(flag)` lookup
+- `Option` - text, short/long flag lists, has_argument, positional, prefix (literal sigil a token must start with for a positional to claim it, e.g. `@` in dig's `[@server]`; restricted to the `OPTION_PREFIX_SIGILS` allowlist `@`/`+`/`:`), nested_cmd
+- `ParsedManpage` - container with options/positionals/prefixed_positionals properties and `find_option(flag)` lookup; `positionals` excludes prefix-bearing options, which are exposed via `prefixed_positionals` (name → (prefix, text))
 
 ### Command Matching (matcher.py)
 
 Uses bashlex AST visitor pattern:
 - `Matcher` inherits from `bashlex.ast.nodevisitor`
 - `visitcommand()` - looks up man page, handles multi-command (e.g., `git commit`)
 - `visitword()` - matches tokens to options (exact match, then fuzzy split for combined short flags like `-abc`)
+- Positional operands use two pools: prefixed positionals are claimed only by tokens starting with their sigil (e.g. `@8.8.8.8` → dig's `server`); remaining tokens consume the non-prefixed positionals in order, reusing the last one (variadic). A token whose sigil no positional declares falls through to ordered consumption; if all positionals are prefixed and none match, the token is unknown.
 - Produces `MatchResult(start, end, text, match)` where start/end are character positions in the original string
 
 ### E2E Tests

Makefile

@@ -8,7 +8,7 @@ E2E_DB := tests/e2e/e2e.db
 
 e2e-db:
 	rm -f $(E2E_DB)
-	python -m explainshell.manager --db $(E2E_DB) extract --mode llm:openai/gpt-5.2 $(E2E_MANPAGES_25)/tar.1.gz $(E2E_MANPAGES_25)/echo.1.gz $(E2E_MANPAGES_25)/grep.1.gz $(E2E_MANPAGES_24)/tar.1.gz $(E2E_MANPAGES_24)/echo.1.gz $(E2E_MANPAGES_24)/grep.1.gz $(E2E_MANPAGES_ARCH)/tar.1.gz $(E2E_MANPAGES_25)/git-rebase.1.gz
+	python -m explainshell.manager --db $(E2E_DB) extract --mode llm:codex/gpt-5.4/medium -j 9 $(E2E_MANPAGES_25)/tar.1.gz $(E2E_MANPAGES_25)/echo.1.gz $(E2E_MANPAGES_25)/grep.1.gz $(E2E_MANPAGES_24)/tar.1.gz $(E2E_MANPAGES_24)/echo.1.gz $(E2E_MANPAGES_24)/grep.1.gz $(E2E_MANPAGES_ARCH)/tar.1.gz $(E2E_MANPAGES_25)/git-rebase.1.gz $(E2E_MANPAGES_25)/dig.1.gz
 
 e2e:
 	@npx playwright --version >/dev/null 2>&1 || (echo "playwright is required. Install with: npm install && npx playwright install chromium"; exit 1)

biome.json

@@ -10,7 +10,9 @@
         "includes": [
             "explainshell/web/static/js/es.js",
             "tests/**/*.js",
-            "playwright.config.js"
+            "playwright.config.js",
+            "!.claude",
+            "!tests/parse_conformance"
         ]
     },
     "formatter": {

explainshell/diff.py

@@ -22,10 +22,10 @@
 )
 
 # Per-option fields to compare in diff mode.
-_OPT_FIELDS = ("has_argument", "positional", "nested_cmd", "text")
+_OPT_FIELDS = ("has_argument", "positional", "prefix", "nested_cmd", "text")
 
 # Fields where None and False should be treated as equivalent.
-_FALSY_EQUIVALENT = {"nested_cmd", "positional"}
+_FALSY_EQUIVALENT = {"nested_cmd", "positional", "prefix"}
 
 # ANSI color helpers.
 _RED = "\033[31m"

explainshell/extraction/llm/prompt.py

@@ -28,6 +28,11 @@
       "positional": null,        // name string for positional operands with no flags (e.g.
                                  // "FILE"). NEVER set this if the option has short or long
                                  // flags. Omit if null.
+      "prefix": null,            // literal sigil character the SYNOPSIS attaches to this
+                                 // positional operand (e.g. "@" in dig's "[@server]").
+                                 // Only valid together with "positional". Do NOT use for
+                                 // optional sub-components like "[user@]hostname" or for
+                                 // placeholder styling like "<FILE>". Omit if none.
       "nested_cmd": false,       // true only when the argument is itself a shell command
                                  // (e.g. find -exec CMD ;). Omit if false.
       "lines": [111, 115]        // [start, end] line range from the left margin, covering the

explainshell/extraction/llm/response.py

@@ -132,6 +132,9 @@ def normalize_option_fields(raw: dict) -> dict:
     - ``has_argument: null`` → ``False`` (8 positional-arg options)
     - ``has_argument`` list with int elements → stringified (hdparm --set-sector-size)
     - ``has_argument`` bare string (e.g. ``"1..99"``) → ``True`` (avrdude range params)
+    - sigil embedded in the positional name (``"positional": "@server"``) →
+      split into ``prefix`` + bare name; only for allowlisted sigil characters,
+      so placeholder styling like ``<FILE>`` is untouched
     """
     raw = dict(raw)  # shallow copy to avoid mutating caller's data
 
@@ -147,6 +150,17 @@ def normalize_option_fields(raw: dict) -> dict:
     if isinstance(ha, str):
         raw["has_argument"] = True
 
+    # Strip a leading sigil from the positional name into prefix.
+    pos = raw.get("positional")
+    if (
+        isinstance(pos, str)
+        and len(pos) > 1
+        and pos[0] in models.OPTION_PREFIX_SIGILS
+        and not raw.get("prefix")
+    ):
+        raw["prefix"] = pos[0]
+        raw["positional"] = pos[1:]
+
     return raw
 
 
@@ -156,21 +170,34 @@ def sanitize_option_fields(
     has_argument: bool | list[str],
     positional: str | None,
     nested_cmd: bool,
-) -> tuple[list[str], list[str], bool | list[str], str | None, bool]:
+    prefix: str | None = None,
+) -> tuple[list[str], list[str], bool | list[str], str | None, bool, str | None]:
     """Fix common LLM mistakes in option fields.
 
-    Returns (short, long, has_argument, positional, nested_cmd).
+    Returns (short, long, has_argument, positional, nested_cmd, prefix).
     """
     if positional and (short or long):
         logger.debug(
             "clearing positional=%r on flagged option %s/%s", positional, short, long
         )
         positional = None
 
+    if prefix and not positional:
+        logger.debug("clearing prefix=%r on non-positional option", prefix)
+        prefix = None
+
+    if prefix and prefix not in models.OPTION_PREFIX_SIGILS:
+        logger.debug(
+            "dropping prefix=%r on positional %r: not in sigil allowlist",
+            prefix,
+            positional,
+        )
+        prefix = None
+
     if nested_cmd and not has_argument:
         has_argument = True
 
-    return short, long, has_argument, positional, nested_cmd
+    return short, long, has_argument, positional, nested_cmd, prefix
 
 
 def llm_option_to_store_option(
@@ -185,6 +212,7 @@ def llm_option_to_store_option(
     long = raw.get("long") or []
     has_argument = raw.get("has_argument", False)
     positional = raw.get("positional") or None
+    prefix = raw.get("prefix") or None
     nested_cmd = bool(raw.get("nested_cmd", False))
 
     if not isinstance(short, list):
@@ -198,8 +226,8 @@ def llm_option_to_store_option(
     start, end = int(lines[0]), int(lines[1])
     text = extract_text_from_lines(original_lines, start, end)
 
-    short, long, has_argument, positional, nested_cmd = sanitize_option_fields(
-        short, long, has_argument, positional, nested_cmd
+    short, long, has_argument, positional, nested_cmd, prefix = sanitize_option_fields(
+        short, long, has_argument, positional, nested_cmd, prefix
     )
 
     return models.Option(
@@ -208,6 +236,7 @@ def llm_option_to_store_option(
         long=long,
         has_argument=has_argument,
         positional=positional,
+        prefix=prefix,
         nested_cmd=nested_cmd,
         meta={"lines": [start, end]},
     )

explainshell/extraction/postprocess.py

@@ -7,7 +7,7 @@
 from dataclasses import dataclass
 
 from explainshell.errors import ExtractionError, FailureReason
-from explainshell.models import Option
+from explainshell.models import OPTION_PREFIX_SIGILS, Option
 
 logger = logging.getLogger(__name__)
 
@@ -34,9 +34,12 @@ def sanitize_option(opt: Option) -> Option:
     """Fix universally-invalid field combinations.
 
     - positional + flags: clear positional
+    - prefix without positional: clear prefix
+    - prefix outside the sigil allowlist: clear prefix
     - nested_cmd without has_argument: set has_argument = True
     """
     positional = opt.positional
+    prefix = opt.prefix
     has_argument = opt.has_argument
     changed = False
 
@@ -50,21 +53,33 @@ def sanitize_option(opt: Option) -> Option:
         positional = None
         changed = True
 
+    if prefix and not positional:
+        logger.debug("clearing prefix=%r on non-positional option", prefix)
+        prefix = None
+        changed = True
+
+    if prefix and prefix not in OPTION_PREFIX_SIGILS:
+        logger.debug(
+            "dropping prefix=%r on positional %r: not in sigil allowlist",
+            prefix,
+            positional,
+        )
+        prefix = None
+        changed = True
+
     if opt.nested_cmd and not has_argument:
         has_argument = True
         changed = True
 
     if not changed:
         return opt
 
-    return Option(
-        text=opt.text,
-        short=opt.short,
-        long=opt.long,
-        has_argument=has_argument,
-        positional=positional,
-        nested_cmd=opt.nested_cmd,
-        meta=opt.meta,
+    return opt.model_copy(
+        update={
+            "has_argument": has_argument,
+            "positional": positional,
+            "prefix": prefix,
+        }
     )
 
 
@@ -73,15 +88,7 @@ def strip_trailing_blanks(opt: Option) -> Option:
     stripped = opt.text.rstrip("\n ")
     if stripped == opt.text:
         return opt
-    return Option(
-        text=stripped,
-        short=opt.short,
-        long=opt.long,
-        has_argument=opt.has_argument,
-        positional=opt.positional,
-        nested_cmd=opt.nested_cmd,
-        meta=opt.meta,
-    )
+    return opt.model_copy(update={"text": stripped})
 
 
 def _subset_has_cross_reference(subset: Option, extra_flags: frozenset[str]) -> bool:
@@ -179,14 +186,8 @@ def dedup_options(options: list[Option]) -> tuple[list[Option], int]:
             sup = options[best_j]
             sub = options[i]
             if len(sub.text) > len(sup.text):
-                options[best_j] = Option(
-                    text=sub.text,
-                    short=sup.short,
-                    long=sup.long,
-                    has_argument=sup.has_argument,
-                    positional=sup.positional,
-                    nested_cmd=sup.nested_cmd,
-                    meta=sub.meta,
+                options[best_j] = sup.model_copy(
+                    update={"text": sub.text, "meta": sub.meta}
                 )
             removed.add(i)
 

explainshell/manager.py

@@ -1090,6 +1090,8 @@ def show_manpage(
         click.echo(f"      has_argument: {opt.has_argument}")
         if opt.positional:
             click.echo(f"      positional: {opt.positional}")
+        if opt.prefix:
+            click.echo(f"      prefix: {opt.prefix}")
         if opt.nested_cmd:
             click.echo(f"      nested_cmd: {opt.nested_cmd}")
         desc = opt.text.strip()

explainshell/matcher.py

@@ -57,6 +57,7 @@ def _option_debug(option):
         "long": option.long,
         "has_argument": option.has_argument,
         "positional": option.positional,
+        "prefix": option.prefix,
         "nested_cmd": option.nested_cmd,
     }
 
@@ -667,7 +668,8 @@ def _visitword(node, word):
                             self.matches.extend(m)
                             return
 
-                    if self.man_page.positionals:
+                    prefixed = self.man_page.prefixed_positionals
+                    if self.man_page.positionals or prefixed:
                         if self.man_page.nested_cmd:
                             logger.info("manpage %r can nest commands", self.man_page)
                             if self.startcommand(
@@ -676,6 +678,33 @@ def _visitword(node, word):
                                 self._current_option = None
                                 return
 
+                        # Prefix pool first: a token carrying a declared
+                        # sigil claims that positional, regardless of order.
+                        # Multiple prefixed tokens may claim the same
+                        # positional; first declaration in document order
+                        # wins when several share a prefix.
+                        for k, (prefix, text) in prefixed.items():
+                            if word.startswith(prefix):
+                                logger.info(
+                                    "word %r starts with prefix %r, using %r",
+                                    word,
+                                    prefix,
+                                    k,
+                                )
+                                mr = MatchResult(
+                                    node.pos[0],
+                                    node.pos[1],
+                                    text,
+                                    None,
+                                    {
+                                        "kind": "argument",
+                                        "positional": k,
+                                        "prefix": prefix,
+                                    },
+                                )
+                                self.matches.append(mr)
+                                return
+
                         d = self.man_page.positionals
                         keys = list(d.keys())
                         group = self.group_stack[-1][1]
@@ -689,22 +718,27 @@ def _visitword(node, word):
                             # Ordered consumption: advance through positionals.
                             k = keys[group.positional_index]
                             group.positional_index += 1
-                        else:
+                        elif keys:
                             # All positionals consumed; reuse the last
                             # (variadic).
                             k = keys[-1]
-
-                        logger.info("got arguments, using %r", k)
-                        text = d[k]
-                        mr = MatchResult(
-                            node.pos[0],
-                            node.pos[1],
-                            text,
-                            None,
-                            {"kind": "argument", "positional": k},
-                        )
-                        self.matches.append(mr)
-                        return
+                        else:
+                            # Every positional is prefix-bearing and this
+                            # token carries none of the prefixes.
+                            k = None
+
+                        if k is not None:
+                            logger.info("got arguments, using %r", k)
+                            text = d[k]
+                            mr = MatchResult(
+                                node.pos[0],
+                                node.pos[1],
+                                text,
+                                None,
+                                {"kind": "argument", "positional": k},
+                            )
+                            self.matches.append(mr)
+                            return
 
                     # if all of that failed, we can't explain it so mark it unknown
                     self.matches.append(self.unknown(word, node.pos[0], node.pos[1]))