Skip to content

fix: count tracked files for test-file metric in /retro and /ship#2308

Open
joshRpowell wants to merge 1 commit into
garrytan:mainfrom
joshRpowell:fix/test-count-scan
Open

fix: count tracked files for test-file metric in /retro and /ship#2308
joshRpowell wants to merge 1 commit into
garrytan:mainfrom
joshRpowell:fix/test-count-scan

Conversation

@joshRpowell

Copy link
Copy Markdown

Fixes #2307.

Problem

The test-file count used by /retro and /ship scans the working tree with find, so untracked build output is counted as test files. On a Rails repo with bundled gems it over-reports by 37x.

Measured on a Rails 8 app with 17 real test files:

$ find . -name '*.test.*' -o -name '*.spec.*' -o -name '*_test.*' -o -name '*_spec.*' 2>/dev/null | grep -v node_modules | wc -l
623

$ git ls-files | grep -E '(\.test\.|\.spec\.|_test\.|_spec\.)' | wc -l
17
433  vendor/bundle/ruby/3.4.0/gems/...     bundled gem test suites
189  .claude/worktrees/agent-*/            agent worktree copies
  4  nested node_modules                   vendor/.../axe-core-api-4.11.1/node_modules
 17  test/                                 the real answer

/retro prints this as "Total test files: N", so the retro is just wrong. /ship uses it for a before/after delta, so the delta survives but both endpoints are inflated.

Two things are broken: grep -v node_modules filters output lines rather than pruning the walk (find still descends into every ignored tree), and no exclusion list can be complete — vendor/ is Ruby, .claude/worktrees/ is gstack's own feature, and every ecosystem adds more (target/, .venv/, Pods/).

Fix

-find . -name '*.test.*' -o -name '*.spec.*' -o -name '*_test.*' -o -name '*_spec.*' 2>/dev/null | grep -v node_modules | wc -l
+git ls-files 2>/dev/null | grep -E '(\.test\.|\.spec\.|_test\.|_spec\.)' | wc -l

Untracked build output is excluded by definition, .gitignore is respected for free, and the full-tree walk goes away. Both skills already require a git repo and run git log origin/<default> in the same step, so no new dependency.

.claude/worktrees/ is untracked in the parent repo, so this also fixes the worktree inflation without special-casing gstack's own directory.

Changes

Sources of truth:

  • retro/SKILL.md.tmpl
  • ship/sections/test-coverage.md (x2)
  • scripts/resolvers/testing.ts (x2, inside a template literal so the escape is \\.)
  • openclaw/skills/gstack-openclaw-retro/SKILL.md

Regenerated / updated:

  • retro/SKILL.md via bun run gen:skill-docs
  • test/fixtures/golden/{codex,factory}-ship-SKILL.md to match the regenerated .agents/ and .factory/ ship skills

Verification

Generated output is correct shell (single backslashes survive the TS template literal):

$ grep -n 'git ls-files' retro/SKILL.md
1029:git ls-files 2>/dev/null | grep -E '(\.test\.|\.spec\.|_test\.|_spec\.)' | wc -l

Behavior on the affected repo: 623 -> 17.

Test suite (bun test test/ with the e2e/eval ignores from package.json):

  • test/host-config.test.ts — 77/77 pass, including all three golden-file regression tests
  • Two failures remain, both reproduced 3/3 on clean main (a325940) with zero changes, so they are pre-existing and unrelated:
    • gstack-model-benchmark --dry-run > NOT READY path fires when auth env vars are stripped
    • session-runner observability > 11: all new I/O is wrapped in try/catch (non-fatal)

Left alone deliberately

test/fixtures/golden-ship-claude.md still contains the old command at lines 1090 and 1250. Nothing reads it — no .ts in the repo references that path, and it is not the fixture used by the golden-file regression suite (that one is test/fixtures/golden/claude-ship-SKILL.md, which does not contain the pattern at all). It looks like an orphaned fixture from an earlier layout, so hand-editing it seemed more likely to entrench drift than fix it. Happy to update or delete it if you'd prefer.

Tradeoff worth a look

Test files written but not yet git add-ed stop counting. Correct for /retro, which reports on merged history. For /ship's before/after delta it is a slight semantic change — if you want untracked-but-not-ignored files included, git ls-files --cached --others --exclude-standard does that while still honoring .gitignore. Happy to switch /ship to that variant.

The test-file count scanned the whole working tree with `find`, so it
counted untracked build output as test files. On a Rails repo using
`bundle install --path vendor/bundle` it over-reported by 37x:

    find . -name '*.test.*' ... | grep -v node_modules | wc -l   -> 623
    git ls-files | grep -E '(\.test\.|...)' | wc -l              ->  17

The 623 breaks down as 433 from vendor/bundle gem suites, 189 from
.claude/worktrees/ agent copies, 4 from nested node_modules, and 17
real files under test/.

`grep -v node_modules` only filters output lines, so find still walks
every ignored tree, and it does nothing for vendor/, worktrees, tmp/,
target/, .venv/, Pods/ and so on. No exclusion list can stay complete,
so count tracked files instead: untracked build output is excluded by
definition, .gitignore is respected for free, and it avoids the
full-tree walk. Both skills already require a git repo and run
`git log origin/<default>` in the same step.

Fixes garrytan#2307

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@trunk-io

trunk-io Bot commented Jul 20, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

retro/ship test-file count scans untracked build output: reports 623 vs 17 real (37x) on a Rails repo

1 participant