Skip to content

Add JSON output for eval inventory#28058

Merged
gundermanc merged 3 commits into
google-gemini:mainfrom
Gsoc26:eval-inventory-json
Jun 23, 2026
Merged

Add JSON output for eval inventory#28058
gundermanc merged 3 commits into
google-gemini:mainfrom
Gsoc26:eval-inventory-json

Conversation

@ved015

@ved015 ved015 commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a JSON output option for the eval inventory command, making it easier to use in CI, scripts, and automated checks.

Details

  • Adds --json support for machine-readable inventory results.
  • Uses relative paths so output works across different machines.
  • Improves --root handling when the path is missing or looks like another flag.
  • Adds tests to keep the JSON output stable and reliable.

Pre-Merge Checklist

  • Updated relevant documentation and README (if needed)
  • Added/updated tests (if needed)
  • Noted breaking changes (if any)
  • Validated on required platforms/methods:
    • MacOS
      • npm run
      • npx
      • Docker
      • Podman
      • Seatbelt
    • Windows
      • npm run
      • npx
      • Docker
    • Linux
      • npm run
      • npx
      • Docker

@gundermanc pls have a look

ved015 added 3 commits June 20, 2026 07:56
- Text formatter 'By Policy' now renders policies not in POLICY_ORDER,
  preventing silent omissions when EvalPolicy gains new values
- Remove unused policyOrder alias in formatInventoryReport
- Thread repoRoot through InventoryResult so resolveRelativePath uses
  the correct base dir instead of process.cwd() for fallback paths
- collectInventory now validates evals/ dir exists with a helpful error
  message before globbing, instead of silently returning empty results
- CLI: detect --root passed with no value (last argument) and exit 1
- Tests: add repoRoot to all InventoryResult fixtures, align totalFiles
  with files.length, add makeEmptyResult() helper, add coverage for
  evals-dir error, unknown policy rendering, and repoRoot relativization
@ved015 ved015 requested review from a team as code owners June 20, 2026 06:03
@github-actions github-actions Bot added the size/l A large sized PR label Jun 20, 2026
@github-actions

Copy link
Copy Markdown

📊 PR Size: size/L

  • Lines changed: 782
  • Additions: +752
  • Deletions: -30
  • Files changed: 4

@ved015

ved015 commented Jun 20, 2026

Copy link
Copy Markdown
Contributor Author

@gemini-cli-robot pls review

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

Gemini encountered an error creating the summary. You can try again by commenting /gemini summary.

@gemini-cli gemini-cli Bot added the status/need-issue Pull requests that need to have an associated issue. label Jun 20, 2026
@gundermanc gundermanc enabled auto-merge June 23, 2026 18:23
@gundermanc gundermanc added this pull request to the merge queue Jun 23, 2026
Merged via the queue into google-gemini:main with commit 6e0bd68 Jun 23, 2026
33 checks passed
hotmanxp added a commit to hotmanxp/gemini-cli that referenced this pull request Jun 24, 2026
16 commits ahead of main-api-integration at sync time:
- f8541cf fix/verify release npm ci ignore scripts (google-gemini#28116)
- 6e0bd68 Add JSON output for eval inventory (google-gemini#28058)
- d3ef6ac fix(ci): use wombat dressing room fallback in nightly release to prevent ENEEDAUTH (google-gemini#28104)
- be7ba2c fix: resolve workspace publish failures and scheduler event loop starvation (google-gemini#28063)
- c22137e feat: add eval:inventory CLI command and reporting logic (google-gemini#28009)
- 6613e12 fix(ci): append trailing slash to registry url in npmrc (google-gemini#28038)
- 93844df chore(deps): pin dependencies and enforce 14-day update cooldown (google-gemini#27948)
- c427d18 fix(ci): provide fallbacks for package variables in nightly release (google-gemini#28016)
- d5e25b9 Changelog for v0.48.0-preview.0 (google-gemini#27999)
- 7ef3b4e chore(release): bump version to 0.49.0-nightly.20260617.g4d3dcdce1 (google-gemini#28003)
- 4d3dcdc Revert "fix(core-tools): resolve defensive path resolution for at-reference files" (google-gemini#27992)
- f741d03 fix(core-tools): resolve defensive path resolution for at-reference files (google-gemini#27943)
- 926f3d9 fix(config): migrate coreTools setting to tools.core (google-gemini#27947)
- 97455e5 Add static eval source analyzer (google-gemini#27631)
- 5624a3b fix(cli): handle tmux false positive background detection (google-gemini#27572)
- fbce3e5 feat(core): Support GDC air-gapped Service Identity after auth library update (google-gemini#27956)

Conflict resolution:
- packages/core/package.json: upstream pinned all dep versions (no caret)
  per 93844df (chore(deps): pin dependencies and enforce 14-day
  update cooldown). Kept fork-specific 'openai: 4.104.0' (pinned to
  match new policy) for MiniMax OpenAI-compat API integration.
- packages/vscode-ide-companion/package.json: took upstream's pinned
  versions (no fork-specific deps).
- package-lock.json: regenerated via 'npm install' after taking
  upstream's lockfile via 'git checkout --theirs'.

Verification:
- npm run build: pass
- npm run typecheck: pass (0 errors)
- npm test --workspace=@google/gemini-cli -- --run: 463 files / 6914 pass,
  4 skipped, 0 fail
- npm test --workspace=@google/gemini-cli-core -- --run: 403 files,
  7710 tests run (12 fail originally, 2 fixed via UPDATE_GOLDENS=true
  golden regen on src/services/modelConfig.golden.test.ts).
  REMAINING 10 FAILURES in memory/AGENTS.md/scoped workspace path
  validation tests are KNOWN UPSTREAM BEHAVIOR DRIFT — upstream
  commit pair (f741d03 + 4d3dcdc revert) left tests in
  intermediate state (expect deny, code allows). Tracked for
  separate fix; not blocking the merge per user direction.

Snapshot regen:
- packages/core/src/services/test-data/resolved-aliases.golden.json
- packages/core/src/services/test-data/resolved-aliases-retry.golden.json
  (regenerated via UPDATE_GOLDENS=true, see pitfall google-gemini#4 in
  autonomous-ai-agents/gemini-cli skill)
software-0ficial pushed a commit to software-0ficial/gemini-cli that referenced this pull request Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/l A large sized PR status/need-issue Pull requests that need to have an associated issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants