build(evaluators): standardize contrib packaging and release flow

[lan17]

Summary

• normalize contrib package wiring around agent-control-evaluators[...] and bring package-local checks into line across the in-repo contrib evaluators
• add contrib package discovery and verification so root packaging metadata, local uv.sources, and extras wiring stay in sync
• make release/build automation and root CI/check flows discovery-driven for all real contrib packages
• refresh the contrib template so new packages follow the current evaluator naming, packaging, and release contract

Review Order

1. build(evaluators): normalize contrib package wiring
2. fix(evaluators): satisfy budget package-local checks
3. build(evaluators): normalize package-local Makefile targets
4. build(infra): add contrib package discovery script
5. ci(infra): drive contrib releases from discovery
6. ci(infra): make root checks contrib-aware
7. chore(evaluators): refresh contrib template packaging scaffold
8. fix(infra): address contrib packaging review feedback
9. fix(ci): harden contrib release and coverage flow
10. fix(ci): stabilize release artifact publishing
11. test(evaluators): cover contrib version fallbacks

Design Context

• Shared design doc and implementation plan: https://gist.github.com/lan17/823617097f4b7e9f4ff717979de7dba4

Validation

• make sync
• make contrib-verify
• make scripts-test
• make test-extras
• make contrib-lint
• make contrib-typecheck
• make contrib-build
• uv run python scripts/build.py all
• make check (stops in server-test locally because Postgres is not listening on localhost:5432)
• pg_isready -h localhost -p 5432 -U agent_control -> no response

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[abhinav-galileo]

Template scaffolds stale >=3.0.0 floors for agent-control-evaluators / agent-control-models

The rest of the PR moves in-repo contrib packages onto >=7.5.0 for agent-control-evaluators / agent-control-models, but evaluators/contrib/template/pyproject.toml.template (lines 10-11) still scaffolds new packages at >=3.0.0. Since scripts/contrib_packages.py verify only checks presence of wiring (not floor values), a template-copied package will silently ship metadata claiming compatibility with 3.x while using 7.5.0+ APIs - the failure surfaces at install/import for downstream users, not in CI.

(Posting as a general PR comment rather than inline because those two dependency lines are unchanged context between diff hunks in this PR, so GitHub will not accept an inline review comment there.)

Consider either (a) a literal placeholder like <minimum-compatible-version> so the choice is explicit (and can be grep-guarded), or (b) extending verify_contrib_packages to assert each contrib pyproject's internal floors parse as >=X.Y.Z above a documented minimum. The placeholder approach is cheaper; the verify-extension approach eliminates the drift class.

[lan17]

Addressing #193 (comment): I took the placeholder route here. The contrib template now uses <minimum-compatible-version> for the internal floors, and the template README calls out that authors need to replace it intentionally based on whether the package rides the shared monorepo release train or is maintained independently.

[lan17]

Since I replied on the review threads, I cleaned up a few more things:

• fixed the CI coverage upload path so it passes Codecov an explicit contrib coverage file list instead of relying on a glob
• removed the agent-control-server[galileo] install alias from the Galileo README since this repo does not publish that package in the in-repo release flow
• hardened the release workflow:
  • stage built dists into a stable release-dists/ tree before upload/download
  • publish builtin + contrib evaluator distributions together from one staged directory, so we do not create a bad builtin/contrib ordering hazard
  • keep SDK publish gated on successful evaluator publishing
  • upload GitHub release assets from the staged tree
• added direct tests for the contrib package-version fallback path and tightened those three evaluators to resolve package version through a small helper

I also ran a fresh review loop after that. No new blocking findings came back. The remaining nits are cleanup-level, not merge blockers.