Skip to content

fix: refresh supported agent model catalog#1574

Open
simple-agent-manager[bot] wants to merge 9 commits into
mainfrom
sam/weekly-model-catalog-update-zy3pn4
Open

fix: refresh supported agent model catalog#1574
simple-agent-manager[bot] wants to merge 9 commits into
mainfrom
sam/weekly-model-catalog-update-zy3pn4

Conversation

@simple-agent-manager

@simple-agent-manager simple-agent-manager Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Refresh the shared Claude Code, OpenAI Codex, Gemini CLI, Mistral Vibe, and OpenCode model catalogs from current primary sources.
  • Remove retired selectors, correct malformed IDs and lifecycle labels, scope GPT-5.3 Codex Spark as ChatGPT-only, and synchronize the OpenCode fallback with active Models.dev entries.
  • Expand exact catalog, lifecycle, duplicate-ID, and platform-proxy invariant coverage without changing the ModelGroup[] consumer contract.

Validation

  • pnpm lint (existing warnings only)
  • pnpm typecheck
  • pnpm test (7,000+ tests)
  • Additional validation: pnpm build; focused shared catalog tests 25/25; exact Models.dev ID/name comparison
  • N/A: no sweep/cron/alarm candidate selection changes

Staging Verification (REQUIRED for all code changes — merge-blocking)

  • Staging deployment green — final-head run 29243127194 passed from the migration-compatible integration branch described below
  • Live app verified via Playwright — the staging workflow authenticated to app.sammy.party and passed all 12 Playwright smoke tests
  • Existing workflows confirmed working — the standard authenticated staging smoke suite passed 12/12
  • New feature/fix verified on staging — authenticated live API responses matched the reviewed static catalogs exactly; OpenCode had zero diff against current Models.dev
  • N/A: no infrastructure paths changed
  • N/A: no UI layout or styling changes

Staging Verification Evidence

The shared staging Worker had already been advanced from main's append-only Durable Object migration v16 to draft PR #1547's v17. Deploying this branch directly therefore failed at Cloudflare migration reconciliation in run 29239922843. For safe verification, temporary branch sam/weekly-model-catalog-staging-zy3pn4 was created from the branch owning live v17, and only catalog commit d6e668d62 was cherry-picked.

Final-head run 29243127194 passed deployment, D1 data-integrity checks, health checks, and 12/12 Playwright smoke tests after one retry for a transient Cloudflare /zones HTTP 500. Authenticated requests to api.sammy.party/api/model-catalog/:agentType matched this PR exactly for Claude (14 models), Codex (19), Gemini (6), and Mistral (10). The deployed OpenCode groups, IDs, and names had no diff from the current live Models.dev response.

UI Compliance Checklist (Required for UI changes)

N/A: no UI code or layout changed; the existing model selector consumes the unchanged catalog contract.

End-to-End Verification (Required for multi-component changes)

N/A: this is a shared data refresh with an unchanged API/UI contract. Focused tests exercise shared catalog lookups and proxy invariants, and staging verified the authenticated API's final serialized responses plus the standard browser smoke suite.

Data Flow Trace

N/A: no multi-component control flow changed.

Untested Gaps

N/A: exact static responses and dynamic OpenCode source parity were verified on live staging.

Post-Mortem (Required for bug fix PRs)

What broke

The user-facing model catalog suggested retired or malformed model IDs and carried stale lifecycle labels; its OpenCode fallback had also drifted from Models.dev.

Root cause

Provider catalogs evolved after the prior static snapshot. This is cumulative external-catalog drift rather than a regression attributable to one repository commit.

Class of bug

External capability-catalog drift.

Why it wasn't caught

Existing tests protected catalog shape and selected invariants but did not assert all newly retired IDs, exact corrected provider IDs, or full fallback parity with the current source snapshot.

Process fix included in this PR

Expanded focused tests in packages/shared/tests/model-catalog.test.ts and packages/shared/tests/unit/model-catalog-alternative-providers.test.ts cover critical exact IDs, removals, lifecycle labels, duplicates, active OpenCode entries, and proxy versus agent-only routing.

Post-mortem file

tasks/archive/2026-07-13-refresh-supported-agent-model-catalog.md

Specialist Review Evidence (Required for agent-authored PRs)

  • All local reviewers completed and findings addressed before merge
  • If any reviewer did NOT complete: needs-human-review label added and merge deferred to human
Reviewer Status Outcome
task-completion-validator subagent FAILED Runtime interrupted before returning a report; the checklist was subsequently completed locally with no gaps, but rule 25 still requires human review
constitution-validator subagent FAILED Runtime interrupted before returning a report; local checklist found no hardcoded configurable values, but rule 25 still requires human review
test-engineer subagent FAILED Runtime interrupted before returning a report; local checklist found focused coverage complete, but rule 25 still requires human review
doc-sync-validator subagent FAILED Runtime interrupted before returning a report; local checklist found no stale documentation, but rule 25 still requires human review

Exceptions (If any)

  • Scope: staging branch ancestry and self-merge
  • Rationale: shared staging already contains unmerged Durable Object migration v17, requiring a temporary integration branch for a valid deployment; interrupted local reviewer reports trigger the mandatory human merge gate
  • Expiration: temporary staging branch can be deleted after this PR; needs-human-review remains until a human decides whether the locally repeated reviews are sufficient

Agent Preflight (Required)

  • Preflight completed before code changes

Classification

  • external-api-change
  • cross-component-change
  • business-logic-change
  • public-surface-change
  • docs-sync-change
  • security-sensitive-change
  • ui-change
  • infra-change

External References

Consulted official documentation before coding: Anthropic models, Codex models, Gemini CLI model selection, Mistral models, and the live Models.dev API. Additional lifecycle URLs and exact findings are recorded in the archived task file.

Codebase Impact Analysis

packages/shared/src/model-catalog.ts and its focused tests changed. Existing API and web consumers were traced and retain the same ModelGroup[] contract.

Documentation & Specs

Updated tasks/archive/2026-07-13-refresh-supported-agent-model-catalog.md with the dated research, current authoritative source URLs, implementation checklist, and acceptance criteria. No user guide or architecture document needed changes because the ModelGroup[] interface and consumer behavior are unchanged.

Constitution & Risk Check

Checked Principle XI (No Hardcoded Values). The modified identifiers and lifecycle metadata are the requested provider-owned static catalog data, not deployment configuration. Main risks were invalid IDs, retired suggestions, proxy-routing leakage, and source drift; exact tests and staging comparisons cover each.

@simple-agent-manager simple-agent-manager Bot added the needs-human-review Agent could not complete all review gates — human must approve before merge label Jul 13, 2026
@codspeed-hq

codspeed-hq Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 6 untouched benchmarks


Comparing sam/weekly-model-catalog-update-zy3pn4 (cfbb638) with main (298d8dd)

Open in CodSpeed

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-human-review Agent could not complete all review gates — human must approve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant