fix(model-benchmark): don't drop the claude provider on macOS subscription installs#1894
Open
jbetala7 wants to merge 1 commit into
Open
fix(model-benchmark): don't drop the claude provider on macOS subscription installs#1894jbetala7 wants to merge 1 commit into
jbetala7 wants to merge 1 commit into
Conversation
…ption installs ClaudeAdapter.available() treated Claude auth as present only when ~/.claude/.credentials.json exists or ANTHROPIC_API_KEY is set. On the default macOS Claude Code subscription install the credential lives in the login Keychain, not that file, so available() returned NOT READY for a provider whose run() path (`claude -p --output-format json`) authenticates fine via the same subscription session. model-benchmark and the LLM-judge path silently dropped the claude provider for that whole population. An absent ~/.claude/.credentials.json on macOS is not evidence of missing auth. Keep the positive signals (creds file or ANTHROPIC_API_KEY) as a fast yes, but on darwin be optimistic when the binary resolves and defer the real auth decision to run(), which already classifies a genuinely logged-out state as an `auth` error. Non-macOS keeps the strict file/key check, where ~/.claude/.credentials.json is the actual credential store. Adds test/claude-adapter-available.test.ts covering: macOS no-creds-no-key now available; non-macOS no-creds-no-key still not available; API key and a present creds file available on any platform; unresolvable binary still reported before the auth sniff. The macOS case fails on main and passes with this change. Fixes garrytan#1890 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
ClaudeAdapter.available()(test/helpers/providers/claude.ts) treats Claude auth as present only when~/.claude/.credentials.jsonexists orANTHROPIC_API_KEYis set:On the default macOS Claude Code subscription install,
~/.claude/.credentials.jsonis absent — the credential lives in the login Keychain — yet the subscription session authenticatesclaude -pfine. So the file-existence check is a proxy that misses a valid auth state, andavailable()reports NOT READY for a provider whoserun()path (claude -p --output-format json) would actually succeed. Any caller gated on thisavailable()(model-benchmark, and the LLM-judge path) silently drops the claude provider for that whole population.On
main(cab774c)on a machine where
~/.claude/.credentials.jsonis absent,claude --versionresolves, andclaude -pworks.Root cause
An absent
~/.claude/.credentials.jsonon macOS is not evidence of missing auth — it is the expected state for a Keychain-backed subscription install.available()rejecting an auth state thatrun()can actually use is an internal inconsistency, not a user-config gap.Fix
ANTHROPIC_API_KEY) as a fast yes.run(), which already classifies a genuinely logged-out state as anautherror (/unauthorized|auth|login/i). So a logged-out mac still surfaces a clear auth error fromrun()instead of being silently dropped atavailable().~/.claude/.credentials.jsonis the actual credential store.Testing
Adds
test/claude-adapter-available.test.ts(resolver satisfied with a real binary viaGSTACK_CLAUDE_BIN,os.homedirpointed at an empty dir,process.platformoverridden — no live CLI):ANTHROPIC_API_KEYset → available on any platform.credentials.json→ available on non-macOSFixes #1890
🤖 Generated with Claude Code