Observed problem
test/helpers/providers/claude.ts ClaudeAdapter.available() treats Claude auth as present only when ~/.claude/.credentials.json exists or ANTHROPIC_API_KEY is set:
const credsPath = path.join(os.homedir(), '.claude', '.credentials.json');
const hasCreds = fs.existsSync(credsPath);
const hasKey = !!process.env.ANTHROPIC_API_KEY;
if (!hasCreds && !hasKey) {
return { ok: false, reason: 'No Claude auth found. Log in via `claude` interactive session, or export ANTHROPIC_API_KEY.' };
}
On a macOS subscription install (the default Claude Code install on Mac), ~/.claude/.credentials.json is absent — the credential lives elsewhere (likely the Keychain) — yet the subscription session authenticates claude -p fine. So the file-existence check is a proxy that misses a valid auth state, and the adapter reports NOT READY for a provider whose run() path (claude -p --output-format json) would actually succeed.
This is internal to the adapter: run() uses the subscription via claude -p, so available() rejecting that same auth state is an inconsistency, not a user-config gap. (ANTHROPIC_API_KEY is a workaround, but macOS subscription users have no key and don't need one for claude -p.)
Current behavior on main (cab774c)
$ gstack-model-benchmark --dry-run --prompt "hi"
claude: NOT READY — No Claude auth found. Log in via `claude` interactive session, or export ANTHROPIC_API_KEY.
On the same machine:
~/.claude/.credentials.json — absent
claude CLI — resolves (claude --version → 2.1.x), logged in via subscription; claude -p works
available() returns {ok:false} purely on the missing file
Any code path gated on this available() (model-benchmark, and the LLM-judge path) silently drops the claude provider for this population.
Expected behavior
A logged-in macOS subscription install (no ANTHROPIC_API_KEY, no .credentials.json, but claude -p working) should report the claude adapter as available.
Possible fix
run() already classifies runtime auth failures gracefully (/unauthorized|auth|login/i → auth). So available() could be optimistic when resolveClaudeCommand() resolves on macOS (defer the real auth decision to run()), or detect the Keychain credential — whichever you prefer.
Duplicate search performed
Observed problem
test/helpers/providers/claude.tsClaudeAdapter.available()treats Claude auth as present only when~/.claude/.credentials.jsonexists orANTHROPIC_API_KEYis set:On a macOS subscription install (the default Claude Code install on Mac),
~/.claude/.credentials.jsonis absent — the credential lives elsewhere (likely the Keychain) — yet the subscription session authenticatesclaude -pfine. So the file-existence check is a proxy that misses a valid auth state, and the adapter reports NOT READY for a provider whoserun()path (claude -p --output-format json) would actually succeed.This is internal to the adapter:
run()uses the subscription viaclaude -p, soavailable()rejecting that same auth state is an inconsistency, not a user-config gap. (ANTHROPIC_API_KEYis a workaround, but macOS subscription users have no key and don't need one forclaude -p.)Current behavior on main (cab774c)
On the same machine:
~/.claude/.credentials.json— absentclaudeCLI — resolves (claude --version→ 2.1.x), logged in via subscription;claude -pworksavailable()returns{ok:false}purely on the missing fileAny code path gated on this
available()(model-benchmark, and the LLM-judge path) silently drops the claude provider for this population.Expected behavior
A logged-in macOS subscription install (no
ANTHROPIC_API_KEY, no.credentials.json, butclaude -pworking) should report the claude adapter as available.Possible fix
run()already classifies runtime auth failures gracefully (/unauthorized|auth|login/i→auth). Soavailable()could be optimistic whenresolveClaudeCommand()resolves on macOS (defer the real auth decision torun()), or detect the Keychain credential — whichever you prefer.Duplicate search performed
model-benchmark auth,No Claude auth,credentials.json,ANTHROPIC_API_KEY subscription,claude adapter available): no existing issue.claude.tsavailable()auth detection. No collision.