[awf] API Proxy: Gemini key injection failing — API_KEY_INVALID despite valid key

[lpcox]

Problem

Gemini API calls fail with API_KEY_INVALID even when GEMINI_API_KEY is correctly set and confirmed working via direct curl. The error persists across firewall image versions including 0.25.26 (post gh-aw-firewall#1995).

Observed log sequence:

[INFO] API proxy enabled: OpenAI=false, Anthropic=false, Copilot=false, Gemini=true
[INFO] API proxy sidecar enabled - API keys will be held securely in sidecar container
...
_ApiError: {"error":{"code":400,"message":"API key not valid.","status":"INVALID_ARGUMENT","reason":"API_KEY_INVALID"}}

Gemini=true confirms GEMINI_API_KEY is received by the CLI and should be passed to the sidecar. The key injection pipeline is therefore broken somewhere between the sidecar receiving the request and forwarding it to Google.

Context

• Original issue: bug: gemini API key rejected by proxy sidecar despite valid key gh-aw#25944
• Reported by: multiple users across v0.68.1 → v0.69.3
• Prior fix attempts: gh-aw-firewall#1944 (routing), gh-aw-firewall#1995 (key stripping/injection)

Root Cause (Hypotheses)

The api-proxy sidecar (containers/api-proxy/server.js) handles Gemini on port 10003. It:

1. Strips x-goog-api-key and ?key= query param from incoming requests (stripGeminiKeyParam, line 204; STRIPPED_HEADERS set, line 44)
2. Injects x-goog-api-key: <real_key> header on forward (line 1076)

Remaining suspects:

A. SDK sends auth via a path not handled by the proxy. The @google/genai SDK may send auth using Authorization: Bearer <key> (which is stripped by shouldStripHeader) but Google may validate the Authorization bearer token differently from the x-goog-api-key header. If Google rejects x-goog-api-key for certain API versions/endpoints, the injected header may silently be ignored.

B. stripGeminiKeyParam misses some auth query param variants. The SDK may append ?apiKey= or ?api_key= instead of ?key=. The current regex in stripGeminiKeyParam (line 204) only removes ?key= — other param names would pass through with the placeholder value, resulting in Google seeing both the placeholder key= param and the real x-goog-api-key header, with the param taking precedence.

C. No sidecar startup validation for Gemini. Unlike Anthropic/OpenAI, there is no pre-flight check confirming GEMINI_API_KEY is non-empty in the sidecar at startup. Failures are silent until the first real API call. The healthResponse() function (line 887) includes gemini: !!GEMINI_API_KEY, but Docker healthcheck only hits port 10000 — Gemini misconfiguration is invisible until inference time.

Proposed Solution

1. Audit stripGeminiKeyParam for completeness (containers/api-proxy/server.js line 204)

Extend to strip all known Gemini auth query param variants:

// Strip ?key=, ?apiKey=, ?api_key= from Gemini request URLs
function stripGeminiKeyParam(reqUrl) {
  const url = new URL(reqUrl, '(localhost/redacted)
  url.searchParams.delete('key');
  url.searchParams.delete('apiKey');
  url.searchParams.delete('api_key');
  return url.pathname + (url.search || '');
}

2. Add Gemini health check pre-flight logging (src/docker-manager.ts lines 1921–1924)

The existing warning is issued at AWF CLI startup but not during container health checks. Add an explicit log inside the sidecar at startup that shows the resolved value of GEMINI_API_KEY (masked, showing only length/prefix):

if (GEMINI_API_KEY) {
  logRequest('info', 'server_start', { message: `GEMINI_API_KEY configured (length=\$\{GEMINI_API_KEY.length})` });
} else {
  logRequest('warn', 'server_start', { message: 'GEMINI_API_KEY not set — Gemini proxy will return 503' });
}

3. Verify x-goog-api-key vs Authorization: Bearer precedence

Add a targeted integration test that sends a request with both x-goog-api-key: placeholder and ?key=placeholder and confirms the proxy forwards only x-goog-api-key: <real_key> to upstream with no ?key= param.

4. Capture a full proxy debug log from a failing run

Request the reporter to run with --log-level debug --keep-containers and share the api-proxy container log (docker logs awf-api-proxy) to confirm whether the sidecar is receiving the real key at request time.

Files to Change

• containers/api-proxy/server.js — stripGeminiKeyParam (line 204), sidecar startup logging
• containers/api-proxy/server.test.js — add stripGeminiKeyParam edge case tests
• src/docker-manager.ts — verify GEMINI_API_KEY env var propagation to sidecar (line 1753)

Generated by Firewall Issue Dispatcher · ● 1.2M · ◷