feat(voice): backend voice picker + advanced controls behind disclosure (#742)

[heavygee]

Reviewers: do not use the default "Files changed" tab on this PR.
This branch is stacked on #692 (feat/pluggable-voice-backend). Against tiann/main GitHub will show the union of #692 + this PR.
Review only: heavygee/hapi@feat/pluggable-voice-backend...feat/voice-selection-all-backends
Do not merge until #692 lands, then rebase onto main so this PR shrinks to its own delta only.

Summary

Two voice features against #742, stacked on #692. The visible default surface stays small (one picker, one toggle); everything else lives behind a single collapsed disclosure so a user who just wants to pick a voice never sees the tuning UI.

Backend-aware voice picker

• shared/voicePickerCatalog.ts - static Gemini/Qwen voice lists, per-backend localStorage keys, resolve helpers
• Hub GET /api/voice/backend returns { backend, backends } when multiple providers are configured
• Hub GET /api/voice/voices - ElevenLabs list available even when default backend is Gemini
• Hub gemini-ws - ?voice= query param wired into setup message
• Web Settings - voice backend chooser (when 2+ backends), voice list follows selection, Gemini/Qwen descriptions, preview-is-EL-only hint
• Voice sessions - VoiceSessionConfig.voiceName + stored preference per backend

Composed system prompt + bootstrap-and-stream context (folded in from feat/voice-advanced-controls)

• Layered prompt in shared/voicePromptLayers.ts: platform fixtures (read-only - tool contracts, routing, TTS rules) + provider guardrails + user-editable identity + user-editable character. composeVoiceAgentPrompt merges them.
• Bootstrap + stream context: small initial conversation payload at handshake (~4 KB) plus streaming chunks via sendContextualUpdate after connect. Honest UI wire-budget hints.
• All three backends: ElevenLabs ConvAI, Gemini Live, and Qwen Realtime each compose + bootstrap + stream.
• ElevenLabs minimal-overrides discipline: empty prefs produce a minimal {agent:{language:'en'}} payload (byte-parity with upstream baseline). Custom layers/sliders/voiceId opt in their respective override fields. Fixes the unauthorized-override crash (Cannot read properties of undefined ('error_type')).
• Hub-side ConvAI override reconciliation: on every /voice/token resolution the hub PATCHes the agent's platform_settings.overrides to match buildVoiceAgentConfig() (agent.prompt + tts.voice_id/stability/similarity_boost/style/speed). Idempotent per-process and best-effort - existing agents that predate the override declaration now self-heal on next session start instead of requiring operator-side console edits.

UI discipline

• New web/src/components/settings/VoiceAdvancedControls.tsx wraps fixtures preview, identity editor, character editor, delivery preset selector, and tuning sliders inside one master Advanced voice settings disclosure (collapsed by default).
• Sub-sections (fixtures / identity / character / delivery / tuning) start collapsed when the master opens.
• A customized badge appears next to the master title if any layer differs from defaults so power users still find their tweaks.
• Backend picker + voice picker remain at the top, outside the disclosure.

Test plan

• bunx tsc --noEmit (hub + web)
• bun test voice routes (hub) - 20 pass including new reconciles platform_settings.overrides on existing agents test
• bun test voice client tests (web) - voicePersonalitySession + voiceContextPlan green
• hapi-driver-rebuild --build-web --verify then dogfood the three backends end-to-end on driver
• Operator dogfood on driver soup (PR review gate)

Merge order

1. Merge feat(voice): pluggable voice backend with Gemini Live & Qwen Realtime #692 (feat/pluggable-voice-backend)
2. Rebase this branch onto upstream/main (PR diff should drop to this PR's own delta), merge feat(voice): backend voice picker + advanced controls behind disclosure (#742) #743

Issues

Ref #742
Blocked by #692

[heavygee]

Stack note: This PR is blocked by #692. For review, prefer the incremental diff on the fork:

heavygee/hapi@feat/pluggable-voice-backend...feat/voice-selection-all-backends

(4 commits: catalog scaffold, backend chooser, Gemini/Qwen descriptions, Playwright dogfood script.)

[github-actions]

Findings

• [Major] Qwen proxy forwards arbitrary client frames with the hub API key — POST /api/voice/qwen-token correctly keeps the DashScope key server-side, but the new /api/voice/qwen-ws proxy then opens DashScope with that key and blindly forwards every browser frame to upstream. Unlike the Gemini path, which rejects client-provided setup frames, any authenticated web client can connect directly and send its own session.update/response.create payloads, turning the hub into a generic DashScope proxy and allowing client-controlled instructions/tools under server credentials. Evidence: hub/src/web/server.ts:159.
  Suggested fix:

  const allowedQwenRuntimeEvents = new Set([
      'input_audio_buffer.append',
      'input_audio_buffer.commit',
      'response.create',
      'conversation.item.create'
  ])
  
  function parseQwenClientEvent(message: string | ArrayBuffer | Uint8Array): { type?: string } | null {
      try {
          return JSON.parse(decodeWsText(message)) as { type?: string }
      } catch {
          return null
      }
  }
  
  message(clientWs: ServerWebSocket<unknown>, message: string | ArrayBuffer | Uint8Array) {
      const event = parseQwenClientEvent(message)
      if (!event?.type || !allowedQwenRuntimeEvents.has(event.type)) {
          try { clientWs.close(1008, 'Client-provided Qwen setup is not allowed') } catch { /* */ }
          return
      }
  
      const upstream = upstreamMap.get(clientWs)
      if (upstream?.readyState === WebSocket.OPEN) {
          upstream.send(message)
      }
  }

  Move the initial session.update construction into the hub proxy, as Gemini does with buildGeminiLiveSetupMessage, and only let the browser send runtime audio/tool-response events afterward.

Questions

• None.

Summary

• Review mode: initial
• One issue found: Qwen realtime proxy needs the same server-owned setup boundary as Gemini before this is safe to merge.

Testing

• Not run (automation; static review only, per PR security instructions).

HAPI Bot

[heavygee]

Noise check (re: default diff vs main)

Compare base	Commits	Files
tiann/main (GitHub default)	29	35
feat/pluggable-voice-backend (#692 tip)	4	23

GitHub would not let us retarget base to feat/pluggable-voice-backend on tiann/hapi (branch only exists on heavygee/hapi). Until #692 merges, use the compare link in the PR description for review.

Automated review note: The github-actions MAJOR on Qwen proxy (hub/src/web/server.ts) is from the #692 stack in this branch, not from the 4 #742 commits — please route that feedback to #692 if still open.

[heavygee]

Status update on the open Major (Qwen WS proxy)

Coordinated with the #692 agent — they're already implementing the fix in their worktree:

• buildQwenSessionUpdateMessage(language) in shared/src/voice.ts (parallel to buildGeminiLiveSetupMessage)
• isQwenSafeClientFrame(message) — allowlist; only session.update frames with only instructions pass through
• Hub createQwenProxyWebSocketHandler sends hub-owned setup on upstream open; rejects with WS close 1008 'Client session.update may only modify instructions'
• QwenVoiceSession.tsx updated to drop the now-redundant client-side initial setup

This is correctly #692 scope — the Qwen proxy message() forwarding-everything behaviour was introduced in #692; my #742 delta to hub/src/web/server.ts is 5 lines, all on the Gemini path (threading voiceName through buildGeminiLiveSetupMessage). No Qwen changes in this branch.

Plan: Stand by until #692 pushes the fix, then rebase feat/voice-selection-all-backends on the new tip. Their hardening flows through to this PR automatically (we stack on it). The bot's Major thread should then be resolvable on #692's commit, not on a duplicate fix here.

CI green; no other open Major findings. Holding for the upstream layer to land.

[heavygee]

Heads-up from a downstream soup-rebuild (heavygee/hapi driver/integration, 2026-05-31):

To get this branch + the voice-advanced layer building cleanly on the current feat/pluggable-voice-backend tip, two pre-existing test bugs needed fixing. Both are local-only commits on the rebased branches and will be lost on the next force-push unless preserved.

1. On feat/voice-selection-all-backends (this PR)

238ad4c test(voice): isolate VOICE_BACKEND fallback test from leaked env vars

hub/src/web/routes/voice.test.ts | 4 ++++

The "falls back to elevenlabs for unknown VOICE_BACKEND values" test never deleted GEMINI_API_KEY / QWEN_API_KEY / OPENAI_API_KEY / QWEN_REALTIME_API_KEY before asserting. When hub/.env has any of those set, gemini-live (or another backend) leaks into the resolved backends list and the test fails. The fix adds four delete process.env.* lines at the top of the test, mirroring the cleanup pattern already used by the other tests in the same describe block. No behavior change.

2. On feat/voice-advanced-controls (sibling branch, stacks on this one)

f3ab0f3 test(web): import from vitest instead of bun:test

web/src/lib/voicePersonalitySession.test.ts        | 2 +-
web/src/realtime/hooks/voiceContextPlan.test.ts    | 2 +-

Both files imported describe/expect/test from 'bun:test'. The web workspace runs vitest run, which cannot bundle the built-in bun:test module and fails the transform during bun run build. Only generic test APIs are used; the import swap is a no-op behavior change.

Surfacing in case you re-do the rebase from scratch and overwrite these locally - they would silently come back as test failures on the next CI/build. No action needed if you keep them.

[github-actions]

Findings

• [Major] Qwen proxy still forwards runtime frames before setup is acknowledged — the browser waits for session.updated before starting audio, but an authenticated client can connect directly and send response.create, conversation.item.create, or instruction-only session.update while pendingSetupMap is still set. Those frames are forwarded immediately at hub/src/web/server.ts:188, before the hub-owned session.update sent at hub/src/web/server.ts:159 is acknowledged. That can run the upstream session under the provider default config or race ahead of HAPI's locked tools/voice/instructions, reopening part of the generic-proxy boundary the previous review called out.
  Suggested fix:

  const pendingClientFrames = new WeakMap<ServerWebSocket<unknown>, Array<string | ArrayBuffer | Uint8Array>>()
  
  // after session.created
  pendingSetupMap.delete(clientWs)
  setupAckMap.set(clientWs, false)
  upstream.send(pendingSetup)
  
  // after session.updated for the hub-owned setup
  setupAckMap.set(clientWs, true)
  for (const queued of pendingClientFrames.get(clientWs) ?? []) upstream.send(queued)
  pendingClientFrames.delete(clientWs)
  
  message(clientWs, message) {
      if (!isQwenSafeClientFrame(message)) { /* close */ return }
      const upstream = upstreamMap.get(clientWs)
      if (upstream?.readyState !== WebSocket.OPEN) return
      if (pendingSetupMap.has(clientWs) || setupAckMap.get(clientWs) !== true) {
          const pending = pendingClientFrames.get(clientWs) ?? []
          pending.push(message)
          pendingClientFrames.set(clientWs, pending)
          return
      }
      upstream.send(message)
  }

Questions

• None.

Summary

• Review mode: follow-up after new commits
• The previous Qwen proxy issue is partly addressed, but the proxy still needs to gate client frames until the hub-owned Qwen setup has been acknowledged.

Testing

• Not run (automation; static review only, per PR security instructions).

HAPI Bot

[github-actions]

Findings

• [Major] Gate Gemini client frames until setupComplete — the proxy sends HAPI's setup on upstream open, but it then flushes queued non-setup frames immediately and forwards later frames whenever the upstream socket is open. A direct authenticated client can therefore send clientContent/tool frames before Google acknowledges the hub-owned setup with setupComplete, racing execution ahead of HAPI's locked model, tools, system prompt, language, and voice config. Evidence: hub/src/web/server.ts:77 and hub/src/web/server.ts:107.
  Suggested fix:

  const setupCompleteMap = new WeakMap<ServerWebSocket<unknown>, boolean>()
  const pendingClientFrames = new WeakMap<ServerWebSocket<unknown>, Array<string | ArrayBuffer | Uint8Array>>()
  
  upstream.onopen = () => {
      setupCompleteMap.set(clientWs, false)
      upstream.send(JSON.stringify(buildGeminiLiveSetupMessage(data.language, data.voiceName)))
  }
  
  upstream.onmessage = (event) => {
      const text = typeof event.data === 'string'
          ? event.data
          : new TextDecoder().decode(new Uint8Array(event.data as ArrayBuffer))
      if (setupCompleteMap.get(clientWs) === false && JSON.parse(text).setupComplete) {
          setupCompleteMap.set(clientWs, true)
          for (const queued of pendingClientFrames.get(clientWs) ?? []) upstream.send(queued)
          pendingClientFrames.delete(clientWs)
      }
      if (clientWs.readyState === 1) clientWs.send(event.data)
  }
  
  message(clientWs, message) {
      if (isGeminiSetupFrame(message)) {
          clientWs.close(1008, 'Client-provided Gemini setup is not allowed')
          return
      }
      const upstream = upstreamMap.get(clientWs)
      if (!upstream || upstream.readyState !== WebSocket.OPEN || setupCompleteMap.get(clientWs) !== true) {
          const pending = pendingClientFrames.get(clientWs) ?? []
          pending.push(message)
          pendingClientFrames.set(clientWs, pending)
          return
      }
      upstream.send(message)
  }

Questions

• None.

Summary

• Review mode: follow-up after new commits
• The previous Qwen ack-gate issue appears addressed in the follow-up, but the newly added Gemini proxy still has the same setup-ack race at its proxy boundary.

Testing

• Not run (automation; static review only, per PR security instructions).

HAPI Bot

[github-actions]

[Major] Gate Gemini client frames until setupComplete

This flushes queued non-setup frames immediately after sending HAPI's setup, before Google has acknowledged it with setupComplete; once the upstream socket is open, message() also forwards later frames directly. A direct authenticated client can send clientContent or tool frames before HAPI's hub-owned model, tools, system prompt, language, and voice setup is installed, so the session can race ahead under default or partially-applied config.

Suggested fix:

const setupCompleteMap = new WeakMap<ServerWebSocket<unknown>, boolean>()
const pendingClientFrames = new WeakMap<ServerWebSocket<unknown>, Array<string | ArrayBuffer | Uint8Array>>()

upstream.onopen = () => {
    setupCompleteMap.set(clientWs, false)
    upstream.send(JSON.stringify(buildGeminiLiveSetupMessage(data.language, data.voiceName)))
}

// On the first upstream setupComplete, flip the gate and flush queued client frames.
if (setupCompleteMap.get(clientWs) === false && parsed.setupComplete) {
    setupCompleteMap.set(clientWs, true)
    for (const queued of pendingClientFrames.get(clientWs) ?? []) upstream.send(queued)
    pendingClientFrames.delete(clientWs)
}

message(clientWs, message) {
    if (isGeminiSetupFrame(message)) {
        clientWs.close(1008, 'Client-provided Gemini setup is not allowed')
        return
    }
    const upstream = upstreamMap.get(clientWs)
    if (upstream?.readyState !== WebSocket.OPEN || setupCompleteMap.get(clientWs) !== true) {
        const pending = pendingClientFrames.get(clientWs) ?? []
        pending.push(message)
        pendingClientFrames.set(clientWs, pending)
        return
    }
    upstream.send(message)
}

[github-actions]

Findings

• [Major] Gate Gemini client frames until setupComplete — the proxy sends HAPI's setup, then immediately flushes queued client frames and forwards later frames as soon as the upstream socket is open. A direct authenticated client can race clientContent or tool frames before Google acknowledges the hub-owned setup, so execution can start before HAPI's model, tools, prompt, language, and voice configuration are installed. Evidence: hub/src/web/server.ts:88 and hub/src/web/server.ts:118.
  Suggested fix:

  const setupCompleteMap = new WeakMap<ServerWebSocket<unknown>, boolean>()
  const pendingClientFrames = new WeakMap<ServerWebSocket<unknown>, Array<string | ArrayBuffer | Uint8Array>>()
  
  upstream.onopen = () => {
      setupCompleteMap.set(clientWs, false)
      upstream.send(JSON.stringify(buildGeminiLiveSetupMessage(data.language, data.voiceName, data.systemInstruction)))
  }
  
  upstream.onmessage = (event) => {
      const text = typeof event.data === 'string'
          ? event.data
          : new TextDecoder().decode(new Uint8Array(event.data as ArrayBuffer))
      const parsed = JSON.parse(text) as { setupComplete?: unknown }
      if (setupCompleteMap.get(clientWs) === false && parsed.setupComplete) {
          setupCompleteMap.set(clientWs, true)
          for (const queued of pendingClientFrames.get(clientWs) ?? []) upstream.send(queued)
          pendingClientFrames.delete(clientWs)
      }
      if (clientWs.readyState === 1) clientWs.send(event.data)
  }
  
  if (upstream?.readyState !== WebSocket.OPEN || setupCompleteMap.get(clientWs) !== true) {
      const pending = pendingClientFrames.get(clientWs) ?? []
      pending.push(message)
      pendingClientFrames.set(clientWs, pending)
      return
  }

• [Major] Pass decoded systemInstruction into Qwen setup — /api/voice/qwen-ws decodes ?systemPrompt= and stores it on ws.data, and the Qwen client sends that param for advanced voice settings, but createQwenProxyWebSocketHandler() casts the data without systemInstruction and calls buildQwenSessionUpdateMessage(data.language, data.voiceName). Qwen sessions therefore ignore the user-edited identity/character/delivery prompt while Gemini uses it. Evidence: hub/src/web/qwenProxyHandler.ts:53 and hub/src/web/qwenProxyHandler.ts:61.
  Suggested fix:

  const data = clientWs.data as {
      apiKey: string
      model: string
      language?: string
      voiceName?: string
      systemInstruction?: string
  }
  
  pendingSetupMap.set(clientWs, JSON.stringify(buildQwenSessionUpdateMessage(
      data.language,
      data.voiceName,
      data.systemInstruction
  )))

Questions

• None.

Summary
Review mode: follow-up after new commits
The previous Gemini setup-ack race remains present, and the new Qwen prompt transport path drops the decoded custom prompt before upstream setup. Add proxy-level tests for both setup contracts.

Testing

• Not run (automation; static review only).

HAPI Bot

[github-actions]

[Major] Gate Gemini client frames until setupComplete

This flushes queued client frames immediately after sending HAPI's setup, before Google acknowledges it with setupComplete; message() also forwards later frames whenever the upstream socket is open. A direct authenticated client can race clientContent or tool frames before HAPI's model, tools, prompt, language, and voice setup is installed.

Suggested fix:

const setupCompleteMap = new WeakMap<ServerWebSocket<unknown>, boolean>()
const pendingClientFrames = new WeakMap<ServerWebSocket<unknown>, Array<string | ArrayBuffer | Uint8Array>>()

upstream.onopen = () => {
    setupCompleteMap.set(clientWs, false)
    upstream.send(JSON.stringify(buildGeminiLiveSetupMessage(data.language, data.voiceName, data.systemInstruction)))
}

if (setupCompleteMap.get(clientWs) !== true) {
    const pending = pendingClientFrames.get(clientWs) ?? []
    pending.push(message)
    pendingClientFrames.set(clientWs, pending)
    return
}
// Flip the gate and flush only after the first upstream setupComplete frame.

[github-actions]

[Major] Pass the decoded prompt into Qwen's hub-owned setup

The server decodes ?systemPrompt= for /api/voice/qwen-ws and stores it on ws.data, and the Qwen client sends that param for the advanced voice prompt. This handler drops it when building the initial session.update, so Qwen ignores the user-edited identity/character/delivery prompt while Gemini applies it.

Suggested fix:

const data = clientWs.data as {
    apiKey: string
    model: string
    language?: string
    voiceName?: string
    systemInstruction?: string
}

pendingSetupMap.set(clientWs, JSON.stringify(buildQwenSessionUpdateMessage(
    data.language,
    data.voiceName,
    data.systemInstruction
)))