feat(vtuber): add OpenAI-compatible adapter#1234
Conversation
Expose POST /v1/chat/completions (SSE) backed by the OAB agent, so any OpenAI-compatible character skin (AniCompanion, Open-LLM-VTuber, …) gets a real agent with zero client changes. messages[] is flattened into the agent prompt; the agent's streamed reply is re-emitted as OpenAI chat.completion.chunk deltas via a per-request channel.id registry drained by the /ws recv loop. Inline [emotion] tags pass through untouched. Tier 1 of RFC openabdev#1233. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NXngQyvmJwQPNYsiRUNU2m
Additional: Frontend WebSocket Client DemoAdded examples/vtuber-demo/index.html — a minimal WebSocket client reference implementation for VTuber skins. What it does:Connects to OAB Gateway via raw WebSocket (ws://host:8080/ws?token=) Usage:Open index.html in browser, configure WS URL and token, connect and chat. Note:Currently the gateway's handle_oab_connection does not forward GatewayEvents from WS clients to OAB core — backend update needed for full functionality. Code: |
Tier-1 complete ✅All validation items checked:
CI: all checks and smoke tests passing. Next: Tier-2Tier-2 RFC opened as #1235 — WebSocket side-channel ( |
This comment has been minimized.
This comment has been minimized.
Tier-2 adds GET /v1/vtuber/ws — a persistent WebSocket that pushes agent_state, emotion, and notification events derived from GatewayReply commands. VTuber skins connect once and receive real-time state updates (thinking/working/idle, tool usage, emotion tags) without polling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NXngQyvmJwQPNYsiRUNU2m
|
Tier-2 WebSocket event stream pushed (acd3e38). Pending: e2e testing with a live VTuber skin. |
This comment has been minimized.
This comment has been minimized.
Replace Vec<WsClient> with HashMap<u64, WsClient> and an AtomicU64 counter. The old Vec+index scheme broke when broadcast() called swap_remove on dead clients — surviving clients' stored indices became stale, routing subscribe/pong to the wrong connection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NXngQyvmJwQPNYsiRUNU2m
This comment has been minimized.
This comment has been minimized.
F2: Cap in-flight /v1/chat/completions at 32 by checking vtuber_pending size before accepting — returns 429 when full. F3: Emit SSE comment `: waiting for agent` after 10s of silence, giving clients an early signal before the 180s hard timeout. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NXngQyvmJwQPNYsiRUNU2m
|
Review findings addressed:
|
This comment has been minimized.
This comment has been minimized.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NXngQyvmJwQPNYsiRUNU2m
|
All findings resolved, clippy CI fixed in c1b9495. Ready for merge — pending e2e testing with a live VTuber skin. |
This comment has been minimized.
This comment has been minimized.
|
Caution This PR has been waiting on the author for more than 2 days (labeled @canyugs — You must add a new comment on this PR to remove the |
Bring the branch up to openab-0.9.0. Conflict resolutions: - Cargo.toml: take main's default feature set (adds `filestore`); the `vtuber` feature and its slot in `unified` are unchanged. - main.rs `has_unified_platform`: keep main's config-first Google Chat resolver and re-attach the env-signaled vtuber arm after it. - main.rs unified routing: adopt main's `gw_state.*_webhook_path` fields and the shared trust-registry gating (openabdev#1391), re-attaching the vtuber `/v1/chat/completions` route after the Google Chat route. - main.rs tests: `has_unified_platform_env` was renamed upstream to `has_unified_platform(&cfg)`; port the two vtuber cases over. - vtuber.rs test AppState: add the three new webhook path fields. Verified: cargo test --workspace --features vtuber (all vtuber tests pass). `secrets::tests::resolve_exec_nonzero_exit` fails identically on clean upstream main — pre-existing, unrelated to this merge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Important CHANGES REQUESTED What This PR DoesThis PR adds an OpenAI-compatible How It WorksThe endpoint accepts streaming chat requests, maps system/developer instructions plus the latest user-like message into a gateway event, and routes gateway replies back as OpenAI-compatible SSE chunks. A shared Findings
Finding Details🔴 F1: Clean up request routing when the client disconnectsThe request stores its sender under the constant Requested change: Make cancellation/disconnect cleanup remove the request's own registry entry, using compare-and-remove/ownership protection so it cannot remove a replacement; also ensure late replies from an abandoned turn cannot match a subsequent request (for example, use a per-request correlation key while retaining the persistent ACP session separately). 🟡 F2: Require authentication for an enabled endpoint
Requested change: Fail startup or refuse requests whenever the adapter is enabled without a non-empty 🟢 F3: Focused protocol and lifecycle testsThe added tests cover the streaming-only contract, busy-request behavior, exact auth-key comparison, prompt shaping, and normal tail-idle registry cleanup. These provide a useful base for adding the missing disconnect/cancellation regression case. Baseline Check
What's Good (🟢)
Addressing External Reviewer FeedbackNo external reviewer feedback was present when this review round was prepared. 5️⃣ Three Reasons We Might Not Need This PR
|
chaodu-agent
left a comment
There was a problem hiding this comment.
Important
CHANGES REQUESTED
Consolidated review: #1234 (comment)
| .insert(axum::http::header::RETRY_AFTER, "3".parse().unwrap()); | ||
| return resp; | ||
| } | ||
| pending.insert(channel_id.clone(), tx); |
There was a problem hiding this comment.
� F1 � Clean up request routing when the client disconnects
If the SSE client disconnects, the stream is dropped before the cleanup at line 430. Since this request uses the fixed vtb_persistent key, the next request can replace this sender and receive late replies from the abandoned turn.
Requested change: Add cancellation-safe, ownership-aware cleanup and a per-request correlation mechanism so an abandoned turn cannot be routed into a later request.
| .into_response(); | ||
| }; | ||
|
|
||
| if let Some(expected) = cfg.auth_key.as_ref() { |
There was a problem hiding this comment.
� F2 � Require authentication when the adapter is enabled
This branch only validates a key when one exists. VTUBER_ENABLED=true with no VTUBER_AUTH_KEY therefore mounts an unauthenticated endpoint that can invoke the agent and its tools.
Requested change: Fail startup/refuse requests without a non-empty key, or require an explicit local-only unauthenticated opt-in, and document that contract.
What problem does this solve?
Desktop character apps such as AniCompanion, Open-LLM-VTuber, and ChatVRM already speak OpenAI chat completions, but they usually connect to a raw LLM. This PR lets those skins point at OpenAB instead, so the same UI gets ACP-backed tool use, code editing, memory, MCP, and existing OpenAB steering.
Closes #1233
Discord Discussion URL: https://discord.com/channels/1491295327620169908/1520790210320011274
Architecture
The VTuber adapter now runs inside the unified OpenAB binary:
No separate gateway process or adapter config block is required for VTuber in unified mode. Set
VTUBER_ENABLED=trueon the OpenAB process, and the unified HTTP listener exposes the OpenAI-compatible endpoint.Proposed Solution
OpenAI-Compatible SSE
POST /v1/chat/completionsstreams OpenAI-compatiblechat.completion.chunkevents.[emotion]tags pass through for skins that already parse and strip them before TTS.vtb_persistentchannel reuses one warm ACP session for VTuber traffic, avoiding per-turn cold starts.429withRetry-After: 3.messages[]; the adapter forwards system/developer instructions plus the latest user turn so assistant history is not duplicated inside the persistent ACP session.Why this approach?
Validation
cargo test -p openab-gateway vtuber— 7 VTuber tests passedcargo test -p openab has_unified_platform_env_checks --features unifiedcargo check -p openab --features unifiedcargo clippy -- -D warningscargo testNotes
docs/vtuber.mddocuments unified-mode setup only.docs/config-reference.mdlists the VTuber unified environment variables.