feat: update dialogue system, audio service and digital human components

Author: shijiashuai

CLAUDE.md

@@ -0,0 +1,124 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Development commands
+
+- Install dependencies: `npm install`
+- Start dev server: `npm run dev`
+- Production build: `npm run build`
+- Alternate builds:
+  - `npm run build:mobile`
+  - `npm run build:desktop`
+  - `npm run build:ar`
+- Preview production build locally: `npm run preview`
+- Serve preview on `0.0.0.0:3000`: `npm run serve`
+- Lint: `npm run lint`
+- Run tests in watch mode: `npm test`
+- Run tests once: `npm run test:run`
+- Run coverage: `npm run test:coverage`
+- Run a single test file: `npx vitest run src/__tests__/digitalHuman.test.tsx`
+- Run tests matching a name: `npx vitest run -t "test name"`
+
+## Stack and build setup
+
+- React 18 + TypeScript app built with Vite.
+- Path alias `@/*` points to `src/*` in both Vite and Vitest configs.
+- Tailwind CSS is used for UI styling; dark-mode class support is enabled in `tailwind.config.js`.
+- Vitest uses the `jsdom` environment with setup from `src/__tests__/setup.ts`.
+- Vite build modes `mobile`, `desktop`, and `ar` only change compile-time flags (`__MOBILE__`, `__DESKTOP__`, `__AR__`) and output directories (`dist-mobile`, `dist-desktop`, `dist-ar`).
+
+## High-level architecture
+
+### App shell and routing
+
+- Entry point is `src/main.tsx`, which renders `src/App.tsx`.
+- `src/App.tsx` sets up React Router with lazy-loaded pages:
+  - `/` and `/advanced` -> `AdvancedDigitalHumanPage`
+  - `/digital-human` -> `DigitalHumanPage`
+- The app is wrapped in a global `ErrorBoundary` and Suspense fallback UI.
+
+### Two page modes
+
+- `src/pages/AdvancedDigitalHumanPage.tsx` is the main experience and the default route. It combines:
+  - full-screen 3D viewer background
+  - settings drawer with tabs for basic controls, expressions, behavior, vision, and voice
+  - chat/session UI
+  - server health checks and reconnect flow
+  - keyboard shortcuts and toast-driven status feedback
+- `src/pages/DigitalHumanPage.tsx` is a simpler demo page with the viewer plus a basic control panel.
+
+### Central state model
+
+- `src/store/digitalHumanStore.ts` is the central Zustand store for nearly all runtime state.
+- It holds playback, recording, mute, speaking, expression/emotion, behavior, connection status, loading/error state, and chat/session history.
+- Session IDs are persisted in `localStorage` under `metahuman_session_id`, with SSR-safe storage access.
+- Core services and pages commonly read/write state directly via `useDigitalHumanStore.getState()` rather than passing state deeply through props.
+
+### Core runtime layers
+
+The app is organized around `src/core/*` services:
+
+- `src/core/avatar/DigitalHumanEngine.ts`
+  - imperative façade over the Zustand store
+  - translates high-level actions like `play`, `reset`, `setEmotion`, `setBehavior`, `playAnimation` into store updates
+  - contains emotion -> expression mapping and timed auto-reset for animations
+- `src/core/audio/audioService.ts`
+  - browser-only audio integration using Web Speech APIs
+  - `TTSService` drives speech synthesis and updates speaking/behavior store state
+  - `ASRService` wraps speech recognition, handles command mode vs dictation mode, and can forward transcripts into dialogue handling
+- `src/core/dialogue/dialogueService.ts`
+  - HTTP client for backend chat requests
+  - sends requests to `${VITE_API_BASE_URL || 'http://localhost:8000'}/v1/chat`
+  - checks `${baseUrl}/health` for connectivity
+  - includes timeout handling, retry logic for retryable failures, friendly error messages, and a local fallback reply when backend calls fail
+- `src/core/dialogue/dialogueOrchestrator.ts`
+  - orchestrates a full dialogue turn
+  - appends user/assistant messages to store history
+  - toggles loading/thinking state
+  - applies backend response emotion/action to the avatar engine
+  - optionally invokes TTS for spoken replies
+- `src/core/vision/visionService.ts`
+  - camera + MediaPipe integration for face/pose analysis
+  - dynamically imports `@mediapipe/face_mesh` and `@mediapipe/pose`
+  - maps face landmarks to emotion and derives motions like nod/shake/raiseHand/waveHand
+  - model files are loaded from jsDelivr CDN at runtime, so vision features depend on camera permission and network access
+- `src/core/vision/visionMapper.ts`
+  - converts raw face landmarks into the app’s higher-level emotion model
+
+### UI/component structure
+
+- `src/components/DigitalHumanViewer.tsx` is the 3D rendering boundary.
+  - Uses React Three Fiber + Drei.
+  - If `modelUrl` loads successfully, it renders the GLTF scene.
+  - If loading fails or no model is supplied, it falls back to an internal procedural “CyberAvatar”.
+  - Viewer behavior is driven from store state (`currentExpression`, `isSpeaking`, `currentAnimation`, `expressionIntensity`).
+- Control panels (`ControlPanel`, `ExpressionControlPanel`, `BehaviorControlPanel`, `VoiceInteractionPanel`, `VisionMirrorPanel`) are mostly thin UI layers that call into the engine/services.
+- Shared UI primitives live under `src/components/ui`.
+
+## Backend/API assumptions
+
+- The frontend expects a separate backend service at `VITE_API_BASE_URL` or `http://localhost:8000`.
+- Chat response shape expected by the frontend:
+  - `replyText: string`
+  - `emotion: string`
+  - `action: string`
+- Health endpoint expected: `GET /health`
+- Chat endpoint expected: `POST /v1/chat`
+
+## Testing notes
+
+- Current test coverage is centered in `src/__tests__/digitalHuman.test.tsx`.
+- Tests heavily mock Three.js, React Three Fiber, and browser speech APIs; follow that pattern when adding UI/runtime tests for viewer or audio behavior.
+- Because the app relies on browser APIs (speech synthesis, speech recognition, camera/media devices), new tests usually need mocks rather than real integrations.
+
+## Practical implementation notes
+
+- Prefer modifying the advanced page flow unless the task is explicitly about the simpler `/digital-human` demo.
+- For behavior changes affecting avatar reactions, inspect the interaction between:
+  - `src/store/digitalHumanStore.ts`
+  - `src/core/avatar/DigitalHumanEngine.ts`
+  - `src/core/dialogue/dialogueOrchestrator.ts`
+  - `src/components/DigitalHumanViewer.tsx`
+- For backend chat issues, check both `dialogueService.ts` retry/fallback behavior and `AdvancedDigitalHumanPage.tsx` health-check/reconnect UI.
+- For speech features, verify whether logic belongs in browser service wrappers (`audioService.ts`) or page-level orchestration.

server/app/services/dialogue.py

@@ -30,6 +30,33 @@ def __init__(self) -> None:
     except ValueError:
       self.max_session_messages = 10
 
+  def _normalize_user_text(self, user_text: str) -> str:
+    return (user_text or "").strip()
+
+  def _append_session_history(self, session_id: str, role: str, content: str) -> None:
+    if not session_id or not content:
+      return
+    session_histories[session_id].append({
+      "role": role,
+      "content": content,
+      "timestamp": datetime.now().isoformat(),
+    })
+    if len(session_histories[session_id]) > MAX_HISTORY_LENGTH * 2:
+      session_histories[session_id] = session_histories[session_id][-MAX_HISTORY_LENGTH * 2:]
+
+  def _append_turn(self, session_id: Optional[str], user_text: str, reply_text: str) -> None:
+    if not session_id:
+      return
+    self._append_session_history(session_id, "user", user_text)
+    self._append_session_history(session_id, "assistant", reply_text)
+    self._append_session_messages(
+      session_id,
+      [
+        {"role": "user", "content": user_text},
+        {"role": "assistant", "content": reply_text},
+      ],
+    )
+
   def _get_smart_mock_reply(self, user_text: str) -> Dict[str, Any]:
     """智能本地 Mock 回复，根据用户输入生成合理的响应"""
     text_lower = user_text.lower()
@@ -69,27 +96,19 @@ async def generate_reply(
     
     支持会话历史管理和 LLM 调用，当 API Key 未配置时使用智能 Mock 回复。
     """
-    # 记录用户消息到会话历史
-    if session_id:
-      session_histories[session_id].append({
-        "role": "user",
-        "content": user_text,
-        "timestamp": datetime.now().isoformat(),
-      })
-      # 限制历史长度
-      if len(session_histories[session_id]) > MAX_HISTORY_LENGTH * 2:
-        session_histories[session_id] = session_histories[session_id][-MAX_HISTORY_LENGTH * 2:]
+    session_id = (session_id or "").strip() or None
+    user_text = self._normalize_user_text(user_text)
+    if not user_text:
+      return {
+        "replyText": "我在听，请告诉我您想聊什么。",
+        "emotion": "neutral",
+        "action": "idle",
+      }
 
     if not self.api_key:
       logger.info("OPENAI_API_KEY 未配置，使用智能 Mock 回复")
       result = self._get_smart_mock_reply(user_text)
-      # 记录助手回复到历史
-      if session_id:
-        session_histories[session_id].append({
-          "role": "assistant",
-          "content": result["replyText"],
-          "timestamp": datetime.now().isoformat(),
-        })
+      self._append_turn(session_id, user_text, result["replyText"])
       return result
 
     system_prompt = (
@@ -141,11 +160,13 @@ async def generate_reply(
         parsed = json.loads(content)
       except json.JSONDecodeError:
         logger.warning("LLM 返回内容不是合法 JSON，将内容作为 replyText 使用: %s", content)
-        return {
+        result = {
           "replyText": content,
           "emotion": "neutral",
           "action": "idle",
         }
+        self._append_turn(session_id, user_text, result["replyText"])
+        return result
 
       reply_text = str(parsed.get("replyText", "")).strip() or f"你刚才说：{user_text}"
       emotion = str(parsed.get("emotion", "neutral")).strip() or "neutral"
@@ -156,20 +177,7 @@ async def generate_reply(
       if action not in {"idle", "wave", "greet", "think", "nod", "shakeHead", "dance", "speak"}:
         action = "idle"
 
-      # 记录消息到会话历史
-      if session_id:
-        session_histories[session_id].append({
-          "role": "assistant",
-          "content": reply_text,
-          "timestamp": datetime.now().isoformat(),
-        })
-        self._append_session_messages(
-          session_id,
-          [
-            {"role": "user", "content": user_text},
-            {"role": "assistant", "content": reply_text},
-          ],
-        )
+      self._append_turn(session_id, user_text, reply_text)
 
       return {
         "replyText": reply_text,
@@ -181,7 +189,9 @@ async def generate_reply(
         "LLM 请求超时 url=%s，将使用智能 Mock 回复",
         self._get_openai_chat_completions_url(),
       )
-      return self._get_smart_mock_reply(user_text)
+      result = self._get_smart_mock_reply(user_text)
+      self._append_turn(session_id, user_text, result["replyText"])
+      return result
     except httpx.HTTPStatusError as exc:
       body_preview = (exc.response.text or "")[:500]
       logger.error(
@@ -190,29 +200,39 @@ async def generate_reply(
         str(exc.request.url),
         body_preview,
       )
-      return self._get_smart_mock_reply(user_text)
+      result = self._get_smart_mock_reply(user_text)
+      self._append_turn(session_id, user_text, result["replyText"])
+      return result
     except httpx.RequestError as exc:
       req_url = str(exc.request.url) if exc.request else self._get_openai_chat_completions_url()
       logger.error(
         "LLM 请求异常 url=%s error=%s，将使用智能 Mock 回复",
         req_url,
         exc,
       )
-      return self._get_smart_mock_reply(user_text)
+      result = self._get_smart_mock_reply(user_text)
+      self._append_turn(session_id, user_text, result["replyText"])
+      return result
     except Exception as exc:
       logger.exception("调用 LLM 失败，将使用智能 Mock 回复: %s", exc)
-      return self._get_smart_mock_reply(user_text)
+      result = self._get_smart_mock_reply(user_text)
+      self._append_turn(session_id, user_text, result["replyText"])
+      return result
 
   def clear_session(self, session_id: str) -> bool:
     """清除指定会话的历史记录"""
+    removed = False
     if session_id in session_histories:
       del session_histories[session_id]
-      return True
-    return False
+      removed = True
+    if session_id in self._session_messages:
+      del self._session_messages[session_id]
+      removed = True
+    return removed
 
   def get_session_history(self, session_id: str) -> List[Dict[str, str]]:
     """获取指定会话的历史记录"""
-    return session_histories.get(session_id, [])
+    return list(session_histories.get(session_id, []))
 
   def _get_openai_chat_completions_url(self) -> str:
     base_url = (self.base_url or "").strip()
@@ -268,7 +288,7 @@ async def _call_llm(self, messages: list[dict[str, str]]) -> Dict[str, Any]:
     return resp.json()
 
   def _get_session_messages(self, session_id: str) -> list[dict[str, str]]:
-    return self._session_messages.get(session_id, [])
+    return list(self._session_messages.get(session_id, []))
 
   def _append_session_messages(
     self,