Skip to content

RECOMMENDATIONS.md

Latest commit

 

History

History
80 lines (59 loc) · 3.46 KB

RECOMMENDATIONS.md

File metadata and controls

80 lines (59 loc) · 3.46 KB
Raw

System Improvement Recommendations

1. Architectural Refactoring (High Priority)

Problem: voice_chat_app.py has grown to ~4,600 lines, becoming a monolithic single point of failure that is hard to test and maintain.

Proposal: Split the application into a modular app/ directory structure:

app/
├── core/
│   ├── orchestration.py    # Main event loop
│   └── lifecycle.py        # Startup/Shutdown logic
├── services/
│   ├── llm.py              # LLM Client wrapper
│   ├── tts.py              # TTS Engine interfaces
│   └── stt.py              # Whisper integration
├── ui/
│   ├── layout.py           # Gradio Blocks definition
│   └── components/         # Reusable UI widgets
├── api/
│   └── routes.py           # Future REST API endpoints
└── main.py                 # New entry point

Benefit:

  • Decoupled logic allows for safer feature additions.
  • Enables independent testing of services (e.g., testing LLM logic without loading TTS).
  • Facilitates multi-agent development.

2. True "Barge-In" Interruptibility

Problem: The user cannot naturally interrupt the agent during long responses. They must wait for the full response or manually click "Stop".

Proposal:

  1. VAD-Triggered Flush: Connect vad_recorder.py output directly to the AudioQueue.
  2. Logic:
    • IF VAD.is_speech() == True
    • AND AudioQueue.is_playing() == True
    • THEN:
      1. Flush AudioQueue (stop playback immediately).
      2. Cancel pending LLM stream generation.
      3. Treat the interruption as a new user turn.

Benefit: Creates a natural, full-duplex conversational flow.

3. Emotional Feedback Loop

Problem: The system detects emotion (Input) and has expressive TTS (Output), but they are not connected. The agent "knows" you are sad but doesn't "sound" empathetic.

Proposal:

  1. Map Emotion to Style: Create a mapping between SER results and TTS parameters.
    • sad -> Slower speed (0.9x), lower pitch, softer voice profile.
    • happy -> Faster speed (1.1x), higher pitch.
  2. LLM Control: Allow the LLM to output style tags based on the detected emotion context.
    • Input: [User is Angry]
    • LLM Output: <voice style="calm_and_assertive">I understand you are frustrated...</voice>

Benefit: deeply enhances the "human-like" quality of the interaction.

4. "Dream Mode" (Offline Consolidation)

Problem: The Knowledge Graph grows indefinitely, potentially leading to noise and performance degradation over time.

Proposal: Implement a background maintenance task that runs when the system is idle or shutting down.

  1. Cluster & Summarize: Group related episodic memories and summarize them into single semantic facts.
  2. Prune: Remove weak or unused graph connections.
  3. Conflict Resolution: Detect contradictory facts in the graph and flag them for the user to resolve in the next session.

Benefit: Keeps the memory system efficient and simulates biological long-term potentiation (learning).

5. Developer Experience: Test Suite

Problem: Lack of automated tests makes refactoring risky.

Proposal: Create a tests/ directory using pytest.

  • Unit Tests: For MemoryManager scoring logic and Tool execution constraints.
  • Mocks: Mock LLM and Audio interfaces to run tests without GPU/Hardware.

Benefit: Stability and confidence in rapid iteration.