Skip to content

AUDIO_CONVERSATION_GUIDE.md

Latest commit

 

History

History
269 lines (216 loc) · 7.68 KB

AUDIO_CONVERSATION_GUIDE.md

File metadata and controls

269 lines (216 loc) · 7.68 KB
Raw

Audio Conversation Implementation Guide

Overview

This implementation provides full audio conversation functionality for the R3F AI Chatbot, including:

  • Voice Input: Users can speak to the AI using their microphone
  • Audio Responses: AI always responds with synthesized speech
  • Avatar Animations: Dynamic avatar animations based on conversation state
  • Real-time Feedback: Visual indicators for listening, thinking, and speaking states

Features Implemented

1. Voice Input System

  • Speech Recognition: Uses browser's Web Speech API for voice-to-text conversion
  • Auto-send: Voice messages are automatically sent after speech detection ends
  • Visual Feedback: Real-time audio level indicators and transcription display
  • Error Handling: Graceful fallback when microphone access is denied

2. Audio Response System

  • Always Audio: All AI responses are converted to speech using ElevenLabs API
  • Audio Caching: Responses are cached to improve performance
  • Queue Management: Multiple audio responses are queued and played sequentially
  • Progress Tracking: Real-time playback progress and loading states

3. Avatar Animation System

  • Dynamic Models: Different 3D models for different states:
    • avatar.glb - Idle state (default)
    • avatar_thinking.glb - When user is speaking OR AI is processing input
    • avatar_talking.glb - When AI is responding with audio
  • Smooth Transitions: Seamless transitions between animation states
  • Visual Indicators: Color-coded status indicators above avatar
  • Model Path: /public/models/avatar/ directory

4. State Management

  • Centralized Store: Zustand store manages all conversation state
  • Real-time Updates: Avatar animations sync with audio playback
  • Error Recovery: Automatic state reset on errors

Usage Flow

Voice Conversation Flow

  1. User Starts Speaking:

    • Click microphone button or press Space
    • Avatar shows thinking animation (avatar_thinking.glb)
    • Real-time transcription appears

  2. User Stops Speaking:

    • Speech recognition automatically stops after silence
    • Message is auto-sent to AI
    • Avatar continues thinking animation while processing

  3. AI Responds:

    • Text response is generated via Gemini API
    • Audio is synthesized via ElevenLabs API
    • Avatar switches to talking animation (avatar_talking.glb)
    • Audio plays with visual progress indicator

  4. Conversation Continues:

    • Avatar returns to idle state after audio completes
    • Ready for next voice input

Text Conversation Flow

  1. User Types Message: Standard text input
  2. AI Responds: Same audio response flow as voice input
  3. Avatar Animations: Same talking animation during audio playback

Technical Implementation

Key Components

VoiceRecorder Component

// Enhanced with auto-send and avatar state management
<VoiceRecorder
  autoSend={true}
  showTranscription={true}
  disabled={isLoading}
/>

Avatar Component

// Automatically uses correct model based on state
<Avatar
  position={[-10, 0, 6]}
  scale={[2, 2, 2]}
  rotation={[0, 90, 0]}
/>

Audio Status Component

// Shows real-time conversation status
<AudioStatus />

State Management

Avatar States

const avatarState = {
  animation: 'idle' | 'thinking' | 'speaking',
  emotion: 'neutral' | 'happy' | 'thinking',
  isSpeaking: boolean,
  isListening: boolean
}

Voice States

const voiceState = {
  isRecording: boolean,
  isProcessing: boolean,
  transcription: string,
  error: string | null
}

Audio States

const audioState = {
  isPlaying: boolean,
  queue: Array,
  volume: number,
  error: string | null
}

API Integration

Voice Synthesis (ElevenLabs)

// Automatic audio generation for all AI responses
const response = await fetch('/api/voice', {
  method: 'POST',
  body: JSON.stringify({
    text: aiResponse,
    voiceSettings: {
      stability: 0.5,
      similarity_boost: 0.75,
      style: 0.0,
      use_speaker_boost: true
    }
  })
});

Speech Recognition (Browser API)

// Automatic transcription with real-time feedback
const recognition = new (window.SpeechRecognition || window.webkitSpeechRecognition)();
recognition.continuous = true;
recognition.interimResults = true;

Testing

Manual Testing

Visit /test-audio-conversation page to test:

  • Voice input functionality
  • Audio response playback
  • Avatar animation transitions
  • Error handling scenarios

Automated Testing

npm test -- audioConversation.test.js

Configuration

Environment Variables

# Required for audio responses
NEXT_PUBLIC_ELEVENLABS_API_KEY=your_api_key
NEXT_PUBLIC_ELEVENLABS_VOICE_ID=your_voice_id

# Required for AI responses
NEXT_PUBLIC_GEMINI_API_KEY=your_api_key

Avatar Models

Place these 3D models in /public/models/avatars/:

  • avatar.glb - Idle animation (already exists)
  • avatar_thinking.glb - Thinking/listening animation (used when user is speaking or AI is processing)
  • avatar_talking.glb - Speaking animation (used when AI is responding with audio)

Browser Compatibility

Voice Input Requirements

  • Chrome/Edge: Full support
  • Firefox: Partial support (may require user gesture)
  • Safari: Limited support
  • Mobile: Varies by device and browser

Audio Playback

  • All Modern Browsers: Full support for audio playback
  • Mobile: Requires user interaction to start audio

Performance Considerations

Audio Caching

  • Responses are cached for 1 hour to reduce API calls
  • Cache automatically cleans expired entries
  • Object URLs are properly revoked to prevent memory leaks

3D Model Loading

  • Models are preloaded for smooth transitions
  • LOD (Level of Detail) optimization for mobile devices
  • Automatic quality adjustment based on performance

Mobile Optimization

  • Touch-optimized controls
  • Reduced model complexity on mobile
  • Adaptive quality settings

Troubleshooting

Common Issues

  1. Microphone Access Denied

    • Check browser permissions
    • Ensure HTTPS connection (required for microphone access)
    • Try refreshing the page

  2. No Audio Playback

    • Check ElevenLabs API key configuration
    • Verify voice ID is valid
    • Check browser audio permissions

  3. Avatar Not Animating

    • Ensure 3D model files are in correct location
    • Check browser WebGL support
    • Verify avatar state updates in developer tools

  4. Poor Performance

    • Reduce 3D model quality settings
    • Disable shadows on mobile devices
    • Clear audio cache if memory usage is high

Debug Tools

  1. Test Page: /test-audio-conversation - Comprehensive testing interface
  2. Browser DevTools: Monitor state changes and API calls
  3. Console Logs: Detailed logging for audio and avatar state changes

Future Enhancements

Planned Features

  • Lip Sync: Synchronize avatar mouth movements with audio
  • Emotion Detection: Analyze voice tone for emotional responses
  • Multi-language Support: Support for different languages and voices
  • Custom Voices: Allow users to select different voice personalities
  • Gesture Recognition: Add hand gesture recognition for interaction

Performance Improvements

  • WebRTC: Direct peer-to-peer audio streaming
  • WebAssembly: Faster audio processing
  • Service Workers: Offline audio caching
  • WebGL2: Enhanced 3D rendering performance

Support

For issues or questions about the audio conversation implementation:

  1. Check the test page for debugging information
  2. Review browser console for error messages
  3. Verify API key configuration
  4. Test with different browsers and devices