Skip to content

Productionize Live Notes with on-device Whisper transcription #248

Description

@ucguy4u

Problem

Live Notes is now merged into Assistant at /agent/live-notes and Pixel 9 has a working Android SpeechRecognizer bridge with offline preference. That is a useful smoke-tested bridge, but it is not the production-grade on-device Whisper path we want for long team conversations, privacy guarantees, diarization-ready chunks, and consistent offline behavior.

Scope

In scope:

  • Replace or augment the Android SpeechRecognizer bridge with a real on-device Whisper/whisper.cpp runtime behind com.airo.meeting/transcription.
  • Add iOS parity behind the same Flutter MeetingTranscriptionService contract.
  • Preserve explicit-start listening only.
  • Keep raw audio and transcript local by default.
  • Emit partial and final transcript chunks with stable timestamps.
  • Preserve pause, resume, stop, and permission-denied error behavior.
  • Validate on Pixel 9 and at least one iOS simulator/device path.

Out of scope:

  • Cloud transcription fallback.
  • Silent/background always-on listening.
  • Uploading raw audio or raw transcript text.
  • Full task/reminder creation workflow beyond surfacing extracted action candidates.

Acceptance criteria

  • Live Notes uses a real on-device Whisper path on Android when the model is available.
  • Missing model, denied microphone permission, low storage, and native failure states are explicit in UI/state.
  • Pixel 9 test confirms real spoken audio produces partial/final transcript chunks.
  • Pause/resume stops and restarts transcript emission without losing final chunks.
  • Stop finalizes the latest pending text into local intelligence.
  • No network upload path exists for meeting audio or transcript text.
  • Focused meeting tests pass.
  • flutter analyze is clean for the touched surface.
  • Android debug build installs and launches on Pixel 9.

Agent

Primary: agent/ai-llm
Supporting: agent/mobile-ui

Estimate

3-5 days depending on model packaging/download strategy.

Risks

  • Battery drain during long conversations.
  • Model size/storage pressure on mid-range devices.
  • Native audio lifecycle bugs around interruption, Bluetooth microphones, and app backgrounding.
  • Privacy regressions if fallback paths accidentally upload audio or transcript text.

Context

Merged baseline: ccec84f feat(agent): merge assistant model setup
Current route: /agent/live-notes
Compatibility redirects: /meetings, /agent/meetings
Current Android bridge: android.speech.SpeechRecognizer with EXTRA_PREFER_OFFLINE.

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent/ai-llmAI/LLM Agent tasksagent/mobile-uiMobile Platform & UI Agent tasksenhancementNew feature or requeston-deviceOn-device AI processingpriority/P1High - Important but not blockingsprint-2Sprint 2 - Mode Switching and Personalization

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions