New open-source app AI Bluetooth Phone Switchboard: using whisper.cpp as a real-time hallucination filter for cloud AI

[antor44]

Hi everyone,

I wanted to share a new open-source Linux project I’ve been working on where I use whisper.cpp to solve the text-transcription deficiencies of cloud-based multimodal models.

I built an AI Bluetooth Phone Switchboard (using Python, oFono, PipeWire, and Gemini Live). It connects to a mobile phone via Bluetooth HFP to act as an autonomous secretary, answering calls and filtering SPAM.

🔗 GitHub Repo: AI-Bluetooth-Phone-Assistant

The Problem

While the cloud API handles the low-latency audio-to-audio conversation quite well, its text-transcription channel is incredibly error-prone. It struggles heavily with general background noise, poor audio quality, or even just regular speech, frequently hallucinating random foreign characters, repetitive loops, or simply getting the transcription completely wrong.

The whisper.cpp Solution

To maintain a clean SQLite database of call transcripts (which the system uses as contextual memory for repeat callers), I integrated whisper.cpp as a local, asynchronous post-processor to "heal" the cloud's output.

Here is the workflow:

1. The daemon captures the raw PCM audio stream from the caller (pw-record).
2. It buffers short audio chunks during the conversation.
3. When the cloud API generates a text transcription, the daemon immediately spawns a local whisper-cli process against the corresponding audio chunk in /tmp.
4. It compares both outputs. Because the local whisper model is vastly more robust against noise and hallucination, if it outputs silence, filters the noise, or provides a clean transcription, the daemon overrides the cloud's text and updates the live transcript database.
5. After the call drops, it can also run a final offline batch alignment for maximum accuracy.

Using whisper-cli on the CPU is fast enough that this healing process happens in the background without blocking the main asynchronous telephony loop.

If anyone is experimenting with Linux audio routing mixed with local/cloud AI, feel free to check the repo. Feedback or suggestions on the architecture are welcome.

Thanks to the contributors for building such a reliable and lightweight tool; it made this hybrid architecture feasible.