Feature request: Add support for OpenAI-compatible API STT/TTS providers

[bsarkisov]

Currently we are limited by using OpenAI via API or local whisper.cpp-powered models.
With a great progress over last 1-2 years there are multiple other models that can be self-hosted and used via exisitng OpenAI-compatible APIs. Using these models we can vastly improve voice recognition and voice generation and thus user experience.

[dharmab]

I have looked at many of these models, however a lot of them struggle with aviation brevity. Many can't properly pronounce things like NATO reporting names (they struggle with "Fishbed" in particular).

I did take a serious look at STT with nvidia parakeet on the v2-dev branch, but the results were really inconsistent compared to whisper.cpp. Sometimes it was better, most of the time it was much worse.

I think the path forward is probably to enable GPU acceleration of whisper.cpp, unfortunately I have limited access to hardware to test on. But I know others have made personal forks of SkyEye that can use CUDA and they're getting a really great experience.

[dharmab]

Another advantage of whisper.cpp: It's very easy to distribute. All of the code gets packaged right in the exe file and the model is a single file. ONNX runtime is a huge pain by comparison, models get distributed as loose collections of weights and random JSON files.

[bsarkisov]

That's valid from both distribution and support standpoints. Sorry, my initial comment is a bit misleading.
Just to clarify - my ask is not to adopt/support another voice engine within current distribution.
Idea is to provide second option for self-hosted OpenAI API provider, in addition to official OpenAI, in the config, to keep change minimal and re-use existing API implementation. That also means that Skyeye isn't responsible for the other voice engine implementation and only expect OpenAI-api compatibility.
So user can have:

• Official OpenAI provider (param: api-key)
• OpenAI compatible (params: base-url, api-key)
• Local whisper.cpp (param: model name)

I can try to implement it myself within PR however that probably would be vibecoding. :)

[bsarkisov]

I can also provide access to hosted service and/or test the change myself.

[dharmab]

That also means that Skyeye isn't responsible for the other voice engine implementation and only expect OpenAI-api compatibility.

Unfortunately this isn't the case; the voice recognition uses whisper-specific features (pkg/speakers uses Whisper's prompting features to improve recognition of aviation brevity) and the parser (pkg/parser) is specifically designed to post-process the Whisper model's output. The API calls SkyEye makes are not generically compatible with other models, and the outputs of a different model will not match well with the current parser's methods. ML models are not like a lego brick you can swap out, the application really does need to built around the model's available calls, capabilities and weaknesses.

The only use case I see here is if you had multiple SkyEye instances and you wanted to share a single whisper.cpp server between them to save on RAM. I might play around with that approach, but generic OpenAI API support is not something I'm interested in doing in the main repo. You are of course welcome to fork SkyEye and implement features in your fork, and in fact I encourage it.; I'm not a big believer that open source means every change has to flow back to the origin, I'm actually quite happy to see people fork and customize software to fit their own needs.

Overall the direction I want to move the project in the longer term is make the local AI models so good that there's no reason to use cloud hosted models. This is already the case on macOS where local recognition is much faster than OpenAI's API. And others have forked SkyEye to add CUDA support on Linux and Windows and gotten a great experience too; the only reason that CUDA isn't in the main repo is I don't have hardware to test on.

[dharmab]

A concrete example: #694 uses a whisper-specific feature not available in many other STT models.

[bsarkisov]

That's a good one, thanks!