[Future] support real-time translation

[JackZhang1994]

How to support real-time translation. Can you help provide a demo?

[Digipom]

Hi, please check this example for real-time translation: https://github.com/ggerganov/whisper.cpp/tree/master/examples/stream

[Digipom]

Please also check this: ggml-org/whisper.cpp#10

[JackZhang1994]

I want to use the offline real-time translation on Android and iOS devices. I have read stream.cpp code, but I don't find how to use it on Android and iOS devices.

[Digipom]

Definitely, this is something I want to do too.

Streaming quality is also gated on this:

ggml-org/whisper.cpp#426

[Digipom]

I think you could adapt the current code by starting to transcribe the audio right away and padding with silence (there's more discussion about it in ggml-org/whisper.cpp#10). So instead of waiting for the recording to complete, start sending it to Whisper right away.

[JackZhang1994]

I find that iOS Demo support real-time transcription. Can Android implement this in a similar way?
Another question, iOS device transcribe the audio faster than Android device. Why is that? Is there a good solution?

Thanks!

[Digipom]

Can Android implement this in a similar way?

It should be able to.

Another question, iOS device transcribe the audio faster than Android device. Why is that? Is there a good solution?

There's two reasons I can think of at the top of my head:

1. iOS uses Accelerate framework: https://developer.apple.com/accelerate/. Possibly the main repo could speed up Android ops using the neural networks API: https://developer.android.com/ndk/guides/neuralnetworks/
2. Apple's CPUs are really good. Most Android devices lag far behind them in raw processing power.

[JackZhang1994]

Thanks for your reply! 🌹
And sorry, my English is bad.

Can Android implement this in a similar way?

It should be able to.

Do you have any plans to improve the real-time function (like the iOS demo) in the near future? I'm not good at Kotlin. And I've been trying for two days, but it still has a lot of bugs, and the transcription is very slow.

[Digipom]

No problem at all, your English is just fine from my side ;) I'm unfortunately not able to work on it for the next few weeks but it's something I want to bring to our production apps so I'm definitely interested in this, too.

I recommend you use "tiny" for Android -- everything else will be too slow.

[Digipom]

Also. please ensure you're using release mode when you're running from Android studio. In debug mode it will take forever. ;)

[JackZhang1994]

All right, thank you~