tests : add WER benchmarks

[ggerganov]

It would be nice to start measuring the word error rate (WER) of whisper.cpp across some representative dataset:

• short audio
• long audio
• english
• non-english
• etc.

This will help us catch regressions in the future. I'm not familiar with what is typically used for TTS WER benchmarks, so looking for help from the community.

[harvestingmoon]

Hi Grigory, perhaps we can use LibriSpeech for measuring long audio (approx ~ 1000 hours but could trim it to fit for requirements). For short audio, we can use Libri-Light

Alternatively, there are other audio datasets for measuring WER: https://github.com/jim-schwoebel/voice_datasets

I could start making small sample scripts to see how whisper.cpp fairs among these datasets

[ggerganov]

Thanks. Yes, I'm not sure what is typically used. But in general, I think any dataset would work. The main goal here is not to compare whisper.cpp numbers with other numbers, but to create a reference set of WER numbers that we track as the development continues. This would allow us to catch regressions when they appear, because the WER scores would get worse in such cases.

Ideally, we can have scripts that perform heavier benchmarks that developers would use locally. But we also need a mode where the scripts run just a few fast benchmarks that can be added to the CI without overloading it, so that these would be computed on every commit.

[foldl]

@harvestingmoon are you working on this?

[harvestingmoon]

@foldl hi yes I'm looking at it, more or less likely to start after 12 as it's currently Chinese new year period...

[foldl]

I think we need a tiny dataset (~10MB) just contained in this repo. WER can then be measured on-the-fly.