Strange results - more details about 'getAudioPCMData' and a reference graph?

[fquirin]

Hi @miaobin

this is very interesting work 👍 and I've been planning to integrate it into an audio library for SEPIA I'm working on as a replacement for Meyda MFCC to gain some WASM speed 😀 ... but the results seem a bit confusing 😅

Here is a screenshot comparing Web-MFCC (1, 3) to Meyda MFCC (2):

In graph 1 and 2 I've used the same chunked buffer to create the MFCC step-by-step. 1 is Web-MFCC and 2 is Meyda. In graph 3 I've used Web-MFCC on the same buffer but as one big step and not in chunks. In graph 3 I've also removed the first and every 13th entry of the result array, which leads me to my first question:

1. What exactly is the result of getAudioPCMData as seen in your example? Looking at my results it almost seems like the first entry of every block (1 block = number of MFCCCoefficients = 13) is the sum of the signal instead of a coefficient.

2. Did you check the results against any reference? Comparing Meyda (2) and the cleaned-up Web-MFCC (3) the Web-MFCC result looks scrambled somehow. Now the parameters might not be exactly the same (I've tried to get as close as possible) but I'm pretty sure there is something else going on.

Btw I've used a recording of "yes ... no ... yes" for my graphs.

Any comment or help to get the results fixed would be appreciated 🙂

Ty,
Florian

[miaobin]

@fquirin I am glad you are interested in my work, and sorry to reply you so late because I am celebrating the Chinese New Year.
First of all, the result of getAudioPCMData is the original pcm value of the audio calculated by Web Audio API. Perhaps due to the update of the chrome version, the sampling rate used when collecting audio has changed from the default 44100 to 48000. You can try to create Context use const audioContext = new AudioContext({sampleRate: 44100}); Because when I calculate the windowSize and windowStride, I use the sampling rate of 44100. This is a hard code and I will correct it in next update.
On the other hand, I have compared the results calculated by Web-MFCC and native TensorFlow, and they are the same. I also used Meyda API to calculate MFCC and got the wrong result. I submitted an issue to them and they also replied to me. I think the biggest problem that causes different results is the different parameter settings.
Please feel free to contact me if you have any questions.

BR,
Bin

[fquirin]

Hi @miaobin thanks for the answer and happy new year 😁

the result of getAudioPCMData is the original pcm value of the audio calculated by Web Audio API [...] I use the sampling rate of 44100. This is a hard code and I will correct it in next update.

Maybe I should have mentioned that I'm using my own code to get the PCM data and that I've refactored your library to call the low level functions directly:

var tensorFlowLiteMfccs = Module.cwrap('tf_mfccs', 'number', ['number', 'number', 'number', 'number', 'number', 'number', 'number', 'number', 'number']);
//(pcmPtr, lenPtr, sampleRate, windowSize, windowStride, upperFrequencyLimit, lowerFrequencyLimit, filterbankChannelCount, dctCoefficientCount);
		
WebMfcc.getCoefficients = function(pcmSamples, sampleRate, windowSize, windowStride, upperFrequencyLimit, lowerFrequencyLimit, filterbankChannelCount, dctCoefficientCount){ ...
// ...this is your ' getAudioMfccs' function

I can adjust all the parameters including sample rate, window size, etc. individually and always use the exact same PCM input for both WebMFCC and Meyda where I'm matching the parameters as follows:

• Meyda.sampleRate = sampleRate = 16000 (I've also used 48000 and 44100 and they all give very similar results just different resolution)
• Meyda.bufferSize = windowSize = windowStride = 512
• Meyda.melBands = filterbankChannelCount = 26 (I'm not entirely sure about this one but it doesn't really change much)
• Meyda.numberOfMFCCCoefficients = dctCoefficientCount = 13
• upperFrequencyLimit = 8000 (this seems to be Meyda default)
• lowerFrequencyLimit = 0 (this seems to be Meyda default)

I also used Meyda API to calculate MFCC and got the wrong result. I submitted an issue to them and they also replied to me

It seems that this has been fixed and I'm using the newest version. At least I cannot see this error in my results anymore.

On the other hand, I have compared the results calculated by Web-MFCC and native TensorFlow, and they are the same

Do you maybe still have the Tensorflow results for your test soundfile? It would be a nice reference!

The strange thing is that the WebMFCC results just look like a scrambled version of Meyda ... IF your remove the first line! The first line is giving me some headaches and I wonder if something goes wrong when converting the Tensorflow "tensor" result into an Javascript array and ultimately into the 2D false-color plot you see above.

What I've done to build the false-color plot:

The result of your getAudioMfccs is an 1D array which should contain each of the 13 coefficients for each window right? So I've binned the array into Math.floor(totalSamples/window) bins of each 13 coefficients.
In plot 1 above you can see the first line and it just doesn't make sense. In addition this line doesn't move when you play with upper/lower frequency limits. If it was real it should dissapear in a 100-4000 limit for example.

It would be really great to get some references of maybe Tensorflow and Librosa to see whats going on. Maybe I'll tryout librosa ... it looks easy ^^ and I'll also get some references using your original soundfile :-)

[fquirin]

I've found something that may support my claim about the first line: https://stackoverflow.com/a/60494490
Basically a user is asking why librosa and 'python_speech_features ' give different results and another user answers:

"python_speech_features by default puts energy as first (index zero) coefficient (appendEnergy is True by default), meaning that when you ask for e.g. 13 MFCC, you effectively get 12 + 1"

Though 'python_speech_features ' is not the same as TensorflowLite it seems likely that they behave the same when you look at the data.

[EDIT]
Ok so I got the weirdest result now. Basically I just wanted to check what happens if I accidentally give a Int16Array as input to Meyda (because I know it wants Float32) and ... the result looks EXACTLY like the Web-MFCC result ... even with the strange first line O_o. I will investigate further ... ^^