Running out of ram

[jesuistay]

I couldnt get HTK to work properly, possibly due to bad installation. But seamed to work fine with librosa.

However when it comes to '===> Reading audio files... it seams like the for loop going over the audio paths just fills up my 8gb of ram and swap. And this is only on the 28539 files from train-clean-100. And it doesn't produce any files at this stage.
Is there a trick I am missing to get the preprocessor going without reading everything into ram all at once?
Eta was over 1 hour and it broke down after 54% of the train-clean-100 dataset.

[hirofumi0810]

Hi, @jesuistay

Only 1 file will be loaded In each loop for the memory efficiency, so I don't know why.
Which loop do you mean?
There are 3 loops in librispeech/inputs/input_data.py.

[jesuistay]

First one: for i, audio_path in enumerate(tqdm(audio_paths)):
To me it looks like it traverses the entire dataset and creating the dict, in order to calculate the mean std I assume.

For now I've just skipped the all the normalization and just writing the npy files after I get the input_data_utt from librosa.

[jesuistay]

I managed to get HTK working, but the ram problem still confuses me. I had to increase my swap partition to 16 gb (+ 8gb of ram) just to manage and preprocess the clean-100