Error loading MIMICNotes

[kiranchari]

I successfully downloaded MIMICNotes following the instructions provided here: https://github.com/YyzHarry/SubpopBench/blob/main/MedicalData.md#mimicnotes

When I try to train a model on MimicNotes, I got the following error when loading the features.npy file due to this line:

SubpopBench/subpopbench/dataset/datasets.py

Line 473 in 4d3dbbe

self.x_array = np.load(os.path.join(data_path, "mimic_notes", 'features.npy'))

raise ValueError("Object arrays cannot be loaded when "
ValueError: Object arrays cannot be loaded when allow_pickle=False

I then added allow_pickle=True in the np.load() statement above, which fixed this error. But then I get a different error due to this line:

SubpopBench/subpopbench/dataset/datasets.py

Line 478 in 4d3dbbe

return self.x_array[int(x), :].astype('float32')

    return self.x_array[int(x), :].astype('float32')
IndexError: too many indices for array: array is 0-dimensional, but 2 were indexed

Upon inspection, self.x_array does not look like a standard numpy ndarray but a sparse matrix in Compressed Sparse Row format.

Could you please advise how to correcty load and index this dataset?

Thanks!