Create converters module and prototype converter for Aeon Bonsai (Sumiya)

[JoeZiminski]

This PR begins to add structure for converter for non-NeuroBlueprint formats into NeuroBlueprint, and adds a converter for Bonsai Aeon format that Sumiya is using.

The idea is to create something general and customizable, that we can also extend to have 'default' converters (e.g. from ONE). For writing customizable converters, the (suggested, not currently fixed) approach is to have a format where the converter must be in a python file and be called convterer_func(project) which takes the datashuttle project. Then the writer can search the project for files to converter, create the folders and do the conversion.

This can be trick in some cases because we want to handle a project that is part NeuroBlueprint, part non-NeuroBlueprint. In this case, data is collected in Aeon bonsai. We want to convert it to NeuroBlueprint, then transfer it. The next day, more data will be written in Aeon Bonsai format for conversion. So the converter needs to be able to determine whether the recent Aeon Bonsai file belongs to an existing subject or not (if it does, then the session must be added to this existing subject).
A new function get_sub_names_from_key_value_pair is introduced for this, and can be used alongside the existing project.get_next_sub() and project.get_next_ses().

The workflow is to call project.convert(path_to_module). The function will load the passed module, and run the converter_func() function. It will also initialise logging, and within the conveter function the user can log with project.log_and_message() or project.log_and_raise_error() which are now exposed on the datashuttle class.

Testing this PR

This PR is still in a prototyping phase. Sumiya if you'd like to test this out, you can run the datashuttle/converters/run_convert_sumiya.py script and it will try and convert any folders in /datashuttle/converters/example_project.
The main script that does the conversion and is passed to project.convert() is /datashuttle/converters/convert_sumiya.py. Just to play around you can run this directly.

However, feel free to move it and we can work on it elsewhere e.g. on vr-abcd, whatever is most convenient. Eventually on the datashuttle side, we will probably expose only the project.convert() function to allow any user to insert their convter of choice (and down the line provide only a few default converters e.g. to/from BIDS. Otherwise if it's easier, we can use this PR for prototype and test the Aeon converter function and remove it right at the end.

TODO

• tidy up TODOs on the PR
• write extensive tests for the Aeon-bonsai converter
• tidy up and document datashuttle-side functions and test
• add this to TUI

[sumiya-kuroda]

Hi @JoeZiminski !

This is such an amazing tool! Very very nice! Thank you very much!

We end up making a Bonsai workflow, based on our discussion and the tool you made, which saves the data in NeuroBlueprint format in the first place by itself (https://github.com/SainsburyWellcomeCentre/vr-abcd/blob/add-sessionid/src/workflows/Extensions/LogController.bonsai). This is because:

• in our project, there will no sessions where more than one animal is performing the task.
• everything is moduraly coded in Bonsai which makes it easier for us to debug.

Please also note that there is a bug in this workflow at the moment. https://github.com/SainsburyWellcomeCentre/vr-abcd/pull/34

[JoeZiminski]

Thanks @sumiya-kuroda that's a neat solution! I think if others ask about using Bonsai and writing NeuroBlueprint, I will suggest this approach. Maybe we can meet to discuss this briefly, I'll message you on slack. I could write up a short blog for NeuroBlueprint on how to do this with your approach if others ask about this.

I'll close this PR now and create an issue on converters in datashuttle, as I'm sure this will come up again some time and now we have a prototype implementation. Cheers!