The current fast data downloader is fine, but only handles recording channels. They're not the only things that are desired for parallel downloading, especially given @rowleya's megamodel!
There are also:
- General Regions (used for weights, provenance, etc.)
- IOBUFs
The downloading of general regions requires a data model update. In particular, we have to think of them as being snapshots of the state (instead of streams) and that requires additional information, especially what the current simulation timestep is. (In theory, it also requires the run number, but I'm not sure if that is not implicit in which database we're writing to.)
IOBUF downloading is mostly there, but I don't know if we need to extend what we've got. There's also the question of whether to pull back data into a pile of files (which are then easily examinable with text editors) or to instead put it into a database. I also don't know whether the code has kept pace with @Christian-B's changes to mapping dictionaries.
Feedback/design assistance would be welcome.
The current fast data downloader is fine, but only handles recording channels. They're not the only things that are desired for parallel downloading, especially given @rowleya's megamodel!
There are also:
The downloading of general regions requires a data model update. In particular, we have to think of them as being snapshots of the state (instead of streams) and that requires additional information, especially what the current simulation timestep is. (In theory, it also requires the run number, but I'm not sure if that is not implicit in which database we're writing to.)
IOBUF downloading is mostly there, but I don't know if we need to extend what we've got. There's also the question of whether to pull back data into a pile of files (which are then easily examinable with text editors) or to instead put it into a database. I also don't know whether the code has kept pace with @Christian-B's changes to mapping dictionaries.
Feedback/design assistance would be welcome.