Multi-file `external_file` on `ImageSeries` cannot represent time alignment with gaps

[h-mayorquin]

The external_file dataset on ImageSeries accepts an array of file paths paired with a starting_frame attribute. I could not find the original discussion around this design (the external_file dataset predates the move to YAML and was already present in the initial JSON schema), but the intent appears to have been covering the common case of consecutive, back-to-back files from a single continuous recording. This is a standard output pattern in camera acquisition software (e.g., Basler Pylon) that automatically splits recordings when files hit a size or container limit. The design only addresses frame access ("which file do I open to get frame N?") and no per-file timing metadata was ever added, which suggests that temporal gaps between files were not a considered use case.

The schema, however, does not state this explicitly. Nothing prevents a writer from listing files with temporal gaps under a single ImageSeries, and some users are already attempting this (see catalystneuro/nwb-video-widgets#34, where a user needs to map a global session time to the right file and local time for each camera across multiple video files with gaps and offsets). When gaps are present, starting_time + rate cannot be used because a single rate implies uniform spacing across all frames, leaving no way to represent the gap between files. The only alternative is to provide the full concatenated timestamps array, where the gap appears as a jump in timestamp values, and look up timestamps[starting_frame[i]] to recover the start time of each file. The closest guidance for this is a comment in pynwb#1318, where Oliver Ruebel confirmed that timestamps should be concatenated and used alongside starting_frame, but this is not documented in the schema itself. It is also problematic from a performance standpoint as even when each segment has a constant sampling rate, the writer is forced to store the full timestamps array (one entry per frame across all files) since starting_time + rate cannot represent gaps.

In summary, the current data type does not have all the necessary information to resolve time alignment of individual videos in an ImageSeries with multiple external files. Time alignment is a critical feature of NWB and we should solve this. One option would be to extend the schema to add per-file timing fields (e.g., a per-file starting_time), but I think this is an undesirable increase in complexity with no real gain, as it would mean maintaining two separate timing mechanisms on the same object: one inherited from TimeSeries for the series as a whole, and a new one for individual files. In my view, multi-file external_file should be scoped to its original intent (gapless continuous recordings split across files for storage reasons), and the schema should guide users toward separate ImageSeries objects when they have multiple videos with gaps or offsets. Each ImageSeries already carries its own starting_time, rate, and timestamps through the TimeSeries inheritance, so separate series handle the gap case naturally without any schema changes.

Even for the gapless case, the schema is not fully self-describing: num_samples is missing for external-mode ImageSeries (#561), so a reader cannot determine the duration of the last file without opening it. Adding num_samples would close this gap and likely make #543 (num_frames_per_external_file) redundant, since frame counts for all but the last file are already inferrable from starting_frame.

Finally, it is true that even in the original intent of automatic file rollover there might be an occasionally drop frame because of a bug. These are recording artifacts, not intentional gaps and crucially the timestamps path is still available to represent the data in that edge case.

[Akseli-Ilmanen]

Hi @h-mayorquin,

Replying to Issue 34 here.

I think the gapless and with actual gaps case ("with-gap") is both common in research. For example, the PAIR-R24M dataset dataset I mentioned in Issue 34 is a case of gapless video files. And in our lab, we have actual gaps between the videos. The reason is two-fold: 1) We record at a very high sampling rate (200Hz), and to prevent the camera from dropping frames, we have individual videos per behavioural trial with (inter-trial-intervals = ~10s). 2) Our video file onsets are initiated by an action of the animal (a crow pulling out a stick tool). I think such event-driven video recordings are very common in naturalistic neuroscience, and it would be great to store those on DANDI.

Regarding timestamps /rate, I think it would be a valid choice to represent each frame (across videos) in one large timestamps vector for the "with-gap" case. In terms of file-size, this would be a bit larger, but compared to the video data itself, quite negligible. Knowing when each video starts in session time and the number of frames per video + rate, it would be fairly simple for the user to construct such a timestamps vector, as done in 1138.

Regarding num_frames_per_external_file vs separate ImageSeries objects. I think a list of num_frames_per_external_file or a tuple (0, num_frames_in_that_video) would be a valid way to do it. I am not sure what a set of ImageSeries objects per file would look like, but I am a bit sceptical. One thing I noticed when adapting the code from nwb-video-widgets is that it's very painful to map video to other nwb objects (e.g. pose data). In the IBL Brain Wide Map, there is "VideoLeftCamera" for the external video, and "LeftCamera" for the PoseEstimationSeries container. The two are only mapped semantically, and there is no reliable automatic mapping. A user can manually map these two together, but mapping a set of external file objects (with again custom semantic naming?) to other NWB objects might get out of hand.

I think when uploading multiple videos per device, it would be great if something like a list of (0, num_frames_in_that_video) tuples would be compulsory.

[h-mayorquin]

I think you are right on the core point. We have actually discussed the trialized data problem before in the NWB ecosystem: pynwb#973 (c.c. @rly). In a nutshell, we don't want people to store one object per trial and instead we should store data as one continuous object instead one per trial. Your data is clearly trialized, and while it can be handled with the concatenated timestamps pattern, it would be good to document this as an exception to my recommendation above. So there are two cases where timestamps is the right approach: (1) the gapless acquisition splits I described originally, and (2) trialized data like yours. That said, your setup makes a strong case for adding full per-file timing metadata to the schema (a rate and starting_time per sub-series), so users don't have to construct the full timestamps array themselves.

On a separate track, I think we can work on better linking between pose estimation data and video regardless of how this is resolved. I have an open PR on ndx-pose (rly/ndx-pose#56) that adds a formal source_video link from PoseEstimation to ImageSeries, replacing the free-form string paths in original_videos which are fragile and break when files are moved (see dandi/dandi-cli#1817). This would address the mapping problem you raised with the IBL example, where "VideoLeftCamera" and "LeftCamera" are only connected by naming convention.

However, your use case exposes a deeper limitation. The source_video link points to the ImageSeries object as a whole, not to a specific file within it. When an ImageSeries has multiple external_file entries (one per trial), there is no way for the pose data to say "I correspond to the 5th video file in this series." One could add a file index to the link, but that would make ndx-pose depend on the internal structure of the external_file array, which is fragile. Alternatively, a reader could cross-reference PoseEstimationSeries timestamps against the ImageSeries timestamps to figure out which file the pose data falls in, but that is a lot of implicit reasoning. The non-external case does not have this problem at all: timestamps just align and a link to the ImageSeries is sufficient. I think this is a sign that external_file was designed for a narrow case (gapless splits) and is reaching its limits as we try to extend it to handle gaps, per-file timing, and linking from other objects to specific files. This probably needs broader schema discussion.

[Akseli-Ilmanen]

Love the ndx-pose extension

However, your use case exposes a deeper limitation. The source_video link points to the ImageSeries object as a whole, not to a specific file within it. When an ImageSeries has multiple external_file entries (one per trial), there is no way for the pose data to say "I correspond to the 5th video file in this series." One could add a file index to the link, but that would make ndx-pose depend on the internal structure of the external_file array, which is fragile.

Regarding this tension, do you think one could address this at construction in the ndx-pose module. Please disagree if there are exceptions but I would assume DLC, Sleap, DANNCE all output one prediction (h5, csv) file per input video file (or for 3D triangulation) per set of synchronized video files. So users should by default have synchronized timestamps for video and pose files.

What if via ndx-pose or ndx-pose + neuroconv, users could upload both video files and pose files together (e.g. list of tuples) and only need to specify per pair of pose and/or video file the starting time of the file and the rate? And then one could create a shared timestamps/ NWB object mappings at construction.

Users would only need to specify starting times and rate per pair. And if videos are gapless, one can could use natsort to order the files chronologically and only specify starting time of the first file (if one reads the number of frames from the video file metadata). This would even work if videos are with gaps, and not "trialized" (e.g. trial start stop time, defined within the video segments themselves).

Also maybe @niksirbi (from movement) would like to chime in.

[niksirbi]

Thanks for tagging me, this was an interesting discussion to read, as we (in movement) are also thinking about ways to align pose tracking data with video, ephys etc.

Please disagree if there are exceptions but I would assume DLC, Sleap, DANNCE all output one prediction (h5, csv) file per input video file (or for 3D triangulation) per set of synchronized video files. So users should by default have synchronized timestamps for video and pose files.

I think that's true in most, but not all, cases. DLC and LightningPose indeed work this way, 1 video → 1 file containing predicted poses, so the number of frames should be the same in both files. I'm less sure about SLEAP's .slp files, though. During training, you can link multiple videos to a single .slp file, and I believe you can also store predictions there. That said, SLEAP can also export an "analysis" .h5 file containing just the pose predictions for a given video, which may be a workable solution.

That's indeed how we handle SLEAP predictions in movement (see docs for this function): we prefer to load the "analysis" .h5 file where possible. If a user supplies an .slp file instead, we load only the poses corresponding to the first video (if multiple are present), since movement datasets (upon loading) typically correspond to contiguous pose tracks generated from a single video file.

Regarding the rest of the discussion, I don't have much to add, as I don't have much experience with trialised data acquisition. I can make two comment though:

• Even for the 'gapless' case, in real datasets we often see camera frames being dropped between the saved video files. This was mentioned above as an edge case, but may be quite common.
• Camera frame rates are not always perfectly constant, meaning the timestamps may not be evenly spaced, even within a video. Reconstructing timestamps from just starting_time + frame_rate assumes a constant frame rate, which is a reasonable assumption for most cases and also what movement does. But in some cases, there may be no way around supplying the entire timestamp array. See this movement issue.

[h-mayorquin]

What if via ndx-pose or ndx-pose + neuroconv, users could upload both video files and pose files together (e.g. list of tuples) and only need to specify per pair of pose and/or video file the starting time of the file and the rate? And then one could create a shared timestamps/ NWB object mappings at construction.

Let me take a look at how far we are from this. I think most of the pieces are there but I need to double-check.

[Akseli-Ilmanen]

Awesome!

[h-mayorquin]

We discussed this at the NWB meeting on 2026-04-14 and we are leaning toward not adding full per-file time alignment metadata to the schema for multiple external files (e.g. an array of starting times). We judge that the added complexity to the schema is not justified at this point. The strongest case we have seen is trialized video data, but it remains an edge case and can be handled with the concatenated timestamps approach (even if it is less convenient than a per-file starting_time would be). The timestamps path also handles other edge cases such as dropped frames during file rollover or non-constant frame rates.

We are willing to reconsider this position if the community surfaces cases where using timestamps is particularly difficult or impractical. We would love to hear more about those.

In the meantime, if no such cases come up, we intend to move forward with documentation and checks:

1. Multi-file external_file with gaps should use timestamps (not starting_time + rate, which cannot represent gaps).
2. Unless there are strong reasons to do otherwise (e.g., trialized data), users should represent separate recording segments as separate ImageSeries objects.

Separately, #678 adds num_samples to external-mode ImageSeries, which is needed even for the gapless case so readers can determine the duration of the last file without opening it.

[h-mayorquin]

Addressing the tooling side of the requests separately from the schema decision. This is what I understood:

• num_frames_per_external_file: This is addressed by #678, which adds num_samples to external-mode ImageSeries. Once num_samples is available, per-file frame counts can be derived from starting_frame: starting_frame[i+1] - starting_frame[i] for all but the last file, and num_samples - starting_frame[-1] for the last. A separate num_frames_per_external_file field is not needed.

• Paired video + pose upload: I will look into supporting this in neuroconv. The ExternalVideoInterface already has set_aligned_segment_starting_times which lets users specify a starting time per video file. The goal would be to make it easy to upload video + pose pairs together, specifying only a starting time and rate per pair, and have the tool construct the concatenated timestamps and starting_frame arrays automatically. See draft PR catalystneuro/neuroconv#1713.

• Better mapping between video and pose objects: I have an open PR on ndx-pose (rly/ndx-pose#56) that adds a formal source_video link from PoseEstimation to ImageSeries, replacing the free-form string paths in original_videos. This would give a reliable, machine-readable connection between pose data and its source video, addressing the problem where objects like "VideoLeftCamera" and "LeftCamera" are only connected by naming convention. I am still unsure what do about the multi-file case there @rly.

Let me know if there is something that I missed.

[Akseli-Ilmanen]

Hi @h-mayorquin,

Thanks for all this! I agree that "per-file time alignment metadata to the schema for multiple external files" can get too complicated.

In a situation where you have 5 consecutive videos (all same camera) and want to jump to t=X, it would be helpful if there was a public API like get_videos in issue 34 which can query video timestamps to figure out the right video and it's local time. Generally I would agree that it's a better to not change the schema too much, but provide more public methods to index external files via timestamps/ starting_time + rate.

Regarding all the other proposals of ndx-pose and neuroconv, they all look great.

Unless there are strong reasons to do otherwise (e.g., trialized data), users should represent separate recording segments as separate ImageSeries objects.

To clarify on this point. If I have multiple video files per camera with real gaps, it would be valid to use a single ImageSeries object (per camera), and provide multiple external files and concatenate timestamps. As a result of the gaps, the timestamps wouldn't be evenly spaced but have time jumps at a file boundaries.