NWB use cases and schema requirements: acquisition, processing, and archival

[h-mayorquin]

@bendichter @rly @oruebel @ehennestad

Today in our CN+LBNL NWB sync (2026-03-17) we were discussing #672 and the team agreed that we should re-frame this in the following way: NWB is a single format used in at least three distinct contexts, and requirements that make sense for one context are inappropriate or impossible in another.

There are at least three distinct contexts in which NWB files are created and used:

• Acquisition. NWB files written directly from hardware during an experiment, by tools like Open Ephys or AqNWB. At this stage, many required schema fields are simply unavailable: the software does not know the anatomical location of the probe, the calcium indicator, or the excitation_lambda. Time alignment between data streams is performed online and is inherently approximate. The session_start_time may be ambiguous. Requiring well-annotated, fully aligned data at this stage is not realistic.

• Processing / conversion. This context covers two related but distinct cases.

  Analysis tools like CaImAn and SpikeInterface operate on already-processed data and offer NWB as an output format. They do not have access to experimental metadata: CaImAn hardcodes location="brain" and indicator="OGB-1" because that information is simply not in its input data and the fields are required. SpikeInterface users may want to write quality metrics or spike sorting results to NWB at some point during a pipeline, again without having the full experimental context available.

  Automated conversion tools like NeuroConv play a different role: they can integrate multiple source files, align timestamps across sources, and build a considerably more complete NWB file than any single analysis tool could. But there are hard limits to what can be inferred automatically. The experimental context that the user holds (what brain region, which subject, what protocol) is often not encoded in any source file and cannot be filled in automatically. Still, we should be able to provide files without those fields so the downstream tools or the user can complete them.

  In both cases, this is where the placeholder problem in Required fields need official placeholders for automated NWB file builders #672 is most acute. Time alignment can be corrected at this stage (post-hoc alignment is more accurate than online), but the file is not yet complete for archival purposes.

• Archival. NWB files published to DANDI or shared for reuse. Here, the bar should be high: accurate time alignment, complete metadata, meaningful values in all required fields. DANDI already enforces stricter requirements on top of the schema via the NWB Inspector, creating a de facto two-tiered system. An archival file with location="unknown" is a quality failure.

When NWB was created, DANDI did not exist, so the schema carried all the validation weight alone. The heavy field requirements were the only filter between a well-annotated file and a poorly-annotated one. Now that DANDI and the NWB Inspector exist, a two-tier system is already in place in practice, even if it was never explicitly designed that way.

Two concrete examples illustrate how the same design question looks different across these contexts. The placeholder problem in #672 is a direct consequence of applying archival-level field requirements to acquisition and processing contexts: a MissingValueEnum or making certain fields optional both address this, but the right approach depends on which context we are designing for. Similarly, @oruebel raised that online time alignment between data streams during acquisition is often less precise than post-hoc correction, so an archival context can reasonably expect well-aligned timestamps while an acquisition context cannot. This has already surfaced concretely in AqNWB: NeurodataWithoutBorders/aqnwb#251 discusses how clock offsets between acquisition devices and the file writer need to be stored precisely because online alignment is approximate and the corrected alignment happens post-hoc.

My preferred direction. The approach I favor is the one @bendichter proposed in the meeting: make the problematic fields non-required in the base schema. I think we should calibrate the requirements to the acquisition context where the fewest fields are available. This is the simplest path and does not force anyone to invent data or adjust their pipelines. It opens NWB to more use cases and makes it easier to write. This does not mean abandoning the idea that NWB enforces structure: the type system, data organization, and semantics are still there. We just become more permissive on required metadata.

The concern @rly raised, that users complain "I have a valid NWB file, why can't I upload it to DANDI?", is real, but I don't think it has a clean solution at the schema level. DANDI's requirements have changed before and may change again, and we cannot play a cat-and-mouse game with the schema chasing them. If other archives emerge with different requirements, the problem compounds: the schema cannot satisfy all of them simultaneously and should not try. The right place to address this is better communication from archives about what they require on top of NWB, combined with deeper integration of the NWB Inspector with the APIs (PyNWB, MatNWB). Perhaps we can add per-archive validation profiles on the inspector.

[h-mayorquin]

For reference, the following is a near-exhaustive list of required fields that are problematic for automated tools. The list is not long, which is the point: the scope of the problem is real but bounded. Fields marked with * are optional in the NWB schema but required by DANDI for publication.

Field	Type	Notes
TimeSeries.data.unit	string	PyNWB defaults to 'unknown'
ElectrodeGroup.location	string
ElectrodesTable.location	string
ImagingPlane.location	string
ImagingPlane.indicator	string
ImagingPlane.excitation_lambda	float	NaN available as sentinel
OpticalChannel.emission_lambda	float	NaN available as sentinel
OptogeneticStimulusSite.location	string
OptogeneticStimulusSite.excitation_lambda	float	NaN available as sentinel
PatchClampSeries.stimulus_description	string	PyNWB defaults to 'N/A'
EventDetection.detection_method	string
NWBFile.session_start_time	datetime	Ambiguous in processing context
NWBFile.timestamps_reference_time	datetime	Ambiguous in processing context
description	string	Required in ProcessingModule, ElectrodeGroup, OpticalChannel, IntracellularElectrode, OptogeneticStimulusSite
Subject.subject_id *	string
Subject.species *	string	Latin binomial or NCBI Taxon URL
Subject.sex *	string	M/F/O/U
Subject.age or Subject.date_of_birth *	string / datetime

[rly]

Following up on our sync — here's my summary of where we landed and the two directions we talked through, plus a few post-meeting notes.

TL;DR: Two directions came up — (1) relax the schema and push strictness into PyNWB/MatNWB/NeuroConv + the Inspector, or (2) keep the schema strict and introduce explicit "stub" files for incomplete data. They're not mutually exclusive. The underlying choice: which artifact gets to be called "NWB"?

Direction 1: Relax the schema, push strictness into tooling

This is what @h-mayorquin proposed, building on @bendichter's earlier suggestion. Make the currently-problematic required fields optional in the base schema — calibrated to what is actually available in the acquisition context — and push archival-level requirements out of the schema and into:

• The NWB Inspector, which already enforces DANDI-specific rules on top of the schema.
• The API defaults in PyNWB, MatNWB, and NeuroConv, so that the common path for a human writing an NWB file still produces a well-annotated, DANDI-compliant result.

Under this model, the schema defines structure and semantics (type system, data organization, linking), while metadata completeness becomes a layered concern enforced per-context.

Why it's attractive:

• The schema was designed before DANDI existed, so it had to carry all validation weight. That's no longer true — a two-tier system (schema + Inspector) already exists in practice and could be made explicit.
• It stops forcing acquisition and analysis tools (CaImAn, SpikeInterface, Open Ephys, AqNWB, MIES) to invent placeholder metadata (location="brain", indicator="OGB-1", "unknown" strings) just to produce a valid file. Invented data that leaks into DANDI is actively harmful.
• It decouples NWB from any single archive's requirements. If DANDI's requirements change, or a specialized archive appears with different ones (e.g., mouse-only, requires CCF atlas), the schema doesn't have to chase them.
• The pain is localized: only writers who explicitly want the relaxed path see it; everything going through standard API defaults still gets archival-grade output.

Opt-in friction sub-idea (@oruebel): gate relaxation behind an explicit API flag so omitting a field is a deliberate act, not silent. @bendichter pushed back that automated tools will set the flag once and forget it, making it effectively equivalent to "optional" at the tool level — though it may still help for hand-written user code.

Concerns

• Branding / user confusion (@rly, @bendichter). The implicit contract "valid NWB = uploadable to DANDI" is something users rely on, even though it's already partially broken. Relaxing the schema makes the two-tier system explicit, which @bendichter worried fragments NWB's unified message.
• Weak incentive in practice (@bendichter, @rly). Any relaxation, however gated, is likely to be adopted wholesale by automated tools, eroding the archival default.
• Inspector can only add, not remove (@oruebel). Today the Inspector can layer additional requirements on top of the schema, but there's no mechanism for it to relax something the schema declares required. Direction 1 implicitly depends on that asymmetry being fine — the schema becomes the floor, the Inspector the ceiling.
• Hard per-field judgment calls. Relaxation has to be decided field by field. Some (e.g. session_start_time, description) should likely stay required; others (unit, location, indicator, excitation_lambda, ElectricalSeries→electrodes) are contested.

Direction 2: Stub / component files

Proposed by @oruebel, reinforced by @ehennestad. Keep the schema strict as it is today. Introduce a separate, explicit concept for incomplete NWB files — variously called "stubs" or "components" in the discussion — which are still structurally NWB but are not expected to satisfy archival requirements. A stub might be distinguished by a different file extension (.nwb.stub, .nwbc, etc.) so it's visually and programmatically distinct from an archive-ready file.

@ehennestad framed it as: an NWB file that contains, say, a single TimeSeries with only the strictly structural fields populated is already technically valid under a minimal interpretation of the schema. The stub concept formalizes that minimal interpretation and gives it a name and an extension so it can't be confused with an archival file.

Why it's attractive:

• Preserves the branding invariant that "an NWB file" (the .nwb extension) means "archive-ready." No user ever has to discover that their valid NWB file is secretly not uploadable.
• Makes the distinction between intermediate and archival artifacts explicit in the filesystem itself, not just in validator output. A tool like SpikeInterface can honestly say "I produce NWB stubs" rather than "I produce NWB files that won't upload."
• Supports Neo-style intermediate-processing use cases (Andrew Davison's arbitrary-annotation extension) without contaminating the archival format.
• Lines up with a longer-term idea @bendichter had floated: writing data in pieces and assembling a complete NWB file later, rather than requiring single-shot complete writes.

Concerns

• Spec proliferation (@oruebel, raising his own concern). If "stub" is just "NWB with most rules turned off," every tool will define its own notion of what a stub looks like. We would either have to define a canonical stub schema — a significant effort comparable to defining a new format — or accept fragmentation under the hood.
• It still requires relaxation somewhere. @ehennestad acknowledged that even the stub concept ultimately depends on being able to write files without required fields, which is the same problem Direction 1 tries to solve. The difference is where the relaxation lives: in a separate stub schema versus in the main schema.
• User resistance to multi-file workflows (@oruebel). Companies like BlackRock Neurotech already push back on splitting their output across files — they want to hand customers one big file they can upload directly. A stub-based workflow that requires later assembly adds friction exactly where users least want it.
• Assembly semantics are undefined. Going from stubs to a complete NWB file is a non-trivial operation (merging timestamps, reconciling metadata, resolving links). None of this exists today and would need to be designed, specified, and implemented across APIs.
• Delayed impact. This is a larger engineering effort than Direction 1 and would not address the immediate Required fields need official placeholders for automated NWB file builders #672 placeholder problem on any short timeline.

How the directions relate

These aren't strictly alternatives — they answer different questions.

• Direction 1 asks: which fields should the schema actually require? The answer reshapes the schema itself.
• Direction 2 asks: what do we call a file that deliberately isn't archive-ready? The answer creates a new first-class artifact alongside the schema.

We could do both: relax a small set of fields in the main schema and introduce a stub concept for the more extreme "I have almost nothing" cases. We could also do neither, and instead invest in better archive-side communication and deeper Inspector integration so the branding mismatch is at least clearly signposted.

A third framing @rly raised captures the choice starkly: which artifact gets to be called "NWB"?

1. NWB = the archival format; stubs/components are a separate, flexible sibling format.
2. NWB = the flexible format; "DANDI-NWB" (and potentially others) are named profiles layered on top.

Either way, there are two tiers. The decision is which tier inherits the name, and therefore which tier inherits the existing brand equity and user expectations.

Merging DANDI requirements into the NWB schema. @bendichter, @oruebel, and @rly discussed at SfN 2025 pulling DANDI's requirements into the schema so "valid NWB" and "DANDI-uploadable" converge rather than diverge. That framing is the inverse of Direction 1 and would preserve single-tier branding, but it would make the #672 placeholder problem worse for acquisition and processing tools. Flagging it here as prior context worth weighing alongside the directions above.

A few other use cases that came up

• Python Neo as an intermediate processing backend. Andrew Davison (Python Neo maintainer) wants an extension for writing arbitrary annotation dictionaries into NWB during Neo pipelines. Consensus: acceptable for library / intermediate use, not for archival. NWB has no native key-value store because HDF5 is an array store.

• BlackRock Neurotech. Wants to ship customers a single large NWB file the customer can upload directly, pushing back against any stub/component workflow that requires later assembly.

  Post-meeting note: I don't think this use case is actually resolvable at the schema level. If the input is multiple unaligned streams, an archive-ready single file requires post-hoc alignment — neither direction changes that. What NWB can offer is a way to store raw clock offsets precisely (cf. aqnwb#251) so the alignment step can happen downstream. That reframes BlackRock's deliverable from "archive-ready file" to "faithful recording with enough information to be aligned later."

• Specialized vs. general archives (@oruebel). DANDI is general-purpose, so NWB should arguably aim to be DANDI-compliant by default. A specialized archive (mouse-only, requires CCF) is a different problem and can layer its own requirements via the Inspector — NWB shouldn't try to satisfy every specialized archive simultaneously.

Open questions

• For each currently-required field, is it realistically available at acquisition time? (@oruebel suggested walking the list field by field — probably the most concrete next step.)
• Should relaxation require an explicit opt-in flag in the API, to preserve archival output on the default path? (@oruebel: yes; @bendichter: users will just set the flag and it won't change behavior.)
• Can the DANDI team commit to clearer public documentation of their on-top-of-NWB requirements? Some documentation already exists: https://docs.dandiarchive.org/user-guide-sharing/validating-files/. Worth linking more prominently from NWB-side docs.
• Bring this to the Technical Advisory Board for feedback on the directions and tradeoffs, especially around branding and user expectations. @oruebel indicated he'd like to attend.