`IntracellularRecordingsTable` does not have columns, but the colnames attribute is required from schema

[ehennestad]

Problem

IntracellularRecordingsTable is an AlignedDynamicTable with three required category sub-tables (electrodes, stimuli,responses). It does not have any column requirements.

However, the colnames attribute is required from the hdmf-schemas DynamicTable.
https://github.com/hdmf-dev/hdmf-common-schema/blob/497bde378749d2e56a30a05d6851885f58e78f75/common/table.yaml#L116-L127

MatNWB currently enforces required attributes, and can therefore not export an NWB file without adding a custom column to the IntracellularRecordingsTable (if colnames is empty, nwbExport fails).

PyNWB allows exporting IntracellularRecordingsTable without any extra columns.

There are some options:

1. colnames should have required: False in DynamicTable
2. colnames in IntracellularRecordingsTable can can be overridden to have required: False
3. patch MatNWB to allow empty colnames if a table does not have any real column.

This is a schema inconsistency, so I lean towards 1 or 2.

@rly @oruebel

[oruebel]

I am not sure I fully understand the issue yet, so let me ask a few questions:

1. The schema only requires that colnames is present, but it is permitted to be an empty array. So I think the correct behavior should be that if a table does not have any custom columns, that it should have a colnames attribute that is an empty array. I assume that is what PyNWB is doing, i.e., when there are no custom columns, we write an empty colnames dateset. Is that correct?
2. Is the issue that MatNWB does not support empty list/arrays as valid values for colnames?
3. All tables have a required id column. I think currently, the id column may not be listed in colnames. However, that may be another option, i.e., include id as the first entry in the colnames, in which case colnames should never be empty. Although that would likely have a bunch of downstream side-effects.

[ehennestad]

1. The schema only requires that colnames is present, but it is permitted to be an empty array.

Then I have misunderstood the meaning of required. Is this the case for all attributes that have the required key set to true - i.e. they can be empty? Is it then correct that an attribute must be present and non-empty only if the shape key is set to a scalar positive value, like shape: [1]

2. Is the issue that MatNWB does not support empty list/arrays as valid values for colnames?

Yes, exactly. If something is required in MatNWB it must be non-empty

3. All tables have a required id column. I think currently, the id column may not be listed in colnames. However, that may be another option, i.e., include id as the first entry in the colnames, in which case colnames should never be empty. Although that would likely have a bunch of downstream side-effects.

Correct, id is not listed in colnames. That sounds like a bigger change where existing NWB files would diverge.

[oruebel]

Is this the case for all attributes that have the required key set to true - i.e. they can be empty? Is it then correct that an attribute must be present and non-empty only if the shape key is set to a scalar positive value, like shape: [1]

Correct 👍 @rly do you agree?

Yes, exactly. If something is required in MatNWB it must be non-empty

Makes sense. How challenging would it be to support this?

Correct, id is not listed in colnames. That sounds like a bigger change where existing NWB files would diverge.

Yes, that would be a bigger change.

[oruebel]

@ehennestad can this issue be closed here?

[ehennestad]

Yes

[ehennestad]

Reopening, because while thinking more about this, it does not make much sense to me.

@oruebel confirmed above that required + unbounded shape (e.g. shape: [null]) means the field must be present, but is allowed to be empty. I'd like to get clarification on whether this is intentionally uniform across all field types, because from a semantic standpoint it seems inconsistent.

The semantic problem

For container/table fields the rule makes sense:

• DynamicTable.colnames — a table with no custom columns is a degenerate but valid state; requiring the field just forces the caller to declare the structure explicitly.

For data array fields the rule seems wrong:

• TimeSeries.data, ElectricalSeries.data, SpikeEventSeries.timestamps, etc. — a TimeSeries with zero data points is not a meaningful neuroscience object. Allowing it to be exported as valid seems like an unintended consequence of applying the rule uniformly rather than a deliberate semantic decision.

Should the spec distinguish between fields where an empty value is semantically valid and fields where it is not? Or is the current behaviour (all required+unbounded fields can be empty) a deliberate design choice with a documented rationale?

The MATLAB problem

This distinction matters practically for matnwb, because MATLAB has no general None type. In Python, None and [] carry different meanings:

• None → field was never provided (triggers "required" violation in pynwb/hdmf)
• [] → field was explicitly provided as empty (passes validation)

In MATLAB, there is no equivalent of None — an unset property is simply []. matnwb therefore cannot distinguish "field was forgotten" from "field was intentionally set to empty." As a consequence, matnwb currently treats an empty value for a required field as missing and blocks export — which is stricter than pynwb but arguably more correct for data array fields.

If the spec intent is that "required" means only "must not be absent/None", then matnwb's current behaviour is technically wrong for fields like colnames (where empty is valid). But if the spec intent is that required data array fields like TimeSeries.data must also be non-empty, then pynwb's behaviour is too permissive.

A clarification on whether this is a uniform rule or a per-field semantic question would help resolve the right behaviour for matnwb.

🤖 Generated with Claude Code

[oruebel]

• TimeSeries with zero data points is not a meaningful neuroscience object.

An empty TimeSeries is in my view technically still a valid object. Whether an empty series is useful or not is a different question. In principle an empty series may indicate that a recording yielded no data or was missing.

Should the spec distinguish between fields where an empty value is semantically valid and fields where it is not? Or is the current behaviour (all required+unbounded fields can be empty) a deliberate design choice with a documented rationale?

Supporting this behavior would require enhancements in the schema language, schema, and APIs. So the question is less whether it was a "deliberate design choice", but rather whether there was a use case and need for this particular feature.

If the spec intent is that "required" means only "must not be absent/None",

This is my current reading of the spec.

TimeSeries.data must also be non-empty,

I think for this behavior, a simple non-empty would probably be too simple, since the real question becomes "along which dimensions should the data be non-empty?" I think rather what this would be asking for is something along the lines of specifying a min_shape.

As a consequence, matnwb currently treats an empty value for a required field as missing and blocks export — which is stricter than pynwb

I think being slightly stricter on write is not necessary a problem, because the files that MatNWB writes would still be valid. If I understand this correctly: 1) what it mainly means is that MatNWB does not allow certain edge-cases (e.g, an empty TimeSeries), requiring the users to specify a non-empty value for required fields and 2) colnamesfor DynamicTable is the main practical issue right now.

In MATLAB, there is no equivalent of None — an unset property is simply []

Just to clarify. It looks like scalars may have "undefined" values in Matlab (e.g,. numeric data has NaN and isnan(), strings have <missing> and ismissing(), datetime has NaT and isnat). Is this mainly an issue for array types?

I am not a Matlab expert, so this may be a naive question, but could this be addressed via a custom classdef NoneType that could serve as the default-value for required fields, such that [] would clearly indicate that the field was set?

[rly]

My reading of the spec language is also that "required" means only "must not be absent/None", and that applies across fields. An empty array is valid though usually not meaningful. In fact, a 3D array with 0 elements is explicitly allowed in https://github.com/NeurodataWithoutBorders/nwb-schema/blob/dev/core/nwb.image.yaml#L66 to get around the fact that ImageSeries.data is required (from the TimeSeries.data requirement) and we want ImageSeries to support data stored in an external file (See also https://nwbinspector.readthedocs.io/en/dev/best_practices/general.html#empty-data). Validating that the data is non-empty should be handled with shape: [a positive integer], a new feature like min_shape as @oruebel proposed or minimum_cardinality from LinkML, or more weakly, inspector checks. We can't redefine required to be non-empty without breaking the ImageSeries.data case above.

NWB also allows empty strings for required string attributes, though this also goes against best practices https://nwbinspector.readthedocs.io/en/dev/best_practices/general.html#empty-strings

If MatNWB detects an empty value for a required field of unbounded shape, could it mark that as valid, and write an empty dataset or attribute?

As an aside for my reference, LinkML treats required: true, multivalued: true means one or more, and the validator will reject empty lists. However, when the LinkML schema is translated to JSON Schema, JSON Schema treats required as "key must be present," and an empty array satisfies that. To block an empty array, set minimum_cardinality: 1 explicitly.

[ehennestad]

Just to clarify. It looks like scalars may have "undefined" values in Matlab (e.g,. numeric data has NaN and isnan(), strings have <missing> and ismissing(), datetime has NaT and isnat). Is this mainly an issue for array types?

Yes, many types have a missing variant (NaN for floats, NaT for date times), but integer and compound types do not.

I am not a Matlab expert, so this may be a naive question, but could this be addressed via a custom classdef NoneType that could serve as the default-value for required fields, such that [] would clearly indicate that the field was set?

Yes, this is a good option for the time being. However, modern MATLAB has great property validation features (https://se.mathworks.com/help/matlab/matlab_oop/validate-property-values.html) and this would be more finicky to implement in MatNWB if every required property should also be allowed to be a custom None type.

@rly suggestion:
If MatNWB detects an empty value for a required field of unbounded shape, could it mark that as valid, and write an empty dataset or attribute?
is also viable, but would not distinguish between:
A. User did not set the value
B. User specifically set it to empty

Thanks for feedback and clarifications.

I'll close this again, and open an issue on MatNWB for working towards a solution.