Optional metric unit for VectorData

[h-mayorquin]

There is no standard, machine-readable way to record the physical unit of a numeric DynamicTable column. A per-row quantity (a stimulus frequency in Hz, a running speed in cm/s) has nowhere structured to declare its unit today, so it gets folded into the column name or the free-text description. The motivating case is NWBEP001's EventsTable, where an event can carry a continuous numeric payload whose value and duration have clean homes but whose unit does not. There is precedent for unit-bearing columns already: TimestampVectorData and DurationVectorData both define a unit attribute fixed to "seconds".

Two options are on the table. Adding an optional unit (text) attribute directly on VectorData or create a subtype like NumericVectorData. I think the first is the best choice: zero new type, zero migration, and a reader just checks whether the attribute is present. I think this is best as it requires less churn because the unit is optional. The only downside is the semantics as unit does not make sense in some VectorData (reference columns for example).

[oruebel]

Adding an optional unit (text) attribute directly on VectorData

I agree, since it is an optional field, and since this is a need that applies broadly across tables, I think it make sense to address this at the level of VectorData via an optional unit attribute.

Note, this would need to be addressed in the hdmf-common-schema rather than in the NWB schema.

[rly]

Related:

• Add Measurements hdmf-dev/hdmf-common-schema#63
• Integrating with MeasurementData #493
• add Measurement neurodata_type #208

I think adding a unit attribute as an optional field to VectorData is a good idea, and I think the simplicity of that is worth the downside that unit doesn't make sense for some VectorData.

A slight tangent: conversion and offset:

hdmf-dev/hdmf-common-schema#63 also suggested adding conversion and offset. That never made it to the finish line, and I don't think we should do it here, but I think it's worth deciding now whether that proposal changes the choice above:

• Option A (my preference): add unit to VectorData, defer conversion and offset. If we later want conversion/offset, add a subtype whose distinguishing feature is scaling, e.g. ScaledVectorData, that inherits unit and adds the two scaling attributes. This is consistent with today's decision and keeps the scaling mechanics off the columns that will never use them.
• Option B: one subtype with all three. Don't touch the base type; create a single subtype QuantityVectorData carrying unit, conversion, and offset.

I prefer A. The only use case I can think of for conversion and offset is storing spike waveforms in the Units table as the original int16 with conversion and offset to recover volts, which saves storage space. Existing solutions seem tolerable and demand for this change seems low.

An aside on naming the type for Option B

"MeasurementData" (from https://github.com/hdmf-dev/hdmf-common-schema/pull/63) is descriptive but not super precise for cases like: - A filter cutoff frequency (e.g., 300 Hz) or an electrode contact width (e.g., 12 um), which are settings/specifications, not strictly measurements. ("specified by the manufacturer" is different from "measured from histology or an instrument.") - Thresholds - Derived values, like a predicted firing rate

If we go with Option B, I think "QuantityVectorData" (quantity is defined in metrology, the study of measurement, as a numerical value and a unit) is technically better and covers measured, configured, and derived values.

[oruebel]

I prefer A.

I agree with Option A.