Mixed types for neurodata_type_inc when defining NWBGroupSpec
#37
-
|
Hi, I'm learning about creating extension and I want to know how to define mixed types in the Minimal exampleStep 1 - Create extension:from pynwb.spec import NWBNamespaceBuilder, NWBGroupSpec, NWBAttributeSpec
from pynwb.core import NWBDataInterface
from hdmf.common import DynamicTable, VectorData
name = 'raw_acquisition_module'
ns_path = name + ".namespace.yaml"
ext_source = name + ".extensions.yaml"
ns_builder = NWBNamespaceBuilder(name + ' extensions', name, version='0.1.0')
ns_builder.include_type('NWBDataInterface', namespace='core')
ns_builder.include_type('DynamicTable', namespace='hdmf-common')
rawacqmod = NWBGroupSpec(
neurodata_type_def='RawAcquisitionModule',
neurodata_type_inc='NWBDataInterface',
attributes=[
NWBAttributeSpec(name='name', doc='Name of this acquisition module', dtype='text'),
NWBAttributeSpec(name='description', doc='Description of this acquisition module', dtype='text'),
],
doc='Module/container to add acqusition raw data into',
quantity='*',
default_name='UntitledRawAcquisitionModule',
groups=[
NWBGroupSpec(
neurodata_type_inc='NWBDataInterface', <------- this is where I have to want mixed types
doc='NWBDataInterface',
quantity='*',
),
]
)
ns_builder.add_spec(ext_source, rawacqmod)
ns_builder.export(ns_path)It's the "arrow-ed" line that I want to add both Step 2 - Import and register class# In a different file from Step 1
import pandas as pd
import numpy as np
from datetime import datetime
from dateutil import tz
from pynwb import NWBFile, TimeSeries, NWBHDF5IO
from pynwb import register_class, load_namespaces, get_class
from pynwb.file import MultiContainerInterface
from pynwb.core import NWBDataInterface
from hdmf.common import DynamicTable, VectorData
from hdmf.utils import docval, popargs_to_dict, get_docval, popargs
ns_path = 'config/raw_acquisition_module.namespace.yaml'
load_namespaces(ns_path)
@register_class('RawAcquisitionModule', 'raw_acquisition_module')
class RawAcquisitionModule(MultiContainerInterface):
__nwbfields__ = ('name', 'description')
__clsconf__ = [
{
'attr': 'nwb_data_interfaces',
'add': 'add',
'type': (NWBDataInterface, DynamicTable),
'get': 'get'
}
]
# I have to manually define these again because to have the ability to `add`, for some reason,
# I'd have to define `__clsconf__` and that somehow erases the attributes defined in the original namespace
@docval(
{
'name': 'name',
'doc': 'Name of this acquisition module',
'type': str,
'default': 'UntitledRawAcquisitionModule'
},
{
'name': 'description',
'doc': 'Description of this acquisition module',
'type': str,
'default': None
},
{
'name': 'nwb_data_interfaces',
'doc': 'NWBDataInterface',
'type': (list, tuple, dict, NWBDataInterface, DynamicTable), <------- this is where I have to declare explicitly 2 types
'default': None
}
)
def __init__(self, **kwargs):
description, nwb_data_interfaces = popargs("description", "nwb_data_interfaces", kwargs)
super().__init__(**kwargs)
self.description = description
self.nwb_data_interfaces = nwb_data_interfacesStep 3 - Usage# In same file as Step 2
nwbfile = NWBFile(
session_description='abc',
identifier='xyz',
session_start_time= datetime(2018, 1, 2, 3, 4, 5, tzinfo=tz.gettz("US/Pacific"))
)
ts = TimeSeries(
name = 'raw_series',
description = 'raw series',
data = np.random.randint(3,size=10),
timestamps = np.arange(10).astype('float'),
unit = '-'
)
tbl = DynamicTable(
name = 'id_lookup',
description = 'lookup table',
columns = [
VectorData(name = 'the_id', description='id', data=[0,1,2]),
VectorData(name = 'the_name', description='name', data=['a','b','c'])
]
)
mod = RawAcquisitionModule(
name = 'raw_mod',
description = 'raw module'
)
mod.add(ts)
mod.add(tbl)
nwbfile.add_acquisition(mod)
print(f'In acquisition: {nwbfile.acquisition}')This is what came out of In acquisition: {'raw_mod': raw_mod __main__.RawAcquisitionModule at 0x140313029216048
Fields:
description: raw module
nwb_data_interfaces: {
id_lookup <class 'hdmf.common.table.DynamicTable'>, <---- this is here when creating
raw_series <class 'pynwb.base.TimeSeries'>
}
}Then save: with NWBHDF5IO('test.nwb', mode='w') as io:
io.write(nwbfile)Step 4 - Open and compareWhen I try to open the file, only the series is there, the table is gone: with NWBHDF5IO('test.nwb', mode='r', load_namespaces=True) as io:
nwbfile_reload = io.read()
print(nwbfile_reload.acquisition)
-------
{'raw_mod': raw_mod __main__.RawAcquisitionModule at 0x140311279003744
Fields:
description: raw module
nwb_data_interfaces: {
raw_series <class 'pynwb.base.TimeSeries'> <------ only series remain, table disappears
}
}System
|
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 7 replies
-
|
@tuanpham96 I'm curious what your use-case is that you want acquisition modules. We considered building that into the standard but didn't have many compelling use-cases. Your code is very close. I would try this: from pynwb.spec import NWBNamespaceBuilder, NWBGroupSpec, NWBAttributeSpec
from pynwb.core import NWBDataInterface
from hdmf.common import DynamicTable, VectorData
name = 'ndx-acquisition-module'
ns_path = name + ".namespace.yaml"
ext_source = name + ".extensions.yaml"
ns_builder = NWBNamespaceBuilder(name + ' extensions', name, version='0.1.0')
ns_builder.include_type('NWBDataInterface', namespace='core')
ns_builder.include_type('DynamicTable', namespace='hdmf-common')
rawacqmod = NWBGroupSpec(
neurodata_type_def='AcquisitionModule',
neurodata_type_inc='NWBDataInterface',
doc='Module/container to add acquisition raw data into',
quantity='*',
groups=[
NWBGroupSpec(
neurodata_type_inc='NWBDataInterface',
doc='NWBDataInterface',
quantity='*',
),
NWBGroupSpec(
neurodata_type_inc='DynamicTable',
doc='DynamicTable',
quantity='*',
),
]
)
ns_builder.add_spec(ext_source, rawacqmod)
ns_builder.export(ns_path)
You might be able to just use |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for your response! My answer about use-case is at the end. Summary of my attempts
Details of my attempts
I had been trying both initially but it hadn't seem to work. But I'd probably missed something fundamental. Anyway, I tried the code you have above, then:
Why
|
Beta Was this translation helpful? Give feedback.
-
NWBAttributeSpec(name='description', doc='Description of this acquisition module', dtype='text'),
mod = AcquisitionModule(
name = 'raw_mod',
# description = 'raw module',
nwb_data_interfaces = dict(), # both dict() and []
dynamic_tables = dict()
)seems like a bug.
|
Beta Was this translation helpful? Give feedback.
-
|
I was actually trying to do I'll try your suggestion (2) and the hook in (4) later this week and see if it works. |
Beta Was this translation helpful? Give feedback.
-
|
OK so this issue: mod = AcquisitionModule(
name = 'raw_mod',
# description = 'raw module',
nwb_data_interfaces = dict(), # both dict() and []
dynamic_tables = dict()
)was resolved 2 weeks ago in the dev branch of hdmf: hdmf-dev/hdmf#779 |
Beta Was this translation helpful? Give feedback.
-
|
I just tried updating to Just to verify one thing, there's no way to have one object |
Beta Was this translation helpful? Give feedback.
-
|
No I don't think so |
Beta Was this translation helpful? Give feedback.
-
This is possible, in the same way that Define the type YAML like so: Define ( Note that Set up a custom mapping from both schema types to the Python attribute There is a similar setup for Let me know if you run into any issues! |
Beta Was this translation helpful? Give feedback.
-
|
For both attempts below, rawacqmod = NWBGroupSpec(
neurodata_type_def='AcquisitionModule',
neurodata_type_inc='NWBDataInterface',
doc='Module/container to add acquisition raw data into',
quantity='*',
attributes = [
NWBAttributeSpec(name='description', doc='Description of this acquisition module', dtype='text'),
],
groups=[
NWBGroupSpec(
neurodata_type_inc='NWBDataInterface',
doc='NWBDataInterface',
quantity='*',
),
NWBGroupSpec(
neurodata_type_inc='DynamicTable',
doc='DynamicTable',
quantity='*',
),
]
)Attempt 1: Defining
|
Beta Was this translation helpful? Give feedback.
-
|
Q1: I agree with @bendichter 's response below. Q3: Yes. Both writing and reading. Q4: That is a good point... This method is redundant. I will create a PR to remove it. Q5: hdmf 3.4.7 was released a week ago with the fix mentioned above. Q6: Generally speaking, you will always be able to read an NWB 2.x file with PyNWB and HDMF. NWB files written with extension schema automatically cached and can be read by PyNWB and MatNWB; HOWEVER, if you use a custom API + mapping and want users to use that API when reading/writing extension data, they will need to install that code separately, e.g., through Q7: There is no strong reason of not having an Q8: In general, I think this is a good idea and something we have briefly considered as an alternative way of identifying data compared to by object hierarchy and by data type. I created a feature request here: NeurodataWithoutBorders/nwb-schema#531 |
Beta Was this translation helpful? Give feedback.
-
|
Thank you both for the updates and answers! And thanks for creating a feature request on tagging as well! I'll mark this as answered and close the discussion now. Thanks so much for your help! |
Beta Was this translation helpful? Give feedback.
-
Yes, this is exactly what I was thinking, but it seems like doing this with the mapper fits your use-case better. Q1: we'll have to discuss the other questions. We'll get back to you asap, but with SfN it might be a little while. |
Beta Was this translation helpful? Give feedback.
For both attempts below,
descriptionis put back toNWBGroupSec(attributes=...)when creating the extension, no need to definename. Usinghdmf 3.4.6.post0.dev4.