What would you like to see added to HDMF?
HERD.add_ref currently models every external reference as a three-way relationship: an object (Container/Data), a key (the user term used within that object, stored in the KeyTable), and an entity (the external resource, stored in the EntityTable). This works well for vocabulary-style references, where a value stored in the file (e.g., the species name "Homo sapiens") is linked to its definition in an external resource (e.g., NCBI Taxonomy).
However, it does not support associating an external resource with a dataset, group, or attribute directly, when there is no specific term in the data that the entity defines. Examples:
- Linking a dataset to the REST API record or DOI it was derived from.
- Linking a group to related metadata stored in an external database.
- Annotating an object with a pointer to a protocol, registry entry, or related publication.
This is the core of the "foreign field" use case in #327, which was closed in favor of incremental improvements to HERD.
What solution would you like?
Allow the key to be omitted when adding a reference, so that an entity can be associated with an object (object + relative_path + field) directly. As suggested in #327 (comment), this can be done without a schema change by storing a sentinel value (e.g., the empty string "") in the KeyTable for such references, since the ObjectKeyTable and EntityKeyTable already provide the object-to-key and key-to-entity links.
Proposed changes:
HERD.add_ref: make key optional in behavior (it already defaults to None in the signature) and map None to the sentinel key.- Retrieval methods (
get_key, get_object_entities, get_entities, the to_dataframe helpers) should handle the sentinel key gracefully, e.g., report it as no key rather than an empty-string term.
- Validation/normalization: ensure a
None/empty key today raises a clear error rather than silently storing an unintended row (currently add_ref with key=None is not validated).
- Documentation and the HERD tutorial should describe when to use a keyed reference vs. a direct object-entity reference.
Alternatives considered:
- Adding a separate
ObjectEntityTable linking objects directly to entities. This is cleaner conceptually but requires a schema change to the HERD type in hdmf-common-schema and a corresponding schema version bump, whereas the sentinel-key approach works with the existing schema.
- Requiring users to invent placeholder keys. This pollutes the key table with meaningless terms and makes the semantics of the reference ambiguous.
Do you have any interest in helping implement the feature?
Yes.
Note
Split off from #327 (foreign fields), following the discussion in #327 (comment). Related: #1485.
What would you like to see added to HDMF?
HERD.add_refcurrently models every external reference as a three-way relationship: an object (Container/Data), a key (the user term used within that object, stored in theKeyTable), and an entity (the external resource, stored in theEntityTable). This works well for vocabulary-style references, where a value stored in the file (e.g., the species name"Homo sapiens") is linked to its definition in an external resource (e.g., NCBI Taxonomy).However, it does not support associating an external resource with a dataset, group, or attribute directly, when there is no specific term in the data that the entity defines. Examples:
This is the core of the "foreign field" use case in #327, which was closed in favor of incremental improvements to HERD.
What solution would you like?
Allow the key to be omitted when adding a reference, so that an entity can be associated with an object (object + relative_path + field) directly. As suggested in #327 (comment), this can be done without a schema change by storing a sentinel value (e.g., the empty string
"") in theKeyTablefor such references, since theObjectKeyTableandEntityKeyTablealready provide the object-to-key and key-to-entity links.Proposed changes:
HERD.add_ref: makekeyoptional in behavior (it already defaults toNonein the signature) and mapNoneto the sentinel key.get_key,get_object_entities,get_entities, theto_dataframehelpers) should handle the sentinel key gracefully, e.g., report it as no key rather than an empty-string term.None/empty key today raises a clear error rather than silently storing an unintended row (currentlyadd_refwithkey=Noneis not validated).Alternatives considered:
ObjectEntityTablelinking objects directly to entities. This is cleaner conceptually but requires a schema change to theHERDtype in hdmf-common-schema and a corresponding schema version bump, whereas the sentinel-key approach works with the existing schema.Do you have any interest in helping implement the feature?
Yes.