Support for non-standard slots (“extensions”)

[gouttegd]

The SSSOM specification has an explicit mechanism to allow the use of extra, non-standard metadata slots beyond those described in the spec (“extension slots”).

Unfortunately, SSSOM-Py is currently completely ignorant of that mechanism. This makes extension slots all but unusable with SSSOM-Py.

Ideally, as the “reference implementation” of SSSOM, SSSOM-Py should have full support at least for defined extensions, which notably means:

• that if a set contains defined extension slots, then those slots should always be preserved;
• that the SSSOM-Py API should allow to define such extension slots.

[joeflack4]

@gouttegd Thanks for opening up. Just adding context.

I originally brought this up on slack.

1. I included an extension_definitions field in my metadata when creating a MappingSetDataFrame.

Code snippet

msdf: MappingSetDataFrame = from_sssom_dataframe(df, prefix_map={
    'COMPLOINC_PROP': '[https://comploinc-props#](https://comploinc-props/#)',
}, meta={
    'mapping_set_date': datetime.now().strftime('%Y-%m-%d'),
    'extension_definitions': [
        {
            'slot_name': 'mapping_set_date',
            'property': 'COMPLOINC_PROP:mapping_set_date',
            'type_hint': 'http://www.w3.org/2001/XMLSchema#date',
        },
        {
            'slot_name': 'PartTypeName',
            'property': 'COMPLOINC_PROP:PartTypeName',
            'type_hint': 'http://www.w3.org/2001/XMLSchema#string',
        },
        {
            'slot_name': 'curator_approved',
            'property': 'COMPLOINC_PROP:curator_approved',
            'type_hint': 'http://www.w3.org/2001/XMLSchema#bool',
        },
        {
            'slot_name': 'subject_dangling',
            'property': 'COMPLOINC_PROP:subject_dangling',
            'type_hint': 'http://www.w3.org/2001/XMLSchema#bool',
        },
        {
            'slot_name': 'object_dangling',
            'property': 'COMPLOINC_PROP:object_dangling',
            'type_hint': 'http://www.w3.org/2001/XMLSchema#bool',
        },
    ],
})

Expected behavior:

• a. Support

• b. If no support, a warning or an error about non-implemented feature.
  Actual behavior:

• Silent failure. My extension columns were removed, and extension metadata field was removed.

• Related: Mirror issue in my repo: Dangling parts: SSSOM fields dropped loinc/comp-loinc#163

2. No error/warning when passing obj instead of list

Of course, (1) needs to be addressed first, but FYI one time I did this and got no warning or error.

    'extension_definitions': {
            'slot_name': 'mapping_set_date',
            'property': 'COMPLOINC_PROP:mapping_set_date',
            'type_hint': 'http://www.w3.org/2001/XMLSchema#date',
    },

3. Inconsistent/incorrect warnings

(1) would need to be addressed first, and then this might automatically go away, but it might not.

WARNING:root:No attr for PartTypeName [32972 instances]
WARNING:root:No attr for subject_dangling [30628 instances]
WARNING:root:No attr for object_dangling [2344 instances]

i. It doesn't give a warning for the curator_approved col, even though it is an extension too. Probably because this is the only column that is completely empty.
ii. Warning instance counts for subject_dangling and object_dangling appear to be based on the true/false values. The values in these columns are the inverse of each other. So if you add them up, they = len(df): 2,344 + 30,628 = 32,972 (same as the number for PartTypeName. So I assume “False” values are being interpreted as “no attr”.
iii. For PartTypeName, IDK why this number is equal to the total number of rows. Almost 100% of the rows have values.

[gouttegd]

By the way, I confirm that the way you defined your extensions:

'extension_definitions': [
        {
            'slot_name': 'mapping_set_date',
            'property': 'COMPLOINC_PROP:mapping_set_date',
            'type_hint': 'http://www.w3.org/2001/XMLSchema#date',
        },
...

is correct according to the SSSOM spec. :) If it does not work, it’s not because you made a mistake, it’s just that it cannot work since SSSOM-Py completely lacks support for the feature.

Though of course SSSOM-Py could (and arguably should) provide an API to encapsulate the details of how extension definitions are actually represented in the model, so that client code does not have to manipulate the extension_definitions directly.