Support for SDMX Metadataflow

[jakubklimek]

By the same logic as for Dataflows, shall we extend StatDCAT-AP to cover the representation of Metadataflows as a new kind of catalogued resource? The specific additions would be:

• Link to the MSD (dct:conformsTo)
• Link to disseminated datasets

Originally posted by @ChLaaboudi in #24

Related 2016 issue: Modelling SDMX Metadataflow in StatDCAT-AP by @duffgui

From SDMX Glossary:
Definition: Collection of metadata concepts, structure and usage when used to collect or disseminate reference metadata.

Context: A reference metadata set also has a set of structural metadata which describes how it is organised. This metadata identifies what reference metadata concepts are being reported, how these concepts relate to each other (typically as hierarchies), what their presentational structure is, how they may be represented (as free text, as coded values, etc.), and with which formal object types they are associated.

[pellemar]

In SDMX 3.1 (as it was in version 2) a Metadataflow is linked to a MSD, just like a Dataflow is linked to a DSD. They move at the same level. In StatDCAT-AP, this should be at the Dataset level, mapping to the attributes defined in the MSD. As metadata are just "data about data", there is no need of duplicating the Dataset element: a Metadataset is just another Dataset, with the difference that it links to attributes only, because in the MSD there is no dimension. But a Metadataset may target a specific DSD. And the advantage is to allow the definitions of a precise set of allowed attributes (such as SIMS).

[jakubklimek]

Could we get a more specific proposal on what we need to know about a Metadataflow, similarly to what we have now for a Dataflow?, as a subclass of Catalog?

@ChLaaboudi You mentioned similarities between Dataflow and Metadataflow. Is a Metadataflow also a special case of a Catalogue, containing datasets? Your original suggestion seems like it. And what should be the connection between a Dataflow and a Metadataflow?

@pellemar you mention that a Metadataflow corresponds to a Dataset (perhaps of a special kind), not a Catalogue? But then there should be a link between the Metadataflow dataset and other datasets related to the Metadataflow?

Also, since in the Data Cube Vocabulary, there is a representation of a DSD, not an MSD, how should the MSD be represented?

[ChLaaboudi]

You mentioned similarities between Dataflow and Metadataflow. Is a Metadataflow also a special case of a Catalogue, containing datasets? Your original suggestion seems like it. And what should be the connection between a Dataflow and a Metadataflow?

The core question a user has when they find a dataset is: "Where can I read about its quality?" They want a direct link to a document describing how the data was collected, its coverage, and its reliability/quality (the MetadataReport).

Since the MetadataReport is a document (mainly a URL) rather than an RDF resource, the most suitable property is foaf:page:

Dataset / Dataflow  ──(foaf:page)──►  <URL of MetadataReport document>
                                                   │
                                        dct:conformsTo SIMS

This pragmatic choice can be revisited once a SIMS ontology is available — at which point the MetadataReport could be treated as a fully described RDF resource, and the full Metadataflow structure introduced accordingly..

[pellemar]

@jakubklimek you mention that "in the Data Cube Vocabulary, there is a representation of a DSD, not an MSD". In 2017, a W3C group led by Franck Cotton (INSEE, at that time) edited "An RDF vocabulary for representing the SDMX metadata model" (you can read it at [https://linked-statistics.github.io/SDMX-Metadata/sdmx-metadata.html]). The group expressed the “metadata” part of the SDMX information model (Metadata Structure Definition and Metadata Set) using RDF and related standards. As far as I know, INSEE accepted it as the basis of its reference metadata standard. So, it is not completely true that an RDF vocabulary for metadata still does not exist. StatDCAT should be a bit more ambitious. A metadata report is "usually" a document and very often can only be retrieved via a URL, but this was also possible in StatDCAT-AP v1 through an annotation.
I am not such a big technical expert of LoD and DCAT, but I would invite you to read the "Metadata Structure Definition" chapter of the W3C document, where they propose a way to model metadata attributes and reports in a way which is consistent with SDMX and with an RDF representation.

[jakubklimek]

@ChLaaboudi Your proposal could be therefore resolved with:

1. Usage note on Dataset.documentation
2. Similarly, added documentation property to Dataflow
3. Added conformsTo to Document with the SIMS URL specified.

Could you phrase the usage note and specify the SIMS URL that would identify it?

@pellemar Thank you for the pointer. We are aware of the existence of that document; however, it is not directly usable in its current form. It seems unfinished, abandoned 9 years ago, the namespace is not functional, missing examples, etc. Nevertheless, it can be taken into account in future work of making SIMS machine-readable (#14).

[Heractos]

Original contents of this comment moved to #24 (comment) by @jakubklimek

[ChLaaboudi]

@jakubklimek

1. Usage note on Dataset.documentation

A document providing reference metadata — such as methodological descriptions and quality information — about this Dataset. In the ESS context, such documents typically conform to the Single Integrated Metadata Structure (SIMS) or a derived reporting structure (ESMS, ESQRS).

3. Added conformsTo to Document with the SIMS URL.

There is currently no stable, dereferenceable URI for SIMS itself
What exists:

• https://cros.ec.europa.eu/book-page/sims — the CROS portal page describing SIMS. I
• The official SIMS 2.0 PDF: https://ec.europa.eu/eurostat/documents/64157/4373903/SIMS-2-0-Revised-standards-November-2015-ESSC-final.pdf

[jakubklimek]

@ChLaaboudi please check

1. Added usage note on Dataset.documentation
2. Added Dataflow.documentation with the same usage note
3. Added Document.conformsTo
4. Added Metadata report example

@Heractos I moved your comment to the Dataflow issue #24

This issue is originally about support for SDMX Metadataflow. We have added support for linking a Metadata report document as a first step, with proper support for Metadataflow pending as future work, waiting for #14.