Aggregation Proof of Concept

[j12y]

What does an asset in one of the collections look like?

Build a proof of concept for how we might store data.

• YAML, JSON, XML, other
• is there a tool to validation data against a schema
• what does submission look like and how much would a contributor need to learn to submit
• what does validation look like and can it be automated through PR process
• how might indexing occur to read data into a solution for quick retrieval
• if somebody wanted to build an api, what might it look like?

This is just to track an exploratory project to form an opinion.

[j12y]

Open Data Examples

Open Exoplanet Catalog (using XML)
https://github.com/OpenExoplanetCatalogue/open_exoplanet_catalogue/blob/master/systems/30%20Ari.xml

United States Congress (using YAML)
https://github.com/unitedstates/congress-legislators/blob/main/executive.yaml

[j12y]

Specifications Examples

Open API Specification
https://github.com/OAI/OpenAPI-Specification/tree/main

WebAssembly Specification
https://github.com/WebAssembly/spec

Schema.org (RDF)
https://github.com/schemaorg/schemaorg

• https://schema.org/Event

[j12y]

Specification Formats

JSON-Schema
https://json-schema.org/

JSON-LD
https://json-ld.org/

RDF
https://www.w3.org/RDF/

[j12y]

Data Quality Checking

Soda
https://www.soda.io/

[j12y]

1. JSON Schema

Description: JSON Schema is a JSON-based format used to define the structure, validation, and constraints for JSON documents.

Pros:

• Widely Supported: Many tools and libraries exist for validating JSON Schema in different languages.
• Easy to Read: Designed for JSON, making it intuitive for developers familiar with JSON.
• Extensive Validation Rules: Offers rich validation capabilities (e.g., pattern matching, range constraints).
• Modular and Reusable: Supports $ref for reusing schemas and creating modular designs.
• Good for APIs: Works seamlessly with OpenAPI (Swagger), making it popular for API contracts.

Cons:

• Verbosity: Can become verbose and complex for larger schemas.
• Limited Hierarchical Mapping: Works primarily with JSON and doesn’t naturally map to complex graphs or hierarchical relationships.
• Strictly JSON-Centric: Not natively suitable for other data formats like XML or RDF.

---

2. RDF and RDF Schema (RDFS)

Description: RDF (Resource Description Framework) and its schema layer, RDFS, are designed for describing and linking resources in a semantic and graph-based manner.

Pros:

• Semantic Web and Linked Data: Ideal for defining relationships between entities in a graph structure.
• Flexible: Schema-less design allows you to describe data that evolves over time.
• Interoperable: Can interlink with other RDF datasets, enabling rich integrations.
• SPARQL Querying: Works seamlessly with SPARQL for querying graph-based data.
• Machine-Readable Metadata: Great for use cases requiring semantic reasoning.

Cons:

• Complexity: RDF and RDFS can be harder to grasp for developers unfamiliar with graph-based models.
• Performance: Querying large RDF datasets can be slower than relational databases.
• Less Familiar: Not as widely adopted as JSON Schema, especially in non-semantic web contexts.

---

3. Avro Schema

Description: Apache Avro is a data serialization system that includes a schema definition format for strongly typed data.

Pros:

• Compact Serialization: Ideal for big data and streaming applications due to its compact binary format.
• Schema Evolution: Supports schema evolution by maintaining compatibility (backward/forward changes).
• Interoperability: Works seamlessly with the Hadoop ecosystem (e.g., Kafka, Spark).
• Language Agnostic: Supports multiple programming languages.

Cons:

• Binary Format: Not human-readable by default, requiring tools to inspect.
• Tied to Avro: Primarily useful in the Avro ecosystem, limiting general-purpose use.
• Validation Overhead: Validation tools may not be as robust or widespread as JSON Schema.

---

4. Protobuf (Protocol Buffers)

Description: Protobuf is a schema definition format for Google’s efficient data serialization protocol.

Pros:

• Compact and Fast: Highly efficient in both serialization size and speed.
• Cross-Language Support: Strongly typed and works across many programming languages.
• Schema Evolution: Built-in support for backward and forward compatibility.
• Good for RPC: Often used in gRPC frameworks for API communication.

Cons:

• Not Readable: Requires tools to generate or interpret .proto files.
• Steeper Learning Curve: Syntax and concepts may be less familiar to new users.
• Binary Output: Less suitable for human-readable outputs compared to JSON.

---

5. XML Schema (XSD)

Description: XSD is an XML-based schema definition format used for validating and structuring XML data.

Pros:

• Mature Standard: Established and well-supported in the industry.
• Rich Features: Strong data typing, constraints, and hierarchical structure.
• Extensible: Can define highly complex structures and rules.
• Interoperable: Works seamlessly with SOAP and enterprise systems.

Cons:

• Verbosity: XML and XSD are verbose, making them harder to read and maintain.
• Complexity: Can become unwieldy for large or intricate schemas.
• Not JSON-Friendly: Requires conversion tools to work with JSON-based ecosystems.

---

6. YAML and OpenAPI Schema

Description: YAML-based schemas (like OpenAPI) define the structure and constraints for API contracts, focusing on REST APIs.

Pros:

• Human-Friendly: YAML’s readability is a key advantage.
• Integrated with APIs: Used extensively with OpenAPI for designing RESTful APIs.
• Extensible: Can define custom types and modularize schema definitions.

Cons:

• Limited Validation Tools: Not as robust as JSON Schema for standalone validation.
• Whitespace Sensitivity: YAML’s reliance on indentation can lead to errors.

---

7. Parquet Schema

Description: Apache Parquet uses a schema to define data in a columnar storage format optimized for analytical queries.

Pros:

• Efficient for Analytics: Columnar format allows fast aggregations and queries.
• Compact: Highly optimized for large-scale data storage.
• Schema Evolution: Supports schema evolution with careful design.

Cons:

• Ecosystem Tied: Works best within the Hadoop or big data ecosystem.
• Not Readable: Not human-readable without tools for inspection.
• Limited General Use: Less applicable outside analytical workloads.

---

8. DDL (Data Definition Language)

Description: Used in relational databases to define schemas for structured data (e.g., SQL CREATE TABLE).

Pros:

• Familiarity: SQL is well-understood and widely used.
• Direct Integration: Works with relational databases seamlessly.
• Rich Constraints: Supports indexes, keys, and advanced constraints.

Cons:

• Database-Centric: Tied to relational databases, limiting portability.
• Less Flexible: Not suitable for semi-structured or unstructured data.

---

When to Use Each Format

• Use JSON Schema for REST APIs or JSON-centric applications where human readability and API tooling are important.
• Use RDF/RDFS for graph-based applications, semantic web, or linked data.
• Use Avro or Protobuf for big data pipelines or when compact, fast serialization is critical.
• Use XML/XSD for legacy enterprise systems or XML-based workflows.
• Use Parquet or Avro for analytics and columnar storage in big data ecosystems.
• Use OpenAPI/YAML when designing and documenting APIs alongside schema validation.

Which formats seem most relevant to your use case?