Spark Data Pipeline (SDP) integration

[Neutrinic]

Investigate Spark Data Pipeline integration for declarative API-to-table pipelines.

Background

SDP allows declarative specification of data ingestion pipelines. Apilytics could provide a native SDP source type for REST APIs.

Scope

• Research SDP architecture and extension points
• Design Apilytics SDP source configuration schema
• Implement SDP source adapter wrapping existing connector
• Add SDP-specific documentation

Example

sources:
  - name: github_issues
    type: apilytics
    config:
      openapi: github.yaml
      endpoint: /repos/{owner}/{repo}/issues
      auth:
        type: bearer
        token: ${GITHUB_TOKEN}

Acceptance

• Users can define Apilytics sources in SDP pipeline YAML
• Automatic schema inference from OpenAPI spec
• Incremental refresh support