Skip to content

plan-spec.md

Latest commit

 

History

History
195 lines (158 loc) · 4.74 KB

plan-spec.md

File metadata and controls

195 lines (158 loc) · 4.74 KB
Raw

📄 Kvatch Plan Specification

This document describes the fields available in a Kvatch plan.yaml or plan.json file, used to define federated queries, connectors, datasets, and output configuration.

🔹 Top-Level Fields

Field Type Description
name string Name of the plan
storage object Storage backend details for metadata & results
connectors array List of connector definitions
datasets array Processing/query definitions
output object Controls output formatting and destination

🗄️ storage

storage:
  type: sqlite
  metadata_store_path: meta.db
  data_store_path: quickstart.db
Field Type Description
type string Backend type (e.g., sqlite, postgres)
data_store_path string Optional override for data store path

🔌 connectors

Each connector represents a data source (e.g., file, database, API).

Example:

- name: books_db
  type: POSTGRES
  connection:
    host: localhost
    port: 5432
    database: books
    username: user
    password: pass
    sslmode: disable

Supported connector types and configs:

🧠 POSTGRES

connection:
  host: localhost
  port: 5432
  database: mydb
  username: user
  password: pass
  sslmode: disable

📁 LOCALFILE

connection:
  file_path: ./data/file.csv

🧱 GIT

connection:
  repo: https://github.com/org/repo.git
  branch: main
  path: data/

🌐 GOOGLESHEET

connection:
  spreadsheet_id: your-id
  api_key: your-key
  read_range: Sheet1!A:B
  header_row_no: 0

📊 datasets

Defines how each dataset should be processed.

Example:

- name: books
  connector_name: books_db
  type: SQL
  query: SELECT * FROM books
  dedupe: [id]
Field Type Description
name string Dataset name
connector_name string Name of connector to use
type string Plugin type (e.g., SQL, CSV, JSON)
query string SQL query, filename, or range
config object Plugin-specific options
dedupe array Keys for deduplication (optional)
children array Dependency datasets for joins

🔧 Plugin Configs

📁 CSV

config:
  has_headers: true
  delimiter: ","
  skip_lines: 0

📜 JSON

config:
  root_path: data.items

📊 GOOGLESHEET

config:
  spreadsheet_id: your-id
  read_range: Sheet1!A:B
  header_row_no: 0

📤 output

Controls output formatting, limits, and destination.

output:
  dataset_name: books
  sink: console         # or 'sqlite' or 'file'
  format: table         # table, json, csv, ndjson
  file_path: output.json
  sqlite_path: output.db
  verbose: true
  include_headers: true
  limit: 100
Field Type Description
dataset_name string Dataset to return
sink string Where to output: console, sqlite, file
format string Format: table, json, csv, ndjson
file_path string Output path when using file sink
sqlite_path string Path to SQLite DB when using sqlite sink
verbose bool Print verbose logs
include_headers bool Include headers in output (for table/csv)
limit int Maximum rows to output (0 = unlimited)

✅ Minimal Example

name: quickstart
storage:
  type: sqlite
  data_store_path: quickstart.db
connectors:
  - name: local_file
    type: LOCALFILE
    connection:
      file_path: ./data/users.csv
datasets:
  - name: users
    connector_name: local_file
    type: CSV
    query: users.csv
    options:
      has_headers: true
output:
  dataset_name: users
  sink: console
  format: table