feat: bundle databricks delta workflow (#19)

* feat: bundle databricks delta workflow

* fix: use direct engine for delta bundle validation

Author: anand-testcompare

README.md

@@ -8,7 +8,7 @@
 Convex CDC sync engine with two supported target families:
 
 - `S3/export`: append-only raw parquet -> current-state staging parquet -> S3 publish
-- `Databricks/native`: bronze Delta CDC landing -> Lakeflow `AUTO CDC` -> silver current-state Delta tables
+- `Databricks Delta`: bronze Delta CDC landing -> Lakeflow `AUTO CDC` -> silver current-state Delta tables
 
 The source-side behavior intentionally stays close to the public Convex/Fivetran
 extraction model:
@@ -29,7 +29,7 @@ flowchart TD
   S3[crates/convex-export-s3]
   AWS[platform/aws]
   DBS3[platform/databricks/s3]
-  DBN[platform/databricks/native]
+  DBN[platform/databricks/delta]
   Root --> Inspect
   Root --> CLI
   Root --> Core
@@ -48,7 +48,7 @@ Read the repo by layer:
 - [`platform/aws/README.md`](platform/aws/README.md): AWS assets for publishing and downstream readers
 - [`platform/databricks/README.md`](platform/databricks/README.md): Databricks target family overview
 - [`platform/databricks/s3/README.md`](platform/databricks/s3/README.md): Databricks consuming the S3 export path
-- [`platform/databricks/native/README.md`](platform/databricks/native/README.md): Databricks-native bronze/silver landing
+- [`platform/databricks/delta/README.md`](platform/databricks/delta/README.md): Databricks Delta bronze/silver landing
 
 ## Install
 
@@ -85,7 +85,7 @@ flowchart LR
   C[Convex]
   E[shared sync semantics]
   S3[S3 export path]
-  DBN[Databricks native path]
+  DBN[Databricks Delta path]
   DBS3[Databricks over S3 path]
   C --> E
   E --> S3
@@ -126,20 +126,28 @@ Or via `just`:
 - `just publish-s3 --bucket your-bucket`
 - `just run --bucket your-bucket`
 
-### `Databricks/native`
+### `Databricks Delta`
 
-Checked-in Databricks-native assets:
+Checked-in Databricks Delta assets:
 
-- `platform/databricks/native/extractor/convex_cdc_job.py`
-- `platform/databricks/native/sql/bootstrap/`
-- `platform/databricks/native/lakeflow/bronze_to_silver_template.sql`
+- `platform/databricks/delta/databricks.yml`
+- `platform/databricks/delta/resources/convex_delta_extract.job.yml`
+- `platform/databricks/delta/extractor/convex_cdc_job.py`
+- `platform/databricks/delta/sql/bootstrap/`
+- `platform/databricks/delta/lakeflow/bronze_to_silver_template.sql`
 
 Runtime split:
 
 1. a Databricks job runs the extractor and appends bronze CDC rows
 2. checkpoint rows land in the control schema
 3. Lakeflow `AUTO CDC` materializes silver current-state tables
 
+Packaged entrypoints:
+
+- `just databricks-delta-deploy`
+- `just databricks-delta-run`
+- `just databricks-delta-smoke <warehouse_id>`
+
 ### `Databricks over S3`
 
 This variation keeps the existing Rust exporter and S3 publish loop, then adds:

docs/architecture.md

@@ -6,7 +6,7 @@ This repo has three explicit layers:
 
 - `core`: Convex extraction and checkpoint semantics
 - `target-s3`: raw parquet, staging parquet, and S3 publish
-- `platform/databricks`: Databricks assets for both S3-backed consumption and Databricks-native landing
+- `platform/databricks`: Databricks assets for both S3-backed consumption and Databricks Delta landing
 
 The shared extraction logic stays target-agnostic. Targets decide how event
 batches are durably written and what downstream shape they expose.
@@ -67,7 +67,7 @@ Use it when you want:
 - a target-agnostic export contract
 - another platform to consume S3 directly
 
-## `Databricks/native`
+## `Databricks/delta`
 
 ```text
 Convex
@@ -87,7 +87,7 @@ Owned pieces:
 Use it when:
 
 - Databricks is the primary serving layer
-- you want Databricks-native CDC reconstruction
+- you want Databricks Delta CDC reconstruction
 - downstream consumers can read Unity Catalog tables directly
 
 ## Data Shapes
@@ -121,7 +121,7 @@ Append-only CDC landing in Delta.
 Current-state Delta tables derived from bronze CDC.
 
 - one current row per source key
-- resolved with Databricks-native CDC semantics
+- resolved with Databricks Delta CDC semantics
 
 ## Checkpoints
 
@@ -140,7 +140,7 @@ Rules:
 Target storage differs:
 
 - `S3/export`: file-backed JSON
-- `Databricks/native`: Delta control table
+- `Databricks/delta`: Delta control table
 
 ## Boundary
 
@@ -149,7 +149,7 @@ This repo should own:
 - Convex extraction
 - checkpoint semantics
 - S3/export target code
-- Databricks-native landing assets
+- Databricks Delta landing assets
 
 Downstream systems should own:
 

docs/public-reference-map.md

@@ -119,4 +119,4 @@ Today we intentionally differ from the Fivetran connector in these ways:
 - we support target-owned landing contracts instead of one destination runtime
 - the S3/export path writes Parquet datasets instead of pushing rows into a Fivetran destination
 - the S3/export path materializes local `staging` tables instead of relying on warehouse-native managed tables
-- the Databricks-native path lands bronze CDC rows directly and relies on Lakeflow `AUTO CDC` for current-state resolution
+- the Databricks Delta path lands bronze CDC rows directly and relies on Lakeflow `AUTO CDC` for current-state resolution

docs/release-artifacts.md

@@ -39,9 +39,11 @@ as separate binary assets:
 
 - `platform/aws/`
 - `platform/databricks/s3/`
-- `platform/databricks/native/extractor/convex_cdc_job.py`
-- `platform/databricks/native/sql/bootstrap/`
-- `platform/databricks/native/lakeflow/`
+- `platform/databricks/delta/databricks.yml`
+- `platform/databricks/delta/resources/`
+- `platform/databricks/delta/extractor/convex_cdc_job.py`
+- `platform/databricks/delta/sql/bootstrap/`
+- `platform/databricks/delta/lakeflow/`
 
 ## Explicit Non-Artifacts
 
@@ -59,4 +61,4 @@ Not in the first release slice:
 
 - wider platform matrix beyond `linux-amd64`
 - container images for the S3/export runtime
-- separate packaging for Databricks-native assets
+- separate packaging for Databricks Delta assets

justfile

@@ -75,6 +75,15 @@ databricks-sync-staging-views *args:
 databricks-apply-sql-dir profile warehouse_id sql_dir:
   ./scripts/apply-databricks-sql-dir.sh {{profile}} {{warehouse_id}} {{sql_dir}}
 
+databricks-delta-deploy profile="DEFAULT" target="dev":
+  ./scripts/deploy-databricks-delta.sh {{profile}} {{target}}
+
+databricks-delta-run profile="DEFAULT" target="dev" job_key="convex_delta_extract":
+  ./scripts/run-databricks-delta-job.sh {{profile}} {{target}} {{job_key}}
+
+databricks-delta-smoke warehouse_id profile="DEFAULT" target="dev":
+  ./scripts/run-databricks-delta-smoke.sh {{profile}} {{target}} {{warehouse_id}}
+
 # Install
 install-dev:
   ./install.sh --mode dev --force

platform/databricks/README.md

@@ -3,16 +3,16 @@
 Databricks assets grouped by target family:
 
 - `s3/`: Databricks consuming the existing S3 export path
-- `native/`: Databricks-first assets where Convex changes land directly in
+- `delta/`: Databricks-first assets where Convex changes land directly in
   Unity Catalog Delta tables
 
 ```mermaid
 flowchart LR
   S3[S3-backed path]
-  Native[Databricks-native path]
+  Delta[Databricks Delta path]
   S3 --> Views[Unity Catalog views over S3 parquet]
-  Native --> Bronze[bronze CDC Delta]
-  Native --> Silver[silver current-state Delta]
+  Delta --> Bronze[bronze CDC Delta]
+  Delta --> Silver[silver current-state Delta]
 ```
 
 Use these assets the same way as the AWS templates:
@@ -42,17 +42,18 @@ external location before `read_files(...)` views are applied.
 
 Read more: [`platform/databricks/s3/README.md`](s3/README.md)
 
-## `native/`
+## `delta/`
 
 The Databricks-first assets are the starting point for direct Delta landing.
 
-- `native/extractor/`: a Databricks job entrypoint that mirrors the current
+- `delta/extractor/`: a Databricks job entrypoint that mirrors the current
   Convex snapshot/delta checkpoint logic and writes bronze CDC tables
-- `native/lakeflow/`: Lakeflow SQL templates that turn bronze CDC tables into
+- `delta/lakeflow/`: Lakeflow SQL templates that turn bronze CDC tables into
   silver current-state tables
-- `native/sql/`: bootstrap DDL for schemas and control tables
+- `delta/sql/`: bootstrap DDL for schemas and control tables
+- `delta/resources/`: Databricks bundle job definitions
 
 The provider is configured from `~/.databrickscfg` by default, typically using
 the `DEFAULT` profile.
 
-Read more: [`platform/databricks/native/README.md`](native/README.md)
+Read more: [`platform/databricks/delta/README.md`](delta/README.md)

platform/databricks/native/README.md platform/databricks/delta/README.mdplatform/databricks/native/README.md renamed to platform/databricks/delta/README.md

@@ -1,4 +1,4 @@
-# Databricks-native Target
+# Databricks Delta Target
 
 Databricks as the primary landing plane:
 
@@ -27,13 +27,47 @@ cannot overwrite key, ordering, or delete semantics.
 
 ## Layout
 
+- `databricks.yml`: Databricks bundle entrypoint
+- `resources/`: Databricks bundle resources for the extractor job
 - `extractor/convex_cdc_job.py`: Databricks job entrypoint
 - `sql/bootstrap/`: ordered bootstrap DDL for configurable control/bronze/silver schemas and checkpoint table
 - `lakeflow/bronze_to_silver_template.sql`: per-table Lakeflow template
 
-Apply the bootstrap directory with:
+## Deploy Surface
+
+Bundle lifecycle:
+
+- `scripts/deploy-databricks-delta.sh <profile> <target>`
+- `scripts/run-databricks-delta-job.sh <profile> <target> [job_key]`
+- `scripts/run-databricks-delta-smoke.sh <profile> <target> <warehouse_id>`
+
+These scripts default `DATABRICKS_BUNDLE_ENGINE=direct` so deployment does not
+depend on Terraform downloads.
+
+Bootstrap SQL can still be applied directly with:
 
 - `scripts/apply-databricks-sql-dir.sh <profile> <warehouse_id> <rendered_sql_dir>`
 
 The extractor mirrors the Rust source/checkpoint logic and does not depend on
 the local parquet/S3 path.
+
+## Typical Flow
+
+```mermaid
+flowchart LR
+  B[bundle validate/deploy]
+  J[convex_delta_extract job]
+  C[checkpoint table]
+  T[bronze CDC tables]
+  B --> J
+  J --> C
+  J --> T
+```
+
+Recommended operator entrypoints:
+
+```bash
+just databricks-delta-deploy
+just databricks-delta-run
+just databricks-delta-smoke <warehouse_id>
+```

platform/databricks/delta/databricks.yml

@@ -0,0 +1,49 @@
+bundle:
+  name: convex-streaming-olap-export-delta
+  uuid: 2d5d8d8e-cdb2-41e4-bc7e-7ffae07f628d
+
+include:
+  - resources/*.yml
+
+variables:
+  convex_deployment_url:
+    description: Convex deployment root URL.
+  convex_deploy_key:
+    description: Convex deploy key used by the extractor.
+  source_id:
+    description: Source identifier stored in the checkpoint table.
+  table_name:
+    description: Optional table filter for smoke/debug runs.
+  catalog:
+    description: Unity Catalog catalog that owns control and bronze tables.
+  control_schema:
+    description: Schema that owns the checkpoint table.
+  bronze_schema:
+    description: Schema that owns bronze CDC tables.
+  checkpoint_table:
+    description: Checkpoint table name.
+
+targets:
+  dev:
+    mode: development
+    default: true
+    workspace:
+      profile: DEFAULT
+    variables:
+      source_id: convex-streaming-olap-export-dev
+      table_name: ""
+      catalog: workspace
+      control_schema: convex_streaming_olap_export_control
+      bronze_schema: convex_streaming_olap_export_bronze
+      checkpoint_table: connector_checkpoint
+  prod:
+    mode: production
+    workspace:
+      profile: DEFAULT
+    variables:
+      source_id: convex-streaming-olap-export
+      table_name: ""
+      catalog: workspace
+      control_schema: convex_streaming_olap_export_control
+      bronze_schema: convex_streaming_olap_export_bronze
+      checkpoint_table: connector_checkpoint

…rm/databricks/native/extractor/README.md …orm/databricks/delta/extractor/README.mdplatform/databricks/native/extractor/README.md renamed to platform/databricks/delta/extractor/README.md platform/databricks/native/extractor/README.md renamed to platform/databricks/delta/extractor/README.md

@@ -1,6 +1,6 @@
 # Convex CDC Job
 
-`convex_cdc_job.py` is the Databricks job entrypoint for the Databricks-native
+`convex_cdc_job.py` is the Databricks job entrypoint for the Databricks Delta
 target family.
 
 It mirrors the current Rust source/checkpoint behavior:
@@ -24,3 +24,6 @@ It mirrors the current Rust source/checkpoint behavior:
 - `DATABRICKS_CONTROL_SCHEMA`: defaults to `control`
 - `DATABRICKS_BRONZE_SCHEMA`: defaults to `bronze`
 - `DATABRICKS_CHECKPOINT_TABLE`: defaults to `connector_checkpoint`
+
+In the bundled Databricks Delta path, these are usually passed as task
+parameters rather than exported manually.