Skip to content

feat: bundle databricks delta workflow#19

Merged
anandpant merged 2 commits into
mainfrom
feat/databricks-delta-bundle
Apr 21, 2026
Merged

feat: bundle databricks delta workflow#19
anandpant merged 2 commits into
mainfrom
feat/databricks-delta-bundle

Conversation

@anandpant

@anandpant anandpant commented Apr 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • rename the Databricks direct-landing path from native to delta
  • add a real Databricks bundle/deploy/run/smoke surface for the Delta path
  • document and expose the Delta operator flow through just and readmes

What Changed

  • rename platform/databricks/native -> platform/databricks/delta
  • add platform/databricks/delta/databricks.yml
  • add platform/databricks/delta/resources/convex_delta_extract.job.yml
  • add scripts:
    • scripts/render-databricks-delta-bootstrap.sh
    • scripts/deploy-databricks-delta.sh
    • scripts/run-databricks-delta-job.sh
    • scripts/run-databricks-delta-smoke.sh
  • add just commands:
    • databricks-delta-deploy
    • databricks-delta-run
    • databricks-delta-smoke
  • update repo/docs naming from native to delta
  • document that the bundle scripts default to the direct deployment engine

Why

This turns the Databricks Delta path from checked-in pieces into a usable operator surface:

  • one bundle config
  • one deploy command
  • one run command
  • one smoke command

Verification

  • cargo test --workspace
  • actionlint -config-file .github/actionlint.yaml .github/workflows/*.yml
  • bash -n scripts/render-databricks-delta-bootstrap.sh scripts/deploy-databricks-delta.sh scripts/run-databricks-delta-job.sh scripts/run-databricks-delta-smoke.sh
  • databricks bundle validate -p DEFAULT -t dev ... from platform/databricks/delta
  • live smoke run:
    • ./scripts/run-databricks-delta-smoke.sh DEFAULT dev 63d28889f3eb3c4b
    • bundle deployed successfully with DATABRICKS_BUNDLE_ENGINE=direct
    • job run succeeded: 402885444842042 / 327152123456035
    • checkpoint row verified in workspace.convex_streaming_olap_export_delta_smoke_20260420204345_control.connector_checkpoint_latest
    • bronze table verified in workspace.convex_streaming_olap_export_delta_smoke_20260420204345_bronze (convex__app__jobs__cdc)

Open in Devin Review

devin-ai-integration[bot]

This comment was marked as resolved.

@anandpant
anandpant merged commit 7b63a3e into main Apr 21, 2026
9 checks passed
@anandpant
anandpant deleted the feat/databricks-delta-bundle branch April 21, 2026 02:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant