feat: add databricks delta monitoring dashboard

Author: anand-testcompare

Cargo.lock

@@ -162,6 +162,26 @@ depot ci run --workflow .depot/workflows/release.yml
 depot ci run --workflow .depot/workflows/release-rc.yml
 ```
 
+## Monitoring
+
+The Delta path now has a Lakeview dashboard template and publish flow for a
+high-level sync view:
+
+```bash
+just databricks-delta-publish-dashboard DEFAULT <warehouse-id>
+```
+
+The first version focuses on:
+
+- latest checkpoint freshness
+- bronze table count
+- silver table count
+- recent checkpoint history
+- bronze and silver table inventory
+
+Silver remaining empty is expected until you deploy a real Lakeflow `AUTO CDC`
+pipeline that reads bronze and materializes silver.
+
 ## Suggested Screenshots
 
 If you want to show this repo working in a talk or video, start with:
@@ -183,6 +203,7 @@ There is a more detailed capture list in [docs/demo-storyboard.md](docs/demo-sto
 
 - [Ask DeepWiki about this repo](https://deepwiki.com/shpitdev/convex-sync-kit)
 - [docs/architecture.md](docs/architecture.md)
+- [docs/monitoring.md](docs/monitoring.md)
 - [docs/public-reference-map.md](docs/public-reference-map.md)
 - [docs/release-artifacts.md](docs/release-artifacts.md)
 - [docs/demo-storyboard.md](docs/demo-storyboard.md)

README.md

@@ -0,0 +1,51 @@
+# Monitoring
+
+## Databricks Delta Dashboard
+
+This repo now includes a Lakeview dashboard template for the Databricks Delta
+path:
+
+- `platform/databricks/delta/dashboards/convex_sync_overview.lvdash.json.tmpl`
+- `platform/databricks/delta/dashboards/dashboards.json`
+
+Publish it with:
+
+```bash
+just databricks-delta-publish-dashboard DEFAULT <warehouse_id>
+```
+
+If you pass an existing dashboard ID to the publish script, it updates and
+re-publishes that dashboard. Otherwise it creates a new one and prints the new
+dashboard ID.
+
+The first dashboard focuses on:
+
+- latest checkpoint freshness
+- bronze table count
+- silver table count
+- recent checkpoint history
+- bronze and silver table inventory
+
+## Why Silver Is Empty
+
+Silver is still expected to be empty until a real Lakeflow `AUTO CDC` pipeline
+is deployed.
+
+What exists today:
+
+- control schema and checkpoint tables
+- bronze CDC landing
+- Delta extractor job
+- Lakeflow SQL template for per-table `AUTO CDC`
+
+What is still missing:
+
+- a source-aware pipeline generation and deploy path that turns the bronze
+  tables for a specific source into a real deployed Lakeflow pipeline
+
+The generic blocker is not Lakeflow itself. It is the missing code path that:
+
+1. enumerates the bronze tables for one source
+2. decides the silver target names consistently
+3. renders the per-table `AUTO CDC` SQL
+4. deploys or updates the pipeline repeatably

docs/monitoring.md

@@ -81,6 +81,12 @@ databricks-delta-sync-secret *args:
 databricks-delta-bootstrap warehouse_id profile="DEFAULT":
   ./scripts/bootstrap-databricks-delta.sh {{profile}} {{warehouse_id}}
 
+databricks-delta-render-dashboard output_file:
+  ./scripts/render-databricks-delta-dashboard.sh {{output_file}}
+
+databricks-delta-publish-dashboard profile warehouse_id dashboard_id="":
+  ./scripts/publish-databricks-delta-dashboard.sh {{profile}} {{warehouse_id}} {{dashboard_id}}
+
 databricks-delta-deploy profile="DEFAULT" target="dev":
   ./scripts/deploy-databricks-delta.sh {{profile}} {{target}}
 

justfile

@@ -29,6 +29,7 @@ cannot overwrite key, ordering, or delete semantics.
 
 - `databricks.yml`: Databricks bundle entrypoint
 - `resources/`: Databricks bundle resources for the extractor job
+- `dashboards/`: Lakeview dashboard templates and metadata
 - `extractor/convex_cdc_job.py`: Databricks job entrypoint
 - `sql/bootstrap/`: ordered bootstrap DDL for configurable control/bronze/silver schemas and checkpoint table
 - `lakeflow/bronze_to_silver_template.sql`: per-table Lakeflow template
@@ -39,6 +40,8 @@ Bundle lifecycle:
 
 - `scripts/ensure-databricks-delta-secret.sh <profile> [scope] [key]`
 - `scripts/bootstrap-databricks-delta.sh <profile> <warehouse_id>`
+- `scripts/render-databricks-delta-dashboard.sh <output_file>`
+- `scripts/publish-databricks-delta-dashboard.sh <profile> <warehouse_id> [dashboard_id]`
 - `scripts/deploy-databricks-delta.sh <profile> <target>`
 - `scripts/run-databricks-delta-job.sh <profile> <target> [job_key]`
 - `scripts/run-databricks-delta-smoke.sh <profile> <target> <warehouse_id>`
@@ -105,6 +108,7 @@ Recommended operator entrypoints:
 ```bash
 just databricks-delta-sync-secret
 just databricks-delta-bootstrap <warehouse_id>
+just databricks-delta-publish-dashboard DEFAULT <warehouse_id>
 just databricks-delta-deploy
 just databricks-delta-run
 just databricks-delta-smoke <warehouse_id>

platform/databricks/delta/README.md

@@ -0,0 +1,19 @@
+# Databricks Delta Dashboards
+
+Lakeview dashboard assets for monitoring the Convex sync at a high level.
+
+Files:
+
+- `dashboards.json`: lightweight manifest for the included dashboards
+- `convex_sync_overview.lvdash.json.tmpl`: renderable Lakeview dashboard template
+
+Use the helper scripts:
+
+```bash
+./scripts/render-databricks-delta-dashboard.sh /tmp/convex-sync-overview.lvdash.json
+./scripts/publish-databricks-delta-dashboard.sh DEFAULT <warehouse_id>
+```
+
+If you pass an existing dashboard ID to the publish script, it updates and
+re-publishes that dashboard. Otherwise it creates a new one and prints the
+dashboard ID.