Fix Databricks dashboard counts and auto-refresh (#28)

Author: anand-testcompare

README.md

@@ -207,14 +207,27 @@ just databricks-delta-publish-dashboard DEFAULT <warehouse-id>
 
 The first version focuses on:
 
-- latest checkpoint freshness
+- checkpoint freshness and write volume
 - bronze table count
 - silver table count
 - recent checkpoint history
-- bronze and silver table inventory
+- per-table bronze vs silver record counts
+- a side-by-side bronze/silver table map
 
-Silver remaining empty is expected until you deploy a real Lakeflow `AUTO CDC`
-pipeline that reads bronze and materializes silver.
+The dashboard now filters out internal Lakeflow objects from the bronze/silver
+counts and uses full-width tables for recent checkpoints, per-table record
+counts, and the bronze/silver map.
+
+To make the pipeline feel truly live:
+
+- the extractor job now deploys with a 5-minute Databricks schedule
+- the Lakeflow pipeline is configured as continuous after its first run
+- after `just databricks-delta-deploy-pipeline DEFAULT prod`, run `just databricks-delta-run-pipeline DEFAULT prod` once to start the continuous update loop
+
+If you want a self-animating demo, add a tiny heartbeat write in the source
+Convex app on a 1-minute cron. That creates a steady stream of real upstream
+changes that show up in checkpoints, bronze, and silver without any manual
+reruns.
 
 ## Suggested Screenshots
 

docs/monitoring.md

@@ -20,11 +20,15 @@ dashboard ID.
 
 The first dashboard focuses on:
 
-- latest checkpoint freshness
+- checkpoint freshness and write volume
 - bronze table count
 - silver table count
 - recent checkpoint history
-- bronze and silver table inventory
+- per-table bronze vs silver record counts
+- a side-by-side bronze/silver table map
+
+The template filters out internal Lakeflow objects from the layer counts and
+uses full-width tables so the dashboard is easier to read in Lakeview.
 
 ## AUTO CDC Status
 
@@ -39,6 +43,15 @@ just databricks-delta-run-pipeline DEFAULT prod
 For a newly onboarded source, silver will stay empty until you run that same
 deploy/run sequence for that source.
 
+Once deployed:
+
+- the extractor job runs every 5 minutes on a Databricks job schedule
+- the Lakeflow pipeline stays continuous after its first `run`
+
+If you want a no-touch proof loop, add a tiny heartbeat write in the upstream
+Convex app on a 1-minute cron. That gives Databricks a steady stream of real
+changes to ingest and makes the dashboard visibly move without manual reruns.
+
 What exists today:
 
 - control schema and checkpoint tables

justfile

@@ -81,8 +81,8 @@ databricks-delta-sync-secret *args:
 databricks-delta-bootstrap warehouse_id profile="DEFAULT":
   ./scripts/bootstrap-databricks-delta.sh {{profile}} {{warehouse_id}}
 
-databricks-delta-render-dashboard output_file:
-  ./scripts/render-databricks-delta-dashboard.sh {{output_file}}
+databricks-delta-render-dashboard output_file profile="":
+  ./scripts/render-databricks-delta-dashboard.sh {{output_file}} {{profile}}
 
 databricks-delta-publish-dashboard profile warehouse_id dashboard_id="":
   ./scripts/publish-databricks-delta-dashboard.sh {{profile}} {{warehouse_id}} {{dashboard_id}}

platform/databricks/delta/README.md

@@ -40,7 +40,7 @@ Bundle lifecycle:
 
 - `scripts/ensure-databricks-delta-secret.sh <profile> [scope] [key]`
 - `scripts/bootstrap-databricks-delta.sh <profile> <warehouse_id>`
-- `scripts/render-databricks-delta-dashboard.sh <output_file>`
+- `scripts/render-databricks-delta-dashboard.sh <output_file> [profile]`
 - `scripts/publish-databricks-delta-dashboard.sh <profile> <warehouse_id> [dashboard_id]`
 - `scripts/render-databricks-delta-pipeline.sh <profile> <output_file>`
 - `scripts/deploy-databricks-delta-pipeline.sh <profile> <target>`
@@ -118,3 +118,10 @@ just databricks-delta-deploy
 just databricks-delta-run
 just databricks-delta-smoke <warehouse_id>
 ```
+
+Auto-update behavior after deployment:
+
+- the extractor job is scheduled every 5 minutes
+- the Lakeflow pipeline is continuous once you start it with the first `run`
+- if you want a hands-free demo, add a simple heartbeat write in the source
+  Convex app so fresh upstream changes keep landing automatically

platform/databricks/delta/dashboards/README.md

@@ -10,10 +10,14 @@ Files:
 Use the helper scripts:
 
 ```bash
-./scripts/render-databricks-delta-dashboard.sh /tmp/convex-sync-overview.lvdash.json
+./scripts/render-databricks-delta-dashboard.sh /tmp/convex-sync-overview.lvdash.json DEFAULT
 ./scripts/publish-databricks-delta-dashboard.sh DEFAULT <warehouse_id>
 ```
 
+Pass the Databricks profile to the render helper if you want the generated
+Lakeview dashboard to include per-table bronze and silver row counts. Without a
+profile, the dashboard still renders, but the row-count dataset is left empty.
+
 If you pass an existing dashboard ID to the publish script, it updates and
 re-publishes that dashboard. Otherwise it creates a new one and prints the
 dashboard ID.