Update Databricks dashboard layout and docs (#30)

Author: anand-testcompare

README.md

@@ -83,7 +83,7 @@ There are two supported Databricks paths:
 
 | Path | What it creates | When to use it | Recommendation |
 |---|---|---|---|
-| Databricks Delta | Unity Catalog control, bronze, and silver schemas | Primary production path | Recommended |
+| Databricks Delta | Unity Catalog control/bronze/silver schemas plus silver current-state tables | Primary production path | Recommended |
 | Databricks over S3 | Unity Catalog views over published Parquet snapshots | Reference example, simpler bridge from the Rust exporter | Supported, but secondary |
 
 Recommended Databricks Delta flow:
@@ -95,20 +95,20 @@ just databricks-delta-bootstrap <warehouse-id>
 just databricks-delta-sync-secret DEFAULT
 just databricks-delta-deploy DEFAULT prod
 just databricks-delta-run DEFAULT prod
+just databricks-delta-deploy-pipeline DEFAULT prod
+just databricks-delta-run-pipeline DEFAULT prod
 ```
 
 The Delta path creates and updates:
 
-- `convex_sync_kit_<source>_delta_control`
-- `convex_sync_kit_<source>_delta_bronze`
-- `convex_sync_kit_<source>_delta_silver`
+- `convex_sync_kit_<source>_delta_control` for checkpoints
+- `convex_sync_kit_<source>_delta_bronze` for append-only CDC landing tables
+- `convex_sync_kit_<source>_delta_silver` for Lakeflow-materialized current-state tables
 
-The silver schema is expected to stay empty until you stand up a Lakeflow `AUTO CDC` pipeline for the tables you actually want to materialize there.
-
-```bash
-just databricks-delta-deploy-pipeline DEFAULT prod
-just databricks-delta-run-pipeline DEFAULT prod
-```
+Bootstrap creates the schemas, the extractor job writes control + bronze, and the
+Lakeflow `AUTO CDC` pipeline materializes silver. The checked-in `meshix-api`
+example already includes a generated bronze-to-silver pipeline file, while new
+sources still need the same deploy/run pipeline step after bronze tables exist.
 
 Reference Databricks over S3 flow:
 
@@ -150,7 +150,7 @@ Relevant Foundry docs:
 | Local recurring analysis | raw change log parquet, staging parquet | user-defined paths |
 | S3/export | `staging/current`, manifests, versioned snapshots | bucket and prefix chosen by operator |
 | Databricks over S3 | Unity Catalog views over published parquet snapshots | `convex_sync_kit_<source>_s3` |
-| Databricks Delta | checkpoint table, bronze CDC tables, silver current-state tables | `convex_sync_kit_<source>_delta_{control,bronze,silver}` |
+| Databricks Delta | checkpoint table, bronze CDC tables, silver current-state tables materialized by Lakeflow `AUTO CDC` | `convex_sync_kit_<source>_delta_{control,bronze,silver}` |
 
 The checked-in [`sources/meshix-api/env.sh`](sources/meshix-api/env.sh) file is only an example source profile, not a repo identity. Add more source directories as you onboard more Convex projects.
 
@@ -215,8 +215,8 @@ The first version focuses on:
 - a side-by-side bronze/silver table map
 
 The dashboard now filters out internal Lakeflow objects from the bronze/silver
-counts and uses full-width tables for recent checkpoints, per-table record
-counts, and the bronze/silver map.
+counts, places recent checkpoints and per-table record counts side by side,
+and keeps the bronze/silver map full width underneath.
 
 To make the pipeline feel truly live:
 
@@ -242,7 +242,8 @@ If you want to show this repo working in a talk or video, start with:
    - `convex_sync_kit_meshix_api_delta_silver`
 4. A query result from `connector_checkpoint_latest` showing `meshix-api / delta_tail`.
 5. A `SHOW TABLES` result for the bronze schema showing many `_cdc` tables.
-6. The S3-backed `__source_map` view so people can see the reference path is real too.
+6. A `SHOW TABLES` result for the silver schema showing materialized current-state tables.
+7. The S3-backed `__source_map` view so people can see the reference path is real too.
 
 There is a more detailed capture list in [docs/demo-storyboard.md](docs/demo-storyboard.md).
 

docs/monitoring.md

@@ -27,8 +27,9 @@ The first dashboard focuses on:
 - per-table bronze vs silver record counts
 - a side-by-side bronze/silver table map
 
-The template filters out internal Lakeflow objects from the layer counts and
-uses full-width tables so the dashboard is easier to read in Lakeview.
+The template filters out internal Lakeflow objects from the layer counts,
+places recent checkpoints and per-table record counts side by side, and keeps
+the bronze/silver map full width below.
 
 ## AUTO CDC Status
 

platform/databricks/delta/README.md

@@ -101,9 +101,14 @@ flowchart LR
   J[convex_delta_extract job]
   C[checkpoint table]
   T[bronze CDC tables]
+  P[Lakeflow AUTO CDC pipeline]
+  S[silver current-state tables]
   B --> J
+  B --> P
   J --> C
   J --> T
+  T --> P
+  P --> S
 ```
 
 Recommended operator entrypoints:
@@ -112,13 +117,17 @@ Recommended operator entrypoints:
 just databricks-delta-sync-secret
 just databricks-delta-bootstrap <warehouse_id>
 just databricks-delta-publish-dashboard DEFAULT <warehouse_id>
+just databricks-delta-deploy DEFAULT prod
+just databricks-delta-run DEFAULT prod
 just databricks-delta-deploy-pipeline DEFAULT prod
 just databricks-delta-run-pipeline DEFAULT prod
-just databricks-delta-deploy
-just databricks-delta-run
 just databricks-delta-smoke <warehouse_id>
 ```
 
+The checked-in `meshix-api` example already has a generated bronze-to-silver
+Lakeflow SQL file under `generated/`. New sources render that file during the
+`deploy-pipeline` step once bronze tables exist.
+
 Auto-update behavior after deployment:
 
 - the extractor job is scheduled every 5 minutes

platform/databricks/delta/dashboards/convex_sync_overview.lvdash.json.tmpl

@@ -263,7 +263,7 @@
           "position": {
             "x": 0,
             "y": 2,
-            "width": 3,
+            "width": 2,
             "height": 3
           }
         },
@@ -301,7 +301,7 @@
             }
           },
           "position": {
-            "x": 3,
+            "x": 2,
             "y": 2,
             "width": 3,
             "height": 3
@@ -341,7 +341,7 @@
             }
           },
           "position": {
-            "x": 6,
+            "x": 5,
             "y": 2,
             "width": 3,
             "height": 3
@@ -381,9 +381,9 @@
             }
           },
           "position": {
-            "x": 9,
+            "x": 8,
             "y": 2,
-            "width": 3,
+            "width": 4,
             "height": 3
           }
         },
@@ -450,7 +450,7 @@
           "position": {
             "x": 0,
             "y": 5,
-            "width": 12,
+            "width": 6,
             "height": 7
           }
         },
@@ -507,10 +507,10 @@
             }
           },
           "position": {
-            "x": 0,
-            "y": 12,
-            "width": 12,
-            "height": 6
+            "x": 6,
+            "y": 5,
+            "width": 6,
+            "height": 7
           }
         },
         {
@@ -583,13 +583,14 @@
           },
           "position": {
             "x": 0,
-            "y": 18,
+            "y": 12,
             "width": 12,
             "height": 12
           }
         }
       ],
-      "pageType": "PAGE"
+      "pageType": "PAGE_TYPE_CANVAS",
+      "layoutVersion": "GRID_V1"
     }
   ]
 }