Feature/builder app skills update (#355)

* Update databricks-builder-app to support lakebase autoscale

added/modified builder-app setup and deployments to support autoscale instance along with provisioned.  In the app.yaml.example provided an update on how to configure autoscale and in the README instructions of how to deploy and setup lakebase permission

* minor change to builder app deploy script

made a change to the deploy script to clean up the SP's src directory of the snapshot App source code.  noticed every deployment would make the copy with no clean on the old versions.  created a step in the deploy to clean up the old inactive directories

* Delete databricks-builder-app/app.yaml

removing file that should have ignored in commit

* builder-app-skills update

Deploy all skills (databricks, MLflow, APX) via install_skills.sh and dynamically populate ENABLED_SKILLS in app.yaml at deploy time instead of maintaining a hardcoded list.

Author: scottDBX1886

.claude/skills/databricks-python-sdk/SKILL.md

@@ -613,3 +613,13 @@ If I'm unsure about a method, I should:
 | Pipelines | https://databricks-sdk-py.readthedocs.io/en/latest/workspace/pipelines/pipelines.html |
 | Secrets | https://databricks-sdk-py.readthedocs.io/en/latest/workspace/workspace/secrets.html |
 | DBUtils | https://databricks-sdk-py.readthedocs.io/en/latest/dbutils.html |
+
+## Related Skills
+
+- **[databricks-config](../databricks-config/SKILL.md)** - profile and authentication setup
+- **[databricks-bundles](../databricks-bundles/SKILL.md)** - deploying resources via DABs
+- **[databricks-jobs](../databricks-jobs/SKILL.md)** - job orchestration patterns
+- **[databricks-unity-catalog](../databricks-unity-catalog/SKILL.md)** - catalog governance
+- **[databricks-model-serving](../databricks-model-serving/SKILL.md)** - serving endpoint management
+- **[databricks-vector-search](../databricks-vector-search/SKILL.md)** - vector index operations
+- **[databricks-lakebase-provisioned](../databricks-lakebase-provisioned/SKILL.md)** - managed PostgreSQL via SDK

databricks-builder-app/.gitignore

@@ -32,5 +32,8 @@ client/.vite/
 .vscode/
 *.swp
 
+# Local docs/notes
+docs/TODO.md
+
 # OS
 .DS_Store

databricks-builder-app/README.md

@@ -503,18 +503,21 @@ databricks auth login --host https://your-workspace.cloud.databricks.com
 # 2. Create the app (first time only)
 databricks apps create my-builder-app
 
-# 3. Add Lakebase as a resource (first time only)
+# 3. Configure app.yaml (copy and edit the example)
+cp app.yaml.example app.yaml
+# Edit app.yaml — set LAKEBASE_ENDPOINT (autoscale) or LAKEBASE_INSTANCE_NAME (provisioned)
+
+# 4. (Provisioned Lakebase only) Add Lakebase as an app resource
+#    Skip this step if using autoscale — it connects via OAuth directly.
 databricks apps add-resource my-builder-app \
   --resource-type database \
   --resource-name lakebase \
   --database-instance <your-lakebase-instance-name>
 
-# 4. Configure app.yaml (copy and edit the example)
-cp app.yaml.example app.yaml
-# Edit app.yaml with your Lakebase instance name and other settings
-
 # 5. Deploy
 ./scripts/deploy.sh my-builder-app
+
+# 6. Grant database permissions to the app's service principal (see Section 7)
 ```
 
 ### Step-by-Step Deployment Guide
@@ -564,6 +567,10 @@ The app requires a PostgreSQL database (Lakebase) for storing projects, conversa
 
 #### 4. Add Lakebase as an App Resource
 
+**Autoscale Lakebase**: Skip this step. Autoscale connects via OAuth using `LAKEBASE_ENDPOINT` — no app resource needed.
+
+**Provisioned Lakebase**: Add the instance as an app resource:
+
 ```bash
 databricks apps add-resource my-builder-app \
   --resource-type database \
@@ -647,25 +654,89 @@ The deploy script will:
 
 #### 7. Grant Database Permissions
 
-After the first deployment, grant table permissions to the app's service principal:
+After the first deployment, the app's service principal needs two things:
+1. A **Lakebase OAuth role** (so it can authenticate via OAuth tokens)
+2. **PostgreSQL grants** on the `builder_app` schema (so it can create/read/write tables)
 
-```sql
--- Run this in a Databricks notebook or SQL editor
--- Replace <service-principal-id> with your app's service principal
+##### Step 7a: Find the service principal's client ID
+
+```bash
+SP_CLIENT_ID=$(databricks apps get my-builder-app --output json | jq -r '.service_principal_client_id')
+echo $SP_CLIENT_ID
+```
+
+##### Step 7b: Create a Lakebase OAuth role for the SP
 
-GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public
-  TO `<service-principal-id>`;
-GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public
-  TO `<service-principal-id>`;
-ALTER DEFAULT PRIVILEGES IN SCHEMA public
-  GRANT ALL ON TABLES TO `<service-principal-id>`;
+> **Important**: Do NOT use PostgreSQL `CREATE ROLE` directly. Lakebase Autoscaling requires
+> roles to be created through the Databricks API so the OAuth authentication layer recognizes them.
+
+```python
+from databricks.sdk import WorkspaceClient
+from databricks.sdk.service.postgres import Role, RoleRoleSpec, RoleAuthMethod, RoleIdentityType
+
+w = WorkspaceClient()
+
+# Replace with your branch path and SP client ID
+branch = "projects/<project-id>/branches/<branch-id>"
+sp_client_id = "<sp-client-id>"
+
+w.postgres.create_role(
+    parent=branch,
+    role=Role(
+        spec=RoleRoleSpec(
+            postgres_role=sp_client_id,
+            auth_method=RoleAuthMethod.LAKEBASE_OAUTH_V1,
+            identity_type=RoleIdentityType.SERVICE_PRINCIPAL,
+        )
+    ),
+).wait()
 ```
 
-To find your app's service principal ID:
+Or via CLI:
+
 ```bash
-databricks apps get my-builder-app --output json | jq '.service_principal_id'
+databricks postgres create-role \
+  "projects/<project-id>/branches/<branch-id>" \
+  --json '{
+    "spec": {
+      "postgres_role": "<sp-client-id>",
+      "auth_method": "LAKEBASE_OAUTH_V1",
+      "identity_type": "SERVICE_PRINCIPAL"
+    }
+  }'
+```
+
+**Provisioned Lakebase**: This step is not needed — adding the instance as an app resource
+(Step 4) automatically configures authentication.
+
+##### Step 7c: Grant PostgreSQL permissions
+
+Connect to your Lakebase database as your own user (via psql or a notebook) and run:
+
+```sql
+-- Replace <sp-client-id> with the service_principal_client_id
+
+-- 1. Allow the SP to create the builder_app schema
+GRANT CREATE ON DATABASE databricks_postgres TO "<sp-client-id>";
+
+-- 2. Create the schema and grant full access
+CREATE SCHEMA IF NOT EXISTS builder_app;
+GRANT USAGE ON SCHEMA builder_app TO "<sp-client-id>";
+GRANT ALL PRIVILEGES ON SCHEMA builder_app TO "<sp-client-id>";
+
+-- 3. Grant access to any existing tables/sequences (needed if you ran migrations locally)
+GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA builder_app TO "<sp-client-id>";
+GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA builder_app TO "<sp-client-id>";
+
+-- 4. Ensure the SP has access to future tables/sequences created by other users
+ALTER DEFAULT PRIVILEGES IN SCHEMA builder_app
+  GRANT ALL ON TABLES TO "<sp-client-id>";
+ALTER DEFAULT PRIVILEGES IN SCHEMA builder_app
+  GRANT ALL ON SEQUENCES TO "<sp-client-id>";
 ```
 
+After granting permissions, redeploy the app so it can run migrations with the new role.
+
 #### 8. Access Your App
 
 After successful deployment, the script will display your app URL:
@@ -706,33 +777,29 @@ Skills are copied from the sibling `databricks-skills/` directory. Ensure:
 2. The skill name in `ENABLED_SKILLS` matches a directory in `databricks-skills/`
 3. The skill directory contains a `SKILL.md` file
 
-#### "Permission denied for table projects" or Database Errors
+#### "password authentication failed" or "Permission denied for table projects"
 
-When using a shared Lakebase instance, you need to grant the app's service principal permissions on the tables:
+See [Section 7: Grant Database Permissions](#7-grant-database-permissions) for the complete setup.
 
-```bash
-# 1. Get your app's service principal ID
-databricks apps get my-builder-app --output json | python3 -c "import sys, json; print(json.load(sys.stdin)['service_principal_id'])"
-```
-
-2. Connect to your Lakebase instance via psql or a Databricks notebook, then run:
+Common causes:
 
-```sql
--- Replace <service-principal-id> with the ID from step 1
-GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO "<service-principal-id>";
-GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO "<service-principal-id>";
-GRANT USAGE ON SCHEMA public TO "<service-principal-id>";
-ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO "<service-principal-id>";
-ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO "<service-principal-id>";
-```
+| Error | Cause | Fix |
+|-------|-------|-----|
+| `password authentication failed` | Lakebase OAuth role missing or created via SQL instead of API | Create the role via `w.postgres.create_role()` with `LAKEBASE_OAUTH_V1` auth (Step 7b) |
+| `permission denied for table` | SP lacks PostgreSQL grants on schema/tables | Run the GRANT statements (Step 7c) |
+| `schema "builder_app" does not exist` | SP lacks `CREATE` on the database | `GRANT CREATE ON DATABASE databricks_postgres TO "<sp-client-id>"` |
+| `relation does not exist` | Migrations haven't run | Redeploy the app, or run `alembic upgrade head` locally |
 
-Alternatively, if you have a fresh/private Lakebase instance, the app's migrations will create the tables with proper ownership automatically.
+> **Autoscale Lakebase pitfall**: Do NOT use `CREATE ROLE ... LOGIN` in PostgreSQL directly.
+> Lakebase Autoscaling requires roles to be created through the Databricks API so that OAuth
+> token authentication works. Manually created roles get `NO_LOGIN` auth and will fail with
+> "password authentication failed".
 
 #### App shows blank page or "Not Found"
 
 Check the app logs in Databricks:
 ```bash
-databricks apps get-logs my-builder-app
+databricks apps logs my-builder-app
 ```
 
 Common causes:

databricks-builder-app/alembic/env.py

@@ -35,9 +35,10 @@
 def get_url_and_connect_args():
   """Get database URL and connect_args from environment.
 
-  Supports two modes:
+  Supports three modes:
   1. Static URL: Uses LAKEBASE_PG_URL directly
-  2. Dynamic OAuth: Builds URL from LAKEBASE_INSTANCE_NAME + generates token
+  2. Autoscale OAuth: Builds URL from LAKEBASE_ENDPOINT + generates token via client.postgres
+  3. Provisioned OAuth: Builds URL from LAKEBASE_INSTANCE_NAME + generates token via client.database
 
   Returns tuple of (url, connect_args) for psycopg2 driver.
   """

databricks-builder-app/app.yaml.example

@@ -28,29 +28,35 @@ env:
   # =============================================================================
   # Skills Configuration
   # =============================================================================
-  # Comma-separated list of skills to enable
+  # Comma-separated list of skills to enable.
+  # AUTO-POPULATED by deploy.sh at deploy time based on installed skills.
+  # To override, set a specific list here before deploying.
   - name: ENABLED_SKILLS
-    value: "databricks-bundles,databricks-agent-bricks,databricks-aibi-dashboards,databricks-app-apx,databricks-app-python,databricks-config,databricks-docs,databricks-jobs,databricks-python-sdk,databricks-unity-catalog,databricks-mlflow-evaluation,databricks-spark-declarative-pipelines,databricks-synthetic-data-gen,databricks-unstructured-pdf-generation"
+    value: ""
   - name: SKILLS_ONLY_MODE
     value: "false"
 
   # =============================================================================
   # Database Configuration (Lakebase)
   # =============================================================================
-  # IMPORTANT: You must add Lakebase as an app resource for database connectivity.
-  # 
-  # Steps:
-  #   1. Create a Lakebase instance in your workspace (if not exists)
-  #   2. Add it as an app resource:
-  #      databricks apps add-resource <app-name> \
-  #        --resource-type database \
-  #        --resource-name lakebase \
-  #        --database-instance <your-lakebase-instance-name>
+  # Choose ONE of the two options below.
   #
-  # When added as a resource, Databricks automatically sets:
-  #   - PGHOST, PGPORT, PGUSER, PGPASSWORD, PGDATABASE
+  # --- Option A: Autoscale Lakebase (recommended) ---
+  # Scales to zero when idle. No add-resource step needed — connects via OAuth.
+  # Find endpoint name in: Catalog → Lakebase → your project → Branches → Endpoints
+  #
+  # - name: LAKEBASE_ENDPOINT
+  #   value: "projects/<project-name>/branches/production/endpoints/<endpoint>"
+  # - name: LAKEBASE_DATABASE_NAME
+  #   value: "databricks_postgres"
+  #
+  # --- Option B: Provisioned Lakebase ---
+  # Fixed-capacity instance. Must add as an app resource:
+  #   databricks apps add-resource <app-name> \
+  #     --resource-type database \
+  #     --resource-name lakebase \
+  #     --database-instance <your-lakebase-instance-name>
   #
-  # You only need to specify the instance name for OAuth token generation:
   - name: LAKEBASE_INSTANCE_NAME
     value: "<your-lakebase-instance-name>"
   - name: LAKEBASE_DATABASE_NAME