Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions skills/databricks-app-resources/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
---
name: databricks-app-resources
description: "Attach, update, and reason about resources on a Databricks App — apps create-update, update_mask/merge semantics, and app-level vs space-level resources. Use when connecting a resource (Lakebase/Postgres, SQL warehouse, serving endpoint, secret, UC volume, Genie space) to an app or changing an app's resources."
compatibility: Requires databricks CLI (>= v0.294.0)
metadata:
version: "0.1.0"
parent: databricks-apps
---

# Databricks App Resources

How a Databricks App gets access to resources, and how to attach or change them. For a resource type's own fields, load that type's skill (e.g. **`databricks-lakebase`** for Lakebase/Postgres).

## Resource levels: app vs space

A resource is usable by an app only once it's **attached**, at one of two levels:

- **App-level** — scoped to this app's service principal; **changeable from here** (with consent).
- **Space-level** — inherited from the app's space, **shared by every app in the space**; change it in the Databricks UI only.

Which levels a type supports is fixed by the type (a type may support one or both); the app's effective access is the **union**. To tell a resource's level, check `databricks apps get <APP_NAME>`:

- listed under `resources` ⇒ **app-level**
- appears only under `effective_resources` (the app+space union) but **not** under `resources` ⇒ **space-level**

Don't assume space-level and refuse without checking.

## Updating an app's resources: `apps create-update`

Always use **`databricks apps create-update`** to change an app's resources or config — for **every** app. The older `databricks apps update` is **legacy** and should not be used (it can't change resources for an app in a space). Pass everything in `--json` (only `APP_NAME` is positional); the body is `{"update_mask": "...", "app": {...}}`:

```bash
databricks apps create-update <APP_NAME> --json @update.json --profile <PROFILE> # waits; --no-wait to return early
```

⚠️ **`update_mask` scopes the change to the fields you list, and each listed field is replaced wholesale (not item-merged).** With `update_mask=resources` the entire `resources` array is replaced — read the app's current resources and **merge** your new entry in, or you'll detach the rest. Fields you don't list (e.g. `user_api_scopes`) are left untouched.

```bash
# read the current resources first, then submit the full (merged) array
databricks apps get <APP_NAME> --profile <PROFILE>
```

Body shape (one entry per attached resource; the typed key + fields depend on the resource type):

```json
{
"update_mask": "resources",
"app": {
"resources": [
{ "name": "<key>", "<resource_type>": { "...type-specific fields...": "", "permission": "<PERMISSION>" } }
]
}
}
```

## Wiring a resource at scaffold time

To create a **new** app already wired to a resource, scaffold with the resource's feature and `--set` keys:

```bash
databricks apps init --name <APP_NAME> --features <feature> \
--set "<key>=<value>" ... --run none --profile <PROFILE>
```

The feature name, the `--set` keys, and how to obtain the identifier values are resource-specific — get them from that resource's skill (for Lakebase, see **`databricks-lakebase`**).

## Resources on a deployed app: ownership & deploy-first

A deployed app runs as its **service principal**, not as you. The SP can only use objects it has been granted or that it **created itself** — so when an app creates objects inside a resource (e.g. a database schema), the SP must own them.

- **Deploy before running locally.** Deploy first so the SP creates and owns its objects; running locally as yourself first can leave objects user-owned, which the SP then can't use — the #1 source of permission errors.
- If an object ends up user-owned, the fix (grant vs. recreate) is resource-specific — see that resource's skill (for Lakebase schemas, **`databricks-lakebase`**).

## Resource-type fields

The typed sub-object differs per resource type. For **Lakebase/Postgres** — the `postgres` resource key, `branch`/`database` paths, scaffold `--set` keys, and schema ownership — see the **`databricks-lakebase`** skill.

## Consent

Never create, modify, or detach a resource without the user's explicit go-ahead. `create-update` runs as the user, so it fails if they lack permission — then give step-by-step Databricks UI instructions instead.
81 changes: 16 additions & 65 deletions skills/databricks-lakebase/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -174,85 +174,36 @@ databricks postgres reset-branch projects/<PROJECT_ID>/branches/<BRANCH_ID> --pr

**Migration:** Manual via `pg_dump`/`pg_restore` (requires pausing writes). Automatic seamless upgrades (seconds of downtime) begin June 2026 -- no customer action required.

## What's Next
## Using Lakebase as an App Resource

### Build a Databricks App
For the **app-side flows** — scaffolding an app wired to a resource, attaching to an existing app via `apps create-update`, and the deployed-app service-principal ownership / deploy-first model — use the **`databricks-app-resources`** skill. This section holds only the Lakebase-specific details those flows pull.

After creating a project, scaffold a connected Databricks App:
**Scaffold keys** (`databricks apps init`): `--features lakebase --set "lakebase.postgres.branch=<BRANCH_NAME>" --set "lakebase.postgres.database=<DATABASE_NAME>"`. Get the names from `list-branches` / `list-databases`.

```bash
# 1. Get branch name
databricks postgres list-branches projects/<PROJECT_ID> --profile <PROFILE>

# 2. Get database name
databricks postgres list-databases projects/<PROJECT_ID>/branches/<BRANCH_ID> --profile <PROFILE>

# 3. Scaffold with lakebase feature
databricks apps init --name <APP_NAME> --features lakebase \
--set "lakebase.postgres.branch=<BRANCH_NAME>" \
--set "lakebase.postgres.database=<DATABASE_NAME>" \
--run none --profile <PROFILE>
```

For the full app workflow, use the **`databricks-apps`** skill.

### Attach Lakebase to an existing app

`apps init --features lakebase` (above) wires the database at scaffold time. To
attach a project to an **existing** app, update its resources.

**Use the `postgres` resource key for an Autoscaling project** — its fields are
`branch` + `database` (full resource paths from the table above). The legacy
`database` key (`instance_name` + `database_name`) is for **Provisioned**
instances only; using it for an Autoscaling project fails with
`Database instance <name> does not exist`. Get the exact paths from
`list-branches` / `list-databases` (the DB name is often hyphenated, e.g.
`databricks-postgres`).

Update the app's resources with **`databricks apps create-update`** — the method to use for any app (the older `databricks apps update` is legacy and can't change resources for an app in a space). `update_mask=resources` replaces the whole `resources` array, so read the app's current resources and **merge** the new one in (or you'll detach the rest). Pass everything in `--json`; only `APP_NAME` is positional:

```bash
databricks apps create-update <APP_NAME> --json @update.json --profile <PROFILE> # waits for completion; --no-wait to return early
```
**Resource key** (attaching / `app.yaml`): use the **`postgres`** key for an Autoscaling project — fields `branch` + `database` (full resource paths). The legacy `database` key (`instance_name` + `database_name`) is **Provisioned-only**; using it for an Autoscaling project fails with `Database instance <name> does not exist` (DB name is often hyphenated, e.g. `databricks-postgres`). The `resources[]` entry:

```json
{
"update_mask": "resources",
"app": {
"resources": [
{
"name": "postgres",
"postgres": {
"branch": "projects/<PROJECT_ID>/branches/<BRANCH_ID>",
"database": "projects/<PROJECT_ID>/branches/<BRANCH_ID>/databases/<DATABASE_ID>",
"permission": "CAN_CONNECT_AND_CREATE"
}
}
]
"name": "postgres",
"postgres": {
"branch": "projects/<PROJECT_ID>/branches/<BRANCH_ID>",
"database": "projects/<PROJECT_ID>/branches/<BRANCH_ID>/databases/<DATABASE_ID>",
"permission": "CAN_CONNECT_AND_CREATE"
}
}
```

**Confirm the branch/database with the user — don't default to `production` silently.** The app's service principal must be able to create and **own** the schema(s) it uses there, so avoid a branch/database where those schema names are already owned by a user (the SP will hit `permission denied … 42501`) — a fresh/dedicated branch, or a new app-owned schema, is cleanest. See **Schema Permissions for Deployed Apps** below for the full ownership model.

### Schema Permissions for Deployed Apps

The app's Service Principal has `CAN_CONNECT_AND_CREATE` -- it can create new objects but **cannot access existing schemas**. The SP must create the schema to become its owner.

**ALWAYS deploy the app before running it locally.** This is the #1 source of Lakebase permission errors.
Confirm the branch/database with the user — don't default to `production` silently.

**Correct workflow:**
1. **Deploy first**: `databricks apps deploy <APP_NAME> --profile <PROFILE>`
2. **Grant local access** *(if needed)*: assign `databricks_superuser` via UI (project creators already have access)
3. **Develop locally**: your credentials get DML access to SP-owned schemas
**Suggest a dev branch off `production`, then ask one focused question — not a plan.** Recommend that iteration run on a copy-on-write **branch off `production`** rather than on `production` directly, give the one-line reason (`production` is what the deployed app reads and writes, so a branch keeps your dev changes off its live data while deploy still targets `production`), and ask a single question: go the recommended route, or point somewhere else? Wait for the go-ahead — never create the branch silently. Keep it to the isolation choice only: don't recite the mechanics below, walk through a multi-step plan, or fold in an unrelated build plan.

**If you already ran locally first** and hit `permission denied`: the schema is owned by your credentials, not the SP. **Do NOT drop the schema without asking the user** -- dropping it deletes all data.
**On go-ahead, do these in order — silently; they're mechanics, not steps to narrate:** attach the postgres resource to the app on `production` (this provisions the app's **service-principal Postgres role** there) → `create-branch` off `production` with a TTL and a session-unique name (copy-on-write, so it inherits the SP role — *only because the attach ran first*) → point your dev iteration at the branch instead of `production`. Forking before attaching leaves the branch with no SP role and the app fails with `28P01 password authentication failed`; **never hand-create SP roles or `GRANT` privileges to patch that** — needing to is the signal the order was wrong, so redo it attach-first.

Ask the user to choose:
- **(A) Drop and redeploy:** `databricks psql --project <PROJECT_ID> -- -c "DROP SCHEMA IF EXISTS <SCHEMA_NAME> CASCADE;"`, then `databricks apps deploy` from the app directory. The SP recreates the schema on startup.
- **(B) Export first, then drop and redeploy:** export via `pg_dump` (use connection details from `databricks postgres get-endpoint`; see **Other Workflows** below for HOST and TOKEN) or copy tables to a temp schema using `databricks psql --project <PROJECT_ID>`, then do option A. After the SP recreates the schema on redeploy, restore with `pg_restore` or re-INSERT from the temp schema.
**Schema ownership:** the app's SP has `CAN_CONNECT_AND_CREATE` — it can create new objects but **cannot use a schema owned by someone else**, so it must create (and thus own) its schema. Deploy first so the SP owns it (for local dev as yourself, assign `databricks_superuser` in the UI). If a schema is already user-owned (you ran locally first), the SP hits `permission denied … 42501`; recreate it under the SP (**ask first — destructive**):
- **Drop and redeploy:** `databricks psql --project <PROJECT_ID> -- -c "DROP SCHEMA IF EXISTS <SCHEMA_NAME> CASCADE;"`, then `databricks apps deploy` the SP recreates the schema on startup.
- **Export first** if you need the data: `pg_dump` (connection details from `get-endpoint`; see **Other Workflows**) or copy to a temp schema via `databricks psql`, then drop + redeploy + restore.

### Other Workflows
## Other Workflows

```bash
# Connect a Postgres client -- get connection string
Expand Down
Loading