Update agent skills to 5443705509e5ac693d61a35ba974cb7f68f444c6 (#206)

Automated update of the agent skills content.

Co-authored-by: databricks-ci-ghec-2[bot] <184307802+databricks-ci-ghec-2[bot]@users.noreply.github.com>

Author: databricks-ci-ghec-2[bot]

.gen.json

@@ -1,3 +1,3 @@
 {
-  "source_commit": "0b79be80b388e6f3bac84bdc719bac68c5204f1e"
+  "source_commit": "5443705509e5ac693d61a35ba974cb7f68f444c6"
 }

skills/databricks-ai-functions/references/1-task-functions.md

@@ -93,7 +93,7 @@ df.withColumn(
     }
     ```
   - Supported types: `string`, `integer`, `number`, `boolean`, `enum`
-  - Max 128 fields, 7 nesting levels, 500 enum values
+  - Max 256 fields, 12 nesting levels, 500 enum values
 - `options`: optional MAP\<STRING, STRING\>:
   - `instructions`: task context to improve extraction quality (max 20,000 chars)
 

skills/databricks-ai-functions/references/4-document-processing-pipeline.md

@@ -11,11 +11,11 @@ When processing documents with AI Functions, apply this order of preference for
 | Stage | Preferred function | Use `ai_query` when... |
 |---|---|---|
 | Parse binary docs (PDF, DOCX, images) | `ai_parse_document` | Need image-level reasoning |
-| Extract fields from text (flat or nested) | `ai_extract` | Schema exceeds 128 fields or 7 nesting levels |
+| Extract fields from text (flat or nested) | `ai_extract` | Schema exceeds 256 fields or 12 nesting levels |
 | Classify document type or status | `ai_classify` | More than 20 categories |
 | Score item similarity / matching | `ai_similarity` | Need cross-document reasoning |
 | Summarize long sections | `ai_summarize` | — |
-| Extract deeply nested JSON | `ai_query` with `responseFormat` | Schema exceeds `ai_extract` limits (128 fields, 7 levels) |
+| Extract deeply nested JSON | `ai_query` with `responseFormat` | Schema exceeds `ai_extract` limits (256 fields, 12 levels) |
 
 ---
 
@@ -173,7 +173,7 @@ def extracted_flat():
 
 
 # ── Stage 3b: Nested JSON extraction (last resort: ai_query) ─────────────────
-# Use ai_query only for deeply nested schemas that exceed ai_extract's 7-level limit
+# Use ai_query only for deeply nested schemas that exceed ai_extract's 12-level limit
 
 @dlt.table(comment="Nested line items extracted — ai_query used for array schema only")
 def extracted_line_items():
@@ -496,7 +496,7 @@ with mlflow.start_run():
 ## Tips
 
 1. **Parse first, enrich second** — always run `ai_parse_document` as the first stage. Feed its text output to task-specific functions; never pass raw binary to `ai_query`.
-2. **Flat or nested fields → `ai_extract`; deeply nested JSON exceeding 7 levels → `ai_query`** — pass `MAP('version', '2.0')` and access results through `:response`.
+2. **Flat or nested fields → `ai_extract`; deeply nested JSON exceeding 12 levels → `ai_query`** — pass `MAP('version', '2.0')` and access results through `:response`.
 3. **`failOnError => false` is mandatory in batch** — write errors to a sidecar `_errors` table rather than crashing the pipeline.
 4. **Truncate before sending to `ai_query`** — use `LEFT(text, 6000)` or chunk long documents to stay within context window limits.
 5. **Prompts belong in `config.yml`** — never hardcode prompt strings in pipeline code. A prompt change should be a config change, not a code change.

skills/databricks-aibi-dashboards/SKILL.md

@@ -90,15 +90,18 @@ Sample rows alone don't tell you what to build. you can write aggregate SQL thro
 - **Trend viability** at daily/weekly/monthly grain → picks the right trend granularity.
 - **Story confirmation** — run the aggregations you plan to put in the dashboard and check they're not flat, empty, or uninteresting. Fix the query or adjust the story before moving on.
 
-Fan out independent probes (state ∈ `PENDING|RUNNING|SUCCEEDED|FAILED|CANCELED|CLOSED`):
+Fan out independent probes in one call — pass several positional SQLs (and/or repeated `--file`) and they run in parallel (default `--concurrency 8`):
 
 ```bash
-submit() { databricks api post /api/2.0/sql/statements --json "$(jq -nc --arg w "$1" --arg s "$2" '{warehouse_id:$w,statement:$s,wait_timeout:"0s",on_wait_timeout:"CONTINUE"}')" | jq -r .statement_id; }
-SIDS=(); for q in "$@"; do SIDS+=( "$(submit "$WH" "$q")" ); done
-for s in "${SIDS[@]}"; do databricks api get "/api/2.0/sql/statements/$s" | jq '{state:.status.state, rows:.result.data_array}'; done
-# cancel: databricks api post "/api/2.0/sql/statements/$SID/cancel"
+DATABRICKS_WAREHOUSE_ID=<WH> databricks experimental aitools tools query --output json \
+  "SELECT COUNT(*) FROM catalog.schema.orders" \
+  "SELECT region, COUNT(*) FROM catalog.schema.orders GROUP BY region ORDER BY 2 DESC LIMIT 10" \
+  "SELECT MIN(ts), MAX(ts) FROM catalog.schema.orders"
 ```
 
+- **`--output json` is mandatory** in multi-query mode. Returns one object per statement: `{sql, state, rows, error}`; failures are per-statement (`state: "FAILED"`), others still succeed.
+- ⚠️ **Don't trust the exit code** (a failed statement can still exit `0`) — gate on each object's `state != "SUCCEEDED"`.
+
 > **Dashboard queries are different** — inside the dashboard JSON, the `FROM` clause must reference ONLY the table name, with no catalog or schema prefix:
 > - ✅ Correct: `FROM trips`
 > - ❌ Wrong: `FROM nyctaxi.trips`
@@ -120,7 +123,9 @@ If values don't match expectations, ensure the query is correct, fix the data if
 
 Before writing JSON, plan your dashboard:
 
-1. You must know the expected specific JSON structure. For this, **Read reference files**: [1-widget-specifications.md](references/1-widget-specifications.md), [3-filters.md](references/3-filters.md), [4-examples.md](references/4-examples.md)
+1. You must know the expected specific JSON structure. For this, **Read reference files**: [1-widget-specifications.md](references/1-widget-specifications.md), [3-filters.md](references/3-filters.md).
+
+Always make sure you read an entire example to understand the structure, like [4-examples.md](references/4-examples.md).
 
 2. Think: **What widgets?** Map each visualization to a dataset:
    | Widget | Type | Dataset | Has filter field? |
@@ -192,6 +197,8 @@ databricks workspace-entity-tag-assignments create-tag-assignment \
 
 Every dashboard's `serialized_dashboard` content must follow this exact structure:
 
+Important: ALWAYS add a space or `\n` at the end of each `queryLines` value as they are concatenated to create the dataset.
+
 ```json
 {
   "datasets": [
@@ -215,7 +222,7 @@ Every dashboard's `serialized_dashboard` content must follow this exact structur
 ```
 
 **Structural rules (violations cause "failed to parse serialized dashboard"):**
-- `queryLines`: Array of strings, NOT `"query": "string"`. Elements are **joined verbatim** with no separator — end each line with `\n` (or strip `-- comments`). A line ending in `-- comment` with no newline swallows the next line.
+- `queryLines`: Array of strings, NOT `"query": "string"`. Elements are **joined verbatim** with no separator — end each line with ` ` or `\n` (or strip `-- comments`). A line ending in `-- comment` with no newline swallows the next line.
 - Widgets: INLINE in `layout[].widget`, NOT a separate `"widgets"` array
 - `pageType`: Required on every page (`PAGE_TYPE_CANVAS` or `PAGE_TYPE_GLOBAL_FILTERS`)
 - Query binding: `query.fields[].name` must exactly match `encodings.*.fieldName`

skills/databricks-ml-training/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: databricks-ml-training
-description: "Train ML or custom-agent models on Databricks with MLflow tracking and Unity Catalog registration. Use when asked to: train classification/regression or deep-learning models (XGBoost, scikit-learn, LightGBM, PyTorch, etc.) with Optuna tuning, register to UC and promote `@prod`/`@challenger` aliases, batch-score via `spark_udf`, build custom PyFunc models, author a custom `ResponsesAgent` with UC Function/Vector Search tools, or submit a training notebook as a serverless one-time job. NOT for: endpoint ops (databricks-model-serving), Knowledge Assistants/Supervisor Agents (databricks-agent-bricks), MLflow evaluation (databricks-mlflow-evaluation)."
+description: "Train ML models on Databricks. Use for: classification/regression/deep-learning (XGBoost, scikit-learn, LightGBM, PyTorch) with Optuna, @prod/@challenger aliases, batch scoring (spark_udf for plain models, fe.score_batch for feature-store-backed), custom PyFunc, custom ResponsesAgent (LangGraph + UC Function/Vector Search); UC feature tables + FeatureLookup + point-in-time joins + Lakebase online store; declarative Feature Views (create_feature, DeltaTableSource, RollingWindow/SlidingWindow/TumblingWindow, materialize_features, streaming Kafka features). NOT for: endpoint ops (databricks-model-serving), MLflow evaluation (databricks-mlflow-evaluation)."
 compatibility: Requires databricks CLI (>= v0.294.0)
 metadata:
   version: "0.1.0"
@@ -19,7 +19,7 @@ If you need to deploy a real time model serving endpoint **after** the model is
 
 | Consumption | When | How |
 |---|---|---|
-| **Batch UDF** | Dashboards, daily/hourly scores, predictions read by Genie/Dashboards or an app (often synced to a Lakebase table) | `mlflow.pyfunc.spark_udf(...)` → `INSERT INTO gold_predictions` |
+| **Batch UDF** | Dashboards, daily/hourly scores, predictions read by Genie/Dashboards or an app (often synced to a Lakebase table) | `mlflow.pyfunc.spark_udf(...)` → `INSERT INTO gold_predictions`. **If the model was logged with `fe.log_model(training_set=...)`, use `fe.score_batch()` instead** — see the [Feature Engineering](#feature-engineering-feature-store--feature-views) section below. |
 | **Real-time endpoint** | Score on a user action (fraud at authorization, rec at page load) — sub-100ms | `mlflow.deployments.get_deploy_client()` (classical) / `agents.deploy()` (agents). Endpoint lifecycle: see [databricks-model-serving](../databricks-model-serving/SKILL.md). |
 
 ## Default Canonical flow
@@ -36,6 +36,8 @@ silver_<features>  +  silver_<labels>
 gold_<entity>_predictions   ◄── dashboards, apps, Genie read this
 ```
 
+> **Feature-store-backed models diverge here.** If training used `fe.log_model(training_set=...)`, replace `spark_udf(@prod)` with `fe.score_batch(model_uri, df=<keys_only>)` — it auto-joins features via the model's registered feature lineage. See the [Feature Engineering](#feature-engineering-feature-store--feature-views) section below.
+
 One notebook, one artifact. Re-running = retraining. Gold is where truth lives — read paths never call the model directly. Keep label-window logic (`failure occurred within 7 days`) in the notebook during dev; once stable, promote to a silver materialized view in SDP.
 
 ---
@@ -125,7 +127,7 @@ client.set_registered_model_alias(FULL_NAME, "prod", info.registered_model_versi
 
 ## Consume: batch scoring over Delta
 
-The cheap, default path. Load the registered model as a Spark UDF and score a Delta table; write predictions to a gold table that downstream consumers read.
+The cheap, default path **for models NOT backed by feature tables**. Load the registered model as a Spark UDF and score a Delta table; write predictions to a gold table that downstream consumers read. **For feature-store-backed models** (logged with `fe.log_model(training_set=...)`), skip `spark_udf` entirely and use `fe.score_batch()` — see the [Feature Engineering](#feature-engineering-feature-store--feature-views) section.
 
 ```python
 # COMMAND ----------
@@ -234,6 +236,19 @@ Prefer no-code authoring via [databricks-agent-bricks](../databricks-agent-brick
 
 ---
 
+## Feature Engineering: Feature Store & Feature Views
+
+When to reach for Feature Engineering (either flavor) instead of a plain Delta table: when the same feature must be computed identically at training and serving time (no training/serving skew), the feature is time-dependent and needs point-in-time joins against labels (no future leakage), the feature needs to be served with <10ms latency via an online store, or the feature is shared across models with lineage tracked in UC. If none apply, a plain UC table is enough — the sections below can be skipped.
+
+**Default: don't use Feature Engineering unless one of the reasons above clearly applies or the user explicitly asked for it.** It adds build time and complexity (an extra Delta table layer for `FeatureLookup`; a materialization pipeline for Feature Views). If you're unsure, use plain UC tables and add Feature Engineering later when a concrete need surfaces.
+
+Two flavors, one train/score path (`create_training_set` → `fe.log_model` → `fe.score_batch`):
+
+- **`FeatureLookup` API** — you own the feature tables (compute, write, refresh); bind them to a training set via `FeatureLookup` + `FeatureEngineeringClient`. Reach for it when the features already exist as Delta tables or you want direct control over how they are computed. See **[references/feature-store.md](references/feature-store.md)**.
+- **Feature Views** (declarative, Public Preview, `databricks-feature-engineering>=0.16.0`) — Databricks owns compute, materialization (Delta offline + Lakebase online), and refresh from a spec you declare (`create_feature` over `DeltaTableSource`; `RollingWindow`/`SlidingWindow`/`TumblingWindow`; `create_stream` for Kafka features). Reach for it when the feature is a formula you want Databricks to keep fresh. See **[references/feature-views.md](references/feature-views.md)**.
+
+---
+
 ## Gotchas (the ones that cost time)
 
 | Trap | Fix |
@@ -255,6 +270,8 @@ Endpoint-lifecycle gotchas (readiness two-state, version-swap, Serving-UI SP fil
 |---|---|
 | [references/custom-pyfunc.md](references/custom-pyfunc.md) | Single end-to-end custom pyfunc example: artifacts, signature, code_paths, log → register → deploy → query. |
 | [references/genai-agents.md](references/genai-agents.md) | Custom LangGraph `ResponsesAgent` with UC Function + Vector Search tools. `create_text_output_item` gotcha and the `resources=[...]` passthrough-auth list. For no-code agents prefer **databricks-agent-bricks**. |
+| [references/feature-store.md](references/feature-store.md) | Feature Engineering in Unity Catalog (standard `FeatureLookup` API): `FeatureEngineeringClient`, `create_table` with `timeseries_column`, `FeatureLookup` with point-in-time joins, `fe.log_model` with lineage, `fe.score_batch` (replaces `spark_udf` for FE models), and Lakebase online store via `publish_table`. Requires `databricks-feature-engineering>=0.16.0`. |
+| [references/feature-views.md](references/feature-views.md) | Feature Views (declarative Feature Engineering, Public Preview): `create_feature` over `DeltaTableSource`, `RollingWindow`/`SlidingWindow`/`TumblingWindow` aggregations, `materialize_features` (offline + online), streaming features off Kafka via `create_stream`, point-in-time training sets, and the Feature Serving Endpoint. Requires `databricks-feature-engineering>=0.16.0`. |
 
 ## Related skills