refactor(serverless-storage-check): nest as sub-skill under databricks-serverless-migration

Moves skills/databricks-serverless-storage-check/ →
skills/databricks-serverless-migration/databricks-serverless-storage-check/
to physically reflect the parent/sub-skill hierarchy.

- Update scripts/skills.py iter_skill_dirs() to yield one level of nested
  skill directories (sub-skills alongside top-level skills)
- Fix relative SKILL.md links: ../databricks-serverless-migration/ → ../
  and ../databricks-{dabs,jobs,core}/ → ../../databricks-{dabs,jobs,core}/
- Fix databricks-serverless-migration/SKILL.md links:
  ../databricks-serverless-storage-check/ → databricks-serverless-storage-check/
- Regenerate manifest.json (skill count unchanged: 9)
- Confirmed: python3 scripts/skills.py validate → Everything is up to date
- Confirmed: stf lint → 0 errors, 0 warnings

Co-authored-by: Isaac

Author: GABRIELLE DOMPREH

.claude-plugin/plugin.json

@@ -20,6 +20,7 @@
     "pipelines",
     "dabs",
     "serverless",
+    "serverless-storage-check",
     "vector-search",
     "data-engineering"
   ],

manifest.json

@@ -390,6 +390,23 @@
         "agents/openai.yaml",
         "assets/databricks.png",
         "assets/databricks.svg",
+        "databricks-serverless-storage-check/SKILL.md",
+        "databricks-serverless-storage-check/agents/openai.yaml",
+        "databricks-serverless-storage-check/assets/databricks.png",
+        "databricks-serverless-storage-check/assets/databricks.svg",
+        "databricks-serverless-storage-check/eval/evaluation_results.json",
+        "databricks-serverless-storage-check/eval/ground_truth.yaml",
+        "databricks-serverless-storage-check/eval/manifest.yaml",
+        "databricks-serverless-storage-check/eval/output_instructions.md",
+        "databricks-serverless-storage-check/eval/report.html",
+        "databricks-serverless-storage-check/eval/tests/conftest.py",
+        "databricks-serverless-storage-check/eval/tests/test_integration.py",
+        "databricks-serverless-storage-check/eval/tests/test_unit.py",
+        "databricks-serverless-storage-check/eval/thinking_instructions.md",
+        "databricks-serverless-storage-check/references/pattern-catalog.md",
+        "databricks-serverless-storage-check/references/remediation-guide.md",
+        "databricks-serverless-storage-check/scripts/preflight.py",
+        "databricks-serverless-storage-check/scripts/test_preflight.py",
         "references/code-patterns.md",
         "references/compatibility-checks.md",
         "references/configuration-guide.md",
@@ -411,13 +428,22 @@
         "agents/openai.yaml",
         "assets/databricks.png",
         "assets/databricks.svg",
+        "eval/evaluation_results.json",
+        "eval/ground_truth.yaml",
+        "eval/manifest.yaml",
+        "eval/output_instructions.md",
+        "eval/report.html",
+        "eval/tests/conftest.py",
+        "eval/tests/test_integration.py",
+        "eval/tests/test_unit.py",
+        "eval/thinking_instructions.md",
         "references/pattern-catalog.md",
         "references/remediation-guide.md",
         "scripts/preflight.py",
         "scripts/test_preflight.py"
       ],
       "repo_dir": "skills",
-      "version": "0.1.0"
+      "version": "0.2.0"
     },
     "databricks-spark-structured-streaming": {
       "description": "Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.",

scripts/skills.py

@@ -40,7 +40,11 @@
 
 
 def iter_skill_dirs(repo_root: Path, parent: str = STABLE_REPO_DIR):
-    """Yield skill directories under `parent` that contain SKILL.md."""
+    """Yield skill directories under `parent` that contain SKILL.md.
+
+    For the stable skills dir, also yields one level of nested sub-skill
+    directories (e.g. skills/databricks-serverless-migration/databricks-serverless-storage-check/).
+    """
     skills_dir = repo_root / parent
     if not skills_dir.exists():
         return
@@ -52,6 +56,15 @@ def iter_skill_dirs(repo_root: Path, parent: str = STABLE_REPO_DIR):
         if not (item / "SKILL.md").exists():
             continue
         yield item
+        if parent == STABLE_REPO_DIR:
+            for subitem in sorted(item.iterdir()):
+                if not subitem.is_dir():
+                    continue
+                if subitem.name.startswith("."):
+                    continue
+                if not (subitem / "SKILL.md").exists():
+                    continue
+                yield subitem
 
 
 def iter_experimental_skill_dirs(repo_root: Path):

skills/databricks-serverless-migration/SKILL.md

@@ -13,6 +13,16 @@ parent: databricks-core
 
 Analyze existing Databricks code for serverless compute compatibility and guide migration from classic clusters. The skill follows a 4-step migration lifecycle: **Ingest** the workload → **Analyze** for compatibility → **Test** via A/B comparison → **Validate** and iterate.
 
+## Sub-skills
+
+This skill is the top of a small hierarchy. Once a single-notebook migration is done, hand off to the niche sub-skill below for multi-task hardening before deploying to serverless:
+
+| Sub-skill | Use when |
+|-----------|----------|
+| [`databricks-serverless-storage-check`](databricks-serverless-storage-check/SKILL.md) | The job has **multiple tasks** (parent/child notebooks via `dbutils.notebook.run`, sibling `notebook_task`s, or a `pipeline_task` downstream of a `notebook_task`) and shares files between them. Detects the `INTERNAL_ERROR: [Errno 13] Permission denied` / `/local_disk0/.../trustedTemp/...` failure class and rewrites the handoff to UC Volumes / `/Workspace` / `taskValues`. Invoke after this skill's single-notebook port, before the Step-3 test run. |
+
+The sub-skill is a **diagnostic + fix** layer, not a replacement — keep this skill's 4-step lifecycle as the spine and call into the sub-skill only when the multi-task triggers fire.
+
 ## When to Use This Skill
 
 - Migrating notebooks, jobs, or pipelines from classic compute to serverless
@@ -198,7 +208,7 @@ Scan the code for patterns that are incompatible with the serverless compute arc
 | Pattern | Severity | Fix |
 |---------|----------|-----|
 | `dbfs:/` or `/dbfs/` paths (persistent data) | Blocker | Replace with `/Volumes/<your_catalog>/schema/volume/path` |
-| `dbfs:/tmp/`, `/dbfs/tmp/`, paths with `cache`/`scratch`/`temp` | Warning | Use `/tmp/` or `/local_disk0/tmp/` (local driver disk) — do not use Volumes for temp files due to performance. **Per-task scratch only**: if another task (child notebook, sibling job task, or pipeline) needs to read the file, use UC Volumes or `/Workspace` — see [`databricks-serverless-storage-check`](../databricks-serverless-storage-check/SKILL.md). |
+| `dbfs:/tmp/`, `/dbfs/tmp/`, paths with `cache`/`scratch`/`temp` | Warning | Use `/tmp/` or `/local_disk0/tmp/` (local driver disk) — do not use Volumes for temp files due to performance. **Per-task scratch only**: if another task (child notebook, sibling job task, or pipeline) needs to read the file, use UC Volumes or `/Workspace` — see [`databricks-serverless-storage-check`](databricks-serverless-storage-check/SKILL.md). |
 | `file:///dbfs/` FUSE mount paths | Warning | Replace persistent paths with `/Volumes/...`; replace temp paths with `/local_disk0/tmp/` |
 | `dbutils.fs.mount(...)` | Blocker | Create UC external location + external volume |
 | `hive_metastore.db.table` | Warning | Migrate to UC or use HMS Federation: `CREATE FOREIGN CATALOG ... USING CONNECTION hms_connection` |

…bricks-serverless-storage-check/SKILL.md …bricks-serverless-storage-check/SKILL.mdskills/databricks-serverless-storage-check/SKILL.md renamed to skills/databricks-serverless-migration/databricks-serverless-storage-check/SKILL.md skills/databricks-serverless-storage-check/SKILL.md renamed to skills/databricks-serverless-migration/databricks-serverless-storage-check/SKILL.md

@@ -3,13 +3,13 @@ name: databricks-serverless-storage-check
 description: "Detect cross-task file-sharing antipatterns in Databricks serverless jobs (writes to /local_disk0, /tmp, or trustedTemp that are read by sibling or child tasks on potentially different compute nodes) and recommend UC Volumes or /Workspace for handoff. Use when a serverless job fails with `INTERNAL_ERROR: [Errno 13] Permission denied` on /local_disk0 paths, when parallel child notebooks fail intermittently, when reviewing a DAB job before deploying to serverless, or when the user mentions trustedTemp, fan-out, or cross-task file handoff. Complements databricks-serverless-migration (which covers single-notebook migration)."
 compatibility: Requires databricks CLI (>= v0.292.0) for --job-id and --run-id modes; --notebook / --dir / --job-yaml modes have no external dependencies.
 metadata:
-  version: "0.1.0"
-parent: databricks-core
+  version: "0.2.0"
+parent: databricks-serverless-migration
 ---
 
 # Serverless Storage Check
 
-**FIRST**: Use the parent `databricks-core` skill for CLI basics, authentication, and profile selection.
+**FIRST**: Use the parent `databricks-serverless-migration` skill for the overall classic-to-serverless migration workflow (Ingest → Analyze → Test → Validate), CLI auth (which it inherits from `databricks-core`), and per-task scratch guidance (`/local_disk0/tmp`). This skill is a niche sub-skill of that workflow — it focuses **only** on the cross-task handoff antipattern that the parent skill explicitly delegates here (see the Category B → DBFS-paths row in `databricks-serverless-migration/SKILL.md`).
 
 This skill detects a specific class of serverless failure: **cross-task file handoffs through local disk**. On serverless compute, each task may run on a different node, so a path written by a parent task to `/local_disk0`, `/tmp`, or a `trustedTemp` directory is not guaranteed to be visible to a child task. The typical symptom is:
 
@@ -32,14 +32,16 @@ Use this skill when any of these triggers appear:
 - The user mentions "trustedTemp", "fan-out", "cross-task file sharing", or `/local_disk0`
 - A new serverless job design needs a sanity check before first run
 
-This skill is **complementary to**, not a replacement for, [`databricks-serverless-migration`](../databricks-serverless-migration/SKILL.md). That skill handles single-notebook migration and explicitly recommends `/local_disk0/tmp` for per-task scratch — which is correct *inside* a task. The boundary between the two skills:
+This skill is a **niche sub-skill** of [`databricks-serverless-migration`](../SKILL.md) — the parent owns single-notebook migration and explicitly recommends `/local_disk0/tmp` for per-task scratch (which is correct *inside* a task). This sub-skill picks up where the parent leaves off: anything that crosses a task boundary. The boundary between the two skills:
 
 | Concern | Use skill |
 |---------|-----------|
-| Migrating one notebook from classic DBR to serverless | `databricks-serverless-migration` |
-| Per-task scratch storage (intra-task) | `databricks-serverless-migration` (recommends `/local_disk0/tmp`) |
-| **Cross-task file handoff between parent/child notebooks or sibling tasks** | **this skill** |
-| Permission-denied on `/local_disk0` during a multi-task run | **this skill** |
+| Migrating one notebook from classic DBR to serverless | `databricks-serverless-migration` (parent) |
+| Per-task scratch storage (intra-task) | `databricks-serverless-migration` (parent — recommends `/local_disk0/tmp`) |
+| **Cross-task file handoff between parent/child notebooks or sibling tasks** | **this skill (sub-skill)** |
+| Permission-denied on `/local_disk0` during a multi-task run | **this skill (sub-skill)** |
+
+Always invoke the parent skill **first** for the overall migration plan, then return here to harden cross-task handoffs before deploying a multi-task serverless job.
 
 ## Quick start
 
@@ -131,10 +133,10 @@ If the scanner emits `ENV001`:
 
 ## Related skills
 
-- [`databricks-serverless-migration`](../databricks-serverless-migration/SKILL.md) — single-notebook classic-to-serverless migration. **Use that skill first** if the workload hasn't been migrated yet.
-- [`databricks-dabs`](../databricks-dabs/SKILL.md) — DAB structure and resource definitions. Use when authoring or fixing the `job.yml` flagged by `FANOUT003` or `FANOUT004`.
-- [`databricks-jobs`](../databricks-jobs/SKILL.md) — Lakeflow Jobs orchestration. Use when restructuring task dependencies to avoid the fan-out antipattern.
-- [`databricks-core`](../databricks-core/SKILL.md) — parent skill for CLI auth and profile selection.
+- [`databricks-serverless-migration`](../SKILL.md) — **parent skill**. Single-notebook classic-to-serverless migration and the overall Ingest → Analyze → Test → Validate lifecycle. **Always invoke first** if the workload hasn't been migrated yet — this sub-skill assumes you've already done the single-notebook port and are now hardening cross-task handoffs.
+- [`databricks-dabs`](../../databricks-dabs/SKILL.md) — DAB structure and resource definitions. Use when authoring or fixing the `job.yml` flagged by `FANOUT003` or `FANOUT004`.
+- [`databricks-jobs`](../../databricks-jobs/SKILL.md) — Lakeflow Jobs orchestration. Use when restructuring task dependencies to avoid the fan-out antipattern.
+- [`databricks-core`](../../databricks-core/SKILL.md) — grandparent skill, inherited via `databricks-serverless-migration`, for CLI auth and profile selection.
 
 ## Reference docs