refactor(serverless-storage-check): nest as sub-skill under databricks-serverless-migration

Moves skills/databricks-serverless-storage-check/ →
skills/databricks-serverless-migration/databricks-serverless-storage-check/
to physically reflect the parent/sub-skill hierarchy.

- Update scripts/skills.py iter_skill_dirs() to yield one level of nested
  skill directories (sub-skills alongside top-level skills)
- Fix relative SKILL.md links: ../databricks-serverless-migration/ → ../
  and ../databricks-{dabs,jobs,core}/ → ../../databricks-{dabs,jobs,core}/
- Fix databricks-serverless-migration/SKILL.md links:
  ../databricks-serverless-storage-check/ → databricks-serverless-storage-check/
- Regenerate manifest.json (skill count unchanged: 9)
- Confirmed: python3 scripts/skills.py validate → Everything is up to date
- Confirmed: stf lint → 0 errors, 0 warnings

Co-authored-by: Isaac

Author: GABRIELLE DOMPREH

manifest.json

@@ -1,6 +1,6 @@
 {
   "version": "2",
-  "updated_at": "2026-05-18T12:22:38Z",
+  "updated_at": "2026-06-08T10:21:37Z",
   "skills": {
     "databricks-apps": {
       "version": "0.1.1",
@@ -152,12 +152,29 @@
       "version": "0.1.0",
       "description": "Migrate Databricks workloads from classic compute to serverless compute, including compatibility checks and concrete fixes",
       "experimental": false,
-      "updated_at": "2026-05-18T12:22:14Z",
+      "updated_at": "2026-06-08T10:21:11Z",
       "files": [
         "SKILL.md",
         "agents/openai.yaml",
         "assets/databricks.png",
         "assets/databricks.svg",
+        "databricks-serverless-storage-check/SKILL.md",
+        "databricks-serverless-storage-check/agents/openai.yaml",
+        "databricks-serverless-storage-check/assets/databricks.png",
+        "databricks-serverless-storage-check/assets/databricks.svg",
+        "databricks-serverless-storage-check/eval/evaluation_results.json",
+        "databricks-serverless-storage-check/eval/ground_truth.yaml",
+        "databricks-serverless-storage-check/eval/manifest.yaml",
+        "databricks-serverless-storage-check/eval/output_instructions.md",
+        "databricks-serverless-storage-check/eval/report.html",
+        "databricks-serverless-storage-check/eval/tests/conftest.py",
+        "databricks-serverless-storage-check/eval/tests/test_integration.py",
+        "databricks-serverless-storage-check/eval/tests/test_unit.py",
+        "databricks-serverless-storage-check/eval/thinking_instructions.md",
+        "databricks-serverless-storage-check/references/pattern-catalog.md",
+        "databricks-serverless-storage-check/references/remediation-guide.md",
+        "databricks-serverless-storage-check/scripts/preflight.py",
+        "databricks-serverless-storage-check/scripts/test_preflight.py",
         "references/code-patterns.md",
         "references/compatibility-checks.md",
         "references/configuration-guide.md",
@@ -166,15 +183,24 @@
       ]
     },
     "databricks-serverless-storage-check": {
-      "version": "0.1.0",
+      "version": "0.2.0",
       "description": "Detect cross-task file-sharing antipatterns in serverless jobs (writes to /local_disk0, /tmp, or trustedTemp that are read by sibling or child tasks) and recommend UC Volumes or /Workspace handoff",
       "experimental": false,
-      "updated_at": "2026-05-18T12:21:52Z",
+      "updated_at": "2026-06-08T10:21:01Z",
       "files": [
         "SKILL.md",
         "agents/openai.yaml",
         "assets/databricks.png",
         "assets/databricks.svg",
+        "eval/evaluation_results.json",
+        "eval/ground_truth.yaml",
+        "eval/manifest.yaml",
+        "eval/output_instructions.md",
+        "eval/report.html",
+        "eval/tests/conftest.py",
+        "eval/tests/test_integration.py",
+        "eval/tests/test_unit.py",
+        "eval/thinking_instructions.md",
         "references/pattern-catalog.md",
         "references/remediation-guide.md",
         "scripts/preflight.py",

scripts/skills.py

@@ -56,7 +56,7 @@
 
 
 def iter_skill_dirs(repo_root: Path):
-    """Yield skill directories that contain SKILL.md."""
+    """Yield skill directories that contain SKILL.md (top-level and one level nested)."""
     skills_dir = repo_root / "skills"
     for item in sorted(skills_dir.iterdir()):
         if not item.is_dir():
@@ -66,6 +66,14 @@ def iter_skill_dirs(repo_root: Path):
         if not (item / "SKILL.md").exists():
             continue
         yield item
+        for subitem in sorted(item.iterdir()):
+            if not subitem.is_dir():
+                continue
+            if subitem.name.startswith("."):
+                continue
+            if not (subitem / "SKILL.md").exists():
+                continue
+            yield subitem
 
 
 def extract_version_from_skill(skill_path: Path) -> str:

skills/databricks-serverless-migration/SKILL.md

@@ -13,6 +13,16 @@ parent: databricks-core
 
 Analyze existing Databricks code for serverless compute compatibility and guide migration from classic clusters. The skill follows a 4-step migration lifecycle: **Ingest** the workload → **Analyze** for compatibility → **Test** via A/B comparison → **Validate** and iterate.
 
+## Sub-skills
+
+This skill is the top of a small hierarchy. Once a single-notebook migration is done, hand off to the niche sub-skill below for multi-task hardening before deploying to serverless:
+
+| Sub-skill | Use when |
+|-----------|----------|
+| [`databricks-serverless-storage-check`](databricks-serverless-storage-check/SKILL.md) | The job has **multiple tasks** (parent/child notebooks via `dbutils.notebook.run`, sibling `notebook_task`s, or a `pipeline_task` downstream of a `notebook_task`) and shares files between them. Detects the `INTERNAL_ERROR: [Errno 13] Permission denied` / `/local_disk0/.../trustedTemp/...` failure class and rewrites the handoff to UC Volumes / `/Workspace` / `taskValues`. Invoke after this skill's single-notebook port, before the Step-3 test run. |
+
+The sub-skill is a **diagnostic + fix** layer, not a replacement — keep this skill's 4-step lifecycle as the spine and call into the sub-skill only when the multi-task triggers fire.
+
 ## When to Use This Skill
 
 - Migrating notebooks, jobs, or pipelines from classic compute to serverless
@@ -118,7 +128,7 @@ Scan the code for patterns that are incompatible with the serverless compute arc
 | Pattern | Severity | Fix |
 |---------|----------|-----|
 | `dbfs:/` or `/dbfs/` paths (persistent data) | Blocker | Replace with `/Volumes/<your_catalog>/schema/volume/path` |
-| `dbfs:/tmp/`, `/dbfs/tmp/`, paths with `cache`/`scratch`/`temp` | Warning | Use `/tmp/` or `/local_disk0/tmp/` (local driver disk) — do not use Volumes for temp files due to performance. **Per-task scratch only**: if another task (child notebook, sibling job task, or pipeline) needs to read the file, use UC Volumes or `/Workspace` — see [`databricks-serverless-storage-check`](../databricks-serverless-storage-check/SKILL.md). |
+| `dbfs:/tmp/`, `/dbfs/tmp/`, paths with `cache`/`scratch`/`temp` | Warning | Use `/tmp/` or `/local_disk0/tmp/` (local driver disk) — do not use Volumes for temp files due to performance. **Per-task scratch only**: if another task (child notebook, sibling job task, or pipeline) needs to read the file, use UC Volumes or `/Workspace` — see [`databricks-serverless-storage-check`](databricks-serverless-storage-check/SKILL.md). |
 | `file:///dbfs/` FUSE mount paths | Warning | Replace persistent paths with `/Volumes/...`; replace temp paths with `/local_disk0/tmp/` |
 | `dbutils.fs.mount(...)` | Blocker | Create UC external location + external volume |
 | `hive_metastore.db.table` | Warning | Migrate to UC or use HMS Federation: `CREATE FOREIGN CATALOG ... USING CONNECTION hms_connection` |

…bricks-serverless-storage-check/SKILL.md …bricks-serverless-storage-check/SKILL.mdskills/databricks-serverless-storage-check/SKILL.md renamed to skills/databricks-serverless-migration/databricks-serverless-storage-check/SKILL.md skills/databricks-serverless-storage-check/SKILL.md renamed to skills/databricks-serverless-migration/databricks-serverless-storage-check/SKILL.md

@@ -3,13 +3,13 @@ name: databricks-serverless-storage-check
 description: "Detect cross-task file-sharing antipatterns in Databricks serverless jobs (writes to /local_disk0, /tmp, or trustedTemp that are read by sibling or child tasks on potentially different compute nodes) and recommend UC Volumes or /Workspace for handoff. Use when a serverless job fails with `INTERNAL_ERROR: [Errno 13] Permission denied` on /local_disk0 paths, when parallel child notebooks fail intermittently, when reviewing a DAB job before deploying to serverless, or when the user mentions trustedTemp, fan-out, or cross-task file handoff. Complements databricks-serverless-migration (which covers single-notebook migration)."
 compatibility: Requires databricks CLI (>= v0.292.0) for --job-id and --run-id modes; --notebook / --dir / --job-yaml modes have no external dependencies.
 metadata:
-  version: "0.1.0"
-parent: databricks-core
+  version: "0.2.0"
+parent: databricks-serverless-migration
 ---
 
 # Serverless Storage Check
 
-**FIRST**: Use the parent `databricks-core` skill for CLI basics, authentication, and profile selection.
+**FIRST**: Use the parent `databricks-serverless-migration` skill for the overall classic-to-serverless migration workflow (Ingest → Analyze → Test → Validate), CLI auth (which it inherits from `databricks-core`), and per-task scratch guidance (`/local_disk0/tmp`). This skill is a niche sub-skill of that workflow — it focuses **only** on the cross-task handoff antipattern that the parent skill explicitly delegates here (see the Category B → DBFS-paths row in `databricks-serverless-migration/SKILL.md`).
 
 This skill detects a specific class of serverless failure: **cross-task file handoffs through local disk**. On serverless compute, each task may run on a different node, so a path written by a parent task to `/local_disk0`, `/tmp`, or a `trustedTemp` directory is not guaranteed to be visible to a child task. The typical symptom is:
 
@@ -32,14 +32,16 @@ Use this skill when any of these triggers appear:
 - The user mentions "trustedTemp", "fan-out", "cross-task file sharing", or `/local_disk0`
 - A new serverless job design needs a sanity check before first run
 
-This skill is **complementary to**, not a replacement for, [`databricks-serverless-migration`](../databricks-serverless-migration/SKILL.md). That skill handles single-notebook migration and explicitly recommends `/local_disk0/tmp` for per-task scratch — which is correct *inside* a task. The boundary between the two skills:
+This skill is a **niche sub-skill** of [`databricks-serverless-migration`](../SKILL.md) — the parent owns single-notebook migration and explicitly recommends `/local_disk0/tmp` for per-task scratch (which is correct *inside* a task). This sub-skill picks up where the parent leaves off: anything that crosses a task boundary. The boundary between the two skills:
 
 | Concern | Use skill |
 |---------|-----------|
-| Migrating one notebook from classic DBR to serverless | `databricks-serverless-migration` |
-| Per-task scratch storage (intra-task) | `databricks-serverless-migration` (recommends `/local_disk0/tmp`) |
-| **Cross-task file handoff between parent/child notebooks or sibling tasks** | **this skill** |
-| Permission-denied on `/local_disk0` during a multi-task run | **this skill** |
+| Migrating one notebook from classic DBR to serverless | `databricks-serverless-migration` (parent) |
+| Per-task scratch storage (intra-task) | `databricks-serverless-migration` (parent — recommends `/local_disk0/tmp`) |
+| **Cross-task file handoff between parent/child notebooks or sibling tasks** | **this skill (sub-skill)** |
+| Permission-denied on `/local_disk0` during a multi-task run | **this skill (sub-skill)** |
+
+Always invoke the parent skill **first** for the overall migration plan, then return here to harden cross-task handoffs before deploying a multi-task serverless job.
 
 ## Quick start
 
@@ -131,10 +133,10 @@ If the scanner emits `ENV001`:
 
 ## Related skills
 
-- [`databricks-serverless-migration`](../databricks-serverless-migration/SKILL.md) — single-notebook classic-to-serverless migration. **Use that skill first** if the workload hasn't been migrated yet.
-- [`databricks-dabs`](../databricks-dabs/SKILL.md) — DAB structure and resource definitions. Use when authoring or fixing the `job.yml` flagged by `FANOUT003` or `FANOUT004`.
-- [`databricks-jobs`](../databricks-jobs/SKILL.md) — Lakeflow Jobs orchestration. Use when restructuring task dependencies to avoid the fan-out antipattern.
-- [`databricks-core`](../databricks-core/SKILL.md) — parent skill for CLI auth and profile selection.
+- [`databricks-serverless-migration`](../SKILL.md) — **parent skill**. Single-notebook classic-to-serverless migration and the overall Ingest → Analyze → Test → Validate lifecycle. **Always invoke first** if the workload hasn't been migrated yet — this sub-skill assumes you've already done the single-notebook port and are now hardening cross-task handoffs.
+- [`databricks-dabs`](../../databricks-dabs/SKILL.md) — DAB structure and resource definitions. Use when authoring or fixing the `job.yml` flagged by `FANOUT003` or `FANOUT004`.
+- [`databricks-jobs`](../../databricks-jobs/SKILL.md) — Lakeflow Jobs orchestration. Use when restructuring task dependencies to avoid the fan-out antipattern.
+- [`databricks-core`](../../databricks-core/SKILL.md) — grandparent skill, inherited via `databricks-serverless-migration`, for CLI auth and profile selection.
 
 ## Reference docs