fix(workflow): 🐛 make stale Snakemake locks recoverable in a multi-run

Snakemake only releases its lock on a clean exit, so a run that was killed --
Ctrl-C, a dropped SSH session, an OOM, or a failed conversion -- leaves the
directory locked. run_multi made that worse: it puts each phase in its own
state directory, including a `.snakemake_convert` the user never named, so
clearing the lock meant reconstructing paths that are not written down
anywhere.

`--unlock` now releases every state directory the script manages, and the
conversion failure message names the exact command instead of leaving the
Snakemake hint to be translated into this layout.

Also documents both this and the glob/sequence_pattern error in the
troubleshooting table, since each cost a cluster round-trip to diagnose.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Author: lguerard

docs/guide/snakemake.md

@@ -346,6 +346,19 @@ the GPU partition stays busy instead of idling through every config's
 `prepare` and multi-hour `merge` in turn. A config that fails does **not**
 abort the others; you get a per-config status and a non-zero exit.
 
+!!! tip "After a killed run"
+    Snakemake only releases its lock on a clean exit, so a run that was killed
+    (Ctrl-C, an SSH drop, an OOM) leaves the directory locked. Each phase has
+    its own state directory, so releasing them by hand means reconstructing
+    several paths — use the flag instead:
+
+    ```bash
+    pixi run multi -- --unlock     # then re-run normally
+    ```
+
+    Running the orchestrator under `tmux` avoids most of these in the first
+    place: it survives a dropped connection.
+
 !!! warning "`jobs:` is per config"
     The profile's `jobs:` caps one Snakemake process. Running three configs
     concurrently can therefore have 3× that many jobs in flight — lower it if
@@ -450,6 +463,8 @@ prologue. The simplest path is a single shared env that the compute nodes see.
 | Symptom | Fix |
 |---------|-----|
 | `snakemake: command not found` | use `python -m snakemake` |
+| `Directory cannot be locked` | a previous run was killed instead of exiting cleanly. For a multi-run: `pixi run multi -- --unlock` (it covers every state directory, including the conversion one). For a single config: add `--unlock --directory <the same one you ran with>` |
+| `BioImage does not support the image: '.../*tif'` | a glob input needs `sequence_pattern` — see [A folder of single-plane TIFFs](ome_zarr_napari.md#a-folder-of-single-plane-tiffs) |
 | Segment jobs pend forever | wrong `slurm_partition`/GPU request; on scicore use `gres: "gpu:1"` |
 | Segment dies, `Network is unreachable` | offline GPU nodes — the `fetch_model` localrule caches the model on the submit host first; if it still fails, your submit host has no network either (pre-download manually) |
 | `cellpose is not installed` in a job | the job's env lacks `patchworks[cellpose]` |

workflow/scripts/run_multi.py

@@ -46,6 +46,7 @@ def _snakemake_cmd(
     dry_run: bool,
     state_dir: Path | None = None,
     targets: list[str] | None = None,
+    extra: list[str] | None = None,
 ) -> list[str]:
     """Build one snakemake invocation.
 
@@ -69,6 +70,8 @@ def _snakemake_cmd(
         cmd += ["--cores", str(cores), "--rerun-triggers", "mtime"]
     if dry_run:
         cmd += ["-n", "-p"]
+    if extra:
+        cmd += extra
     if targets:
         # "--" ends option parsing: --rerun-triggers takes a variable number
         # of values and would otherwise swallow the target path.
@@ -172,6 +175,15 @@ def main() -> None:
         action="store_true",
         help="pass -n -p to every Snakemake run; skips relations",
     )
+    parser.add_argument(
+        "--unlock",
+        action="store_true",
+        help=(
+            "release stale Snakemake locks in every state directory this "
+            "script manages, then exit. Needed after a run was killed or "
+            "died: the lock is only released on a clean exit."
+        ),
+    )
     args = parser.parse_args()
 
     workflow_dir = Path(__file__).resolve().parent.parent
@@ -185,6 +197,32 @@ def main() -> None:
     work_dir = _validate_configs(seg_config_paths, seg_cfgs)
     image_store = f"{work_dir}/image.zarr"
 
+    # Each phase gets its own Snakemake state directory (the lock lives in the
+    # working directory, not the config), so unlocking has to cover all of
+    # them -- and nobody should have to reconstruct these paths by hand.
+    state_dirs = [Path(work_dir) / ".snakemake_convert"] + [
+        Path(cfg["work_dir"]) / cfg["label_name"] / ".snakemake"
+        for cfg in seg_cfgs
+    ]
+    if args.unlock:
+        for state_dir in state_dirs:
+            if not state_dir.exists():
+                continue
+            _run(
+                _snakemake_cmd(
+                    seg_config_paths[0],
+                    workflow_dir=workflow_dir,
+                    profile=args.profile,
+                    cores=args.cores,
+                    dry_run=False,
+                    state_dir=state_dir,
+                    extra=["--unlock"],
+                ),
+                workflow_dir,
+            )
+        print("[run_multi] unlocked; re-run without --unlock", flush=True)
+        return
+
     # Phase A: convert exactly once. The three runs are about to go concurrent
     # and `convert` writes with overwrite=True, so letting them race on it
     # would have them clobbering one store. Ask for its marker explicitly.
@@ -201,7 +239,13 @@ def main() -> None:
         workflow_dir,
     )
     if rc != 0:
-        print("[run_multi] ERROR: conversion failed", file=sys.stderr)
+        print(
+            "[run_multi] ERROR: conversion failed.\n"
+            "  If the log says the directory cannot be locked, a previous run "
+            "was killed rather than exiting cleanly; release it with:\n"
+            f"      {Path(sys.argv[0]).name} --config {args.config} --unlock",
+            file=sys.stderr,
+        )
         sys.exit(rc)
 
     # Still phase A: build the occupancy map here too. Every config's `prepare`