feat: Ship a debugging Agent Skill in the shiny package (#2341)

Author: schloerke

.claude/references/agent-skills.md

@@ -0,0 +1,129 @@
+# Bundled Agent Skills
+
+The shiny package ships [Agent Skills](https://agentskills.io) under
+`shiny/.agents/skills/` — one directory per skill, each with a `SKILL.md`,
+following the [Agent Skills specification](https://agentskills.io/specification).
+They are **package data**: they ship in the wheel and are discovered by
+installers such as [library-skills](https://library-skills.io) (which symlinks
+them into a project's `.agents/skills/` or `.claude/skills/`). A `shiny skills
+list|get` CLI subcommand is planned as a zero-dependency way to read them.
+
+**Audience:** coding agents *using* shiny to build, test, and debug apps — not
+contributors to shiny itself. Contributor-facing guidance belongs in
+`.claude/references/`, not here. If a topic serves both audiences, the bundled
+skill documents the public API and the reference file documents the internals.
+
+## Layout
+
+Per the spec, a skill is a directory containing at minimum a `SKILL.md`, plus
+three conventional optional directories:
+
+```
+shiny/.agents/skills/
+  <skill-name>/
+    SKILL.md          # required: frontmatter + instructions
+    scripts/          # optional: self-contained runnable code
+    references/       # optional: on-demand docs (loaded only when needed)
+    assets/           # optional: templates, images, data files
+```
+
+- Skill names are short topics: `debugging`, `testing`, `modules`, `express`, ...
+- Names are **unprefixed** (no `shiny-` prefix): installers namespace skills
+  by package, and the description scopes the skill to Shiny for Python.
+- Keep everything in `SKILL.md` until a file earns its keep. Heavy reference
+  material (100+ lines) goes in `references/` — agents load those on demand,
+  so smaller focused files cost less context. Link with relative paths from
+  the skill root (`references/REFERENCE.md`) and keep references one level
+  deep — no chains of files pointing at files.
+
+## Frontmatter
+
+```yaml
+---
+name: <skill-name>
+description: <what the skill covers and when to use it>
+---
+```
+
+Required fields (spec constraints):
+
+- `name` — 1-64 characters; lowercase letters, numbers, and hyphens only; no
+  leading, trailing, or consecutive hyphens; **must match the directory name**.
+- `description` — 1-1024 characters, non-empty. Covers both *what the skill
+  does* and *when to use it*, with keywords agents would match on.
+
+How to write the description (this is the only part of a skill loaded at
+startup — agents decide from it alone whether to activate the skill):
+
+- Lead with a one-clause summary of what the skill covers, then "Use when…"
+  listing triggering conditions, symptoms, and search phrases — including the
+  *wrong* approach an agent might be about to take (e.g. "…or when tempted to
+  add print statements or hidden outputs").
+- Do **not** summarize the skill's workflow or step-by-step process: agents
+  that get the recipe from the description follow it and skip the body.
+- Third person. Descriptions across skills must not overlap: agents pick a
+  skill by description alone, so two skills matching the same symptom means
+  the wrong one gets loaded. When adding a skill, re-read the other
+  descriptions and sharpen the boundaries.
+
+Optional fields (`license`, `compatibility`, `metadata`, `allowed-tools`) are
+defined by the spec but usually unnecessary here: skills inherit the package's
+MIT license, and `compatibility` is only for skills with environment
+requirements beyond shiny itself (e.g. requires Playwright installed).
+
+The name/directory match and description presence are enforced by
+`tests/pytest/test_packaging.py`.
+
+## Body content
+
+The body is loaded in full once a skill activates, so size it for progressive
+disclosure: keep `SKILL.md` under ~500 lines (well under 5000 tokens), moving
+detail to `references/`. Skills are reference docs, not tutorials. The shape
+that works:
+
+1. **Overview** — the core principle in 1-3 sentences, including what NOT to
+   do (the workaround the skill replaces).
+2. **Task-oriented sections** — one per job the agent came to do, each with a
+   short runnable example against the **public API only** (imports included,
+   copy-paste ready). One excellent example per pattern; no variations.
+3. **Quick-reference table** for enumerable facts (env vars, flags, marker
+   strings).
+4. **Common mistakes** — concrete symptom → fix pairs, especially the
+   non-obvious gotchas (e.g. module namespacing of ids, 404 when a feature
+   flag is off).
+
+Keep a skill focused on one topic and roughly 400-800 words. If a section
+outgrows that, it is probably a second skill. Prefer extending an existing
+skill over creating a new one; create a new directory only for a genuinely
+separate topic with its own triggering conditions.
+
+Anything in `scripts/` must be self-contained (or clearly document its
+dependencies), emit helpful error messages, and handle edge cases — agents
+run these directly.
+
+## Maintenance
+
+- Bundled skills are **user-facing documentation**. When a PR changes an API
+  that a skill documents, update the skill in the same PR — grep
+  `shiny/.agents/skills/` for the API name as part of the change.
+- Skills describe behavior as shipped, not as planned: never document an API
+  before it is merged.
+- Adding or materially changing a skill warrants a CHANGELOG entry.
+
+## Validation
+
+- `pytest tests/pytest/test_packaging.py` — checks the frontmatter contract
+  and that every file under `shiny/.agents/` is matched by the
+  `[tool.setuptools.package-data]` globs in `pyproject.toml`. The explicit
+  `.agents/**` glob is required because `**` does not match hidden
+  directories; without it skills silently drop out of the wheel.
+- The spec's reference validator checks frontmatter and naming conventions:
+  `uvx --from skills-ref agentskills validate shiny/.agents/skills/<name>`
+  (see [skills-ref](https://github.com/agentskills/agentskills/tree/main/skills-ref)).
+- To confirm against a real build:
+  `uv build --wheel && unzip -l dist/shiny-*.whl | grep .agents`
+- For a new skill, do a before/after check: give a fresh agent (no skill) a
+  task the skill targets and note what it reaches for; then repeat with the
+  skill available. If the "after" run doesn't change the approach, the
+  description isn't triggering or the body isn't teaching — fix before
+  shipping.

CHANGELOG.md

@@ -19,6 +19,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 * Test-mode snapshot values can now be preprocessed before they are written to the snapshot, e.g. to scrub timestamps or temp paths: `input.set_snapshot_preprocess(id, fn)` (or `shiny.testmode.snapshot_preprocess_input()`) for inputs and `my_output.snapshot_preprocess(fn)` for outputs. Handlers may be synchronous or asynchronous. File inputs automatically scrub each file's `datapath` to its basename, matching Shiny for R. `export_test_values()` moved from `shiny.session` to the new `shiny.testmode` module. (#2282)
 
+* The shiny package now ships [Agent Skills](https://agentskills.io) under `shiny/.agents/skills/` (the [library-skills](https://library-skills.io) convention), starting with a `debugging` skill that teaches coding agents to inspect running apps via test mode, `shiny.testmode.export_test_values()`, and the test snapshot endpoint. (#2339)
+
 * Added `offcanvas()` for creating sliding Bootstrap Offcanvas panels that appear from a viewport edge. Panels can be triggered by a UI element, revealed programmatically with `show_offcanvas()`, or controlled by id with `hide_offcanvas()` and `toggle_offcanvas()`. (#2279)
 
 ### Improvements

CLAUDE.md

@@ -107,6 +107,16 @@ Before implementing a UI component or output renderer, read
 (file layout, exports, examples, controllers, tests) and the rules for pairing
 output components with renderers.
 
+### Bundled Agent Skills
+
+The shiny package ships Agent Skills under `shiny/.agents/skills/` — reference
+docs that teach coding agents how to use shiny's public APIs (debugging,
+testing, etc.). Treat them like user-facing documentation: when a change
+touches an API that a skill documents, update the skill in the same PR. Before
+adding or editing a skill, read `.claude/references/agent-skills.md` for the
+layout, frontmatter contract, content shape, scoping rules, and validation
+steps.
+
 ### Type Checking Notes
 
 - Pyright is the primary type checker (not mypy)
@@ -163,3 +173,4 @@ handling before merging one.
 - **Reactive graph debugging**: Use `reactive.flush()` to force synchronous execution in tests
 - **Playwright timing**: Use `.expect_*()` methods which auto-wait; avoid manual `sleep()`
 - **Asset updates**: After running `make upgrade-html-deps`, verify theme preset files were updated
+- **Stale bundled skills**: When changing a public API, grep `shiny/.agents/skills/` for it — bundled Agent Skills document public APIs and must be updated in the same PR

pyproject.toml

@@ -9,8 +9,11 @@ build-backend = "setuptools.build_meta"
 # `**` does not match hidden files (the package templates ship .gitignore files).
 include-package-data = false
 
+# The `.agents/**` glob is needed because `**` does not match hidden
+# directories: Agent Skills ship inside the package under `shiny/.agents/skills/`
+# (the library-skills convention). Guarded by tests/pytest/test_packaging.py.
 [tool.setuptools.package-data]
-shiny = ["**", "**/.gitignore"]
+shiny = ["**", "**/.gitignore", ".agents/**"]
 
 [tool.setuptools.packages.find]
 where = ["."]

shiny/.agents/skills/debugging/SKILL.md

@@ -0,0 +1,127 @@
+---
+name: debugging
+description: Use when debugging a Shiny for Python (py-shiny) app or testing one - inspecting reactive/calc values in a running session, capturing current input/output state, exposing internal values to a test harness, writing snapshot tests, or when tempted to add print statements or hidden outputs to see server-side values.
+---
+
+# Debugging Shiny for Python apps
+
+## Overview
+
+py-shiny has a built-in **test mode**: every session can serve a read-only JSON
+snapshot of its `input`, `output`, and `export` values over HTTP. Use it instead
+of print statements, hidden outputs, or websocket spelunking to observe
+server-side state.
+
+## Enable test mode
+
+- Env var: `SHINY_TESTMODE=1 shiny run app.py`
+- Or in code: `App(app_ui, server, test_mode=True)`
+- The pytest app-launch fixtures (`local_app`, `create_app_fixture`) enable it
+  automatically.
+
+When off, the snapshot endpoint returns 404 and the APIs below are cheap no-ops,
+so calls can be left in production code.
+
+## Read a session snapshot
+
+Each session serves `GET /session/{id}/dataobj/shinytest` returning
+`{"input": {...}, "output": {...}, "export": {...}}` with sorted keys.
+
+**Getting the URL:** the session id only exists after the client connects — do
+not guess it or scrape it from the page. Obtain the URL through one of two
+paths:
+
+1. **From the browser** (devtools console, browser automation) — ask the Shiny
+   client, then GET the result from the page context (so any auth
+   cookies/headers come along):
+
+   ```js
+   window.Shiny.shinyapp.getTestSnapshotBaseUrl({ fullUrl: true })
+   // -> "http://127.0.0.1:8000/session/<id>/dataobj/shinytest?w=...&nonce=..."
+   ```
+
+   This is undefined until the app has connected; wait for it to exist before
+   calling. (In Playwright, prefer `AppTestValues` below, which handles the
+   waiting and fetching.)
+
+2. **From server code** (inside the `server` function, where `session` is in
+   scope) — `session.get_test_snapshot_url()` returns the relative URL with a
+   cache-busting nonce; log it or surface it to wherever you need it.
+
+Select blocks with query params: `?input=1` (whole block), `?output=a,b`
+(specific keys); no params returns all three blocks.
+
+Values that fail to serialize appear as visible markers instead of erroring:
+`{"__shiny_serialization_error__": ...}`, output render errors as
+`{"__shiny_output_error__": ...}`, preprocessor failures as
+`{"__shiny_snapshot_preprocess_error__": ...}`.
+
+## Expose internal reactive values: `export_test_values()`
+
+To let a test harness read a `reactive.calc` or other internal value (do NOT
+render it into a hidden output):
+
+```python
+from shiny import App, Inputs, Outputs, Session, reactive, ui
+from shiny.testmode import export_test_values
+
+app_ui = ui.page_fluid(ui.input_slider("n", "n", 0, 100, 20))
+
+def server(input: Inputs, output: Outputs, session: Session):
+    @reactive.calc
+    def doubled():
+        return input.n() * 2
+
+    # Appears under "export" in the snapshot. No-op unless test mode is on.
+    export_test_values(doubled=doubled, plus_one=lambda: doubled() + 1)
+
+app = App(app_ui, server)
+```
+
+Each value is a zero-argument callable, evaluated lazily (in a reactive
+isolate) when a snapshot is requested. Inside a module, export names are
+namespaced with the module's `ns` prefix. Re-registering a name overwrites it.
+
+## Assert snapshot values in Playwright tests
+
+```python
+from shiny.playwright import controller
+
+app_values = controller.AppTestValues(page)
+app_values.expect_input("n", 20)
+app_values.expect_output("greeting", "Hello abc")
+app_values.expect_export("doubled", 40)
+snapshot = app_values.get()  # full {"input", "output", "export"} dict
+```
+
+Requires the app loaded in the page and test mode on (a `RuntimeError` with a
+fix hint is raised otherwise).
+
+## Scrub unstable values from snapshots
+
+Timestamps, temp paths, etc. make snapshots non-deterministic. Preprocess
+before they are written (sync or async functions both work):
+
+- Inputs: `shiny.testmode.snapshot_preprocess_input("secret", lambda v: "<redacted>")`
+  or `session.input.set_snapshot_preprocess(id, fn)`
+- Outputs: call `.snapshot_preprocess(fn)` on the `@render.*` object
+- File inputs automatically scrub each file's `datapath` to its basename
+
+## Other debugging tools
+
+| Need | Tool |
+|---|---|
+| Verbose framework logs | `SHINY_LOG_LEVEL=DEBUG shiny run app.py` |
+| Echo client/server websocket messages | `App(..., debug=True)` |
+| Auto-reload during development | `shiny run app.py --reload --launch-browser` |
+
+## Common mistakes
+
+- Hitting the snapshot endpoint without test mode enabled → 404. Set
+  `SHINY_TESTMODE=1` or `App(test_mode=True)`.
+- Rendering values into hidden outputs just so tests can read them → use
+  `export_test_values()` instead.
+- Expecting un-namespaced export names inside modules → exports are prefixed
+  with the module namespace, like inputs and outputs.
+- Snapshot diffs churn on paths/timestamps → add a snapshot preprocessor
+  instead of post-processing the JSON.

tests/pytest/test_packaging.py

@@ -0,0 +1,80 @@
+"""Guard that bundled Agent Skills stay covered by the wheel packaging globs.
+
+Agent Skills ship as package data under ``shiny/.agents/skills/`` (the
+library-skills convention). Because ``**`` in setuptools package-data globs does
+not match hidden directories, the ``.agents`` tree needs its own glob in
+``pyproject.toml`` -- and nothing at build time fails if that glob is dropped,
+the files just silently disappear from the wheel. These tests replicate
+setuptools' matching (``glob.glob(os.path.join(src_dir, pattern),
+recursive=True)`` in ``setuptools.command.build_py``) against the declared
+patterns so a regression fails loudly here instead.
+"""
+
+from __future__ import annotations
+
+import glob
+import os
+from pathlib import Path
+
+import pytest
+
+tomllib = pytest.importorskip("tomllib")  # Stdlib in Python 3.11+
+
+REPO_ROOT = Path(__file__).parents[2]
+PACKAGE_DIR = REPO_ROOT / "shiny"
+SKILLS_DIR = PACKAGE_DIR / ".agents" / "skills"
+
+
+def package_data_files() -> set[Path]:
+    """Files shipped as `shiny` package data, per pyproject.toml globs."""
+    with open(REPO_ROOT / "pyproject.toml", "rb") as f:
+        pyproject = tomllib.load(f)
+    patterns: list[str] = pyproject["tool"]["setuptools"]["package-data"]["shiny"]
+
+    matched: set[Path] = set()
+    for pattern in patterns:
+        for match in glob.glob(os.path.join(str(PACKAGE_DIR), pattern), recursive=True):
+            if os.path.isfile(match):
+                matched.add(Path(match))
+    return matched
+
+
+def skill_dirs() -> list[Path]:
+    return sorted(p for p in SKILLS_DIR.iterdir() if p.is_dir())
+
+
+def test_skills_directory_has_skills() -> None:
+    assert SKILLS_DIR.is_dir()
+    assert len(skill_dirs()) > 0
+
+
+def test_every_skill_has_a_skill_md() -> None:
+    for skill_dir in skill_dirs():
+        assert (skill_dir / "SKILL.md").is_file(), f"{skill_dir} is missing SKILL.md"
+
+
+def test_skill_files_are_covered_by_package_data_globs() -> None:
+    shipped = package_data_files()
+    skill_files = [p for p in SKILLS_DIR.rglob("*") if p.is_file()]
+    assert len(skill_files) > 0
+    missing = [p for p in skill_files if p not in shipped]
+    assert not missing, (
+        "Files under shiny/.agents/ are not matched by any "
+        "[tool.setuptools.package-data] glob in pyproject.toml and would be "
+        f"silently dropped from the wheel: {missing}"
+    )
+
+
+def test_skill_frontmatter_has_name_and_description() -> None:
+    # Minimal SKILL.md frontmatter contract (agentskills.io specification);
+    # also what `shiny skills list` will surface.
+    for skill_dir in skill_dirs():
+        text = (skill_dir / "SKILL.md").read_text()
+        assert text.startswith("---\n"), f"{skill_dir}: SKILL.md missing frontmatter"
+        frontmatter = text.split("---", 2)[1]
+        assert (
+            f"name: {skill_dir.name}" in frontmatter
+        ), f"{skill_dir}: frontmatter `name` must match its directory name"
+        assert (
+            "description:" in frontmatter
+        ), f"{skill_dir}: frontmatter is missing `description`"