Skip to content

Commit 8aaaa3f

Browse files
happyhumanhappyhuman
committed
docs: update ADK snippet verification documentation and ignore coverage reports in .gitignore
1 parent fd5123f commit 8aaaa3f

2 files changed

Lines changed: 59 additions & 76 deletions

File tree

‎.agents/skills/adk-verify-snippets/SKILL.md‎

Lines changed: 55 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -10,119 +10,98 @@ metadata:
1010

1111
# Verify Markdown Snippets Skill
1212

13-
This skill allows you to systematically verify the correctness, compile-readiness, and line coverage of Python code snippets embedded within a Markdown document (like tutorials, guides, or READMEs).
14-
15-
It extracts all ` ```python ` blocks, executes them in process-isolated environments using the bundled `run.py` harness, and generates a structured test report.
13+
This skill extracts all ` ```python ` blocks from a Markdown file, executes each one in a process-isolated environment using the bundled `run.py` harness, and generates a structured report covering load status, run status, and line coverage.
1614

1715
> [!CAUTION]
1816
> **STRICT READ-ONLY CONSTRAINT — READ THIS BEFORE DOING ANYTHING ELSE**
1917
>
20-
> When executing this skill, the agent operates in a **strictly read-only** mode with respect to the repository. The agent **MUST NOT**, under any circumstances:
21-
> - **Modify** any existing source file, test file, configuration file, documentation file, or skill file (including this SKILL.md).
18+
> This skill is **read-only**. The agent **MUST NOT**:
19+
> - **Modify** any file in the repository (source, test, config, docs, or skill files — including this SKILL.md).
2220
> - **Delete** any file in the repository.
23-
> - **Create** any new file in the repository other than the two explicit exceptions listed below.
21+
> - **Create** any new file in the repository.
2422
>
25-
> The **only** two write operations permitted are:
26-
> 1. Writing temporary snippet `.py` files to a uniquely-named **system temp directory** (outside the repository).
27-
> 2. Writing the final `<filename>_REPORT.md` report file into the **same directory as the source Markdown file**.
23+
> The **only two write operations permitted** are:
24+
> 1. Writing temporary `.py` snippet files to a **system temp directory outside the repository**.
25+
> 2. Writing the final `<filename>_REPORT.md` into the **same directory as the source Markdown file**.
2826
>
29-
> Any other file system mutation is a violation of this skill's contract. If in doubt, do not write.
27+
> If in doubt, do not write. Any other mutation is a violation of this skill's contract.
3028
3129
---
3230

3331
## 🔧 Prerequisites
3432

35-
Before running this skill, ensure the following are in place:
36-
37-
1. **ADK Python environment**: The skill must be run from the repository root with the project's virtual environment active (via `uv`).
38-
2. **`coverage` package**: Install it to enable per-snippet coverage reporting:
39-
```bash
40-
uv pip install coverage
41-
```
42-
If `coverage` is not installed, the runner degrades gracefully — verification still works, but coverage columns in the report will show `—`.
43-
3. **Gemini API key**: Snippets that instantiate an `Agent`, `App`, or `Workflow` make live calls to the Gemini API. Set one of the following environment variables before running:
44-
```bash
45-
export GEMINI_API_KEY="your-key-here"
46-
# or
47-
export GOOGLE_API_KEY="your-key-here"
48-
```
49-
If both are set, `GEMINI_API_KEY` takes precedence. Snippets that do not expose a runnable ADK component (loadability-only checks) do not require an API key.
33+
1. **ADK Python environment**: Run from the repository root with the `uv` virtual environment active.
34+
2. **`coverage` package** *(optional)*: Enables per-snippet coverage reporting. Without it, coverage columns show `—`.
35+
```bash
36+
uv pip install coverage
37+
```
38+
3. **Gemini API key**: Required only for snippets that instantiate an `Agent`, `App`, or `Workflow` (which make live Gemini API calls). Set one of:
39+
```bash
40+
export GEMINI_API_KEY="your-key-here"
41+
# or
42+
export GOOGLE_API_KEY="your-key-here"
43+
```
44+
If both are set, `GEMINI_API_KEY` takes precedence.
5045

5146
---
5247

53-
## 🛠️ How to Use the Skill
54-
55-
To verify a Markdown file (e.g. `docs/my_guide.md`) and generate its runnability report:
48+
## 🛠️ Usage
5649

5750
```bash
5851
uv run --no-sync python .agents/skills/adk-verify-snippets/scripts/verify_md.py <path_to_markdown_file.md>
5952
```
6053

61-
For example:
62-
```bash
63-
uv run --no-sync python .agents/skills/adk-verify-snippets/scripts/verify_md.py docs/my_guide.md
64-
```
54+
The script prints progress for each snippet, then writes a report to **`<filename>_REPORT.md`** in the same directory as the source file and prints the full path on completion.
6555

66-
This will automatically create a detailed report file called **`docs/my_guide_REPORT.md`** in the same directory as the source file. The full path is printed on completion.
56+
**Report contents:**
57+
- **Executive Summary table** — one row per snippet: preceding heading, Load phase status, Run phase status, coverage %, and error detail.
58+
- **Detailed section** — for each snippet: the extracted code block, full execution logs (stdout + stderr/traceback), and the coverage report.
6759

6860
---
6961

70-
## 📝 Snippet Code Conventions
71-
72-
Each Python code block in the Markdown file is verified using the `run.py` contract. Blocks fall into one of four categories:
62+
## 📝 How Snippets Are Classified
7363

74-
1. **Expose a Global ADK Component (Runnability Test)**:
75-
If the snippet instantiates a global `Workflow`, `Agent`, or `App`, the runner will automatically execute it and measure its execution against the Gemini API.
64+
Each ` ```python ` block falls into one of these categories:
7665

77-
**Requirement**: The component must be assigned to a **module-level variable** (i.e., defined at the top level of the snippet, not inside a function or class). The variable name does not matter — the runner scans `vars(module)` to find it automatically. For multi-agent snippets, the runner identifies the root agent by filtering out any agents that are listed as `sub_agents` of another agent in the same snippet.
66+
### 1. Runnability Test (has a module-level ADK component)
67+
If the snippet assigns a `Workflow`, `Agent`, or `App` to a **module-level variable**, the runner executes it against the Gemini API.
7868

79-
**If this requirement is not met** (i.e., no module-level `Workflow`, `Agent`, or `App` instance exists), the runnability test is skipped. The report will show `➖ NO ADK COMPONENT` in the Run Phase column and explain that no runnable component was found at module level.
69+
- The variable name does not matter — the runner finds it automatically via `vars(module)`.
70+
- For multi-agent snippets, the runner identifies the root agent by excluding any agent that appears in another agent's `sub_agents` list.
71+
- To use a custom test prompt instead of the default `"Test input topic"`, define a module-level `test_input` string in the snippet.
8072

81-
2. **Provide a Custom Test Input (Optional)**:
82-
If the snippet defines a global `test_input` variable, the runner will use that string as the user message during execution instead of the default `"Test input topic"`.
83-
3. **Basic Python Snippets (Loadability Only)**:
84-
If a code block does not define any runnable ADK components, the runner will verify that it compiles and loads without error. No API call is made, and coverage is reported as 100% for the lines present.
85-
4. **Skipping Illustrative / Pseudo-code Blocks**:
86-
Place the HTML comment `<!-- verify-snippets: ignore -->` on the line **immediately before** the opening ` ```python ` fence to exclude that block from execution entirely. It will appear in the report as `⏭️ SKIPPED`. Use this for conceptual examples, incomplete snippets, or blocks that intentionally require external setup unavailable in CI.
73+
If no module-level ADK component is found, the run phase is skipped and the report shows `➖ NO ADK COMPONENT`.
8774

88-
Example — place the annotation on the line directly above the fence, with no blank line between them:
75+
### 2. Loadability-Only (no ADK component)
76+
The runner verifies the snippet compiles and imports without error. No API call is made.
8977

90-
````markdown
91-
<!-- verify-snippets: ignore -->
92-
```python
93-
# This is pseudo-code — not runnable as-is
94-
my_agent = Agent(model="gemini-ultra-hypothetical", ...)
95-
```
96-
````
78+
### 3. Skipped (annotated with ignore)
79+
Place `<!-- verify-snippets: ignore -->` immediately before the opening ` ```python ` fence to exclude a block entirely. Use this for pseudo-code, illustrative examples, or snippets that require external setup.
9780

98-
---
99-
100-
## ⚠️ Known Limitations
81+
````markdown
82+
<!-- verify-snippets: ignore -->
83+
```python
84+
# pseudo-code — not runnable as-is
85+
my_agent = Agent(model="gemini-ultra-hypothetical", ...)
86+
```
87+
````
10188

102-
* **No cumulative / dependent snippet support**: Each snippet is executed in a fully isolated subprocess with no shared state. If a snippet depends on imports, variables, or definitions introduced by a *previous* snippet in the same document, it will fail with a `NameError` or `ImportError`. Author each code block to be self-contained, or annotate dependent snippets with `<!-- verify-snippets: ignore -->`.
103-
* **120-second execution timeout**: Each snippet subprocess is killed after 120 seconds. Snippets that intentionally block (e.g., long-running servers) must be annotated with `<!-- verify-snippets: ignore -->`.
104-
* **`<!-- verify-snippets: ignore -->` must be close to the fence**: Blank lines between the annotation and the ` ```python ` fence are tolerated, but any non-blank line (prose, another heading, etc.) between them cancels the annotation. Place the annotation on the line immediately before the opening fence to be safe.
105-
* **No nested fences inside a code block**: The parser closes a Python block only on a bare ` ``` ` line (no language tag). Lines like ` ```bash ` or ` ```text ` inside the block are treated as literal content. However, a bare ` ``` ` that is the closing fence of a *non-Python* outer block will prematurely terminate any open Python block if the nesting depth is mismatched. Annotate such snippets with `<!-- verify-snippets: ignore -->`.
89+
The report shows these as `⏭️ SKIPPED`.
10690

10791
---
10892

109-
## 📊 The Generated Report Structure
93+
## ⚠️ Known Limitations
11094

111-
The generated `<filename>_REPORT.md` will contain:
112-
1. **Executive Summary**: A high-level table listing every discovered snippet, its preceding Markdown heading, its Load phase status, Run phase status, and line coverage percentage.
113-
2. **Detailed Breakdown**:
114-
* The exact extracted Python code block.
115-
* The detailed execution stdout logs (phases 1–3).
116-
* Any stderr traceback/exceptions (pointing directly to the breaking line of code).
117-
* A full `coverage report` output showing exactly which lines of the snippet were executed.
95+
- **No shared state between snippets**: Each snippet runs in a fresh subprocess with no imports or variables carried over from previous snippets. A snippet that depends on code from an earlier block will fail with `NameError` or `ImportError`. Make each snippet self-contained, or annotate it with `<!-- verify-snippets: ignore -->`.
96+
- **120-second timeout**: Each snippet is killed after 120 seconds. Annotate long-running or blocking snippets with `<!-- verify-snippets: ignore -->`.
97+
- **Ignore annotation placement**: The `<!-- verify-snippets: ignore -->` annotation applies to the next ` ```python ` fence encountered. Blank lines between the annotation and the fence are tolerated, but any non-blank line (prose or a heading) cancels the annotation.
98+
- **Bare ` ``` ` closes the block**: The parser closes a Python block on the first bare ` ``` ` line (no language tag). A bare ` ``` ` appearing as content inside a snippet (e.g. to demonstrate Markdown syntax) will prematurely close the block. Annotate such snippets with `<!-- verify-snippets: ignore -->`.
11899

119100
---
120101

121-
## ⚠️ Crucial Behavioral Constraints (For AI Agents)
102+
## ⚠️ Behavioral Constraints (For AI Agents)
122103

123-
* **Read-Only**: See the caution block at the top of this document. The constraint is absolute — no modifications, no deletions, no new files beyond the two permitted exceptions.
124-
* **Strictly Report, Do Not Fix**: The sole purpose of this skill is to **identify and report** compile-time and run-time issues within the Markdown document's snippets.
125-
* **No Unsolicited Patches**: When executing this skill, the agent **MUST NOT** attempt to rewrite the source Markdown file, modify its code blocks, or automatically generate code fixes/patches.
126-
* **Present the Summary Table Verbatim**: After the script completes, the agent MUST read the generated `<filename>_REPORT.md` file and copy the Executive Summary table to the user **exactly as written** — same columns, same column names, same order. The table has exactly these six columns:
127-
`Snippet | Preceding Heading | Load Phase | Run Phase | Coverage | Details`
128-
The agent MUST NOT rename, reorder, merge, or drop any column when presenting results. Do not offer solutions or rewrite recommendations unless the user explicitly asks for them.
104+
- **Read-only**: See the caution block at the top. The constraint is absolute.
105+
- **Report only, do not fix**: The agent MUST NOT rewrite the source Markdown, modify code blocks, or generate patches. Present the summary table to the user and stop.
106+
- **Present the summary table verbatim**: After the script completes, read the generated `_REPORT.md` and copy the Executive Summary table to the user **exactly as written** — same six columns, same order, no renaming or dropping:
107+
`Snippet | Preceding Heading | Load Phase | Run Phase | Coverage | Details`

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,3 +121,7 @@ CLAUDE.md
121121

122122
# Conformance test outputs (timestamped folders from --test mode)
123123
**/conformance/20*-*-*_*-*-*/
124+
125+
# adk-verify-snippets skill — generated report files (never commit these)
126+
*_REPORT.md
127+
*_report.md

0 commit comments

Comments
 (0)