Skip to content

Commit 68566a9

Browse files
webmaxruwebmaxru
committed
Publish complete book: all 14 chapters authored + verified
1 parent fa3227f commit 68566a9

11 files changed

Lines changed: 1269 additions & 66 deletions

‎chapters/continuous-review-and-testing.html‎

Lines changed: 119 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -99,37 +99,150 @@ <h2 id="depends-heading">Prerequisites</h2>
9999
<section class="chapter-section" data-section="objective">
100100
<h2 id="objective"><a class="anchor-link" href="#objective" aria-label="Link to Objective section">Objective</a></h2>
101101
<div class="section-content" data-content-slot="objective">
102-
<p class="pending-note"><strong>Content pending.</strong> Reserved for the chapter author. Fill the <code>data-slot="objective"</code> block in <code>content/chapters/continuous-review-and-testing.html</code>.</p>
102+
<p>By the end of this chapter you can close the repository's <strong>quality loop</strong> with four more patterns &mdash; <strong>Review</strong>, <strong>Testing</strong>, <strong>CI-Doctor</strong>, and <strong>Refactoring</strong> &mdash; while keeping humans firmly on the merge decision. This is the second half of the Continuous-X library and the close of Part II.</p>
103+
<p>Everything targets <strong>gh aw v0.81.6</strong>. The Repo Assistant graduates from tending the issue tracker to helping tend the <em>code</em>.</p>
103104
</div>
104105
</section>
105106
<section class="chapter-section" data-section="concept-closing-the-quality-loop">
106107
<h2 id="concept-closing-the-quality-loop"><a class="anchor-link" href="#concept-closing-the-quality-loop" aria-label="Link to Concept: closing the quality loop section">Concept: closing the quality loop</a></h2>
107108
<div class="section-content" data-content-slot="concept-closing-the-quality-loop">
108-
<p class="pending-note"><strong>Content pending.</strong> Reserved for the chapter author. Fill the <code>data-slot="concept-closing-the-quality-loop"</code> block in <code>content/chapters/continuous-review-and-testing.html</code>.</p>
109+
<p>A repository's quality has a <strong>loop</strong>: code is proposed (a PR), reviewed, tested, merged, and &mdash; when something slips &mdash; fixed. Traditional CI automates the <em>deterministic</em> checks in that loop: does it compile, do the tests pass, does the linter approve. But the <em>judgement</em> steps &mdash; is this a good change? is this test worth adding? why did CI actually break? &mdash; still wait on a human.</p>
110+
<p>Continuous-X patterns fill exactly those judgement gaps. Where <a class="xref" href="continuous-triage-and-docs.html">Chapter 9</a> kept the <em>inbox</em> honest, these keep the <em>codebase</em> honest &mdash; each one a mini-product owning one link in the quality loop.</p>
111+
112+
<h3>The one rule that makes it safe: humans keep the merge</h3>
113+
<p>The defining constraint of quality automation is that <strong>the agent proposes; a human disposes</strong>. A review agent <em>comments</em>, it doesn't approve. A test-improver <em>opens a draft PR</em>, it doesn't push to main. This isn't timidity &mdash; it's what lets you run these patterns at all. The agent accelerates the work up to the decision point and stops, leaving the irreversible call to a person. That's the human-in-the-loop principle, and it's why the safe-outputs boundary from <a class="xref" href="safe-outputs.html">Chapter 6</a> matters most here.</p>
114+
<aside class="callout callout--leader" aria-label="Real adopters">
115+
<p class="callout-title">Measured: quality agents at scale</p>
116+
<p>A single PR-review agent originating in <code>clash-verge-rev</code> has been cloned across <strong>215+ repositories</strong>; <code>backend.ai-webui</code> runs both a daily test-improver and an e2e-healer; <code>camunda</code> runs CI cost analysis. Quality is the category where Continuous-X adoption has spread fastest &mdash; because the human-keeps-the-merge rule makes it low-risk to try.</p>
117+
</aside>
109118
</div>
110119
</section>
111120
<section class="chapter-section" data-section="in-gh-aw-the-review-testing-ci-doctor-and-refactoring-recipes">
112121
<h2 id="in-gh-aw-the-review-testing-ci-doctor-and-refactoring-recipes"><a class="anchor-link" href="#in-gh-aw-the-review-testing-ci-doctor-and-refactoring-recipes" aria-label="Link to In gh-aw: the Review, Testing, CI-Doctor, and Refactoring recipes section">In gh-aw: the Review, Testing, CI-Doctor, and Refactoring recipes</a></h2>
113122
<div class="section-content" data-content-slot="in-gh-aw-the-review-testing-ci-doctor-and-refactoring-recipes">
114-
<p class="pending-note"><strong>Content pending.</strong> Reserved for the chapter author. Fill the <code>data-slot="in-gh-aw-the-review-testing-ci-doctor-and-refactoring-recipes"</code> block in <code>content/chapters/continuous-review-and-testing.html</code>.</p>
123+
<p>Four patterns, each triggered by a different moment in the quality loop &mdash; and each writing through a safe output that stops short of merging.</p>
124+
<table>
125+
<thead>
126+
<tr><th>Pattern</th><th>Trigger</th><th>Writes via</th></tr>
127+
</thead>
128+
<tbody>
129+
<tr><td><strong>Review</strong></td><td><code>pull_request</code></td><td><code>submit-pull-request-review</code> (COMMENT only)</td></tr>
130+
<tr><td><strong>Testing</strong></td><td><code>schedule</code></td><td><code>create-pull-request</code> (draft)</td></tr>
131+
<tr><td><strong>CI-Doctor</strong></td><td><code>workflow_run</code> (CI failed)</td><td><code>add-comment</code> / <code>create-issue</code></td></tr>
132+
<tr><td><strong>Refactoring</strong></td><td><code>schedule</code> or command</td><td><code>create-pull-request</code> (draft)</td></tr>
133+
</tbody>
134+
</table>
135+
136+
<h3>Review: comment, never approve</h3>
137+
<p>The Review pattern reads a PR diff and leaves inline feedback. The critical setting is <code>allowed-events: [COMMENT]</code>, which &ldquo;prevents the agent from submitting APPROVE reviews regardless of what the agent attempts to output&rdquo; &mdash; the docs explicitly recommend it as &ldquo;the default for automated review workflows&hellip; without creating a persistent merge-blocking state&rdquo; (<a href="https://github.github.com/gh-aw/reference/safe-outputs/">Safe Outputs</a>). Infrastructure enforces the human-keeps-the-merge rule.</p>
138+
139+
<h3>CI-Doctor: react to the failure</h3>
140+
<p>CI-Doctor is the elegant use of the <code>workflow_run</code> trigger from <a class="xref" href="triggers.html">Chapter 4</a> with <strong>conclusion filtering</strong>: fire only when a named CI workflow finishes with <code>failure</code>, read the logs, and post a diagnosis. Because <code>workflow_run</code> is hardened against cross-repo abuse, this stays safe even on public repos.</p>
141+
<figure class="code">
142+
<figcaption>The CI-Doctor trigger &mdash; wake only on a real CI failure</figcaption>
143+
<pre><code class="language-yaml">on:
144+
workflow_run:
145+
workflows: ["CI"]
146+
types: [completed]
147+
conclusion: [failure] # only when CI actually broke
148+
branches: [main]</code></pre>
149+
</figure>
150+
151+
<h3>Testing &amp; Refactoring: propose a diff</h3>
152+
<p>Both run on a schedule, do focused work, and open a <strong>draft</strong> <code>create-pull-request</code>. Testing adds coverage without touching production code; Refactoring makes a small, behavior-preserving cleanup. Draft PRs keep the human on the merge, exactly as in the Docs pattern.</p>
153+
<aside class="callout callout--builder" aria-label="Builder detail">
154+
<p class="callout-title">Builder detail: give the tester a scoped shell</p>
155+
<p>A test-improver has to <em>run</em> the suite, so it needs <code>bash</code> &mdash; but scope it. Prefer an explicit allowlist like <code>bash: ["npm ci", "npm test", "npx jest"]</code> over the unrestricted <code>bash: [":*"]</code>, and add only the ecosystem the build needs to <code>network.allowed</code> (e.g. <code>node</code>). Capability where required, tight everywhere else.</p>
156+
</aside>
115157
</div>
116158
</section>
117159
<section class="chapter-section" data-section="when-to-automate-quality-and-where-humans-stay-in-the-loop">
118160
<h2 id="when-to-automate-quality-and-where-humans-stay-in-the-loop"><a class="anchor-link" href="#when-to-automate-quality-and-where-humans-stay-in-the-loop" aria-label="Link to When to automate quality (and where humans stay in the loop) section">When to automate quality (and where humans stay in the loop)</a></h2>
119161
<div class="section-content" data-content-slot="when-to-automate-quality-and-where-humans-stay-in-the-loop">
120-
<p class="pending-note"><strong>Content pending.</strong> Reserved for the chapter author. Fill the <code>data-slot="when-to-automate-quality-and-where-humans-stay-in-the-loop"</code> block in <code>content/chapters/continuous-review-and-testing.html</code>.</p>
162+
<p>Quality patterns pay off when they act as a <strong>tireless first pass</strong> &mdash; catching the obvious before a human spends attention, never replacing the human's final say. The line to hold: <em>automate the noticing and the drafting; reserve the deciding.</em></p>
163+
<table>
164+
<thead>
165+
<tr><th>Agent may&hellip;</th><th>Human keeps&hellip;</th></tr>
166+
</thead>
167+
<tbody>
168+
<tr><td>comment on a PR, flag risks</td><td>approve / request changes / merge</td></tr>
169+
<tr><td>open a draft test or refactor PR</td><td>review and merge that PR</td></tr>
170+
<tr><td>diagnose a CI failure, file an issue</td><td>decide the fix and ship it</td></tr>
171+
</tbody>
172+
</table>
173+
174+
<h3>When <em>not</em> to</h3>
175+
<ul>
176+
<li><strong>Don't let a review agent block merges.</strong> Auto <code>REQUEST_CHANGES</code> creates a persistent merge-blocking state from a fallible model. Keep <code>allowed-events: [COMMENT]</code> unless a human explicitly wants gating.</li>
177+
<li><strong>Don't let the test-improver edit production code.</strong> Instruct it to add tests only; a PR that &ldquo;fixes&rdquo; code to make a test pass is the opposite of what you want.</li>
178+
<li><strong>Don't auto-merge agent PRs.</strong> The draft PR is the human's decision point &mdash; automating the merge throws away the one safeguard that makes this safe.</li>
179+
<li><strong>Don't run a refactoring agent on a repo without good tests.</strong> &ldquo;Behavior-preserving&rdquo; is only verifiable if the tests can prove it. Ship Testing before Refactoring.</li>
180+
</ul>
121181
</div>
122182
</section>
123183
<section class="chapter-section" data-section="worked-example-a-pr-review-plus-daily-test-improver-pair">
124184
<h2 id="worked-example-a-pr-review-plus-daily-test-improver-pair"><a class="anchor-link" href="#worked-example-a-pr-review-plus-daily-test-improver-pair" aria-label="Link to Worked example: a PR-review plus daily-test-improver pair section">Worked example: a PR-review plus daily-test-improver pair</a></h2>
125185
<div class="section-content" data-content-slot="worked-example-a-pr-review-plus-daily-test-improver-pair">
126-
<p class="pending-note"><strong>Content pending.</strong> Reserved for the chapter author. Fill the <code>data-slot="worked-example-a-pr-review-plus-daily-test-improver-pair"</code> block in <code>content/chapters/continuous-review-and-testing.html</code>.</p>
186+
<p>Two complementary quality agents: one reacts to every PR, the other proactively strengthens the tests. Both compile cleanly, and both stop short of the merge.</p>
187+
188+
<figure class="code">
189+
<figcaption>examples/ch10/continuous-review.md &mdash; comment-only PR review (compiles: 0/0)</figcaption>
190+
<pre><code class="language-yaml">on:
191+
pull_request: { types: [opened, synchronize] }
192+
permissions: { contents: read, pull-requests: read }
193+
engine: copilot
194+
network: { allowed: [defaults, github] }
195+
tools:
196+
github: { toolsets: [pull_requests] }
197+
safe-outputs:
198+
create-pull-request-review-comment: { max: 10 }
199+
submit-pull-request-review:
200+
allowed-events: [COMMENT] # can never approve or block
201+
max: 1</code></pre>
202+
</figure>
203+
204+
<figure class="code">
205+
<figcaption>examples/ch10/daily-test-improver.md &mdash; proposes tests as a draft PR (compiles: 0/0)</figcaption>
206+
<pre><code class="language-yaml">on:
207+
schedule: daily
208+
workflow_dispatch:
209+
permissions: { contents: read }
210+
engine: copilot
211+
network: { allowed: [defaults, github, node] }
212+
tools:
213+
github: { toolsets: [repos] }
214+
bash: ["npm ci", "npm test", "npx jest", "npx vitest run"] # scoped shell
215+
edit:
216+
safe-outputs:
217+
create-pull-request: { title-prefix: "[tests] ", labels: [tests, automated], draft: true }</code></pre>
218+
</figure>
219+
220+
<p>The review agent holds <strong>read-only</strong> PR access and can only emit a <code>COMMENT</code> review &mdash; the <code>allowed-events</code> setting makes &ldquo;never block a merge&rdquo; an infrastructural guarantee, not a hope. The test-improver gets a <strong>scoped shell</strong> to run the suite and <code>edit</code> to write tests, but its sole output is a <strong>draft</strong> PR a human reviews. Both accelerate the work right up to the human's decision, then hand it over.</p>
221+
222+
<figure class="code">
223+
<figcaption>Verifying both examples</figcaption>
224+
<pre><code class="language-text">gh aw compile examples/ch10/continuous-review.md
225+
# ✓ examples\ch10\continuous-review.md (101.8 KB) — 0 error(s), 0 warning(s)
226+
gh aw compile examples/ch10/daily-test-improver.md
227+
# ✓ examples\ch10\daily-test-improver.md (103.8 KB) — 0 error(s), 0 warning(s)</code></pre>
228+
</figure>
229+
<aside class="callout callout--note" aria-label="The safe output enforces the principle">
230+
<p class="callout-title">The principle is enforced, not just documented</p>
231+
<p>Notice how <code>allowed-events: [COMMENT]</code> and <code>draft: true</code> turn &ldquo;humans keep the merge&rdquo; from a guideline into a compiled constraint. Even a hijacked agent can't approve a PR or merge one &mdash; the safe-outputs layer simply won't let the request through. That's the quality loop closed <em>safely</em>.</p>
232+
</aside>
127233
</div>
128234
</section>
129235
<section class="chapter-section" data-section="recap-and-what-s-next">
130236
<h2 id="recap-and-what-s-next"><a class="anchor-link" href="#recap-and-what-s-next" aria-label="Link to Recap &amp; what&#x27;s next section">Recap &amp; what&#x27;s next</a></h2>
131237
<div class="section-content" data-content-slot="recap-and-what-s-next">
132-
<p class="pending-note"><strong>Content pending.</strong> Reserved for the chapter author. Fill the <code>data-slot="recap-and-what-s-next"</code> block in <code>content/chapters/continuous-review-and-testing.html</code>.</p>
238+
<p>You've closed the quality loop &mdash; and Part II:</p>
239+
<ul>
240+
<li>Quality automation fills the <strong>judgement gaps</strong> CI can't: is this change good, is this test worth adding, why did CI break.</li>
241+
<li>Four patterns &mdash; <strong>Review</strong> (PR, comment-only), <strong>Testing</strong> (scheduled draft PR), <strong>CI-Doctor</strong> (<code>workflow_run</code> on failure), <strong>Refactoring</strong> (scheduled draft PR).</li>
242+
<li>The unbreakable rule is <strong>humans keep the merge</strong> &mdash; enforced by <code>allowed-events: [COMMENT]</code> and <code>draft: true</code>, not just by convention.</li>
243+
<li>Automate the noticing and drafting; reserve the deciding. Ship Testing before Refactoring, and never auto-merge an agent's PR.</li>
244+
</ul>
245+
<p><strong>What's next.</strong> You now have a shelf of patterns &mdash; and you're about to notice how much they repeat. <strong>Part III</strong> scales from one repo to an org. <a class="xref" href="reuse-and-memory.html">Chapter 11: Reuse &amp; Memory</a> factors the shared parts into imported components and gives the Repo Assistant memory that persists across runs.</p>
133246
</div>
134247
</section>
135248

0 commit comments

Comments
 (0)