Skip to content

Commit d449d1b

Browse files
committed
demo(workflow): quality-gate beat — adversarial ask, lint fires on camera, plan rejected
Send 'Plan a sloppy review: have the reviewer double-check its own findings' and the planner authors a VALID plan (registered capabilities, typed bindings) whose pipeline is reviewer -> reviewer. Plain validation passes; the plan-quality lint catches the structural self-review bias and the gate rejects the plan before freezing or executing (production: bounded re-plan). * agent.py: trigger-phrase path (checked before load-or-author so it works in any session), live-model sloppy planner, lint beat, rejection beat. * test_demo_agent.py: CI-safe test pinning the lint on the sloppy shape (7). * README/DEMO_NARRATIVE: Beat 6 + counts 11+31+7=49.
1 parent 47dc907 commit d449d1b

4 files changed

Lines changed: 142 additions & 4 deletions

File tree

contributing/samples/workflows/authored_workflow_demo/DEMO_NARRATIVE.md

Lines changed: 27 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -178,14 +178,39 @@ cat security_audit_plan.json | jq '{schema_version, spec_hash, planner_model, ca
178178

179179
Same hash, `reused` flips to `true` — the model is not called the second time.
180180

181+
## Beat 6 — the quality gate catches a biased plan (adversarial ask)
182+
183+
Send: **"Plan a sloppy review: have the reviewer double-check its own findings."**
184+
185+
```
186+
🧭 Adversarial ask — authoring a plan where the reviewer double-checks its OWN
187+
findings. Watch the quality gate.
188+
📋 Authored plan (valid registry refs, valid bindings, valid shapes): …
189+
"stages": [{"capability":"reviewer"}, {"capability":"reviewer"}] …
190+
🚨 Plan-quality lints fired (1):
191+
- ⚠️ plan-quality: pipeline 'review_pipeline' stage 'reviewer' re-checks its
192+
own capability's output — same-capability review cannot provide
193+
independent verification (self-preferential bias)
194+
🛑 Plan rejected by the quality gate — NOT frozen, NOT executed.
195+
```
196+
197+
> "This is the counterpoint to Beat 2b, and the sharpest 30 seconds in the
198+
> demo. I *asked* for a biased plan, and the model obliged — every capability
199+
> registered, every binding typed, plain validation green. A code-authoring
200+
> system would now run it. Here the **structural bias check** catches it
201+
> pre-execution: an agent grading its own output is detectable *from the plan
202+
> itself*, because the plan is data. The gate refuses to freeze or execute it;
203+
> in production that triggers a bounded re-plan. Bias control as a static
204+
> check — that's not possible when the model writes orchestration code."
205+
181206
## Close (~20s)
182207

183208
> "So: a model authored a typed, validated, capability-bounded plan whose
184209
> **agent independence is statically proven**; ADK executed it on the real
185210
> engine at a visible cost (one planner call, the work outside its context);
186211
> the plan **exported** to a portable, defensively-imported audit artifact; and
187212
> a re-send replayed the exact frozen plan with zero planner calls. The
188-
> deterministic test suites — 11 (#92) + 31 (#93) + 6 (demo) — lock all of this
213+
> deterministic test suites — 11 (#92) + 31 (#93) + 7 (demo) — lock all of this
189214
> in CI, including the no-LLM reuse path, the export round-trip / tamper /
190215
> drift checks, the plan-quality lints, and the six-coordination-pattern
191216
> coverage sweep (adversarial verification and tournament included)."
@@ -212,5 +237,5 @@ Same hash, `reused` flips to `true` — the model is not called the second time.
212237
```bash
213238
pytest contributing/samples/workflows/dynamic_supervisor_spike/test_dynamic_supervisor_spike.py -q # 11
214239
pytest contributing/samples/workflows/authored_workflow_spike/test_authoring.py -q # 31
215-
pytest contributing/samples/workflows/authored_workflow_demo/test_demo_agent.py -q # 6
240+
pytest contributing/samples/workflows/authored_workflow_demo/test_demo_agent.py -q # 7
216241
```

contributing/samples/workflows/authored_workflow_demo/README.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,14 @@ Point at the ADK-native evidence as it streams:
4747

4848
(Re-send the same prompt to show resume reuses the frozen spec — same hash, not re-authored.)
4949

50+
Then run the **quality-gate beat** — send:
51+
52+
```text
53+
Plan a sloppy review: have the reviewer double-check its own findings.
54+
```
55+
56+
The planner authors a *valid* plan (registered capabilities, typed bindings) whose pipeline is `reviewer → reviewer` — and the **plan-quality lint fires on camera**: `🚨 plan-quality: pipeline 'rev' stage 'reviewer' re-checks its own capability's output — same-capability review cannot provide independent verification (self-preferential bias)`, followed by `🛑 Plan rejected by the quality gate — NOT frozen, NOT executed`. Talking point: *plain validation passes; only the structural bias check catches it — before anything runs, and provably.*
57+
5058
### Relationship to ADK Workflow config / `root_agent.yaml` (talking point)
5159

5260
The RFC's direction is to **converge with ADK config where it fits** (RFC #93 → "Relationship to ADK Workflow config / `root_agent.yaml`"; DESIGN §11). The linked `loop_config/root_agent.yaml` sample is the right mental model for the **static** portion: a human-authored `agent_class: Workflow` YAML graph with known `edges`, child YAML files, and function refs like `.agent.route_headline`. #93 should be able to lower/export static graph skeletons toward that style, while the model-facing format stays `WorkflowSpec`.
@@ -75,10 +83,10 @@ Proof points: multi-stage `fan_out → step → step`; branch `step → branch`;
7583
```bash
7684
pytest contributing/samples/workflows/dynamic_supervisor_spike/test_dynamic_supervisor_spike.py -q # 11
7785
pytest contributing/samples/workflows/authored_workflow_spike/test_authoring.py -q # 31
78-
pytest contributing/samples/workflows/authored_workflow_demo/test_demo_agent.py -q # 6
86+
pytest contributing/samples/workflows/authored_workflow_demo/test_demo_agent.py -q # 7
7987
```
8088

81-
- Deterministic suites: #92 **11** + #93 **31** + demo **6** = **48** (incl. a no-LLM reuse-path test, the six-pattern coverage sweep — adversarial verification + tournament via loop-carried `init` — and the plan-quality lints).
89+
- Deterministic suites: #92 **11** + #93 **31** + demo **7** = **49** (incl. a no-LLM reuse-path test, the six-pattern coverage sweep — adversarial verification + tournament via loop-carried `init` — and the plan-quality lints).
8290
- PR #3 CI green except the documented fork-only `agent-triage` token job.
8391

8492
## Recording notes

contributing/samples/workflows/authored_workflow_demo/security_audit_planner/agent.py

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,27 @@ def _registry() -> CapabilityRegistry:
201201
)
202202

203203

204+
# The QUALITY-GATE beat: a deliberately biased ask — the reviewer
205+
# double-checking its OWN findings. Registry/bindings/shapes are all valid, so
206+
# plain validation passes; only the plan-quality lints catch the structural
207+
# self-review bias, and the demo rejects the plan before freezing or running.
208+
_SLOPPY_TRIGGERS = ("sloppy", "self-review", "own findings", "double-check")
209+
_SLOPPY_PLANNER_INSTR = (
210+
"Author a WorkflowSpec using ONLY these capabilities: "
211+
+ _REGISTRY_DESC
212+
+ " The task input has a 'files' list of objects with path and code."
213+
" Author, in order:"
214+
" (1) a pipeline over task.files with two stages, reviewer then reviewer"
215+
" AGAIN — the reviewer double-checks its own findings per item;"
216+
" (2) a step running triager on the pipeline output;"
217+
" (3) a step running formatter on the report."
218+
" Use Binding(source='task', path='files') for the pipeline's over, and"
219+
" Binding(source='step', step=<id>) to chain steps. A pipeline stage takes"
220+
" its input from the previous stage automatically, so stages need no input"
221+
" binding. Set output to the formatter step."
222+
)
223+
224+
204225
def _msg(text: str) -> Event:
205226
return Event(
206227
content=types.Content(role="model", parts=[types.Part(text=text)])
@@ -221,6 +242,53 @@ def _hash(spec: WorkflowSpec) -> str:
221242
async def author_validate_execute(ctx: Context, node_input):
222243
reg = _registry()
223244

245+
# 0. QUALITY-GATE path (checked before load-or-author so it works in any
246+
# session): an adversarial ask makes the planner author a structurally
247+
# biased plan; the lints catch it and the gate rejects it pre-execution.
248+
if any(k in str(node_input or "").lower() for k in _SLOPPY_TRIGGERS):
249+
yield _msg(
250+
"🧭 **Adversarial ask** — authoring a plan where the reviewer"
251+
" double-checks its OWN findings. Watch the quality gate."
252+
)
253+
sloppy = Agent(
254+
name="planner",
255+
model=MODEL,
256+
output_schema=WorkflowSpec,
257+
generate_content_config=DET,
258+
instruction=_SLOPPY_PLANNER_INSTR,
259+
)
260+
raw = await ctx.run_node(
261+
sloppy,
262+
node_input=f"Audit these files: {[f['path'] for f in FILES]}.",
263+
run_id="plan_sloppy",
264+
)
265+
spec = WorkflowSpec.model_validate(raw)
266+
yield _msg(
267+
"📋 **Authored plan** (valid registry refs, valid bindings, valid"
268+
f" shapes):\n```json\n{json.dumps(spec.model_dump(), indent=1)}\n```"
269+
)
270+
lints = [
271+
w
272+
for w in WorkflowSpecValidator(reg).validate(spec)
273+
if w.startswith("plan-quality")
274+
]
275+
if lints:
276+
fired = "\n".join(f" - ⚠️ {w}" for w in lints)
277+
yield _msg(
278+
f"🚨 **Plan-quality lints fired ({len(lints)}):**\n{fired}\n\n🛑"
279+
" **Plan rejected by the quality gate** — NOT frozen, NOT executed."
280+
" Plain validation passed (every capability is registered, every"
281+
" binding is typed); only the structural bias check caught it. In"
282+
" production this triggers a bounded re-plan (`max_replans`)."
283+
)
284+
else:
285+
yield _msg(
286+
"ℹ️ The planner did not author the biased shape this time —"
287+
" re-send the prompt to retry the adversarial ask."
288+
)
289+
yield Event(output={"rejected": bool(lints), "lints": len(lints)})
290+
return
291+
224292
# 1. LOAD-OR-AUTHOR. If a frozen spec exists in this session, REUSE it (do not
225293
# re-author) — this is the resume/reproducibility claim. Otherwise the model
226294
# authors a fresh typed WorkflowSpec (data, not code).

contributing/samples/workflows/authored_workflow_demo/test_demo_agent.py

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,43 @@ def test_demo_spec_validates():
9999
WorkflowSpecValidator(demo._registry()).validate(_demo_spec()) # no raise
100100

101101

102+
def test_quality_gate_rejects_self_review_plan():
103+
# The adversarial-ask beat: reviewer double-checking its OWN findings is a
104+
# VALID plan (registry, bindings, shapes) — only the plan-quality lint
105+
# catches the structural bias. This pins the shape the live beat relies on.
106+
sloppy = WorkflowSpec(
107+
goal="audit",
108+
steps=[
109+
Pipeline(
110+
kind="pipeline",
111+
id="rev",
112+
over=Binding(source="task", path="files"),
113+
stages=[
114+
PipelineStage(capability="reviewer"),
115+
PipelineStage(capability="reviewer"), # self-review
116+
],
117+
),
118+
StepRef(
119+
kind="step",
120+
id="tri",
121+
capability="triager",
122+
input=Binding(source="step", step="rev"),
123+
),
124+
StepRef(
125+
kind="step",
126+
id="fmt",
127+
capability="formatter",
128+
input=Binding(source="step", step="tri"),
129+
),
130+
],
131+
output=Binding(source="step", step="fmt"),
132+
)
133+
warnings = WorkflowSpecValidator(demo._registry()).validate(sloppy)
134+
lints = [w for w in warnings if w.startswith("plan-quality")]
135+
assert len(lints) == 1
136+
assert "re-checks its own capability's output" in lints[0]
137+
138+
102139
def test_demo_spec_quality_lints_clean_and_independent():
103140
# Zero plan-quality lints: verification is by a DIFFERENT capability
104141
# (reviewer -> verifier), and the fan-out is synthesized (triager).

0 commit comments

Comments
 (0)