Skip to content

Commit 1a2957e

Browse files
committed
spike+demo(workflow): contract-hash drift, auditable lint waivers, free-authoring beat
Implements the three maintainer-review items that strengthen gate evidence (the rest stay RFC-design-level): * Contract-hash drift (review #3): Capability.contract_hash() = sha256(input_kind + output schema), recorded per referenced capability in FrozenWorkflowRecord; import rejects contract drift even when the manual version string was never bumped (manual versions demoted to secondary). * Auditable lint suppression (review #6): allow_self_chain capability policy opts legitimate draft->critique->redraft refinement out of the self-review lint; per-plan lint_waivers (node id -> justification) suppress lints AND are recorded in the frozen record/export envelope. * Free-authoring beat (demo review): 'freely/decompose' trigger gives the planner ONLY goal + capability descriptions (no recipe) — the honest model-authored claim; recipe-free instruction pinned by a CI test. Suites: 11 + 34 + 8 = 53 deterministic tests.
1 parent d449d1b commit 1a2957e

8 files changed

Lines changed: 250 additions & 30 deletions

File tree

contributing/samples/workflows/authored_workflow_demo/DEMO_NARRATIVE.md

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,23 @@ Send: **"Plan and run a codebase security review."** The chat streams:
4242
> construct that lets each file flow review→verify independently — item A can be
4343
> verifying while item B is still being reviewed."
4444
45+
## Beat 1b — free authoring (the honest "model-authored" claim)
46+
47+
In a new session, send: **"Freely plan a security review of the files — decompose it yourself."**
48+
49+
```
50+
🧭 Free authoring — the planner receives ONLY the goal + capability
51+
descriptions (reviewer, verifier, triager, formatter); no plan recipe.
52+
The shape below is the model's own decomposition (it may differ run to
53+
run — and the freeze beat then makes THIS run replayable).
54+
```
55+
56+
> "Beat 1 was instruction-guided so the recording is reproducible — the model
57+
> filled in a known shape. This beat is the real claim: goal + capabilities
58+
> in, plan out, no recipe. Whatever it authors, the same validator, lints,
59+
> freeze, and replay machinery apply — free authoring composes with
60+
> everything you're about to see, and the frozen hash pins *this* run."
61+
4562
## Beat 2 — validate (capability allow-list)
4663

4764
```
@@ -210,7 +227,7 @@ Send: **"Plan a sloppy review: have the reviewer double-check its own findings."
210227
> engine at a visible cost (one planner call, the work outside its context);
211228
> the plan **exported** to a portable, defensively-imported audit artifact; and
212229
> a re-send replayed the exact frozen plan with zero planner calls. The
213-
> deterministic test suites — 11 (#92) + 31 (#93) + 7 (demo) — lock all of this
230+
> deterministic test suites — 11 (#92) + 34 (#93) + 8 (demo) — lock all of this
214231
> in CI, including the no-LLM reuse path, the export round-trip / tamper /
215232
> drift checks, the plan-quality lints, and the six-coordination-pattern
216233
> coverage sweep (adversarial verification and tournament included)."
@@ -236,6 +253,6 @@ Send: **"Plan a sloppy review: have the reviewer double-check its own findings."
236253

237254
```bash
238255
pytest contributing/samples/workflows/dynamic_supervisor_spike/test_dynamic_supervisor_spike.py -q # 11
239-
pytest contributing/samples/workflows/authored_workflow_spike/test_authoring.py -q # 31
240-
pytest contributing/samples/workflows/authored_workflow_demo/test_demo_agent.py -q # 7
256+
pytest contributing/samples/workflows/authored_workflow_spike/test_authoring.py -q # 34
257+
pytest contributing/samples/workflows/authored_workflow_demo/test_demo_agent.py -q # 8
241258
```

contributing/samples/workflows/authored_workflow_demo/README.md

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,14 @@ Point at the ADK-native evidence as it streams:
4747

4848
(Re-send the same prompt to show resume reuses the frozen spec — same hash, not re-authored.)
4949

50+
Then run the **free-authoring beat** — in a **new session**, send:
51+
52+
```text
53+
Freely plan a security review of the files — decompose it yourself.
54+
```
55+
56+
The planner receives ONLY the goal + capability descriptions (no plan recipe — `test_free_planner_instruction_is_recipe_free` pins this). The shape may differ run to run; that's the point — and the freeze beat then makes *this* run replayable. Talking point: *the default walkthrough shows the mechanics on a scripted plan; this beat is the honest "model-authored" claim.*
57+
5058
Then run the **quality-gate beat** — send:
5159

5260
```text
@@ -82,11 +90,11 @@ Proof points: multi-stage `fan_out → step → step`; branch `step → branch`;
8290

8391
```bash
8492
pytest contributing/samples/workflows/dynamic_supervisor_spike/test_dynamic_supervisor_spike.py -q # 11
85-
pytest contributing/samples/workflows/authored_workflow_spike/test_authoring.py -q # 31
86-
pytest contributing/samples/workflows/authored_workflow_demo/test_demo_agent.py -q # 7
93+
pytest contributing/samples/workflows/authored_workflow_spike/test_authoring.py -q # 34
94+
pytest contributing/samples/workflows/authored_workflow_demo/test_demo_agent.py -q # 8
8795
```
8896

89-
- Deterministic suites: #92 **11** + #93 **31** + demo **7** = **49** (incl. a no-LLM reuse-path test, the six-pattern coverage sweep — adversarial verification + tournament via loop-carried `init`and the plan-quality lints).
97+
- Deterministic suites: #92 **11** + #93 **34** + demo **8** = **53** (incl. a no-LLM reuse-path test, the six-pattern coverage sweep — adversarial verification + tournament via loop-carried `init` — the plan-quality lints with `allow_self_chain` policy + recorded waivers, contract-hash drift rejection, and the recipe-free free-authoring instruction pin).
9098
- PR #3 CI green except the documented fork-only `agent-triage` token job.
9199

92100
## Recording notes

contributing/samples/workflows/authored_workflow_demo/security_audit_planner/agent.py

Lines changed: 35 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -222,6 +222,26 @@ def _registry() -> CapabilityRegistry:
222222
)
223223

224224

225+
# The FREE-AUTHORING beat: the planner receives ONLY the goal + capability
226+
# descriptions — no plan recipe. This is the honest "model-authored" claim
227+
# (the default _PLANNER_INSTR dictates the shape for recording reliability;
228+
# the spike's demand gate also used free authoring).
229+
_FREE_TRIGGERS = ("freely", "free-form", "your own plan", "decompose")
230+
_FREE_PLANNER_INSTR = (
231+
"Author a WorkflowSpec using ONLY these capabilities: "
232+
+ _REGISTRY_DESC
233+
+ " The task input has a 'files' list of objects with path and code."
234+
" GOAL: audit the files for security issues and produce a one-line"
235+
" report note. Decompose the goal into a plan YOURSELF — no recipe is"
236+
" provided. Choose whichever control blocks fit (step / fan_out /"
237+
" pipeline / branch / loop_until). Binding rules:"
238+
" Binding(source='task', path='files') reads the file list;"
239+
" Binding(source='step', step=<id>) reads a prior step's output; a"
240+
" pipeline stage takes the previous stage's per-item output"
241+
" automatically. Set output to the final step."
242+
)
243+
244+
225245
def _msg(text: str) -> Event:
226246
return Event(
227247
content=types.Content(role="model", parts=[types.Part(text=text)])
@@ -305,18 +325,27 @@ async def author_validate_execute(ctx: Context, node_input):
305325
)
306326
else:
307327
reused = False
328+
free = any(k in str(node_input or "").lower() for k in _FREE_TRIGGERS)
308329
cap_list = ", ".join(f"`{n}`" for n in reg.names())
309-
yield _msg(
310-
"🧭 **Model-authored Workflow** — planning a security audit over "
311-
f"{len(FILES)} files using only registered capabilities "
312-
f"({cap_list})."
313-
)
330+
if free:
331+
yield _msg(
332+
"🧭 **Free authoring** — the planner receives ONLY the goal +"
333+
f" capability descriptions ({cap_list}); no plan recipe. The shape"
334+
" below is the model's own decomposition (it may differ run to"
335+
" run — and the freeze beat then makes THIS run replayable)."
336+
)
337+
else:
338+
yield _msg(
339+
"🧭 **Model-authored Workflow** — planning a security audit over "
340+
f"{len(FILES)} files using only registered capabilities "
341+
f"({cap_list})."
342+
)
314343
planner = Agent(
315344
name="planner",
316345
model=MODEL,
317346
output_schema=WorkflowSpec,
318347
generate_content_config=DET,
319-
instruction=_PLANNER_INSTR,
348+
instruction=_FREE_PLANNER_INSTR if free else _PLANNER_INSTR,
320349
)
321350
raw = await ctx.run_node(
322351
planner,

contributing/samples/workflows/authored_workflow_demo/test_demo_agent.py

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,18 @@ def test_quality_gate_rejects_self_review_plan():
136136
assert "re-checks its own capability's output" in lints[0]
137137

138138

139+
def test_free_planner_instruction_is_recipe_free():
140+
# The honesty contract behind the free-authoring beat: the default
141+
# instruction dictates the plan; the free instruction must NOT — only the
142+
# goal, the capability descriptions, and the binding rules.
143+
assert "reviewer then verifier" in demo._PLANNER_INSTR # scripted (default)
144+
assert "reviewer then verifier" not in demo._FREE_PLANNER_INSTR
145+
assert "(1)" not in demo._FREE_PLANNER_INSTR # no step recipe
146+
assert "YOURSELF" in demo._FREE_PLANNER_INSTR
147+
# trigger sets must not overlap (a prompt can't be both free and sloppy).
148+
assert not set(demo._FREE_TRIGGERS) & set(demo._SLOPPY_TRIGGERS)
149+
150+
139151
def test_demo_spec_quality_lints_clean_and_independent():
140152
# Zero plan-quality lints: verification is by a DIFFERENT capability
141153
# (reviewer -> verifier), and the fan-out is synthesized (triager).

contributing/samples/workflows/authored_workflow_spike/DESIGN.md

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,8 @@ Then **`Graph.validate_graph()`** (reused) handles duplicate names, `START`/reac
131131
- **self-review**: a node (or pipeline stage) consuming output produced by the *same capability* — same-capability review cannot provide independent verification;
132132
- **unsynthesized fan-out**: the terminal output binds a bare per-item `fan_out` never combined or verified downstream.
133133

134+
**Suppression** (so the lints stay credible instead of globally disabled): a capability registered with `allow_self_chain=True` opts out of the self-review lint (legitimate `draft → critique → redraft` refinement), and per-plan `lint_waivers` (node id → justification) are **recorded in the `FrozenWorkflowRecord`** — a suppressed lint is an auditable decision, not a silenced one.
135+
134136
The complementary positive facts (`independence_facts`) are derivable from the frozen spec — e.g. *"stage `verifier` sees ONLY stage `reviewer`'s per-item output"* — which is what lets the frozen record **prove** structural bias controls to an auditor, not just assert them.
135137

136138
## 4. Semantics
@@ -152,7 +154,9 @@ class FrozenWorkflowRecord(BaseModel):
152154
spec_hash: str # sha256(canonical_json(spec)) — see §10
153155
planner_model: str
154156
registry_version: str
155-
capability_versions: dict[str, str]
157+
capability_versions: dict[str, str] # manual bumps — coarse SECONDARY signal
158+
capability_contract_hashes: dict[str, str] # DERIVED sha256(input_kind+output schema) — primary drift signal
159+
lint_waivers: dict[str, str] # node id -> justification; auditable lint suppression
156160
validation: ValidationResult # {passed: bool, warnings: [...]}
157161
created_at: str # ISO-8601, stamped at freeze
158162
task_input_schema: dict | None # expected root task-input schema (enables template reuse)
@@ -243,7 +247,10 @@ Re-runnable: `contributing/samples/workflows/authored_workflow_spike/` (31 deter
243247
# INTEGRITY (never trust the envelope's own `validation`):
244248
# 1. recompute sha256(canonical_json(spec)); REJECT if != envelope["spec_hash"]
245249
# 2. re-run WorkflowSpecValidator against the CURRENT registry
246-
# 3. registry/capability drift -> fail loudly (or explicit migration)
250+
# 3. registry/capability drift -> fail loudly (or explicit migration);
251+
# capability drift = manual version (secondary) AND derived contract
252+
# hash sha256(input_kind + output schema) (primary — catches schema
253+
# changes nobody versioned)
247254
# EXECUTION-INPUT:
248255
# replay : task_input digest must match envelope["task_input_digest"] (else audit-only)
249256
# template : task_input validated against envelope["task_input_schema"] before execution

contributing/samples/workflows/authored_workflow_spike/README.md

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ behind the RFC's "can a model author good plans?" question.
2626
pytest contributing/samples/workflows/authored_workflow_spike/test_authoring.py -q
2727
```
2828

29-
Expected: **31 passed**`Binding` invariant, `max_iters>=1`, validator accepts a
29+
Expected: **34 passed**`Binding` invariant, `max_iters>=1`, validator accepts a
3030
valid spec and rejects unknown capability / non-preceding binding / duplicate id,
3131
the open-map warning, and interpreter execution of fan_out→aggregate, **pipeline (barrier-free per-item review→verify, plus per-stage `max_fan_out` enforcement)**, branch
3232
(correct route), and loop_until (stops + correct output); plus **plan export/import**
@@ -38,7 +38,10 @@ shapes: sequence/loop/leaf by capability name; runtime fan_out/pipeline/branch
3838
flagged no-equivalent rather than fabricated); plus **pattern coverage**
3939
(adversarial verification and tournament via loop-carried `init`, incl. the
4040
no-`init` validation error) and **plan-quality lints** (same-capability
41-
self-review and unsynthesized fan-out warn; an independent plan lints clean).
41+
self-review and unsynthesized fan-out warn; an independent plan lints clean;
42+
`allow_self_chain` policy and recorded per-plan waivers suppress auditably);
43+
plus **contract-hash drift** (import rejects a changed capability schema even
44+
when the manual version string was never bumped).
4245

4346
## Pattern coverage — the six coordination shapes
4447

0 commit comments

Comments
 (0)