|
| 1 | +# Authored Workflow Spike — demand gate for RFC #93 |
| 2 | + |
| 3 | +Reference spike for **agent-authored typed Workflows** (RFC #93): a model emits |
| 4 | +a declarative, validated `WorkflowSpec` (typed data, **not** code) that the |
| 5 | +framework validates and executes on the real ADK Workflow engine via the #92 |
| 6 | +`DynamicNodeSupervisor`. This directory is the re-runnable demand-gate artifact |
| 7 | +behind the RFC's "can a model author good plans?" question. |
| 8 | + |
| 9 | +## Environment |
| 10 | + |
| 11 | +- ADK: `2.1.0` |
| 12 | +- Built against `google/adk-python` upstream `main`. |
| 13 | +- Python 3.11+ (recursive `kind`-tagged unions; `asyncio.TaskGroup` in #92). |
| 14 | + |
| 15 | +## Files |
| 16 | + |
| 17 | +| File | Purpose | |
| 18 | +| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | |
| 19 | +| `authoring.py` | `WorkflowSpec` (plain `kind`-tagged recursive tree), `CapabilityRegistry`, `WorkflowSpecValidator`, `SpecInterpreter` (step / fan_out / branch / loop_until). | |
| 20 | +| `test_authoring.py` | Deterministic, CI-safe tests (no LLM). The trustworthy artifact. | |
| 21 | +| `test_live_planner_sweep.py` | OPTIONAL env-gated live planner sweep across plan shapes. | |
| 22 | + |
| 23 | +## Deterministic tests (CI-safe, no network) |
| 24 | + |
| 25 | +```bash |
| 26 | +pytest contributing/samples/workflows/authored_workflow_spike/test_authoring.py -q |
| 27 | +``` |
| 28 | + |
| 29 | +Expected: **10 passed** — `Binding` invariant, `max_iters>=1`, validator accepts a |
| 30 | +valid spec and rejects unknown capability / non-preceding binding / duplicate id, |
| 31 | +the open-map warning, and interpreter execution of fan_out→aggregate, branch |
| 32 | +(correct route), and loop_until (stops + correct output). |
| 33 | + |
| 34 | +## Live planner sweep (optional evidence) |
| 35 | + |
| 36 | +Skipped unless configured — no hardcoded project/model: |
| 37 | + |
| 38 | +```bash |
| 39 | +export SPIKE_LIVE=1 GOOGLE_GENAI_USE_VERTEXAI=1 |
| 40 | +export GOOGLE_CLOUD_PROJECT=<project> GOOGLE_CLOUD_LOCATION=global |
| 41 | +export SPIKE_GEMINI_MODEL=gemini-3.5-flash # 3.5 serves from `global` |
| 42 | +pytest contributing/samples/workflows/authored_workflow_spike/test_live_planner_sweep.py -q -s |
| 43 | +``` |
| 44 | + |
| 45 | +## Gate results (run on `gemini-3.5-flash`) |
| 46 | + |
| 47 | +**Initial gate (codebase audit):** planner authored a valid, sensible, executable |
| 48 | +plan (`fan_out reviewer → triager`) matching a hand-wired baseline. **PASS.** |
| 49 | + |
| 50 | +**Shape sweep (this directory):** the planner authored + validated + executed all |
| 51 | +three shapes: |
| 52 | + |
| 53 | +| Shape | Authored steps | Result | |
| 54 | +| ----------- | ----------------------- | --------------------------------------- | |
| 55 | +| multi-stage | `fan_out → step → step` | report → formatted note | |
| 56 | +| branch | `step → branch` | took the matched route, produced a note | |
| 57 | +| loop_until | `loop_until` | iterated to a headline | |
| 58 | + |
| 59 | +## Findings that fell out (and shaped the RFC) |
| 60 | + |
| 61 | +1. **Open-ended `dict[str, X]` maps are a structured-output reliability hazard.** |
| 62 | + Surfaced **twice**: a capability's `counts: dict[str,int]` came back empty, and |
| 63 | + the spec's own `Branch.routes: dict[str, list]` came back empty. **Both fixed by |
| 64 | + using enumerated/list structures** — capability outputs use fixed severity |
| 65 | + fields; `Branch.routes` is now a `list[Route]`, not a map. The validator also |
| 66 | + warns on open-map capability outputs. |
| 67 | +1. **The strict `unmatched=fail` branch contract earns its keep** — when the planner |
| 68 | + bound a branch switch to a whole object instead of its field, execution failed |
| 69 | + loudly instead of silently mis-routing. |
| 70 | +1. **Gemini `response_schema` rejects Pydantic's `Field(discriminator=...)`.** The |
| 71 | + plan vocabulary is a PLAIN union of models that each carry a `kind` literal (a |
| 72 | + *structurally-tagged* union). The strict discriminated form emits a |
| 73 | + `discriminator` keyword that genai's `response_schema` refuses |
| 74 | + (`Schema: extra_forbidden`, verified on `gemini-3.5-flash`); the `kind` tags |
| 75 | + still make parsing and switching unambiguous. |
| 76 | +1. **Planning vs capability quality are separable** — authoring/structure was |
| 77 | + reliably good; the residual variance was per-capability output quality |
| 78 | + (prompts/schemas/retries), not planning. |
| 79 | + |
| 80 | +This is a demand-gate artifact, not production code. |
0 commit comments