Add capture: response to store an agent's final response text as output

Tasks can set `capture: response` to publish the agent's final response
text (the prose after its last tool call) as their named `outputs.<id>`
instead of their final tool result. This applies uniformly to a plain
task's scalar output and to each branch's result in a fan-out task's
fan-in list, and composes with the `outputs` JSON Schema. The default,
`capture: tool_result`, is unchanged.

The stream driver accumulates response text and resets it on each tool
call, then forwards the final segment through a new record_message sink
threaded from deploy_task_agents; the capture layer wraps it as a
synthetic tool result so response and tool-result capture share one
decode/validate path. Response capture is rejected on shell tasks.

Add an example_model_comparison taskflow that fans a question across
models, captures each model's answer, and has a judge task compare them,
with runnable walkthrough docs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: anticomputer

README.md

@@ -237,6 +237,28 @@ The framework includes a [CodeQL](https://codeql.github.com/) MCP server that ca
 
 Instead of generating CodeQL queries itself, the CodeQL MCP Server is used to provide CodeQL-query based MCP tools that allow an Agent to navigate and explore code. It leverages templated CodeQL queries to provide targeted context for model driven code analysis.
 
+### Model comparison and evaluation
+
+The framework can also run the same prompt across several models and compare their answers, which is useful for evaluating models against each other on a given task. The [example_model_comparison](examples/taskflows/example_model_comparison.yaml) taskflow shows the pattern: a multi-model task fans the same question out across two models, captures each model's prose answer with `capture: response`, and a second "judge" task reads every captured answer by name (`outputs.answers`) and picks the best one.
+
+Run it against your endpoint like any other taskflow:
+
+```bash
+python -m seclab_taskflow_agent -t examples.taskflows.example_model_comparison
+```
+
+Each model answers in its own labelled output block, and the judge task then compares them, for example:
+
+```
+** 🤖✏️ Output for gpt_alt
+A use-after-free bug is a type of software vulnerability that occurs when a program continues to use a memory location after it has been freed ...
+** 🤖✏️ Output for gpt_fast
+A use-after-free bug occurs when a program continues to use memory after it has been freed ...
+The gpt_fast answer is best because it is more concise while still clearly explaining ...
+```
+
+The two models are defined in [multi_model.yaml](examples/model_configs/multi_model.yaml); swap the provider IDs there to compare whatever models your endpoint exposes. `capture: response` is what makes the models' prose answers (rather than a tool result) available to the judge; see the "Capturing the response instead of a tool result" section of [doc/GRAMMAR.md](doc/GRAMMAR.md) for details.
+
 ## Requirements
 
 Python >= 3.10 or Docker

doc/GRAMMAR.md

@@ -488,6 +488,43 @@ Notes:
   task does not feed it: there is no single "last" result across models, so a
   downstream task must consume its output by name via `id`/`over`.
 
+### Capturing the response instead of a tool result (`capture`)
+
+By default a task's named output (`outputs.<id>`) is its final **tool result**.
+Some tasks have no meaningful tool result to capture: an evaluation or
+comparison flow just wants the model's **prose answer**. Set `capture: response`
+to store the agent's final response text instead.
+
+- `capture: tool_result` (default) captures the task's last tool result, as
+  described above.
+- `capture: response` captures the agent's final response text: the prose it
+  emits after its last tool call (or the whole answer when it calls no tools).
+  This applies uniformly to the scalar output of a plain task and to each
+  branch's `result` in a fan-out task's fan-in list.
+
+`capture` composes with everything else: an `outputs` schema still validates the
+captured value (the response text is decoded as JSON first, so a schema-typed
+response task must emit JSON), and `id` still names it for later tasks. It does
+not apply to shell (`run`) tasks, which have no agent response.
+
+```yaml
+  - task:
+      # ask several models the same question; capture each prose answer
+      id: answers
+      models: [gpt_fast, gpt_alt]
+      capture: response
+      agents: [seclab_taskflow_agent.personalities.assistant]
+      user_prompt: "In one sentence, what is a use-after-free bug?"
+  - task:
+      # a judge model reads every captured answer by name
+      agents: [seclab_taskflow_agent.personalities.assistant]
+      user_prompt: |
+        Rank these answers and explain your pick:
+        {% for a in outputs.answers %}
+        - {{ a.model }}: {{ a.result }}
+        {% endfor %}
+```
+
 ### Toolboxes / MCP Servers
 
 Toolboxes are MCP server configurations. They can be defined at the Agent level or overridden at the task level. These MCP servers are started and made available to the Agents in the Agents list during a Task. The `toolboxes` field should contain a list of files for the `toolboxes` that are available for the task:

examples/taskflows/example_model_comparison.yaml

@@ -0,0 +1,50 @@
+# SPDX-FileCopyrightText: GitHub, Inc.
+# SPDX-License-Identifier: MIT
+
+# Example: a simple model evaluation / comparison flow.
+#   1. Ask several models the SAME question in parallel and capture each
+#      model's prose answer (capture: response).
+#   2. A single "judge" model reads every captured answer by name and picks
+#      the best one.
+# This showcases multi-model fan-out, response capture, fan-in via
+# outputs.<id>, and an `if` gate. See doc/GRAMMAR.md "Multiple Models",
+# "Typed named outputs", and "Capturing the response instead of a tool result".
+
+seclab-taskflow-agent:
+  version: "1.0"
+  filetype: taskflow
+
+model_config: examples.model_configs.multi_model
+
+globals:
+  question: "In one sentence, what is a use-after-free bug?"
+
+taskflow:
+  # Fan out the same question across both models. capture: response stores each
+  # model's final prose answer, so outputs.answers becomes a fan-in list of
+  # {model, item, result} records -- one per model -- ready to compare.
+  - task:
+      id: answers
+      models: [gpt_fast, gpt_alt]
+      completion: all
+      capture: response
+      agents:
+        - seclab_taskflow_agent.personalities.assistant
+      user_prompt: "{{ globals.question }}"
+
+  # A single judge model compares the captured answers by name. Runs only if at
+  # least one model answered.
+  - task:
+      if: "outputs.answers | length > 0"
+      agents:
+        - seclab_taskflow_agent.personalities.assistant
+      user_prompt: |
+        You are judging candidate answers to this question:
+        "{{ globals.question }}"
+
+        The candidates, one per model:
+        {% for a in outputs.answers %}
+        - {{ a.model }}: {{ a.result }}
+        {% endfor %}
+
+        In two sentences, say which answer is best and why.

src/seclab_taskflow_agent/_stream.py

@@ -74,6 +74,7 @@ async def drive_backend_stream(
     max_rate_limit_backoff: int,
     record_tool_result: Any = None,
     record_usage: Any = None,
+    record_message: Any = None,
 ) -> None:
     """Run the backend's event stream to completion with retry/backoff.
 
@@ -85,12 +86,24 @@ async def drive_backend_stream(
     :class:`BackendTimeoutError`, and applies exponential backoff up to
     *max_rate_limit_backoff* seconds on :class:`BackendRateLimitError`
     before giving up with a :class:`BackendTimeoutError`.
+
+    When *record_message* is provided, the agent's final response text is
+    forwarded to it once the stream completes successfully. "Final response"
+    is the prose emitted after the last tool call: text deltas are accumulated
+    and reset on each ``ToolEnd``, so a task that ends on a tool call yields
+    only that trailing summary, and a plain question-and-answer turn yields the
+    whole answer. This is what ``capture: response`` tasks store as their named
+    output.
     """
     max_retry = max_api_retry
     rate_limit_backoff = initial_rate_limit_backoff
     last_rate_limit_exc: BackendRateLimitError | None = None
 
     while rate_limit_backoff:
+        # Accumulate the agent's final response text per attempt (reset on a
+        # retry, and on each tool call, so we keep only the prose after the
+        # last tool result -- the model's final answer).
+        final_message_parts: list[str] = []
         try:
             stream = backend_impl.run_streamed(
                 agent_handle, prompt, max_turns=max_turns
@@ -110,10 +123,12 @@ async def drive_backend_stream(
                         ) from exc
                     watchdog_ping()
                     if isinstance(event, TextDelta):
+                        final_message_parts.append(event.text)
                         await render_model_output(
                             event.text, async_task=async_task, task_id=task_id
                         )
                     elif isinstance(event, ToolEnd):
+                        final_message_parts.clear()
                         await handle_tool_end_event(event, run_hooks, record_tool_result)
                     elif isinstance(event, TokenUsage):
                         logging.info(
@@ -138,6 +153,8 @@ async def drive_backend_stream(
                     except Exception:  # noqa: BLE001 - best-effort cleanup
                         logging.exception("Failed to aclose backend stream iterator")
             await render_model_output("\n\n", async_task=async_task, task_id=task_id)
+            if record_message is not None:
+                await record_message("".join(final_message_parts))
             return
         except BackendTimeoutError:
             if not max_retry:

src/seclab_taskflow_agent/models.py

@@ -43,6 +43,12 @@
 # single branch succeeding is enough.
 CompletionPolicy = Literal["all", "any"]
 
+# Which produced value a task captures as its named ``outputs.<id>``:
+# ``tool_result`` (default) captures the task's final tool result; ``response``
+# captures the agent's final response text (the prose after its last tool
+# call), which is what model-comparison / evaluation flows want.
+CaptureSource = Literal["tool_result", "response"]
+
 
 # ---------------------------------------------------------------------------
 # Header
@@ -125,6 +131,11 @@ class TaskDefinition(BaseModel):
     # value; when set, the captured output is validated and exposed to
     # later tasks as ``outputs.<id>``.
     outputs: dict[str, Any] = Field(default_factory=dict)
+    # Which produced value feeds ``outputs.<id>``: the task's final tool result
+    # (default) or the agent's final response text (``response``). Response
+    # capture is what side-by-side model comparison / evaluation flows want,
+    # since the value to compare is the model's prose answer, not a tool result.
+    capture: CaptureSource = "tool_result"
     # Explicit iterable selector for repeat_prompt: a Jinja expression
     # evaluated against the template context (e.g. ``outputs.list_fns.items``).
     over: str = ""
@@ -183,6 +194,11 @@ def _coerce_models(cls, v: Any) -> list[Any]:
     def _run_xor_prompt(self) -> TaskDefinition:
         if self.run and self.user_prompt:
             raise ValueError("shell task ('run') and prompt task ('user_prompt') are mutually exclusive")
+        if self.capture == "response" and self.run:
+            raise ValueError(
+                "capture: response captures an agent's final response text and does not "
+                "apply to a shell task ('run'); use the default capture: tool_result"
+            )
         return self
 
     @model_validator(mode="after")