Skip to content

Commit 900f8ec

Browse files
authored
feat: add task compatibility preflight (#46)
## What - adds side-effect-free `TaskSupportReport` preflight for built-in and third-party runtimes - keeps `AgentRuntime` migration-compatible through the optional `TaskSupportProvider` protocol - reports granular task capability gaps, configured model allow-list mismatches, and static Antigravity constraints - makes built-in dispatch consume the same support report used by public preflight - adds a machine-readable compatibility manifest tied to `pyproject.toml` and exact `uv.lock` versions ## Why Capability booleans and per-adapter rejection branches had drifted apart. Callers could not discover all incompatible fields before starting a run, provider-specific rules were delayed until dispatch, and compatibility ranges/tested runtime binaries existed only as duplicated prose and package metadata. ## Root cause Task support was encoded in several imperative adapter paths rather than one additive preflight contract. An initial implementation also made `validate_task` mandatory on `AgentRuntime`, which would have broken existing structural third-party runtimes; this PR uses an optional extension plus a declared-capability fallback instead. ## Checks - `ruff check src tests` - `mypy` - `pytest -q --cov=agent_runtime_kit --cov-report=term-missing --cov-fail-under=85` (340 passed, 3 skipped; 90.86%) - installed SDK contracts: 19 passed - `uv lock --check` - `uv build` ## Stack - Base: #45 - Next: #47
1 parent 1863b05 commit 900f8ec

20 files changed

Lines changed: 774 additions & 162 deletions

CHANGELOG.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1616
- All built-in adapters and public fake runtimes now validate returned values
1717
against `output_schema` locally, including textual JSON fallbacks.
1818
- `AgentResult.is_success` provides one canonical success predicate.
19+
- `TaskSupportReport`, `TaskSupportProvider`, and `validate_task()` provide
20+
side-effect-free task preflight without adding a requirement to the
21+
`AgentRuntime` protocol. Registry and `AgentKit` helpers preserve custom
22+
runtime validation and capability-based fallback for older runtimes.
23+
- `COMPATIBILITY_MANIFEST` records each built-in adapter's install extra,
24+
import module, accepted SDK range, tested lockfile version, and runtime binary
25+
versions such as `openai-codex-cli-bin`.
1926

2027
### Changed
2128

@@ -26,6 +33,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
2633
duplicate MCP names, and conflicting tool filters.
2734
- BREAKING: unknown `Usage` values, including cost, are `None` rather than zero;
2835
an explicit provider-reported zero remains `0`/`0.0`.
36+
- Capability declarations now distinguish budget, reasoning-effort, network,
37+
tool-filter, and per-MCP-server environment support. Built-in `run()` methods
38+
consume the same support report used by preflight, eliminating divergent
39+
rejection paths.
2940

3041
### Fixed
3142

@@ -39,6 +50,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
3950
- Permission- and budget-critical vendor options must now be explicit,
4051
introspectable SDK parameters; opaque `**kwargs`, positional-only options,
4152
and uninspectable callables fail closed.
53+
- Configured model allow-lists, Antigravity MCP name syntax, disjoint
54+
allow/deny lists, and Antigravity's legacy reasoning-effort alias are rejected
55+
during static preflight instead of surfacing later or being silently ignored.
4256

4357
## 0.4.0 - 2026-07-02
4458

README.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,14 @@ a field (for example only Claude maps `budget_usd`; Codex and Antigravity reject
141141
it with a typed `UnsupportedTaskInputError`) the adapter raises rather than
142142
silently dropping it.
143143

144+
Call `validate_task(runtime, task)` (or `kit.validate_task("codex", task)`) to
145+
inspect every statically detectable incompatibility before dispatch. The
146+
returned `TaskSupportReport` is side-effect-free and lists source fields such as
147+
`budget_usd` and `permissions.network`; `run()` still fails closed on the first
148+
issue. Third-party runtimes can opt into provider-specific checks with the
149+
`TaskSupportProvider` protocol, while older `AgentRuntime` implementations
150+
continue to work through capability-based fallback checks.
151+
144152
`AgentResult` returns output, finish reason (see `FinishReason`), locally
145153
validated structured output, usage, cost, session id, tool-call audits, and
146154
provider metadata. `is_success` is true only for a natural, error-free

docs/api-stability.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,12 @@ import the names from the top-level package instead.
4444
a task field raises `UnsupportedTaskInputError`; the one exception is
4545
vendor-option drift, which is recorded in
4646
`AgentResult.metadata["dropped_options"]` instead.
47+
- **Task support can be inspected without dispatch.** `validate_task(runtime,
48+
task)`, `RuntimeRegistry.validate_task_for(...)`, and `AgentKit.validate_task(...)`
49+
return every statically detectable incompatibility as a `TaskSupportReport`.
50+
`TaskSupportProvider` is an optional extension, not a new requirement on the
51+
`AgentRuntime` protocol, so existing third-party runtimes retain structural
52+
compatibility and use capability-based fallback checks.
4753
- **`AgentKit` is sugar, not a second API.** The hub assembles the same frozen
4854
`AgentTask` and returns the same `AgentResult` the runtimes produce
4955
(`ParsedResult` is a runtime-identical subclass adding only the typed
@@ -72,6 +78,13 @@ it reaches installed users; a release above a cap is by design invisible to that
7278
lane until the cap is raised. A separate lane installs every direct dependency at
7379
its declared floor so a stale minimum cannot sit undetected in the metadata.
7480

81+
`COMPATIBILITY_MANIFEST` is the machine-readable form of that policy. Each entry
82+
records the install extra, import module, accepted SDK range, exact lockfile
83+
version exercised by the repository, and any separately versioned runtime binary
84+
(currently `openai-codex-cli-bin`). The manifest is tested against both
85+
`pyproject.toml` and `uv.lock`; it is evidence of the committed test baseline,
86+
not a claim that every version in the accepted range was exhaustively tested.
87+
7588
## Deprecation
7689

7790
When a public name is slated for removal it will be kept working for at least one

docs/capability-matrix.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,13 @@ The matrix is intentionally not a lowest-common-denominator contract. Adapters
2020
reject unsupported inputs (see below) when silently dropping them would be
2121
misleading.
2222

23+
The public `AgentCapabilities` value also advertises the granular task controls
24+
used by preflight: `budget`, `reasoning_effort`, `network_control`,
25+
`tool_filters`, and `mcp_server_env`. Use `validate_task(runtime, task)` when a
26+
concrete task is available; unlike a raw capability flag, its
27+
`TaskSupportReport` preserves instance-specific rules such as a configured model
28+
allow-list and reports all detectable issues at once.
29+
2330
For every provider, malformed schemas are rejected locally before dispatch and
2431
returned structured values are validated locally after the SDK completes.
2532
`parsed_output_available` distinguishes valid JSON `null` from no parsed value.
@@ -63,6 +70,13 @@ tool, uses the SDK's `disabled_tools` route).
6370
Each adapter raises `UnsupportedTaskInputError` for task fields it has no SDK
6471
surface to honor, rather than dropping them silently.
6572

73+
The same fields can be checked before dispatch through `validate_task`,
74+
`RuntimeRegistry.validate_task_for`, or `AgentKit.validate_task`. Built-in
75+
adapters additionally preflight configured model allow-lists. Antigravity also
76+
checks its MCP server-name syntax and the SDK's prohibition on combining an
77+
allow-list with a deny-list. SDK-version-dependent Antigravity tool vocabulary
78+
is still validated at dispatch, after the installed SDK supplies its enum.
79+
6680
| Field | Claude | Codex | Antigravity |
6781
|-------|--------|-------|-------------|
6882
| `budget_usd` | Mapped (`max_budget_usd`, fails closed under SDK drift) | Rejected | Rejected |

docs/providers.md

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,8 @@ All three adapters map the task's system prompt (Claude `system_prompt`, Codex
3131
field (falling back to the `metadata` aliases of the same names).
3232
`reasoning_effort` maps to the Claude and Codex `effort` options; Antigravity
3333
has no reasoning-effort control and rejects the first-class field with a typed
34-
error (its legacy `metadata["reasoning_effort"]` alias stays ignored, as it
35-
always has been). Effort values are passed through to the vendor SDK rather
34+
error. Its legacy `metadata["reasoning_effort"]` alias is rejected too, rather
35+
than being silently ignored. Effort values are passed through to the vendor SDK rather
3636
than validated by this library — each vendor defines its own accepted
3737
vocabulary (for example `claude-agent-sdk` 0.2.x accepts
3838
`low`/`medium`/`high`/`xhigh`/`max`), and an SDK too old to accept `effort` at
@@ -42,6 +42,12 @@ Usage fields are `None` when a provider omits its usage breakdown or cost.
4242
Provider-reported zeros remain numeric zero, so callers can distinguish a free
4343
or zero-token run from missing telemetry.
4444

45+
Every built-in adapter exposes a pure `validate_task()` extension. The public
46+
`validate_task(runtime, task)`, registry, and `AgentKit` helpers call that richer
47+
validator when present and otherwise fall back to declared capabilities for
48+
older third-party runtimes. A report is a static preflight, not an availability
49+
or credential probe; installed-SDK drift can still make dispatch fail closed.
50+
4551
Claude uses the `claude-agent-sdk` package and maps working directory,
4652
permissions, filesystem access (a `READ_ONLY` filesystem forces `plan` mode),
4753
MCP servers, sessions, structured output, tool allow/deny lists, runtime

pyproject.toml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ dev = [
7070
"pytest-asyncio>=0.23",
7171
"pytest-cov>=5.0",
7272
"ruff>=0.8",
73+
"tomli>=2; python_version < '3.11'",
7374
"types-jsonschema>=4.18",
7475
]
7576

src/agent_runtime_kit/__init__.py

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,10 +27,19 @@
2727
PermissionProfile,
2828
RuntimeAvailability,
2929
SessionResumeState,
30+
TaskSupportIssue,
31+
TaskSupportProvider,
32+
TaskSupportReport,
3033
ToolCallAudit,
3134
Usage,
3235
runtime_kind_value,
3336
)
37+
from agent_runtime_kit.compatibility import (
38+
COMPATIBILITY_MANIFEST,
39+
PackageVersion,
40+
RuntimeCompatibility,
41+
compatibility_for,
42+
)
3443
from agent_runtime_kit.events import (
3544
output_delta_event,
3645
safe_emit,
@@ -42,6 +51,7 @@
4251
vendor_turn_event,
4352
)
4453
from agent_runtime_kit.registry import RuntimeRegistry, create_default_registry
54+
from agent_runtime_kit.support import validate_task
4555

4656
__all__ = [
4757
"AgentCapabilities",
@@ -59,20 +69,27 @@
5969
"FilesystemAccess",
6070
"FinishReason",
6171
"KIND_ALIASES",
72+
"COMPATIBILITY_MANIFEST",
6273
"McpServerConfig",
6374
"OutputSchemaError",
6475
"OutputTypeError",
76+
"PackageVersion",
6577
"ParsedResult",
6678
"PermissionMode",
6779
"PermissionProfile",
6880
"RuntimeAvailability",
81+
"RuntimeCompatibility",
6982
"RuntimeNotRegisteredError",
7083
"RuntimeRegistry",
7184
"SessionResumeState",
85+
"TaskSupportIssue",
86+
"TaskSupportProvider",
87+
"TaskSupportReport",
7288
"ToolCallAudit",
7389
"UnsupportedTaskInputError",
7490
"Usage",
7591
"create_default_registry",
92+
"compatibility_for",
7693
"runtime_kind_value",
7794
"output_delta_event",
7895
"safe_emit",
@@ -82,4 +99,5 @@
8299
"tool_completed_event",
83100
"tool_requested_event",
84101
"vendor_turn_event",
102+
"validate_task",
85103
]

src/agent_runtime_kit/_kit.py

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,10 @@
2626
PermissionProfile,
2727
RuntimeAvailability,
2828
SessionResumeState,
29+
TaskSupportReport,
2930
)
3031
from agent_runtime_kit.registry import RuntimeFactory, RuntimeRegistry, create_default_registry
32+
from agent_runtime_kit.support import validate_task as validate_runtime_task
3133

3234
_T = TypeVar("_T")
3335
# An event handler receives one normalized event dict; sync or async.
@@ -109,6 +111,21 @@ def availability_for(self, kind: AgentRuntimeKind | str) -> RuntimeAvailability:
109111
def capabilities_for(self, kind: AgentRuntimeKind | str) -> AgentCapabilities:
110112
return self._registry.capabilities_for(self._normalize_kind(kind))
111113

114+
def validate_task(
115+
self,
116+
runtime: AgentRuntimeKind | str | AgentRuntime,
117+
task: AgentTask,
118+
) -> TaskSupportReport:
119+
"""Report unsupported task fields without starting a run."""
120+
121+
if isinstance(runtime, str):
122+
kind = self._normalize_kind(runtime)
123+
cached = self._runtimes.get(kind)
124+
if cached is None:
125+
return self._registry.validate_task_for(kind, task)
126+
runtime = cached
127+
return validate_runtime_task(runtime, task)
128+
112129
def on(self, event: str = "*") -> Callable[[_HandlerT], _HandlerT]:
113130
"""Register an event handler for tasks assembled by this hub.
114131

src/agent_runtime_kit/_runtime.py

Lines changed: 9 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -5,14 +5,14 @@
55
from collections.abc import Mapping
66
from typing import Any
77

8-
from agent_runtime_kit._errors import UnsupportedTaskInputError
98
from agent_runtime_kit._schema import resolve_structured_output
109
from agent_runtime_kit._types import (
1110
AgentCapabilities,
1211
AgentResult,
1312
AgentRuntimeKind,
1413
AgentTask,
1514
RuntimeAvailability,
15+
TaskSupportReport,
1616
ToolCallAudit,
1717
)
1818
from agent_runtime_kit.events import (
@@ -25,6 +25,7 @@
2525
tool_requested_event,
2626
vendor_turn_event,
2727
)
28+
from agent_runtime_kit.support import _validate_declared_task_support, require_task_support
2829

2930

3031
class FakeAgentRuntime:
@@ -47,6 +48,7 @@ def __init__(
4748
streaming=False,
4849
tool_audit=True,
4950
cancellation=True,
51+
mcp_server_env=True,
5052
)
5153
self._output = output
5254
self._metadata = dict(metadata or {})
@@ -57,12 +59,17 @@ def availability(self) -> RuntimeAvailability:
5759

5860
return RuntimeAvailability.ok(self.kind, package="agent-runtime-kit")
5961

62+
def validate_task(self, task: AgentTask) -> TaskSupportReport:
63+
"""Report unsupported fields without side effects."""
64+
65+
return _validate_declared_task_support(self.kind, self.capabilities, task)
66+
6067
async def run(self, task: AgentTask) -> AgentResult:
6168
"""Return a deterministic result after validating capabilities."""
6269

6370
await safe_emit(task, task_started_event(task, self.kind))
6471
try:
65-
_ensure_supported(self.kind, self.capabilities, task)
72+
require_task_support(self.validate_task(task))
6673
output = self._output if self._output is not None else f"Fake result for: {task.goal}"
6774
parsed = {"output": output} if task.output_schema is not None else None
6875
parsed_available = parsed is not None
@@ -139,30 +146,3 @@ async def __aenter__(self) -> FakeAgentRuntime:
139146

140147
async def __aexit__(self, _exc_type: object, _exc: object, _tb: object) -> None:
141148
await self.aclose()
142-
143-
144-
def _ensure_supported(
145-
kind: AgentRuntimeKind,
146-
capabilities: AgentCapabilities,
147-
task: AgentTask,
148-
) -> None:
149-
if task.mcp_servers and not capabilities.mcp_support:
150-
raise UnsupportedTaskInputError(kind, "mcp_servers", "runtime does not support MCP")
151-
if task.working_directory is not None and not capabilities.working_directory:
152-
raise UnsupportedTaskInputError(
153-
kind,
154-
"working_directory",
155-
"runtime does not support per-task working directories",
156-
)
157-
if (task.session_id or task.resume_from) and not capabilities.session_resume:
158-
raise UnsupportedTaskInputError(
159-
kind,
160-
"session_id",
161-
"runtime does not support session resume",
162-
)
163-
if task.output_schema is not None and not capabilities.structured_output:
164-
raise UnsupportedTaskInputError(
165-
kind,
166-
"output_schema",
167-
"runtime does not support structured output",
168-
)

src/agent_runtime_kit/_types.py

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -226,6 +226,56 @@ class AgentCapabilities:
226226
streaming: bool = False
227227
tool_audit: bool = False
228228
cancellation: bool = False
229+
budget: bool = False
230+
reasoning_effort: bool = False
231+
network_control: bool = False
232+
tool_filters: bool = False
233+
mcp_server_env: bool = False
234+
235+
236+
@dataclass(frozen=True)
237+
class TaskSupportIssue:
238+
"""One task field a runtime cannot honor faithfully."""
239+
240+
field: str
241+
message: str
242+
243+
def __post_init__(self) -> None:
244+
_require_nonblank(self.field, "TaskSupportIssue.field")
245+
_require_nonblank(self.message, "TaskSupportIssue.message")
246+
247+
248+
@dataclass(frozen=True)
249+
class TaskSupportReport:
250+
"""Pure compatibility result for one task/runtime pair."""
251+
252+
kind: AgentRuntimeKind | str
253+
issues: tuple[TaskSupportIssue, ...] = ()
254+
255+
def __post_init__(self) -> None:
256+
issues = _tuple_value(self.issues, "TaskSupportReport.issues")
257+
if not all(isinstance(issue, TaskSupportIssue) for issue in issues):
258+
raise ValueError(
259+
"TaskSupportReport.issues must contain only TaskSupportIssue values"
260+
)
261+
object.__setattr__(self, "kind", AgentRuntimeKind.coerce(self.kind))
262+
object.__setattr__(self, "issues", issues)
263+
264+
@property
265+
def supported(self) -> bool:
266+
return not self.issues
267+
268+
269+
@runtime_checkable
270+
class TaskSupportProvider(Protocol):
271+
"""Optional extension for runtimes with provider-specific task checks.
272+
273+
``AgentRuntime`` deliberately does not require this protocol: third-party
274+
runtimes written before task preflight was introduced remain compatible.
275+
"""
276+
277+
def validate_task(self, task: AgentTask) -> TaskSupportReport:
278+
"""Purely report whether this runtime can honor the task."""
229279

230280

231281
@dataclass(frozen=True)

0 commit comments

Comments
 (0)