Skip to content

Rectify: Codex L2 Liveness Policy#4217

Merged
Trecek merged 12 commits into
developfrom
codex-l2-liveness-supervision-4202
Jul 9, 2026
Merged

Rectify: Codex L2 Liveness Policy#4217
Trecek merged 12 commits into
developfrom
codex-l2-liveness-supervision-4202

Conversation

@Trecek

@Trecek Trecek commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Codex L2 food trucks can be killed by the raw stdout byte-idle watchdog while doing legitimate long MCP work. The immediate failure is not just a low timeout; it is an architectural mismatch between independent timeout scalars and actual backend liveness.

Today, Codex builders copy run_skill.stream_idle_timeout_ms into CmdSpec.process_idle_timeout_ms, and _execute_claude_headless() treats that value as a lower cap on the outer AutoSkillit process idle watchdog. For Codex, that collapses default L2 food-truck idle from fleet's intended 1800s to 600s, while run_skill.mcp_tool_timeout_sec and fleet session deadlines allow much longer blocking calls. Codex item.started / item.updated status=in_progress MCP records are ignored as liveness, so a healthy in-flight MCP call can look identical to a hung stdout stream.

The immunity plan is to replace raw timeout-field propagation with one resolved session liveness contract plus a process-level liveness supervisor. The contract must be resolved once per backend and session kind, then consumed by every watcher. Codex operation lifecycle events must become first-class liveness inputs, bounded by MCP/session deadlines rather than by an arbitrary marker-suppression cap.

Implementation Plan

Plan file: /home/talon/projects/autoskillit-runs/remediation-20260708-205019-355728/.autoskillit/temp/rectify/rectify_codex_l2_liveness_policy_2026-07-08_211800.md

Closes #4202

🤖 Generated with Claude Code via AutoSkillit

@Trecek Trecek left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AutoSkillit PR Review — Verdict: changes_requested (own-PR guard applied: posted as COMMENT event). 1 critical + 14 warning + 2 decision-required findings posted as inline comments; 11 info findings omitted from inline posts (no line-rendered bug; surface in summary). Resolve and re-request review.

Comment thread tests/execution/process/test_liveness_supervisor.py Outdated
Comment thread tests/contracts/test_headless_liveness_contracts.py
Comment thread tests/execution/test_process_idle_watchdog.py
Comment thread tests/execution/test_process_deadline_extension.py
Comment thread tests/execution/test_headless_provider_fallback.py
Comment thread src/autoskillit/execution/headless/_headless_liveness.py Outdated
Comment thread src/autoskillit/execution/process/_process_race.py Outdated
Comment thread src/autoskillit/execution/process/_liveness_supervisor.py Outdated
Comment thread src/autoskillit/execution/process/_liveness_supervisor.py
Comment thread src/autoskillit/execution/headless/_headless_liveness.py

@Trecek Trecek left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AutoSkillit PR Review — Verdict: changes_requested (own-PR guard applied: posted as COMMENT event). 1 critical + 14 warning + 2 decision-required findings posted as inline comments; 11 info findings omitted from inline posts (no line-rendered bug; surface in summary). Resolve and re-request review.

Comment thread tests/execution/process/test_liveness_supervisor.py Outdated
Comment thread tests/contracts/test_headless_liveness_contracts.py
Comment thread tests/execution/test_process_idle_watchdog.py
Comment thread tests/execution/test_process_deadline_extension.py
Comment thread tests/execution/test_headless_provider_fallback.py
Comment thread src/autoskillit/execution/headless/_headless_liveness.py Outdated
Comment thread src/autoskillit/execution/process/_process_race.py Outdated
Comment thread src/autoskillit/execution/process/_liveness_supervisor.py Outdated
Comment thread src/autoskillit/execution/process/_liveness_supervisor.py
Comment thread src/autoskillit/execution/headless/_headless_liveness.py

@Trecek Trecek left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AutoSkillit PR Review — Verdict: approved_with_comments

Comment thread tests/execution/test_headless_provider_fallback.py
Comment thread tests/execution/test_process_session_log_monitor.py
Comment thread tests/contracts/test_headless_liveness_contracts.py
Comment thread tests/execution/test_process_session_log_monitor.py
if explicit_idle_disabled:
return None, True
if value is not None and value > 0:
return float(value), False

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[warning] bugs: _normalize_caller_idle with value=None, default=0 returns (None, False). The is_idle_disabled property returns True because stdout_idle_timeout_sec is None, which is correct — but explicit_idle_disabled=False is misleading. The field semantics say 'explicit caller disabled' but a None config default causes the same is_idle_disabled=True outcome without setting the explicit flag. Data model conflates two conditions.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid observation — flagged for design decision.

pytestmark = [pytest.mark.layer("contracts"), pytest.mark.small]


def _resolver_inputs_from_config(

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[warning] cohesion: Production/test wiring duplication — resolver_inputs_from_config re-declares the same ResolverInputs(run_skill, fleet_, enable_deadline_extension, caller_idle_output_timeout, caller_session_id, is_food_truck) wiring that resolve_session_liveness_spec builds from cfg. The two copies will drift the moment a new ResolverInputs field is added. The contract test should call a single ResolverInputs.from_config(cfg, *, is_food_truck, caller_idle_output_timeout, ...) factory.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid observation — flagged for design decision.

Comment thread src/autoskillit/execution/headless/_headless_liveness.py
Comment thread src/autoskillit/execution/headless/_headless_liveness.py Outdated
Comment thread src/autoskillit/execution/backends/codex.py


@pytest.mark.anyio
async def test_operation_deadline_clamps_child_activity_extension(monkeypatch) -> None:

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[warning] tests: test_operation_deadline_clamps_child_activity_extension — the 10ms slack on 500ms deadline is workable but the 'unclamped_deadline' assertion is heuristic (relies on probe_times[-1] + poll_interval * 2). Under xdist the recorded probe_times[-1] may not represent the latest probe. Tighten by checking the supervisor floor exactly.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid observation — flagged for design decision.

@Trecek Trecek left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AutoSkillit review: warning-only findings detected. See inline comments — no blocking changes required.

⚠️ Outside Diff Range

These findings target lines not in the diff and could not be posted as inline comments:

tests/execution/test_liveness_supervisor.py

  • L0 [warning/tests]: tests/execution/process/test_liveness_supervisor.py — missing coverage: test_publish_event_failed_drops_operation and te

tests/execution/test_process_session_log_monitor_dispatch_marker.py

  • L0 [info/bugs]: test_supervisor_suppresses_stale_before_dispatch_marker_check expects STALE — but the test name says 'suppresses stale b

@Trecek Trecek added this pull request to the merge queue Jul 9, 2026
Merged via the queue into develop with commit f5148ae Jul 9, 2026
5 of 6 checks passed
@Trecek Trecek deleted the codex-l2-liveness-supervision-4202 branch July 9, 2026 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant