Skip to content

eager session-id persistence via on_session_id_resolved callback#4078

Merged
Trecek merged 1 commit into
developfrom
dispatched-session-id-not-persisted-on-signal-interruption-d/4072
Jun 12, 2026
Merged

eager session-id persistence via on_session_id_resolved callback#4078
Trecek merged 1 commit into
developfrom
dispatched-session-id-not-persisted-on-signal-interruption-d/4072

Conversation

@Trecek

@Trecek Trecek commented Jun 12, 2026

Copy link
Copy Markdown
Collaborator

The dispatch system has a structural weakness where runtime-discovered identity fields (dispatched_session_id, session_chain, dispatched_session_log_dir, token_usage, resume_checkpoint, branch_name) are only written to state on the normal completion path (_api.py:927 via upsert_dispatch_record_by_name). On signal interruption (CancelledError), only status, reason, and ended_at are written via mark_dispatch_interrupted(). This leaves the manifest non-resumable because downstream consumers (state recovery, fleet session, label cleanup) rely on dispatched_session_id being non-empty.

Part A addresses the core architectural immunity: eager session-identity persistence via an on_session_id_resolved callback fired from inside the live-execution task group (mirroring the existing on_pid_resolved pattern), mark_dispatch_interrupted signature extension for defense-in-depth identity capture, and test coverage that makes this bug class instantly caught. Part B (separate task) will cover R3 (sessions.jsonl fallback lookup), R4 (L3 orchestrator dispatch history visibility), and the remaining deferred-write fields (token_usage, resume_checkpoint, branch_name).

Requirements

dispatched_session_id is recorded as "" (empty string) in the dispatch manifest when a signal interrupts the dispatch — even when the headless session completed all pipeline steps successfully. This makes the dispatch non-resumable and, combined with the stale in-progress GitHub label, prevents the orchestrator from resuming the interrupted session. Recovery requires manual reset_dispatch(force=True) followed by a fresh dispatch, losing the ability to continue from where the session left off.

R1: Eager dispatched_session_id Write on Interruption

Extend mark_dispatch_interrupted() to accept an optional dispatched_session_id parameter. In the except asyncio.CancelledError handler at _api.py:731:

  1. Guard against NameError: skill_result is assigned inside the try block at line 698. If CancelledError fires BEFORE dispatch_food_truck() returns, skill_result is never assigned. The handler must use skill_result = locals().get("skill_result") or pre-initialize skill_result = None before the try block.
  2. If skill_result is available and has a non-empty session_id, pass it to mark_dispatch_interrupted().
  3. Also capture session_chain — extend mark_dispatch_interrupted() to accept an optional session_chain parameter, or add a separate set_dispatch_session_identity() state mutation function.

This follows the same eager-write pattern used for PID capture via mark_dispatch_running() and for issue_url capture added in PR #3828.

R2: Intermediate Session ID Write at Process Layer

When Channel B or stdout extraction succeeds during the race monitoring phase, write dispatched_session_id to the manifest immediately — before SubprocessResult is fully constructed. This covers the case where a signal arrives during the session (before dispatch_food_truck() returns) but the session ID is already known.

Architectural constraint: The execution package is IL-1; fleet state is IL-2. Direct imports from executionfleet would violate import layering. This must be implemented via a callback pattern: _run_dispatch passes a on_session_id_resolved: Callable[[str], None] callback down to the process runner, which invokes it when Channel B or stdout extraction succeeds. The callback closes over state_path and effective_name and calls a new state mutation function. This mirrors the existing on_pid_resolved callback pattern at _api.py:657-679.

R3: sessions.jsonl Fallback Lookup

When dispatched_session_id is empty on a resumable/interrupted dispatch that has a non-empty dispatch_id, search sessions.jsonl for entries matching that dispatch_id to recover the session_id. Defense-in-depth, not a substitute for R1/R2.

R4: L3 Orchestrator Dispatch History Visibility

Give the L3 fleet dispatch orchestrator read access to the history of dispatches it launched and their current status. Currently, the L3 has no mechanism to query dispatch state — it relies entirely on the return value of dispatch_food_truck, which is lost on interruption. The orchestrator should be able to:

  • Query dispatches by caller_session_id (its own session) or campaign_id
  • See the current status, dispatched_session_id, dispatched_pid, and branch_name for each dispatch
  • Use this information to make informed resume/reset/skip decisions when re-entering after a crash

This would eliminate the class of bugs where the L3 orchestrator has no context about what happened during an interrupted dispatch and must rely on indirect signals (labels, sidecar files) that may themselves be stale.

R5: Test Coverage

  • Add assertion on dispatched_session_id in TestCancelledErrorRecordsInterruptedState
  • Add test for post-completion race window: CancelledError after dispatch_food_truck() returns but before upsert_dispatch_record_by_name()
  • Add test verifying session_chain is also captured on the interruption path
  • Add test for the resume-blocked scenario: INTERRUPTED + empty dispatched_session_id + stale in-progress label → verify reset_dispatch recovery works

Closes #4072

Implementation Plan

Plan file: /home/talon/projects/autoskillit-runs/remediation-20260611-155102-572761/.autoskillit/temp/rectify/rectify_dispatch_session_id_signal_persistence_2026-06-11_155102.md

🤖 Generated with Claude Code via AutoSkillit

Token Usage Summary

Step Model count uncached output cache_read peak_ctx turns cache_write time
resolve_review* sonnet 3 535 25.6k 3.6M 80.0k 150 209.4k 14m 2s
dispatch:44d3705c-3f62-459b-af5b-9678f2bbcb64* sonnet 1 114 3.6k 1.2M 94.6k 28 94.9k 23m 9s
plan* opus[1m] 5 2.3k 90.0k 5.3M 115.0k 188 374.7k 53m 29s
verify* sonnet 5 1.2k 58.2k 2.4M 74.1k 123 213.1k 28m 36s
implement* MiniMax-M3 4 6.1M 43.5k 0 0 227 0 20m 34s
audit_impl* sonnet 4 4.8k 29.3k 869.8k 43.8k 63 113.0k 16m 32s
prepare_pr* MiniMax-M3 3 741.4k 7.9k 0 0 44 0 3m 14s
compose_pr* MiniMax-M3 3 604.9k 4.9k 0 0 39 0 2m 18s
review_pr* sonnet 5 758 99.0k 4.7M 92.8k 218 250.3k 26m 56s
dispatch:0145c757-e69c-4878-929a-ba4acb07e3d1* sonnet 1 322 13.7k 3.0M 86.3k 84 62.0k 42m 39s
fix* sonnet 2 332 16.9k 2.6M 98.2k 109 117.9k 13m 42s
dispatch:341f411a-3d76-4fcf-88a3-15a64c6d7bd4* sonnet 1 354 10.0k 3.4M 86.4k 89 62.0k 1h 18m
dispatch:7e27c189-e656-4ec2-b528-b128157c59a8* sonnet 1 490 17.2k 5.0M 98.3k 122 73.9k 1h 26m
Total 7.4M 419.6k 31.9M 115.0k 1.6M 6h 50m

* Step used a non-Anthropic provider; caching behavior may differ.

Token Efficiency

Step LoC Changed cache_read/LoC cache_write/LoC output/LoC
resolve_review 48 74473.8 4362.5 533.1
dispatch:44d3705c-3f62-459b-af5b-9678f2bbcb64 0
plan 0
verify 0
implement 368 0.0 0.0 118.3
audit_impl 0
prepare_pr 0
compose_pr 0
review_pr 0
dispatch:0145c757-e69c-4878-929a-ba4acb07e3d1 360 8363.5 172.2 38.2
fix 14 182708.4 8419.4 1204.3
dispatch:341f411a-3d76-4fcf-88a3-15a64c6d7bd4 446 7548.5 139.0 22.5
dispatch:7e27c189-e656-4ec2-b528-b128157c59a8 235 21444.0 314.5 73.2
Total 1471 21705.6 1068.1 285.3

Model Usage Breakdown

Model steps uncached output cache_read cache_write time
sonnet 9 8.9k 273.4k 26.7M 1.2M 5h 30m
opus[1m] 1 2.3k 90.0k 5.3M 374.7k 53m 29s
MiniMax-M3 3 7.4M 56.3k 0 0 26m 7s

…#4072)

The dispatch system silently dropped dispatched_session_id (and related
identity fields) on CancelledError: only status/reason/ended_at were
written via mark_dispatch_interrupted(). The session ID was known in
acc.stdout_session_id/acc.channel_b_session_id inside the task group
but never persisted to state.

Add an on_session_id_resolved callback fired from _extract_stdout_session_id
inside the live-execution task group, mirroring the existing on_pid_resolved
pattern. The closure calls mark_dispatch_session_identity to eagerly persist
the session ID while the dispatch is still RUNNING.

Three layers of defense:
1. Eager callback writes to state during execution
2. CancelledError handler passes captured_session_id + marker_dir to
   mark_dispatch_interrupted (additive — only writes empty fields)
3. Normal completion path prefers eagerly-captured value over skill_result

mark_dispatch_interrupted now accepts optional dispatched_session_id,
session_chain, and dispatched_session_log_dir. New mark_dispatch_session_identity
function is idempotent and status-guarded (RUNNING only).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Trecek Trecek force-pushed the dispatched-session-id-not-persisted-on-signal-interruption-d/4072 branch from 4c2e32d to fafa177 Compare June 12, 2026 00:19
@Trecek Trecek added this pull request to the merge queue Jun 12, 2026
Merged via the queue into develop with commit 75c72dd Jun 12, 2026
3 checks passed
@Trecek Trecek deleted the dispatched-session-id-not-persisted-on-signal-interruption-d/4072 branch June 12, 2026 00:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant