Skip to content

fix(agentchat): persist GraphFlow triggered_activation_groups across save/load_state#7916

Open
nolanchic wants to merge 1 commit into
microsoft:mainfrom
nolanchic:fix/graphflow-state-triggered-activation-groups
Open

fix(agentchat): persist GraphFlow triggered_activation_groups across save/load_state#7916
nolanchic wants to merge 1 commit into
microsoft:mainfrom
nolanchic:fix/graphflow-state-triggered-activation-groups

Conversation

@nolanchic

Copy link
Copy Markdown

Closes #7043.

Why are these changes needed?

GraphFlow workflows become unrecoverable when paused mid-traversal in a cyclic graph (e.g. A → B with B looping back to A). After save_stateload_state into a fresh team, the resume terminates immediately with "Digraph execution is complete" and never reaches the exit node.

Root cause

GraphFlowManager.save_state / load_state (_digraph_group_chat.py, lines 512-529) persisted message_thread, current_turn, remaining, enqueued_any, and ready — but not _triggered_activation_groups.

_triggered_activation_groups (a Dict[str, Set[str]]) records which activation group caused a node to be enqueued. _reset_triggered_activation_groups (line 438) uses it during select_speaker (line 466) to restore a re-entry node's _remaining count to its original value, so the node can be re-triggered on the next loop iteration.

On resume, _triggered_activation_groups is {} (only initialized in __init__), so _reset_triggered_activation_groups becomes a no-op for the re-entry target. Its _remaining stays at 0 (the value saved mid-traversal) and gets decremented to -1 on the next loop iteration → the node is never re-enqueued → _ready drains → _apply_termination_condition sees not self._ready and emits "Digraph execution is complete" (lines 488-501).

Failure trace (cyclic A→B, B loops to A)

  1. Phase 1: run A → B="loop". update_message_thread decrements _remaining["A"]["A"] to 0, enqueues A, records _triggered_activation_groups["A"]={"A"}.
  2. MaxMessageTermination(3) fires before select_speaker runs, so _reset_triggered_activation_groups("A") is never called. In-memory state: _triggered={"A":{"A"}}, _remaining["A"]["A"]=0, _ready=["A"].
  3. save_state writes _remaining["A"]["A"]=0 and _ready=["A"] but drops _triggered_activation_groups.
  4. load_state into a fresh team: _triggered={}, _remaining["A"]["A"]=0, _ready=["A"].
  5. Resume: select_speaker pops A, calls _reset_triggered("A")no-op (A not in the empty dict) → _remaining["A"]["A"] stays 0 → next loop iteration decrements it to -1 → A never re-enqueued → premature "Digraph execution is complete".

Fix

Persist _triggered_activation_groups in save_state (serializing each Set as a list) and restore it in load_state (rebuilding the sets):

# save_state
"triggered_activation_groups": {
    target: list(groups) for target, groups in self._triggered_activation_groups.items()
},
# load_state
self._triggered_activation_groups = {
    target: set(groups) for target, groups in state["triggered_activation_groups"].items()
}

Fields derived purely from graph structure (_activation, _origin_remaining, _edges, _parents) are already reconstructed by __init__ on the new manager and need no persistence — verified they are never mutated at runtime.

Scope

Only cyclic / looping graphs are affected. Linear graphs are unaffected because the reset mechanism only matters for re-entry nodes. The existing test_graph_flow_stateful_pause_and_resume_with_termination (linear A→B) already passed and continues to pass.

Checks

  • I've included any doc changes needed for https://microsoft.github.io/autogen/. (None — internal state-serialization fix.)
  • I've added tests (if relevant) corresponding to the changes introduced in this PR. (New test_graph_flow_cyclic_pause_resume_reaches_exit: runs one loop iteration → save_stateload_state into a fresh team with fresh agent instances → asserts the exit node C is reached. Verified the test fails on the unfixed code — sources [A, B] with stop_reason \"Digraph execution is complete\" — and passes with the fix — sources [A, B, A, B, C].)
  • I've made sure all auto checks have passed. (Local verification below.)

Local verification

cd python
uv run pytest packages/autogen-agentchat/tests/test_group_chat_graph.py -q   # 77 passed (was 76)
uv run ruff format --check packages/autogen-agentchat/src packages/autogen-agentchat/tests   # already formatted
uv run ruff check packages/autogen-agentchat/src packages/autogen-agentchat/tests            # all checks passed
uv run pyright packages/autogen-agentchat/src packages/autogen-agentchat/tests               # 0 errors
uv run mypy --config-file pyproject.toml packages/autogen-agentchat/src packages/autogen-agentchat/tests  # no issues

…save/load_state

GraphFlowManager.save_state/load_state did not persist
_triggered_activation_groups, which records how a re-entry node was
queued during graph execution. On resume in a cyclic graph (e.g. A->B
with B looping back to A), _reset_triggered_activation_groups became a
no-op for the re-entry target, so its _remaining count was never
restored and got decremented below zero on the next loop iteration.
This silently drained the ready queue, so _apply_termination_condition
emitted 'Digraph execution is complete' and the workflow terminated
without reaching the exit node.

Persist triggered_activation_groups (Dict[str, Set[str]]) in save_state
as lists and rebuild it as sets in load_state. Fields derived purely
from the graph structure (_activation, _origin_remaining, _edges,
_parents) are already reconstructed by __init__ on the new manager and
need no persistence.

Adds a regression test that runs one loop iteration, save_state, then
load_state into a fresh team with fresh agent instances and asserts the
exit node is reached. Verified the test fails on the unfixed code
([A, B] then 'Digraph execution is complete') and passes with the fix
([A, B, A, B, C]).

Closes microsoft#7043.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GraphFlow State Persistence Bug: Workflow Gets Stuck After Interruption During Agent Transitions

1 participant