fix(agentchat): persist GraphFlow triggered_activation_groups across save/load_state#7916
Open
nolanchic wants to merge 1 commit into
Open
Conversation
…save/load_state GraphFlowManager.save_state/load_state did not persist _triggered_activation_groups, which records how a re-entry node was queued during graph execution. On resume in a cyclic graph (e.g. A->B with B looping back to A), _reset_triggered_activation_groups became a no-op for the re-entry target, so its _remaining count was never restored and got decremented below zero on the next loop iteration. This silently drained the ready queue, so _apply_termination_condition emitted 'Digraph execution is complete' and the workflow terminated without reaching the exit node. Persist triggered_activation_groups (Dict[str, Set[str]]) in save_state as lists and rebuild it as sets in load_state. Fields derived purely from the graph structure (_activation, _origin_remaining, _edges, _parents) are already reconstructed by __init__ on the new manager and need no persistence. Adds a regression test that runs one loop iteration, save_state, then load_state into a fresh team with fresh agent instances and asserts the exit node is reached. Verified the test fails on the unfixed code ([A, B] then 'Digraph execution is complete') and passes with the fix ([A, B, A, B, C]). Closes microsoft#7043.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #7043.
Why are these changes needed?
GraphFlowworkflows become unrecoverable when paused mid-traversal in a cyclic graph (e.g.A → BwithBlooping back toA). Aftersave_state→load_stateinto a fresh team, the resume terminates immediately with"Digraph execution is complete"and never reaches the exit node.Root cause
GraphFlowManager.save_state/load_state(_digraph_group_chat.py, lines 512-529) persistedmessage_thread,current_turn,remaining,enqueued_any, andready— but not_triggered_activation_groups._triggered_activation_groups(aDict[str, Set[str]]) records which activation group caused a node to be enqueued._reset_triggered_activation_groups(line 438) uses it duringselect_speaker(line 466) to restore a re-entry node's_remainingcount to its original value, so the node can be re-triggered on the next loop iteration.On resume,
_triggered_activation_groupsis{}(only initialized in__init__), so_reset_triggered_activation_groupsbecomes a no-op for the re-entry target. Its_remainingstays at 0 (the value saved mid-traversal) and gets decremented to -1 on the next loop iteration → the node is never re-enqueued →_readydrains →_apply_termination_conditionseesnot self._readyand emits "Digraph execution is complete" (lines 488-501).Failure trace (cyclic A→B, B loops to A)
A → B="loop".update_message_threaddecrements_remaining["A"]["A"]to 0, enqueues A, records_triggered_activation_groups["A"]={"A"}.MaxMessageTermination(3)fires beforeselect_speakerruns, so_reset_triggered_activation_groups("A")is never called. In-memory state:_triggered={"A":{"A"}},_remaining["A"]["A"]=0,_ready=["A"].save_statewrites_remaining["A"]["A"]=0and_ready=["A"]but drops_triggered_activation_groups.load_stateinto a fresh team:_triggered={},_remaining["A"]["A"]=0,_ready=["A"].select_speakerpops A, calls_reset_triggered("A")→ no-op (A not in the empty dict) →_remaining["A"]["A"]stays 0 → next loop iteration decrements it to -1 → A never re-enqueued → premature "Digraph execution is complete".Fix
Persist
_triggered_activation_groupsinsave_state(serializing eachSetas alist) and restore it inload_state(rebuilding thesets):Fields derived purely from graph structure (
_activation,_origin_remaining,_edges,_parents) are already reconstructed by__init__on the new manager and need no persistence — verified they are never mutated at runtime.Scope
Only cyclic / looping graphs are affected. Linear graphs are unaffected because the reset mechanism only matters for re-entry nodes. The existing
test_graph_flow_stateful_pause_and_resume_with_termination(linear A→B) already passed and continues to pass.Checks
test_graph_flow_cyclic_pause_resume_reaches_exit: runs one loop iteration →save_state→load_stateinto a fresh team with fresh agent instances → asserts the exit node C is reached. Verified the test fails on the unfixed code — sources[A, B]with stop_reason\"Digraph execution is complete\"— and passes with the fix — sources[A, B, A, B, C].)Local verification