feat: BAVT budget-aware iteration controller (arXiv:2603.12634)#251
Closed
mlbrilliance wants to merge 3 commits into
Closed
feat: BAVT budget-aware iteration controller (arXiv:2603.12634)#251mlbrilliance wants to merge 3 commits into
mlbrilliance wants to merge 3 commits into
Conversation
Implements a training-free budget management system for the agent loop inspired by 'Spend Less, Reason Better: Budget-Aware Value Tree Search for LLM Agents' (Li et al., arXiv:2603.12634, March 2026). ## What this adds ### agent/core/bavt.py (new) Three classes implementing the core BAVT logic: - BudgetTracker: tracks remaining_iters/max_iters as a continuous ratio (1.0 = fresh, 0.0 = exhausted). When max_iterations == -1 (unlimited) the ratio stays at 1.0 so all BAVT logic is a no-op. - ResidualProgressScorer: scores *relative* progress delta over the last 12 messages using lightweight heuristics (no LLM calls, zero latency). Tracks unique content fingerprints + tool result quality to avoid the LLM self-evaluation overconfidence flaw the paper identifies. - BudgetConditionedController: combines tracker + scorer into BudgetSignal output (corrective_message, effort_hint, prune_warning). A 5-iteration cooldown prevents back-to-back injections. Budget thresholds: ratio > 0.50 → no action 0.25-0.50 → nudge if progress stalled 0.10-0.25 → redirect message + effort downgrade hint < 0.10 → 'wrap up NOW' directive + max effort downgrade ### agent/core/doom_loop.py - check_for_doom_loop() gains a budget_ratio=1.0 parameter - Identical-call threshold tightens 3→2 when budget_ratio < 0.25 - Sequence-repetition min_reps drops 2→1 when budget_ratio < 0.10 - detect_repeating_sequence() gains a min_reps parameter ### agent/core/agent_loop.py - BudgetConditionedController instantiated once per run_agent() call - budget_ratio computed from iteration / max_iterations each loop turn - budget_ratio forwarded to check_for_doom_loop() - BudgetSignal.corrective_message injected as a user message (same pattern as the existing doom-loop injection) - BudgetSignal.effort_hint forwarded to _resolve_llm_params() so reasoning_effort is dynamically downgraded when budget is tight ### tests/test_bavt.py (new) 21 unit tests covering all three classes and key edge cases. ### README.md Architecture diagram updated + new 'Budget-Aware Loop Control' section with paper attribution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Adds a Budget-Aware Value Tree (BAVT)-style controller to make the agent loop adapt behavior (nudges, wrap-up directives, effort downgrades) as the iteration budget depletes, and tightens doom-loop detection under low budget.
Changes:
- Introduces
BudgetTracker,ResidualProgressScorer, andBudgetConditionedControllerto compute per-iteration budget/progress signals. - Makes doom-loop detection budget-aware by tightening thresholds based on
budget_ratio. - Wires BAVT signals into
run_agent()and documents the new control flow; adds unit tests for BAVT.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
agent/core/bavt.py |
New BAVT controller + heuristics producing BudgetSignal each iteration. |
agent/core/doom_loop.py |
Adds budget_ratio to doom-loop checks; introduces min_reps for sequence detection. |
agent/core/agent_loop.py |
Instantiates BAVT controller, injects budget corrective messages, and forwards effort hints. |
tests/test_bavt.py |
New unit tests for BAVT components. |
README.md |
Updates loop diagram + adds “Budget-Aware Loop Control (BAVT)” section. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
135
to
154
| @@ -143,24 +150,37 @@ def detect_repeating_sequence( | |||
| else: | |||
| break | |||
|
|
|||
| if reps >= 2: | |||
| if reps >= min_reps: | |||
| return pattern | |||
Comment on lines
+1180
to
+1185
| # Doom-loop detection: break out of repeated tool call patterns. | ||
| # Pass budget_ratio so thresholds tighten as iterations deplete. | ||
| budget_ratio = budget_controller._tracker.ratio(iteration) | ||
| doom_prompt = check_for_doom_loop( | ||
| session.context_manager.items, budget_ratio=budget_ratio | ||
| ) |
Comment on lines
1246
to
1251
| llm_params = _resolve_llm_params( | ||
| session.config.model_name, | ||
| session.hf_token, | ||
| reasoning_effort=session.effective_effort_for( | ||
| session.config.model_name | ||
| ), | ||
| reasoning_effort=bavt_effort_hint | ||
| or session.effective_effort_for(session.config.model_name), | ||
| ) |
Comment on lines
+54
to
+55
| # Effort level ordering for downgrade logic (worst→best). | ||
| _EFFORT_ORDER = ["low", "minimal", "medium", "high", "xhigh", "max"] |
Comment on lines
+16
to
+26
| identifies as a core flaw of naive self-evaluation. This implementation is | ||
| entirely heuristic (no LLM calls) so it adds zero latency. | ||
|
|
||
| 3. **Budget-conditioned signals** — thresholds on (budget_ratio, progress_score) | ||
| produce at most one injected message per check plus an optional effort hint | ||
| that the caller can forward to the next ``_resolve_llm_params`` call. | ||
|
|
||
| HuggingFace Pro note: The scorer uses only local heuristics by default. Callers | ||
| that want semantic scoring can optionally supply ``hf_token`` for a lightweight | ||
| HF Inference API call; this path is gated behind ``use_hf_scorer=True`` and | ||
| is never exercised in the default agent loop so it never adds latency. |
| |---|---| | ||
| | ratio < 50 % and progress stalling | Inject a gentle nudge message | | ||
| | ratio < 25 % and progress negative | Inject redirect + downgrade `reasoning_effort` one level | | ||
| | ratio < 10 % | Inject a "wrap up now" directive + max effort downgrade | |
Comment on lines
+1
to
+14
| """Unit tests for the BAVT (Budget-Aware Value Tree) module. | ||
|
|
||
| Covers BudgetTracker, ResidualProgressScorer, and BudgetConditionedController. | ||
| """ | ||
|
|
||
| from __future__ import annotations | ||
|
|
||
| from litellm import Message | ||
|
|
||
| from agent.core.bavt import ( | ||
| BudgetConditionedController, | ||
| BudgetTracker, | ||
| ResidualProgressScorer, | ||
| ) |
Comment on lines
264
to
+274
| ╔═══════════════════════════════════════════╗ | ||
| ║ Iteration Loop (max 300) ║ | ||
| ║ ║ | ||
| ║ BAVT budget check (ratio + progress) ║ | ||
| ║ ↓ ║ | ||
| ║ Doom loop check (budget-aware) ║ | ||
| ║ ↓ ║ | ||
| ║ Get messages + tool specs ║ | ||
| ║ ↓ ║ | ||
| ║ litellm.acompletion() ║ | ||
| ║ (effort may be downgraded by BAVT) ║ |
- tests/unit/test_bavt_integration.py: 17 behavioural tests covering all
BAVT thresholds (nudge/downgrade/wrap-up), cooldown mechanics, doom-loop
interaction, effort-hint flow-through, and residual scorer realism.
- tests/integration/test_live_bavt_hf.py: 4 live tests gated by
ML_INTERN_LIVE_LLM_TESTS=1, proving end-to-end with real HF Inference API:
1. effort_hint → correct HF router params (reasoning_effort header)
2. real API call with BAVT-downgraded effort succeeds
3. ResidualProgressScorer correctly ranks success vs error model output
4. full stalling-agent scenario: BAVT fires → corrective message sent
to real HF model → model responds with a coherent wrap-up pivot
All tests pass. Live tests verified against Qwen/Qwen2.5-72B-Instruct
via the HF Inference Router (https://router.huggingface.co/v1).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…3B proof
- Replace ml-intern streaming call with direct litellm.acompletion (no-tools)
to avoid Qwen3/vllm rejection of empty tools=[] array
- Add _LLMResult dataclass + _live_call() helper (non-streaming, model-agnostic)
- Remove _mock_session (no longer needed), keep _IS_THINKING_MODEL + max_tokens
- All 4 live tests PASS on huggingface/Qwen/Qwen3.6-35B-A3B:
test 1: effort_hint → valid HF router params (model, api_base, extra_body)
test 2: real API call with BAVT effort='medium' → '2 + 2 equals 4.'
test 3: ResidualProgressScorer success=1.00, error=-0.20 on real output
test 4: end-to-end stalling → BAVT CRITICAL → LLM delivers Python scraper
- Add BAVT_PROOF_Qwen3.6-35B-A3B.txt evidence file (21s run, all green)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements a training-free Budget-Aware Value Tree controller for the agent loop, adapted from:
Why this paper?
ml-intern already has
max_iterations(default 300),reasoning_effort, and a hash-based doom-loop detector — but no principled mechanism to adapt behaviour as the budget depletes. BAVT fills that gap without any training or LLM-call overhead.Changes
agent/core/bavt.py(new)Three classes, zero external deps:
BudgetTrackerremaining/maxratio; suggests effort downgradeResidualProgressScorerBudgetConditionedControllerBudgetSignalper iterationBudget thresholds:
budget_ratio5-iteration cooldown prevents back-to-back injections.
max_iterations == -1→ ratio stays 1.0, all BAVT logic is a no-op.agent/core/doom_loop.pycheck_for_doom_loop()gainsbudget_ratio=1.0param (backward-compatible)detect_repeating_sequence()gainsmin_repsparam; drops to 1 cycle at ratio < 0.10agent/core/agent_loop.pyBudgetConditionedControllerinstantiated once perrun_agent()callbudget_ratioforwarded tocheck_for_doom_loop()each iterationBudgetSignal.corrective_messageinjected as ausermessage (same pattern as existing doom-loop injection)BudgetSignal.effort_hintforwarded to_resolve_llm_params()for dynamic effort downgradetests/test_bavt.py(new)21 unit tests — all pass, no regressions in the full suite.
README.mdArchitecture diagram updated + new "Budget-Aware Loop Control" section.
Test results
Backward compatibility
--max-iterations -1(unlimited): BAVT ratio stays 1.0, zero effectcheck_for_doom_loopsignature: new param has a default, all callers unaffected