From 108acabc0494e1b40fadf0e3b683ffa8f0444bdb Mon Sep 17 00:00:00 2001 From: cemde Date: Sun, 18 Jan 2026 23:25:56 +0100 Subject: [PATCH 01/17] initial strategy document --- STRATEGY.md | 478 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 478 insertions(+) create mode 100644 STRATEGY.md diff --git a/STRATEGY.md b/STRATEGY.md new file mode 100644 index 00000000..2b680335 --- /dev/null +++ b/STRATEGY.md @@ -0,0 +1,478 @@ +# MultiAgentBench Integration Strategy + +This document outlines strategies for integrating MultiAgentBench (MARBLE) into MASEval. It provides background context, analyzes three integration approaches, and recommends an implementation path. + +## Table of Contents + +1. [Background](#background) +2. [Integration Approaches](#integration-approaches) +3. [Recommendation](#recommendation) +4. [Implementation Roadmap](#implementation-roadmap) + +--- + +## Background + +### What is MultiAgentBench? + +MultiAgentBench (implemented through the MARBLE framework) is a comprehensive benchmark for evaluating LLM-based multi-agent systems across diverse, interactive scenarios. Key characteristics: + +- **6 Scenarios**: Research Collaboration, Minecraft Building, Database Anomaly Analysis, Coding Challenges, Werewolf Game, Bargaining +- **Dual Metrics**: Task completion (KPI, task score) + Coordination quality (communication, planning) +- **Multi-Agent Focus**: Unlike single-agent benchmarks, tests collaboration, competition, and emergent behaviors +- **Coordination Protocols**: Star, Chain, Tree, Graph-mesh topologies +- **Planning Strategies**: Vanilla, CoT, Group Discussion, Cognitive Self-Evolving + +### Source Materials + +- **Repository**: https://github.com/ulab-uiuc/MARBLE +- **License**: MIT License (Copyright 2024 Haofei Yu) +- **Paper**: Available in repository's `paper/` directory + +### MARBLE Architecture Overview + +``` +MARBLE Framework +├── marble/ # Core engine +│ ├── agent/ # Agent implementations +│ ├── configs/ # YAML configuration +│ ├── engine/ # Simulation engine +│ ├── environments/ # Domain environments (Research, Coding, DB, Minecraft, Werewolf, Bargaining) +│ ├── evaluator/ # Metrics and evaluation +│ ├── graph/ # Agent relationship graphs +│ ├── llms/ # LLM integration (uses litellm) +│ ├── memory/ # Shared/individual memory +│ └── main.py # Entry point +├── multiagentbench/ # Benchmark datasets (JSONL + converter) +└── scripts/ # Domain-specific runners +``` + +### MASEval Architecture (for context) + +``` +maseval/ +├── core/ # Framework-agnostic abstractions +│ ├── benchmark.py # Benchmark base class +│ ├── environment.py # Environment base +│ ├── agent.py # AgentAdapter (wraps external frameworks) +│ ├── evaluator.py # Evaluator base +│ ├── model.py # ModelAdapter (LLM abstraction) +│ └── tracing.py # TraceableMixin, ConfigurableMixin +├── interface/ # External framework adapters +│ ├── agents/ # smolagents, langgraph, llamaindex +│ └── inference/ # OpenAI, Anthropic, Google, etc. +└── benchmark/ # Benchmark implementations + ├── macs/ # LLM-simulated tools, multi-agent hierarchy + └── tau2/ # Real tools, deterministic evaluation +``` + +### Key Design Considerations + +1. **Replication Fidelity**: Must closely replicate MARBLE's evaluation methodology to reproduce paper results +2. **Framework Agnosticism**: MASEval supports multiple agent frameworks (smolagents, langgraph) +3. **Tool Execution Model**: MARBLE uses LLM-based tool simulation (similar to MACS, not Tau2) +4. **Multi-Agent Coordination**: MARBLE has its own coordination engine; MASEval's `run_agents()` is more minimal +5. **Memory Management**: MARBLE has sophisticated shared/individual memory; MASEval leaves this to agent frameworks +6. **Evaluation Complexity**: MARBLE evaluates via LLM (communication/planning scores) + domain-specific metrics + +--- + +## Integration Approaches + +### Approach 1: Port MARBLE Code (Recommended) + +**Description**: Copy relevant MARBLE code into `maseval/benchmark/multiagentbench/`, adapting it to use MASEval's abstractions where beneficial while preserving MARBLE's core logic. + +**What to Port**: +- Environments (`marble/environments/`) → Adapt to extend MASEval's `Environment` +- Evaluator logic (`marble/evaluator/`) → Implement as MASEval `Evaluator` subclasses +- Dataset loading (`multiagentbench/`) → Create `data_loader.py` similar to MACS/Tau2 +- Prompt templates (evaluation prompts) → Store in `prompt_templates/` +- Domain-specific configurations + +**What NOT to Port** (use MASEval instead): +- LLM integration → Use MASEval's `ModelAdapter` (LiteLLM, OpenAI, etc.) +- Agent framework wrappers → Use MASEval's `AgentAdapter` for smolagents/langgraph +- Tracing infrastructure → Use MASEval's `TraceableMixin`, `ConfigurableMixin` +- Configuration loading → Use MASEval's `Task`, `TaskQueue` patterns + +**Structure**: +``` +maseval/benchmark/multiagentbench/ +├── __init__.py # Public API +├── multiagentbench.py # MultiAgentBenchmark class(es) +├── data_loader.py # Task loading, data management +├── environments/ +│ ├── base_env.py # BaseMultiAgentEnvironment +│ ├── research_env.py # Research collaboration +│ ├── minecraft_env.py # Minecraft building +│ ├── database_env.py # Database anomaly analysis +│ ├── coding_env.py # Coding challenges +│ ├── werewolf_env.py # Werewolf game +│ └── bargaining_env.py # Bargaining scenarios +├── evaluator.py # KPI, Task Score, Coordination Score +├── coordination/ # Agent coordination protocols +│ ├── star.py +│ ├── chain.py +│ ├── tree.py +│ └── graph_mesh.py +├── prompt_templates/ # LLM evaluation prompts +│ ├── communication_eval.txt +│ ├── planning_eval.txt +│ └── task_eval/ # Per-domain task evaluation +└── data/ # Downloaded benchmark data +``` + +**Pros**: +- Full control over integration +- Can optimize for MASEval patterns +- No external dependency management +- Easier debugging and modification +- Clear licensing (MIT allows copying with attribution) +- Can selectively port scenarios (start with 2-3, expand later) + +**Cons**: +- Significant initial porting effort (~2-4 weeks) +- Must maintain parity with upstream changes +- Risk of subtle divergence from paper methodology +- Need deep understanding of MARBLE internals + +**Effort Estimate**: High initial, low ongoing + +--- + +### Approach 2: Install as Dependency + +**Description**: Add `marble` as an optional dependency in `pyproject.toml` and create thin adapter classes that delegate to MARBLE. + +**Implementation**: +```python +# pyproject.toml +[project.optional-dependencies] +multiagentbench = ["marble @ git+https://github.com/ulab-uiuc/MARBLE.git"] + +# maseval/benchmark/multiagentbench/multiagentbench.py +from maseval import Benchmark, Environment, Task + +class MultiAgentBenchmark(Benchmark): + def setup_environment(self, agent_data, task): + # Import MARBLE at runtime + from marble.environments import get_environment + from marble.configs import Config + + # Create MARBLE environment, wrap for MASEval + marble_config = Config.from_dict(task.environment_data) + marble_env = get_environment(marble_config) + return MultiAgentBenchEnvironmentWrapper(marble_env) + + def run_agents(self, agents, task, environment, query): + # Delegate to MARBLE's coordination engine + from marble.engine import Engine + engine = Engine(environment.marble_env) + return engine.run() +``` + +**Pros**: +- Minimal code to write +- Automatically gets upstream bug fixes +- Guaranteed methodology parity +- Faster initial implementation (~1 week) + +**Cons**: +- **Heavy dependency**: MARBLE has many dependencies (litellm, beautifulsoup4, flask, psycopg2, etc.) +- **Version pinning complexity**: Must track MARBLE versions for reproducibility +- **API surface mismatch**: MARBLE's coordination engine doesn't fit MASEval's `run_agents()` model cleanly +- **Framework conflict**: Both use litellm differently; potential version conflicts +- **Installation friction**: Users must install from git URL (not PyPI) +- **Limited customization**: Hard to use MASEval's smolagents/langgraph adapters + +**Effort Estimate**: Low initial, medium ongoing (dependency management) + +--- + +### Approach 3: Hybrid Dynamic Import + +**Description**: Port the essential components (environments, evaluation) but dynamically load MARBLE's coordination engine when available, falling back to a simplified implementation otherwise. + +**Implementation**: +```python +# maseval/benchmark/multiagentbench/coordination.py + +def get_coordination_engine(protocol: str): + """Get coordination engine, preferring MARBLE if available.""" + try: + from marble.engine import EnginePlanner + from marble.graph import AgentGraph + return MARBLECoordinationAdapter(protocol) + except ImportError: + # Fall back to simplified implementation + return SimpleCoordinationEngine(protocol) + +class SimpleCoordinationEngine: + """Simplified coordination for basic usage without full MARBLE.""" + + def __init__(self, protocol: str): + self.protocol = protocol + + def coordinate(self, agents, task, environment): + if self.protocol == "star": + return self._star_coordinate(agents, task, environment) + elif self.protocol == "chain": + return self._chain_coordinate(agents, task, environment) + # ... etc +``` + +**Structure**: +``` +maseval/benchmark/multiagentbench/ +├── multiagentbench.py # Benchmark class +├── data_loader.py # Task loading (ported) +├── environments/ # Ported from MARBLE +├── evaluator.py # Ported from MARBLE +├── coordination/ +│ ├── __init__.py # get_coordination_engine() +│ ├── simple.py # Fallback implementations +│ └── marble_adapter.py # MARBLE delegation (if available) +└── data/ +``` + +**Pros**: +- Works with or without MARBLE installed +- Can start simple, add MARBLE compatibility later +- Best of both worlds for evaluation fidelity +- Users who don't need full coordination can use lightweight mode + +**Cons**: +- Two code paths to maintain +- Potential behavior differences between modes +- More complex testing (need to test both paths) +- "Simplified" coordination may not replicate paper results accurately + +**Effort Estimate**: Medium initial, high ongoing (maintaining two implementations) + +--- + +## Recommendation + +**Recommended Approach: Port MARBLE Code (Approach 1)** + +### Rationale + +1. **Replication Accuracy**: The primary goal is to replicate paper results. Porting gives full control over the evaluation methodology and ensures we understand every detail of how metrics are computed. + +2. **MASEval Alignment**: MARBLE's architecture differs significantly from MASEval's patterns: + - MARBLE has its own coordination engine → We can adapt this to work with MASEval's `run_agents()` + - MARBLE uses litellm directly → We can use MASEval's `ModelAdapter` for unified provider support + - MARBLE has YAML configs → We can convert to MASEval's `Task` objects + +3. **Dependency Management**: MARBLE has 15+ dependencies, some with specific version requirements. Adding it as a dependency creates potential conflicts with MASEval's existing dependencies. + +4. **Framework Flexibility**: Porting allows us to properly support MASEval's agent framework adapters (smolagents, langgraph), rather than being locked to MARBLE's agent implementation. + +5. **MIT License**: MARBLE's MIT license explicitly permits copying, modification, and redistribution with attribution. + +6. **MACS Precedent**: The MACS benchmark followed a similar pattern—porting core logic while adapting to MASEval's abstractions. This approach has proven successful. + +### Phased Implementation + +Rather than porting all 6 scenarios at once, implement in phases: + +**Phase 1 (MVP)**: Research Collaboration + Bargaining +- Both are text-only, no external services needed +- Research tests collaborative task completion +- Bargaining tests competitive interaction +- Validates core infrastructure (agent graph, coordination, evaluation) + +**Phase 2**: Coding + Database +- More complex tool usage +- Tests multi-role collaboration (debugger, reviewer, etc.) +- Deterministic evaluation possible for coding (execution success) + +**Phase 3**: Minecraft + Werewolf +- Minecraft requires understanding block placement semantics +- Werewolf requires complex game state management +- Most complex coordination patterns + +--- + +## Implementation Roadmap + +### Prerequisites + +- [ ] Confirm MARBLE repository access and license compliance +- [ ] Review MARBLE test suite for validation criteria +- [ ] Identify minimal set of MARBLE dependencies to preserve + +### Phase 1: Foundation (Research + Bargaining) + +**Week 1: Infrastructure** +- [ ] Create `maseval/benchmark/multiagentbench/` directory structure +- [ ] Port data loading (`data_loader.py`) with GitHub download support +- [ ] Implement base environment class +- [ ] Create Task/TaskQueue conversion from MARBLE JSONL format + +**Week 2: Environments** +- [ ] Port ResearchEnvironment with tools (paper fetching, etc.) +- [ ] Port BargainingEnvironment with negotiation tools +- [ ] Adapt tools to work with MASEval's tracing infrastructure + +**Week 3: Coordination & Agents** +- [ ] Implement Agent Graph construction from configs +- [ ] Implement Star and Graph-mesh coordination protocols +- [ ] Create `MultiAgentBenchmark` class with `setup_agents()`, `run_agents()` +- [ ] Framework-agnostic agent wrapper (similar to MACSGenericTool pattern) + +**Week 4: Evaluation** +- [ ] Port LLM-based communication score evaluation +- [ ] Port LLM-based planning score evaluation +- [ ] Implement domain-specific task scores (Research: innovation/safety/feasibility, Bargaining: negotiation metrics) +- [ ] Port Milestone-based KPI calculation +- [ ] Create `compute_benchmark_metrics()` aggregation function + +**Week 5: Testing & Validation** +- [ ] Write unit tests for environments, evaluation +- [ ] Run baseline tests with GPT-4o-mini +- [ ] Compare results to paper Table 1 values +- [ ] Document any deviations and their causes + +### Phase 2: Coding + Database (TBD after Phase 1) + +### Phase 3: Minecraft + Werewolf (TBD after Phase 2) + +--- + +## Key Implementation Details + +### Agent Coordination Pattern + +MARBLE's coordination differs from MASEval's single-agent assumption. The recommended pattern: + +```python +class MultiAgentBenchmark(Benchmark): + def setup_agents(self, agent_data, environment, task, user): + """Build agent graph from task configuration.""" + agent_configs = task.environment_data.get("agents", []) + coordination_mode = task.environment_data.get("coordinate_mode", "graph") + + # Build agents with relationships + agents = self._build_agent_graph(agent_configs, environment) + + # Store coordination mode for run_agents + self._coordination_mode = coordination_mode + + # Return primary agent(s) for execution + primary_agents = [a for a in agents if a.is_primary] + agents_dict = {a.name: a for a in agents} + + return primary_agents, agents_dict + + def run_agents(self, agents, task, environment, query): + """Execute agents using appropriate coordination protocol.""" + if self._coordination_mode == "star": + return self._run_star(agents, query, environment) + elif self._coordination_mode == "graph": + return self._run_graph_mesh(agents, query, environment) + # ... etc +``` + +### Evaluation Metrics Structure + +```python +@dataclass +class MultiAgentBenchEvalResult: + # Task completion + task_score: float # Domain-specific (0-5 scale, normalized) + kpi: float # Milestone-based KPI (0-1) + + # Coordination quality + communication_score: float # LLM-evaluated (1-5 scale) + planning_score: float # LLM-evaluated (1-5 scale) + coordination_score: float # Average of above (1-5 scale) + + # Per-agent breakdown + agent_kpis: Dict[str, float] # Individual agent KPIs + + # Raw data + milestone_report: List[Dict] # Milestone tracking details + eval_report: Dict # Full LLM evaluation outputs +``` + +### Data Loading Pattern + +```python +def load_tasks( + scenario: str, # "research", "bargaining", "coding", etc. + data_dir: Optional[Path] = None, + limit: Optional[int] = None, +) -> TaskQueue: + """Load MultiAgentBench tasks for a scenario.""" + + ensure_data_exists(scenario, data_dir) + + # Load JSONL and convert + raw_tasks = _load_jsonl(data_dir / scenario / "tasks.jsonl") + + tasks = [] + for raw in raw_tasks: + task = Task( + id=raw.get("task_id"), + query=raw["task"]["content"], + environment_data={ + "scenario": scenario, + "agents": raw["agents"], + "relationships": raw["relationships"], + "coordinate_mode": raw.get("coordinate_mode", "graph"), + "tools": raw.get("environment", {}).get("tools", []), + }, + evaluation_data={ + "milestones": raw.get("milestones", []), + "task_rubric": raw.get("evaluation", {}), + }, + metadata={ + "scenario": scenario, + "difficulty": raw.get("difficulty", "medium"), + }, + ) + tasks.append(task) + + return TaskQueue(tasks[:limit] if limit else tasks) +``` + +--- + +## Risk Mitigation + +| Risk | Mitigation | +|------|------------| +| Subtle methodology differences causing result divergence | Carefully validate against paper results; document any deviations | +| MARBLE updates breaking compatibility | Pin to specific commit hash; periodically review upstream changes | +| LLM evaluation non-determinism | Use fixed seeds where possible; report variance across runs | +| Minecraft/Werewolf complexity | Defer to Phase 3; may require simplified versions initially | +| Token cost for LLM-based evaluation | Provide cost estimates in documentation; support cheaper evaluation models | + +--- + +## License Compliance + +MARBLE is released under the MIT License. To comply: + +1. Include MIT license text in `maseval/benchmark/multiagentbench/LICENSE` +2. Add attribution in module docstrings: + ```python + """ + MultiAgentBench integration for MASEval. + + Based on MARBLE: https://github.com/ulab-uiuc/MARBLE + Copyright (c) 2024 Haofei Yu + Licensed under the MIT License + """ + ``` +3. Note original authors in CHANGELOG when adding the benchmark + +--- + +## Conclusion + +Porting MARBLE code (Approach 1) provides the best balance of replication fidelity, MASEval integration, and long-term maintainability. The phased implementation approach allows incremental validation while managing complexity. + +Starting with Research Collaboration and Bargaining scenarios provides a solid foundation that exercises all key components (agent graphs, coordination, LLM evaluation) while avoiding the complexity of Minecraft's spatial reasoning or Werewolf's game state management. From 18c288d4b9640dcc0f3ba78a9559f272700161e1 Mon Sep 17 00:00:00 2001 From: cemde Date: Mon, 19 Jan 2026 12:38:21 +0100 Subject: [PATCH 02/17] removed strategy --- STRATEGY.md | 478 ---------------------------------------------------- 1 file changed, 478 deletions(-) delete mode 100644 STRATEGY.md diff --git a/STRATEGY.md b/STRATEGY.md deleted file mode 100644 index 2b680335..00000000 --- a/STRATEGY.md +++ /dev/null @@ -1,478 +0,0 @@ -# MultiAgentBench Integration Strategy - -This document outlines strategies for integrating MultiAgentBench (MARBLE) into MASEval. It provides background context, analyzes three integration approaches, and recommends an implementation path. - -## Table of Contents - -1. [Background](#background) -2. [Integration Approaches](#integration-approaches) -3. [Recommendation](#recommendation) -4. [Implementation Roadmap](#implementation-roadmap) - ---- - -## Background - -### What is MultiAgentBench? - -MultiAgentBench (implemented through the MARBLE framework) is a comprehensive benchmark for evaluating LLM-based multi-agent systems across diverse, interactive scenarios. Key characteristics: - -- **6 Scenarios**: Research Collaboration, Minecraft Building, Database Anomaly Analysis, Coding Challenges, Werewolf Game, Bargaining -- **Dual Metrics**: Task completion (KPI, task score) + Coordination quality (communication, planning) -- **Multi-Agent Focus**: Unlike single-agent benchmarks, tests collaboration, competition, and emergent behaviors -- **Coordination Protocols**: Star, Chain, Tree, Graph-mesh topologies -- **Planning Strategies**: Vanilla, CoT, Group Discussion, Cognitive Self-Evolving - -### Source Materials - -- **Repository**: https://github.com/ulab-uiuc/MARBLE -- **License**: MIT License (Copyright 2024 Haofei Yu) -- **Paper**: Available in repository's `paper/` directory - -### MARBLE Architecture Overview - -``` -MARBLE Framework -├── marble/ # Core engine -│ ├── agent/ # Agent implementations -│ ├── configs/ # YAML configuration -│ ├── engine/ # Simulation engine -│ ├── environments/ # Domain environments (Research, Coding, DB, Minecraft, Werewolf, Bargaining) -│ ├── evaluator/ # Metrics and evaluation -│ ├── graph/ # Agent relationship graphs -│ ├── llms/ # LLM integration (uses litellm) -│ ├── memory/ # Shared/individual memory -│ └── main.py # Entry point -├── multiagentbench/ # Benchmark datasets (JSONL + converter) -└── scripts/ # Domain-specific runners -``` - -### MASEval Architecture (for context) - -``` -maseval/ -├── core/ # Framework-agnostic abstractions -│ ├── benchmark.py # Benchmark base class -│ ├── environment.py # Environment base -│ ├── agent.py # AgentAdapter (wraps external frameworks) -│ ├── evaluator.py # Evaluator base -│ ├── model.py # ModelAdapter (LLM abstraction) -│ └── tracing.py # TraceableMixin, ConfigurableMixin -├── interface/ # External framework adapters -│ ├── agents/ # smolagents, langgraph, llamaindex -│ └── inference/ # OpenAI, Anthropic, Google, etc. -└── benchmark/ # Benchmark implementations - ├── macs/ # LLM-simulated tools, multi-agent hierarchy - └── tau2/ # Real tools, deterministic evaluation -``` - -### Key Design Considerations - -1. **Replication Fidelity**: Must closely replicate MARBLE's evaluation methodology to reproduce paper results -2. **Framework Agnosticism**: MASEval supports multiple agent frameworks (smolagents, langgraph) -3. **Tool Execution Model**: MARBLE uses LLM-based tool simulation (similar to MACS, not Tau2) -4. **Multi-Agent Coordination**: MARBLE has its own coordination engine; MASEval's `run_agents()` is more minimal -5. **Memory Management**: MARBLE has sophisticated shared/individual memory; MASEval leaves this to agent frameworks -6. **Evaluation Complexity**: MARBLE evaluates via LLM (communication/planning scores) + domain-specific metrics - ---- - -## Integration Approaches - -### Approach 1: Port MARBLE Code (Recommended) - -**Description**: Copy relevant MARBLE code into `maseval/benchmark/multiagentbench/`, adapting it to use MASEval's abstractions where beneficial while preserving MARBLE's core logic. - -**What to Port**: -- Environments (`marble/environments/`) → Adapt to extend MASEval's `Environment` -- Evaluator logic (`marble/evaluator/`) → Implement as MASEval `Evaluator` subclasses -- Dataset loading (`multiagentbench/`) → Create `data_loader.py` similar to MACS/Tau2 -- Prompt templates (evaluation prompts) → Store in `prompt_templates/` -- Domain-specific configurations - -**What NOT to Port** (use MASEval instead): -- LLM integration → Use MASEval's `ModelAdapter` (LiteLLM, OpenAI, etc.) -- Agent framework wrappers → Use MASEval's `AgentAdapter` for smolagents/langgraph -- Tracing infrastructure → Use MASEval's `TraceableMixin`, `ConfigurableMixin` -- Configuration loading → Use MASEval's `Task`, `TaskQueue` patterns - -**Structure**: -``` -maseval/benchmark/multiagentbench/ -├── __init__.py # Public API -├── multiagentbench.py # MultiAgentBenchmark class(es) -├── data_loader.py # Task loading, data management -├── environments/ -│ ├── base_env.py # BaseMultiAgentEnvironment -│ ├── research_env.py # Research collaboration -│ ├── minecraft_env.py # Minecraft building -│ ├── database_env.py # Database anomaly analysis -│ ├── coding_env.py # Coding challenges -│ ├── werewolf_env.py # Werewolf game -│ └── bargaining_env.py # Bargaining scenarios -├── evaluator.py # KPI, Task Score, Coordination Score -├── coordination/ # Agent coordination protocols -│ ├── star.py -│ ├── chain.py -│ ├── tree.py -│ └── graph_mesh.py -├── prompt_templates/ # LLM evaluation prompts -│ ├── communication_eval.txt -│ ├── planning_eval.txt -│ └── task_eval/ # Per-domain task evaluation -└── data/ # Downloaded benchmark data -``` - -**Pros**: -- Full control over integration -- Can optimize for MASEval patterns -- No external dependency management -- Easier debugging and modification -- Clear licensing (MIT allows copying with attribution) -- Can selectively port scenarios (start with 2-3, expand later) - -**Cons**: -- Significant initial porting effort (~2-4 weeks) -- Must maintain parity with upstream changes -- Risk of subtle divergence from paper methodology -- Need deep understanding of MARBLE internals - -**Effort Estimate**: High initial, low ongoing - ---- - -### Approach 2: Install as Dependency - -**Description**: Add `marble` as an optional dependency in `pyproject.toml` and create thin adapter classes that delegate to MARBLE. - -**Implementation**: -```python -# pyproject.toml -[project.optional-dependencies] -multiagentbench = ["marble @ git+https://github.com/ulab-uiuc/MARBLE.git"] - -# maseval/benchmark/multiagentbench/multiagentbench.py -from maseval import Benchmark, Environment, Task - -class MultiAgentBenchmark(Benchmark): - def setup_environment(self, agent_data, task): - # Import MARBLE at runtime - from marble.environments import get_environment - from marble.configs import Config - - # Create MARBLE environment, wrap for MASEval - marble_config = Config.from_dict(task.environment_data) - marble_env = get_environment(marble_config) - return MultiAgentBenchEnvironmentWrapper(marble_env) - - def run_agents(self, agents, task, environment, query): - # Delegate to MARBLE's coordination engine - from marble.engine import Engine - engine = Engine(environment.marble_env) - return engine.run() -``` - -**Pros**: -- Minimal code to write -- Automatically gets upstream bug fixes -- Guaranteed methodology parity -- Faster initial implementation (~1 week) - -**Cons**: -- **Heavy dependency**: MARBLE has many dependencies (litellm, beautifulsoup4, flask, psycopg2, etc.) -- **Version pinning complexity**: Must track MARBLE versions for reproducibility -- **API surface mismatch**: MARBLE's coordination engine doesn't fit MASEval's `run_agents()` model cleanly -- **Framework conflict**: Both use litellm differently; potential version conflicts -- **Installation friction**: Users must install from git URL (not PyPI) -- **Limited customization**: Hard to use MASEval's smolagents/langgraph adapters - -**Effort Estimate**: Low initial, medium ongoing (dependency management) - ---- - -### Approach 3: Hybrid Dynamic Import - -**Description**: Port the essential components (environments, evaluation) but dynamically load MARBLE's coordination engine when available, falling back to a simplified implementation otherwise. - -**Implementation**: -```python -# maseval/benchmark/multiagentbench/coordination.py - -def get_coordination_engine(protocol: str): - """Get coordination engine, preferring MARBLE if available.""" - try: - from marble.engine import EnginePlanner - from marble.graph import AgentGraph - return MARBLECoordinationAdapter(protocol) - except ImportError: - # Fall back to simplified implementation - return SimpleCoordinationEngine(protocol) - -class SimpleCoordinationEngine: - """Simplified coordination for basic usage without full MARBLE.""" - - def __init__(self, protocol: str): - self.protocol = protocol - - def coordinate(self, agents, task, environment): - if self.protocol == "star": - return self._star_coordinate(agents, task, environment) - elif self.protocol == "chain": - return self._chain_coordinate(agents, task, environment) - # ... etc -``` - -**Structure**: -``` -maseval/benchmark/multiagentbench/ -├── multiagentbench.py # Benchmark class -├── data_loader.py # Task loading (ported) -├── environments/ # Ported from MARBLE -├── evaluator.py # Ported from MARBLE -├── coordination/ -│ ├── __init__.py # get_coordination_engine() -│ ├── simple.py # Fallback implementations -│ └── marble_adapter.py # MARBLE delegation (if available) -└── data/ -``` - -**Pros**: -- Works with or without MARBLE installed -- Can start simple, add MARBLE compatibility later -- Best of both worlds for evaluation fidelity -- Users who don't need full coordination can use lightweight mode - -**Cons**: -- Two code paths to maintain -- Potential behavior differences between modes -- More complex testing (need to test both paths) -- "Simplified" coordination may not replicate paper results accurately - -**Effort Estimate**: Medium initial, high ongoing (maintaining two implementations) - ---- - -## Recommendation - -**Recommended Approach: Port MARBLE Code (Approach 1)** - -### Rationale - -1. **Replication Accuracy**: The primary goal is to replicate paper results. Porting gives full control over the evaluation methodology and ensures we understand every detail of how metrics are computed. - -2. **MASEval Alignment**: MARBLE's architecture differs significantly from MASEval's patterns: - - MARBLE has its own coordination engine → We can adapt this to work with MASEval's `run_agents()` - - MARBLE uses litellm directly → We can use MASEval's `ModelAdapter` for unified provider support - - MARBLE has YAML configs → We can convert to MASEval's `Task` objects - -3. **Dependency Management**: MARBLE has 15+ dependencies, some with specific version requirements. Adding it as a dependency creates potential conflicts with MASEval's existing dependencies. - -4. **Framework Flexibility**: Porting allows us to properly support MASEval's agent framework adapters (smolagents, langgraph), rather than being locked to MARBLE's agent implementation. - -5. **MIT License**: MARBLE's MIT license explicitly permits copying, modification, and redistribution with attribution. - -6. **MACS Precedent**: The MACS benchmark followed a similar pattern—porting core logic while adapting to MASEval's abstractions. This approach has proven successful. - -### Phased Implementation - -Rather than porting all 6 scenarios at once, implement in phases: - -**Phase 1 (MVP)**: Research Collaboration + Bargaining -- Both are text-only, no external services needed -- Research tests collaborative task completion -- Bargaining tests competitive interaction -- Validates core infrastructure (agent graph, coordination, evaluation) - -**Phase 2**: Coding + Database -- More complex tool usage -- Tests multi-role collaboration (debugger, reviewer, etc.) -- Deterministic evaluation possible for coding (execution success) - -**Phase 3**: Minecraft + Werewolf -- Minecraft requires understanding block placement semantics -- Werewolf requires complex game state management -- Most complex coordination patterns - ---- - -## Implementation Roadmap - -### Prerequisites - -- [ ] Confirm MARBLE repository access and license compliance -- [ ] Review MARBLE test suite for validation criteria -- [ ] Identify minimal set of MARBLE dependencies to preserve - -### Phase 1: Foundation (Research + Bargaining) - -**Week 1: Infrastructure** -- [ ] Create `maseval/benchmark/multiagentbench/` directory structure -- [ ] Port data loading (`data_loader.py`) with GitHub download support -- [ ] Implement base environment class -- [ ] Create Task/TaskQueue conversion from MARBLE JSONL format - -**Week 2: Environments** -- [ ] Port ResearchEnvironment with tools (paper fetching, etc.) -- [ ] Port BargainingEnvironment with negotiation tools -- [ ] Adapt tools to work with MASEval's tracing infrastructure - -**Week 3: Coordination & Agents** -- [ ] Implement Agent Graph construction from configs -- [ ] Implement Star and Graph-mesh coordination protocols -- [ ] Create `MultiAgentBenchmark` class with `setup_agents()`, `run_agents()` -- [ ] Framework-agnostic agent wrapper (similar to MACSGenericTool pattern) - -**Week 4: Evaluation** -- [ ] Port LLM-based communication score evaluation -- [ ] Port LLM-based planning score evaluation -- [ ] Implement domain-specific task scores (Research: innovation/safety/feasibility, Bargaining: negotiation metrics) -- [ ] Port Milestone-based KPI calculation -- [ ] Create `compute_benchmark_metrics()` aggregation function - -**Week 5: Testing & Validation** -- [ ] Write unit tests for environments, evaluation -- [ ] Run baseline tests with GPT-4o-mini -- [ ] Compare results to paper Table 1 values -- [ ] Document any deviations and their causes - -### Phase 2: Coding + Database (TBD after Phase 1) - -### Phase 3: Minecraft + Werewolf (TBD after Phase 2) - ---- - -## Key Implementation Details - -### Agent Coordination Pattern - -MARBLE's coordination differs from MASEval's single-agent assumption. The recommended pattern: - -```python -class MultiAgentBenchmark(Benchmark): - def setup_agents(self, agent_data, environment, task, user): - """Build agent graph from task configuration.""" - agent_configs = task.environment_data.get("agents", []) - coordination_mode = task.environment_data.get("coordinate_mode", "graph") - - # Build agents with relationships - agents = self._build_agent_graph(agent_configs, environment) - - # Store coordination mode for run_agents - self._coordination_mode = coordination_mode - - # Return primary agent(s) for execution - primary_agents = [a for a in agents if a.is_primary] - agents_dict = {a.name: a for a in agents} - - return primary_agents, agents_dict - - def run_agents(self, agents, task, environment, query): - """Execute agents using appropriate coordination protocol.""" - if self._coordination_mode == "star": - return self._run_star(agents, query, environment) - elif self._coordination_mode == "graph": - return self._run_graph_mesh(agents, query, environment) - # ... etc -``` - -### Evaluation Metrics Structure - -```python -@dataclass -class MultiAgentBenchEvalResult: - # Task completion - task_score: float # Domain-specific (0-5 scale, normalized) - kpi: float # Milestone-based KPI (0-1) - - # Coordination quality - communication_score: float # LLM-evaluated (1-5 scale) - planning_score: float # LLM-evaluated (1-5 scale) - coordination_score: float # Average of above (1-5 scale) - - # Per-agent breakdown - agent_kpis: Dict[str, float] # Individual agent KPIs - - # Raw data - milestone_report: List[Dict] # Milestone tracking details - eval_report: Dict # Full LLM evaluation outputs -``` - -### Data Loading Pattern - -```python -def load_tasks( - scenario: str, # "research", "bargaining", "coding", etc. - data_dir: Optional[Path] = None, - limit: Optional[int] = None, -) -> TaskQueue: - """Load MultiAgentBench tasks for a scenario.""" - - ensure_data_exists(scenario, data_dir) - - # Load JSONL and convert - raw_tasks = _load_jsonl(data_dir / scenario / "tasks.jsonl") - - tasks = [] - for raw in raw_tasks: - task = Task( - id=raw.get("task_id"), - query=raw["task"]["content"], - environment_data={ - "scenario": scenario, - "agents": raw["agents"], - "relationships": raw["relationships"], - "coordinate_mode": raw.get("coordinate_mode", "graph"), - "tools": raw.get("environment", {}).get("tools", []), - }, - evaluation_data={ - "milestones": raw.get("milestones", []), - "task_rubric": raw.get("evaluation", {}), - }, - metadata={ - "scenario": scenario, - "difficulty": raw.get("difficulty", "medium"), - }, - ) - tasks.append(task) - - return TaskQueue(tasks[:limit] if limit else tasks) -``` - ---- - -## Risk Mitigation - -| Risk | Mitigation | -|------|------------| -| Subtle methodology differences causing result divergence | Carefully validate against paper results; document any deviations | -| MARBLE updates breaking compatibility | Pin to specific commit hash; periodically review upstream changes | -| LLM evaluation non-determinism | Use fixed seeds where possible; report variance across runs | -| Minecraft/Werewolf complexity | Defer to Phase 3; may require simplified versions initially | -| Token cost for LLM-based evaluation | Provide cost estimates in documentation; support cheaper evaluation models | - ---- - -## License Compliance - -MARBLE is released under the MIT License. To comply: - -1. Include MIT license text in `maseval/benchmark/multiagentbench/LICENSE` -2. Add attribution in module docstrings: - ```python - """ - MultiAgentBench integration for MASEval. - - Based on MARBLE: https://github.com/ulab-uiuc/MARBLE - Copyright (c) 2024 Haofei Yu - Licensed under the MIT License - """ - ``` -3. Note original authors in CHANGELOG when adding the benchmark - ---- - -## Conclusion - -Porting MARBLE code (Approach 1) provides the best balance of replication fidelity, MASEval integration, and long-term maintainability. The phased implementation approach allows incremental validation while managing complexity. - -Starting with Research Collaboration and Bargaining scenarios provides a solid foundation that exercises all key components (agent graphs, coordination, LLM evaluation) while avoiding the complexity of Minecraft's spatial reasoning or Werewolf's game state management. From 2f99d80565430ed695ec2c39ce9f7e9954b47dc5 Mon Sep 17 00:00:00 2001 From: cemde Date: Mon, 19 Jan 2026 13:45:06 +0100 Subject: [PATCH 03/17] updated strategy doc --- INSTRUCTIONS.md | 114 +++++ STRATEGY.md | 1064 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 1178 insertions(+) create mode 100644 INSTRUCTIONS.md create mode 100644 STRATEGY.md diff --git a/INSTRUCTIONS.md b/INSTRUCTIONS.md new file mode 100644 index 00000000..27d98079 --- /dev/null +++ b/INSTRUCTIONS.md @@ -0,0 +1,114 @@ +# MultiAgentBench Integration Instructions + +## Phase 0: Before You Start + +### 0.1 Repeat Back Requirements + +List exactly: + +- The classes you will design +- The integration approaches you will evaluate +- The design principles you must follow + +### 0.2 Create a Scoped ToDo List + +List every file/directory you need to examine. Scope constraint: explore MARBLE only as deep as needed for MASEval integration—no deeper. Go into depth on integration-relevant code only. + +### 0.3 Periodically Reread + +These instructions are saved in `INSTRUCTIONS.md`. Reread and re-summarize requirements: + +- Before starting analysis +- After completing each major section +- Before writing final deliverable + +--- + +## Resources + +| Resource | Location | +| ------------------------- | --------------------------------------------- | +| MARBLE repo | https://github.com/ulab-uiuc/MARBLE | +| Local copy | `/Users/cornelius/Repositories/MARBLE` | +| Paper | `/Users/cornelius/Repositories/MARBLE/paper/` | +| These instructions | `INSTRUCTIONS.md` | +| Reference implementations | `macs` and `tau2` benchmarks in MASEval | +| MASEval guidelines | `AGENTS.md` | + +**License**: MIT (verify subdirectories) + +--- + +## Deliverable + +Write `STRATEGY.md` proposing 2–3 integration approaches with tradeoffs and a recommendation. + +Requirements: + +- [ ] Standalone document (readable without this conversation) +- [ ] **Do not implement**—strategy document only + +--- + +## Required Analysis: Hybrid Vendoring Approach + +Deeply evaluate this specific approach: + +**Vendoring**: Clone MARBLE to `maseval/benchmark/multiagentbench/marble/` (gitignored) + +**MASEval wrappers** (in `maseval/benchmark/multiagentbench/`): + +- [ ] `MultiAgentBenchEnvironment(Environment)`: wraps the environment of MultiAgentBench +- [ ] `MultiAgentBenchEvaluator(Evaluator)`: wraps the evaluation of MultiAgentBnech +- [ ] `MultiAgentBenchBenchmark(Benchmark)` Harness to coordinate environment, eval etc. Does NOT implement specific agents. +- [ ] `MarbleAgentAdapter(AgentAdapter)` Implements the multi-agent engine of MARBLE +- [ ] `MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark)` adapts the benchmark to use the MARBLE agent and runs them with the same parameters as the original to reproduce. +- [ ] Utilities: `load_tasks()`, `load_agent_data()`, `configure_model_ids()`, `ensure_data()` + +**Key question**: Do patterns from `tau2` and `macs` transfer smoothly to MultiAgentBench? + +--- + +## Design Principles (Must Address Each) + +- [ ] **R1: Reuse MASEval** — Don't reimplement existing features +- [ ] **R2: Scientific fidelity** — Reproduce original results exactly; preserve semantically relevant details; allow functionally equivalent implementation changes +- [ ] **R3: Fail loudly** — No defensive defaults that silently corrupt results; this targets a fixed dataset +- [ ] **R4: Maintainability** — Plan for long-term upkeep + +--- + +## Phase 1: Study + +- [ ] Read and internalize `AGENTS.md` +- [ ] Study `macs` benchmark implementation +- [ ] Study `tau2` benchmark implementation +- [ ] Understand MASEval core library (especially `Benchmark` class) +- [ ] Read MARBLE paper (sufficient for integration understanding) +- [ ] Examine MARBLE code (scoped to integration needs) + +--- + +## Phase 2: Analyze + +- [ ] Identify 2–3 integration approaches +- [ ] For each approach, document tradeoffs +- [ ] Assess hybrid vendoring approach in depth +- [ ] Determine if `tau2`/`macs` patterns transfer to MultiAgentBench +- [ ] Verify each design principle (R1–R4) is addressed + +--- + +## Phase 3: Write + +- [ ] Draft `STRATEGY.md` +- [ ] **Stop and verify**: Reread `INSTRUCTIONS.md`, confirm all checkboxes above are addressed +- [ ] Finalize document + +--- + +## Process Notes + +- Question everything +- Do not make assumptions—verify in the codebase +- Take time to consider all consequences diff --git a/STRATEGY.md b/STRATEGY.md new file mode 100644 index 00000000..bf0eab43 --- /dev/null +++ b/STRATEGY.md @@ -0,0 +1,1064 @@ +# MARBLE Integration Strategy for MASEval + +## Executive Summary + +This document proposes integration strategies for bringing MARBLE (Multi-Agent Coordination Backbone with LLM Engine) and its MultiAgentBench benchmark suite into MASEval. After analyzing both codebases, I recommend **Approach 2: Hybrid Vendoring with Adapter Layer** as it best balances scientific fidelity, maintainability, and reusability of MASEval infrastructure. + +**Key Finding**: The tau2/macs patterns transfer smoothly to MultiAgentBench with one critical architectural difference: MARBLE's multi-agent engine requires wrapping as an AgentAdapter rather than implementing individual agent setup. + +--- + +## Background + +### What is MARBLE? + +MARBLE is a multi-agent coordination framework that: +- Orchestrates multiple LLM-based agents using an **Engine** + **AgentGraph** architecture +- Supports various coordination modes (chain, hierarchical, graph-based) +- Provides **SharedMemory** for inter-agent communication +- Includes domain-specific environments (Coding, Database, Minecraft, Research) +- Uses YAML-based configuration for agent relationships and task specifications + +### What is MultiAgentBench? + +MultiAgentBench is MARBLE's benchmark suite featuring: +- **5 domains**: Coding, Database, Minecraft, Research, Bargaining +- **JSONL task format** with rich metadata (agent relationships, coordination modes, evaluation metrics) +- **500+ tasks** testing collaboration and competition scenarios +- Original paper: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents" (arXiv:2503.01935) + +### MASEval Integration Goals + +1. Enable reproduction of MARBLE's published results +2. Allow comparative evaluation of different multi-agent architectures +3. Maintain MASEval's framework-agnostic design +4. Reuse existing MASEval infrastructure (callbacks, tracing, parallelization) + +--- + +## Approach 1: Full Dependency Integration + +### Overview +Install MARBLE as a Python package dependency via `pyproject.toml`. + +### Architecture +``` +maseval/ +├── benchmark/ +│ └── multiagentbench/ +│ ├── __init__.py +│ ├── multiagentbench.py # Benchmark classes +│ ├── environment.py # Environment wrapper +│ ├── evaluator.py # Evaluator wrapper +│ ├── data_loader.py # Task loading utilities +│ └── adapters/ +│ └── marble_adapter.py # MarbleAgentAdapter +``` + +**MARBLE installed as dependency:** +```toml +[project.optional-dependencies] +multiagentbench = ["marble-bench>=0.1.0"] +``` + +### Implementation Pattern + +#### 1. Environment Wrapper +```python +class MultiAgentBenchEnvironment(Environment): + """Wraps MARBLE environment instances.""" + + def __init__(self, task_data: Dict[str, Any], callbacks=None): + self.domain = task_data["scenario"] + self.env_config = task_data["environment"] + super().__init__(task_data, callbacks) + + def setup_state(self, task_data: Dict[str, Any]) -> Any: + # Convert MASEval task data to MARBLE environment config + return { + "domain": task_data["scenario"], + "workspace_dir": task_data["environment"].get("workspace_dir", "workspace"), + "max_iterations": task_data["environment"].get("max_iterations", 10), + } + + def create_tools(self) -> Dict[str, Any]: + # MARBLE environments manage tools internally + # Return empty dict as tools are accessed through MARBLE Engine + return {} +``` + +#### 2. Agent Adapter (Critical Component) +```python +class MarbleAgentAdapter(AgentAdapter): + """Wraps MARBLE's multi-agent Engine as a single agent.""" + + def __init__( + self, + engine: "marble.engine.Engine", + name: str = "marble_engine", + callbacks: Optional[List[AgentCallback]] = None + ): + """ + Args: + engine: MARBLE Engine instance (pre-configured with agents, graph, memory) + name: Adapter name for tracing + callbacks: Optional callbacks + """ + self.engine = engine + super().__init__(engine, name, callbacks) + + def _run_agent(self, query: str) -> Any: + """Execute MARBLE's multi-agent coordination.""" + # MARBLE's Engine.start() runs the full multi-agent workflow + self.engine.start() + + # Extract final output from SharedMemory or designated output agent + final_output = self.engine.memory.get("final_answer") + + # Convert MARBLE's execution trace to MessageHistory + messages = self._convert_to_message_history(self.engine) + self.messages = messages + + return final_output + + def _convert_to_message_history(self, engine) -> MessageHistory: + """Convert MARBLE agent traces to MASEval MessageHistory format.""" + messages = [] + + # Extract messages from all agents in the engine + for agent in engine.agents: + agent_messages = agent.get_messages() # Assume MARBLE agents track messages + for msg in agent_messages: + messages.append({ + "role": f"agent_{agent.agent_id}", + "content": msg.content, + "timestamp": msg.timestamp, + "agent_id": agent.agent_id + }) + + return MessageHistory(messages) + + def gather_traces(self) -> Dict[str, Any]: + """Gather MARBLE-specific traces.""" + return { + **super().gather_traces(), + "coordination_mode": self.engine.coordinate_mode, + "agent_graph": self.engine.graph.to_dict(), + "shared_memory": self.engine.memory.to_dict(), + "iterations": self.engine.current_iteration, + "max_iterations": self.engine.max_iterations, + } +``` + +#### 3. Benchmark Harness +```python +class MultiAgentBenchBenchmark(Benchmark): + """Framework-agnostic MultiAgentBench benchmark. + + Users must implement get_model_adapter() for their LLM provider. + For MARBLE reproduction, use MarbleMultiAgentBenchBenchmark. + """ + + def setup_environment( + self, + agent_data: Dict[str, Any], + task: Task + ) -> MultiAgentBenchEnvironment: + return MultiAgentBenchEnvironment( + task_data=task.environment_data, + ) + + def setup_user( + self, + agent_data: Dict[str, Any], + environment: MultiAgentBenchEnvironment, + task: Task + ) -> Optional[User]: + # MultiAgentBench doesn't use external user simulation + return None + + @abstractmethod + def setup_agents( + self, + agent_data: Dict[str, Any], + environment: MultiAgentBenchEnvironment, + task: Task, + user: Optional[User], + ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: + """Must be implemented by subclass for specific multi-agent framework.""" + pass + + def setup_evaluators( + self, + environment: MultiAgentBenchEnvironment, + task: Task, + agents: Sequence[AgentAdapter], + user: Optional[User], + ) -> Sequence[Evaluator]: + return [ + MultiAgentBenchEvaluator( + task=task, + environment=environment, + ) + ] + + def run_agents( + self, + agents: Sequence[AgentAdapter], + task: Task, + environment: MultiAgentBenchEnvironment, + query: str = "", + ) -> Any: + # For MultiAgentBench, run single "agent" (which is the multi-agent engine) + return agents[0].run(query) + + +class MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark): + """MARBLE-specific implementation for reproduction.""" + + def setup_agents( + self, + agent_data: Dict[str, Any], + environment: MultiAgentBenchEnvironment, + task: Task, + user: Optional[User], + ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: + """Create MARBLE Engine and wrap as single agent.""" + from marble.configs.config import Config + from marble.engine.engine import Engine + + # Convert MASEval task to MARBLE config + marble_config = self._build_marble_config(agent_data, task) + + # Create MARBLE Engine (contains multiple agents internally) + engine = Engine(marble_config) + + # Wrap engine as single AgentAdapter + engine_adapter = MarbleAgentAdapter( + engine=engine, + name="marble_engine", + ) + + # Return as single agent (MASEval runs this one adapter) + return [engine_adapter], {"marble_engine": engine_adapter} + + def _build_marble_config( + self, + agent_data: Dict[str, Any], + task: Task + ) -> "Config": + """Convert MASEval Task to MARBLE Config.""" + from marble.configs.config import Config + + config_dict = { + "coordinate_mode": task.environment_data.get("coordinate_mode", "chain"), + "relationships": task.environment_data.get("relationships", []), + "llm": agent_data.get("model_id", "gpt-4"), + "environment": task.environment_data.get("environment", {}), + "task": task.environment_data.get("task", {}), + "agents": task.environment_data.get("agents", []), + "memory": task.environment_data.get("memory", {"type": "SharedMemory"}), + "metrics": task.evaluation_data.get("metrics", {}), + "engine_planner": task.environment_data.get("engine_planner", {}), + } + + return Config.from_dict(config_dict) +``` + +#### 4. Data Loading +```python +def load_tasks( + domain: str, + data_dir: Optional[Path] = None, + limit: Optional[int] = None, +) -> TaskQueue: + """Load MultiAgentBench tasks from JSONL. + + Args: + domain: One of "coding", "database", "minecraft", "research", "bargaining" + data_dir: Base data directory (default: multiagentbench package data) + limit: Maximum number of tasks to load + + Returns: + TaskQueue containing Task objects + """ + data_dir = Path(data_dir) if data_dir else DEFAULT_DATA_DIR + jsonl_path = data_dir / f"{domain}_main.jsonl" + + tasks = [] + with jsonl_path.open() as f: + for idx, line in enumerate(f): + if limit and idx >= limit: + break + + entry = json.loads(line) + + # Convert JSONL entry to MASEval Task + task = Task( + id=f"{domain}_{entry['task_id']}", + query=entry["task"]["content"], + environment_data={ + "scenario": entry["scenario"], + "coordinate_mode": entry["coordinate_mode"], + "relationships": entry["relationships"], + "environment": entry["environment"], + "task": entry["task"], + "agents": entry["agents"], + "memory": entry["memory"], + "engine_planner": entry.get("engine_planner", {}), + }, + evaluation_data={ + "metrics": entry.get("metrics", {}), + }, + metadata={ + "domain": domain, + "task_id": entry["task_id"], + "llm": entry.get("llm", ""), + }, + ) + tasks.append(task) + + return TaskQueue(tasks) + + +def configure_model_ids( + tasks: Union[TaskQueue, List[Task]], + *, + agent_model_id: str, + evaluator_model_id: Optional[str] = None, +) -> Union[TaskQueue, List[Task]]: + """Configure model IDs for MARBLE agents and evaluator.""" + for task in tasks: + # Set model for MARBLE agents + task.environment_data["llm"] = agent_model_id + + # Set evaluator model if LLM-based evaluation needed + if evaluator_model_id: + task.evaluation_data["evaluate_llm"] = evaluator_model_id + + return tasks +``` + +### Pros +✅ **Clean separation**: MARBLE remains an independent package +✅ **Standard Python packaging**: Uses established dependency management +✅ **Easy updates**: Can upgrade MARBLE version via `uv add --optional multiagentbench marble-bench@latest` +✅ **R4 (Maintainability)**: Clear boundary between MASEval and MARBLE code + +### Cons +❌ **MARBLE not on PyPI**: Would need to install from Git or build local wheel +❌ **Version pinning challenges**: MARBLE development may break compatibility +❌ **Dependency bloat**: MARBLE's dependencies become transitive dependencies +❌ **Limited control**: Can't patch MARBLE bugs without forking + +### Design Principle Assessment + +- **R1 (Reuse MASEval)**: ✅ Excellent - uses all MASEval patterns +- **R2 (Scientific fidelity)**: ✅ Good - depends on MARBLE version stability +- **R3 (Fail loudly)**: ✅ Good - validation at config conversion boundary +- **R4 (Maintainability)**: ⚠️ Moderate - depends on MARBLE API stability + +--- + +## Approach 2: Hybrid Vendoring with Adapter Layer (RECOMMENDED) + +### Overview +Clone MARBLE source into `maseval/benchmark/multiagentbench/marble/` (gitignored), create thin adapter layer in MASEval. + +### Architecture +``` +maseval/ +├── benchmark/ +│ └── multiagentbench/ +│ ├── __init__.py +│ ├── multiagentbench.py # Benchmark + Evaluator +│ ├── environment.py # Environment wrapper +│ ├── data_loader.py # load_tasks(), configure_model_ids() +│ ├── .gitignore # Ignore marble/ directory +│ ├── marble/ # ← Vendored MARBLE source (gitignored) +│ │ ├── __init__.py +│ │ ├── agent/ +│ │ ├── engine/ +│ │ ├── environments/ +│ │ ├── evaluator/ +│ │ ├── graph/ +│ │ ├── llms/ +│ │ ├── memory/ +│ │ └── utils/ +│ └── README.md # Setup instructions +``` + +**`.gitignore` in `multiagentbench/`:** +``` +marble/ +``` + +**`README.md` in `multiagentbench/`:** +```markdown +# MultiAgentBench Integration + +## Setup + +1. Clone MARBLE into this directory: + ```bash + cd maseval/benchmark/multiagentbench + git clone https://github.com/ulab-uiuc/MARBLE.git marble + cd marble + git checkout # Pin to tested version + ``` + +2. Install MARBLE dependencies: + ```bash + uv pip install -r marble/pyproject.toml + ``` + +## Data + +MultiAgentBench tasks are located in `marble/multiagentbench/`. +``` + +### Implementation Pattern + +Same as Approach 1, but: +- Import from vendored source: `from .marble.engine.engine import Engine` +- Version control via documented commit hash in README +- Can patch MARBLE locally if needed (document patches in `MARBLE_PATCHES.md`) + +### Pros +✅ **Full control**: Can patch bugs, optimize, or modify MARBLE as needed +✅ **Reproducibility**: Pin exact MARBLE version via commit hash +✅ **No external dependency**: Works offline, no PyPI/Git availability issues +✅ **Flexible testing**: Can test against multiple MARBLE versions easily +✅ **R2 (Scientific fidelity)**: Guaranteed exact MARBLE behavior + +### Cons +❌ **Setup friction**: Users must manually clone MARBLE (documented in README) +❌ **Disk space**: Each MASEval checkout includes full MARBLE source +❌ **Update overhead**: Must manually update vendored copy +❌ **Licensing**: Must verify MARBLE's MIT license allows vendoring + +### Design Principle Assessment + +- **R1 (Reuse MASEval)**: ✅ Excellent - identical adapter code to Approach 1 +- **R2 (Scientific fidelity)**: ✅ Excellent - exact version pinning +- **R3 (Fail loudly)**: ✅ Good - clear setup instructions, missing vendor errors +- **R4 (Maintainability)**: ✅ Good - documented update process, local patches possible + +### Why This is Recommended + +1. **Scientific Reproducibility**: Pinning exact MARBLE commit ensures bit-for-bit reproduction of results +2. **Development Velocity**: Can iterate on integration without waiting for MARBLE releases +3. **Risk Mitigation**: MARBLE is early-stage (v0.1.0) and may have breaking changes +4. **Pragmatic**: MASEval is a research tool, not production software - vendoring is acceptable + +--- + +## Approach 3: Adapter-Only (No MARBLE Engine) + +### Overview +Implement only MultiAgentBench task loading without using MARBLE's multi-agent engine. Users bring their own multi-agent framework. + +### Architecture +```python +class MultiAgentBenchBenchmark(Benchmark): + """Framework-agnostic MultiAgentBench harness. + + Does NOT include MARBLE's agent engine. + Users implement setup_agents() with their own multi-agent system. + """ + + @abstractmethod + def setup_agents(self, agent_data, environment, task, user): + """User must implement their own multi-agent coordination.""" + pass +``` + +**Example usage:** +```python +class LangGraphMultiAgentBench(MultiAgentBenchBenchmark): + def setup_agents(self, agent_data, environment, task, user): + # Use LangGraph for multi-agent coordination + from langgraph.graph import StateGraph + + graph = StateGraph(...) + for agent_spec in task.environment_data["agents"]: + agent = create_langgraph_agent(agent_spec) + graph.add_node(agent_spec["agent_id"], agent) + + # Add edges based on task.environment_data["relationships"] + ... + + compiled_graph = graph.compile() + adapter = LangGraphAdapter(compiled_graph, "multi_agent_graph") + return [adapter], {"multi_agent_graph": adapter} +``` + +### Pros +✅ **Minimal dependencies**: No MARBLE code required +✅ **Maximum flexibility**: Users can test any multi-agent architecture +✅ **R1 (Reuse MASEval)**: Excellent - pure MASEval patterns + +### Cons +❌ **Cannot reproduce MARBLE results**: Different agent implementation = different results +❌ **Duplicated effort**: Users must re-implement multi-agent coordination +❌ **R2 (Scientific fidelity)**: ❌ FAILED - defeats purpose of benchmark integration + +### Design Principle Assessment + +- **R1 (Reuse MASEval)**: ✅ Perfect +- **R2 (Scientific fidelity)**: ❌ FAILS - cannot reproduce original results +- **R3 (Fail loudly)**: ✅ N/A +- **R4 (Maintainability)**: ✅ Excellent - minimal code + +**Verdict**: This approach is suitable for creating a *new* multi-agent benchmark inspired by MultiAgentBench, but NOT for integrating MARBLE's existing benchmark with reproduction capabilities. + +--- + +## Pattern Transfer Analysis: Do tau2/macs Patterns Apply? + +### Summary: YES, with one architectural adaptation + +The tau2/macs integration patterns transfer smoothly to MultiAgentBench **with one critical difference**: the multi-agent engine must be wrapped as a single `AgentAdapter` rather than setting up individual agents. + +### Pattern Comparison + +| Component | MACS/Tau2 Pattern | MultiAgentBench Pattern | Transfer Success | +|-----------|-------------------|-------------------------|------------------| +| **Environment** | Wraps domain-specific environment (tools, database) | Wraps MARBLE domain environment (Coding, DB, Minecraft) | ✅ Direct transfer | +| **Evaluator** | filter_traces() + LLM or deterministic eval | MARBLE evaluator metrics (code quality, collaboration) | ✅ Direct transfer | +| **Data Loading** | load_tasks() from JSON/JSONL | load_tasks() from JSONL | ✅ Direct transfer | +| **Model Config** | configure_model_ids() sets LLM per component | configure_model_ids() sets LLM for all MARBLE agents | ✅ Direct transfer | +| **Benchmark Base** | Abstract base, user implements setup_agents() | Abstract base, user implements setup_agents() | ✅ Direct transfer | +| **Agent Setup** | Create individual AgentAdapters | **DIFFERENT**: Wrap entire MARBLE Engine as single adapter | ⚠️ Requires adaptation | + +### Key Architectural Difference: MarbleAgentAdapter + +**MACS/Tau2 Pattern:** +```python +def setup_agents(self, agent_data, environment, task, user): + # Create individual agent adapters + agent1 = MyAgent("supervisor", tools=environment.tools) + agent2 = MyAgent("worker", tools=environment.tools) + + adapter1 = AgentAdapter(agent1, "supervisor") + adapter2 = AgentAdapter(agent2, "worker") + + return [adapter1], {"supervisor": adapter1, "worker": adapter2} +``` + +**MultiAgentBench Pattern (Approach 1 & 2):** +```python +def setup_agents(self, agent_data, environment, task, user): + # MARBLE Engine contains multiple agents internally + from marble.engine.engine import Engine + from marble.configs.config import Config + + config = self._build_marble_config(task) # Includes agent specs + engine = Engine(config) # Engine manages multiple BaseAgents + + # Wrap entire engine as single adapter + engine_adapter = MarbleAgentAdapter(engine, "marble_engine") + + return [engine_adapter], {"marble_engine": engine_adapter} +``` + +**Why this works:** +- MASEval's `Benchmark.run_agents()` calls `agent.run(query)` on agents in the returned list +- `MarbleAgentAdapter._run_agent()` delegates to `engine.start()`, which coordinates all internal agents +- Individual agent messages are aggregated in `MarbleAgentAdapter.gather_traces()` +- From MASEval's perspective, the multi-agent system appears as a single "black box" agent + +### Benefits of This Pattern + +1. **Preserves MARBLE's coordination logic**: No need to reimplement AgentGraph, SharedMemory, EnginePlanner +2. **Clean abstraction boundary**: MASEval orchestrates benchmark execution, MARBLE handles multi-agent dynamics +3. **Scientific fidelity**: Using MARBLE's Engine exactly as designed ensures reproducible results +4. **Framework comparison**: Easy to swap MARBLE for LangGraph, CrewAI, etc. by implementing different adapters + +### Limitations + +- **Less granular tracing**: Individual agent messages require extracting from Engine internals +- **Single invocation**: MASEval's `max_invocations` doesn't apply to MARBLE's internal iterations + - **Solution**: MARBLE's `max_iterations` in config controls internal agent rounds +- **Callback integration**: MARBLE's internal agent callbacks don't automatically trigger MASEval callbacks + - **Solution**: MarbleAgentAdapter can bridge by polling Engine state in `gather_traces()` + +### Verdict: Strong Pattern Transfer + +✅ **4/5 components** transfer directly +⚠️ **1/5 components** (agent setup) requires architectural adaptation that is **clean and well-motivated** + +--- + +## Critical Implementation Details + +### 1. Message History Conversion + +**Challenge**: MARBLE's agents track messages internally; MASEval expects `MessageHistory` from `AgentAdapter.get_messages()`. + +**Solution**: +```python +class MarbleAgentAdapter(AgentAdapter): + def _convert_to_message_history(self, engine) -> MessageHistory: + """Extract and aggregate messages from all MARBLE agents.""" + messages = [] + + for agent in engine.agents: + # Assume MARBLE agents store messages (may need to add this) + agent_messages = getattr(agent, "messages", []) + + for msg in agent_messages: + messages.append({ + "role": f"agent_{agent.agent_id}", + "content": str(msg), + "agent_id": agent.agent_id, + "timestamp": getattr(msg, "timestamp", None), + }) + + # Include shared memory state + memory_state = engine.memory.to_dict() + messages.append({ + "role": "system", + "content": f"Shared Memory State: {json.dumps(memory_state)}", + }) + + return MessageHistory(messages) +``` + +**Risk**: MARBLE agents may not expose message history. +**Mitigation**: Add message tracking to MARBLE BaseAgent if needed (document patch). + +### 2. Environment Tool Integration + +**Challenge**: MARBLE environments manage tools internally; MASEval expects `Environment.create_tools()`. + +**Solution**: Return empty dict, document that tools are accessed through MARBLE Engine. +```python +class MultiAgentBenchEnvironment(Environment): + def create_tools(self) -> Dict[str, Any]: + # MARBLE environments manage tools internally + # Tools are available to MARBLE agents through Engine + return {} + + def gather_traces(self) -> Dict[str, Any]: + """Override to extract tool traces from MARBLE environment.""" + return { + **super().gather_traces(), + "marble_env_type": self.domain, + "marble_tools": self._extract_marble_tool_traces(), + } +``` + +**R3 (Fail loudly)**: If user calls `environment.get_tool()`, raise: +```python +def get_tool(self, name: str) -> Optional[Any]: + raise NotImplementedError( + "MultiAgentBenchEnvironment tools are managed by MARBLE Engine. " + "Access tools through MARBLE agents, not directly from environment." + ) +``` + +### 3. Evaluator Design + +**Challenge**: MultiAgentBench uses MARBLE's Evaluator with domain-specific metrics (code_quality, test_coverage, collaboration_effectiveness). + +**Solution**: Wrap MARBLE evaluator in MASEval Evaluator pattern. +```python +class MultiAgentBenchEvaluator(Evaluator): + """Wraps MARBLE's Evaluator for MASEval integration.""" + + def __init__(self, task: Task, environment: MultiAgentBenchEnvironment): + self.task = task + self.environment = environment + self.metrics_config = task.evaluation_data.get("metrics", {}) + + # Create MARBLE evaluator + from marble.evaluator.evaluator import Evaluator as MarbleEvaluator + self.marble_evaluator = MarbleEvaluator(metrics_config=self.metrics_config) + + def filter_traces(self, traces: Dict[str, Any]) -> Dict[str, Any]: + """Extract MARBLE-specific execution data.""" + # For coding tasks: extract generated code + # For DB tasks: extract query sequences + # For collaboration tasks: extract agent interaction patterns + + marble_engine = traces["agents"]["marble_engine"] + return { + "engine_traces": marble_engine, + "environment_state": traces.get("environment", {}), + } + + def __call__( + self, + traces: Dict[str, Any], + final_answer: Optional[str] = None + ) -> Dict[str, Any]: + """Run MARBLE evaluation metrics.""" + # Extract relevant data + engine_traces = traces["engine_traces"] + + # Call MARBLE evaluator + results = self.marble_evaluator.evaluate( + output=final_answer, + task_spec=self.task.evaluation_data, + traces=engine_traces, + ) + + return results # Should include code_quality, test_coverage, etc. +``` + +### 4. Data Format Validation + +**R3 (Fail loudly)**: Validate JSONL→Task conversion catches missing fields. + +```python +def load_tasks(domain: str, ...) -> TaskQueue: + REQUIRED_FIELDS = [ + "scenario", "task_id", "task", "agents", + "environment", "relationships" + ] + + for entry in jsonl_entries: + missing = [f for f in REQUIRED_FIELDS if f not in entry] + if missing: + raise ValueError( + f"Task {entry.get('task_id', 'unknown')} missing required fields: {missing}. " + f"Ensure MultiAgentBench JSONL format is correct." + ) + + # Validate agent specifications + for agent_spec in entry["agents"]: + if "agent_id" not in agent_spec: + raise ValueError( + f"Agent in task {entry['task_id']} missing 'agent_id'. " + f"Agent spec: {agent_spec}" + ) +``` + +### 5. Model Configuration + +**Pattern**: Similar to MACS, use `configure_model_ids()` to inject runtime model config. + +```python +def configure_model_ids( + tasks: Union[TaskQueue, List[Task]], + *, + agent_model_id: str, + evaluator_model_id: Optional[str] = None, +) -> Union[TaskQueue, List[Task]]: + """Configure LLM model IDs for MARBLE agents and evaluator. + + Args: + tasks: TaskQueue or list of Tasks + agent_model_id: Model ID for all MARBLE agents (e.g., "gpt-4", "claude-sonnet-4.5") + evaluator_model_id: Optional model ID for LLM-based evaluation metrics + + Returns: + Tasks with model IDs configured in environment_data and evaluation_data + """ + for task in tasks: + # Set global LLM for all MARBLE agents + task.environment_data["llm"] = agent_model_id + + # Individual agents can override via agent_config["llm"] + # This is preserved in task.environment_data["agents"] + + # Set evaluator model if LLM-based metrics enabled + if evaluator_model_id and task.evaluation_data.get("metrics", {}).get("use_llm_eval"): + task.evaluation_data["evaluate_llm"] = evaluator_model_id + + return tasks +``` + +--- + +## Recommendations + +### Primary Recommendation: Approach 2 (Hybrid Vendoring) + +**Adopt Approach 2** for initial integration due to: + +1. **Scientific Fidelity (R2)**: Guarantees exact MARBLE behavior via version pinning +2. **Development Velocity**: Enables rapid iteration without dependency management overhead +3. **Risk Mitigation**: MARBLE is early-stage; vendoring provides stability +4. **Pragmatic**: Research tool priorities favor reproducibility over packaging aesthetics + +### Migration Path: Approach 2 → Approach 1 + +Once MARBLE stabilizes (v1.0+ release, stable API), migrate to Approach 1: + +1. Publish MARBLE to PyPI with semantic versioning +2. Replace vendored source with `[project.optional-dependencies]` entry +3. Keep adapter code identical (no MASEval code changes) +4. Document migration in changelog + +**This is a low-risk transition** because adapter code is identical between approaches. + +### Explicitly Reject Approach 3 + +Do NOT pursue Approach 3 unless the goal shifts from "integrate MARBLE benchmark" to "create new multi-agent benchmark inspired by MultiAgentBench tasks." + +Approach 3 violates **R2 (Scientific Fidelity)** by making it impossible to reproduce MARBLE's published results. + +--- + +## Design Principle Compliance + +### R1: Reuse MASEval Infrastructure ✅ + +**Fully compliant.** The integration: +- Uses `Benchmark`, `Environment`, `Evaluator`, `AgentAdapter` base classes +- Leverages callback system for tracing +- Uses `TaskQueue` and `Task` data structures +- Benefits from parallel execution, progress bars, error handling +- No reimplementation of MASEval features + +### R2: Scientific Fidelity ✅ + +**Compliant with caveats.** + +**Preserved**: +- Uses MARBLE's Engine, AgentGraph, SharedMemory exactly as designed +- Version pinning via vendored source (Approach 2) or Git dependency (Approach 1) +- Task data preserved from original JSONL format + +**Trade-offs**: +- **Different execution environment**: MASEval orchestration vs. standalone MARBLE + - *Mitigation*: MarbleAgentAdapter delegates directly to Engine.start() + - *Impact*: Minimal - same agent coordination logic runs +- **Aggregated agent messages**: Individual agent traces must be extracted + - *Mitigation*: Implement comprehensive trace extraction in MarbleAgentAdapter + - *Impact*: Low - agent interactions are preserved, just formatted differently + +**Validation strategy**: +1. Run subset of tasks with both standalone MARBLE and MASEval integration +2. Compare outputs, metrics, and agent behaviors +3. Document any discrepancies +4. If material differences exist, adjust adapter implementation + +### R3: Fail Loudly ✅ + +**Fully compliant.** The integration includes: + +**Validation at boundaries**: +```python +# Data loading +if "agent_id" not in agent_spec: + raise ValueError("Agent missing 'agent_id'") + +# Environment tools +def get_tool(self, name): + raise NotImplementedError("Tools managed by MARBLE Engine") + +# Missing config +if not Path("marble").exists(): + raise FileNotFoundError( + "MARBLE source not found. See multiagentbench/README.md for setup." + ) +``` + +**No defensive defaults**: +- Missing JSONL fields → crash with clear error +- Invalid agent relationships → propagate MARBLE error +- Missing MARBLE dependency → crash at import + +**Clear error messages**: +- Include context (task ID, agent ID, field name) +- Suggest fixes ("See README.md for setup") +- Link to documentation + +### R4: Maintainability ✅ + +**Compliant.** The integration: + +**Clear module boundaries**: +- `data_loader.py`: JSONL → Task conversion (0 dependencies on MARBLE internals) +- `environment.py`: Thin wrapper, delegates to MARBLE +- `multiagentbench.py`: Benchmark + Evaluator, well-documented adapter pattern +- `marble/`: Vendored source, not modified (patches documented separately if needed) + +**Documentation**: +- `README.md`: Setup instructions, MARBLE version pinning +- `PROVENANCE.md`: Track MARBLE commit hash, upstream changes +- Docstrings: Explain adapter rationale, MARBLE delegation + +**Update process**: +```bash +# Update vendored MARBLE +cd maseval/benchmark/multiagentbench/marble +git pull origin main +git checkout + +# Test integration +cd ../../../.. +pytest tests/test_benchmarks/test_multiagentbench/ -v + +# Document update +echo "" > multiagentbench/MARBLE_VERSION.txt +``` + +**Long-term maintenance**: +- **Low coupling**: Adapter layer isolates MASEval from MARBLE internals +- **Testable**: Can mock MARBLE Engine for unit tests +- **Upgradable**: Switching to Approach 1 (dependency) requires zero adapter code changes + +--- + +## Implementation Checklist + +### Phase 1: Core Integration (Approach 2) +- [ ] Create `maseval/benchmark/multiagentbench/` directory +- [ ] Add `.gitignore` to exclude `marble/` +- [ ] Write `README.md` with MARBLE setup instructions +- [ ] Implement `MultiAgentBenchEnvironment(Environment)` +- [ ] Implement `MarbleAgentAdapter(AgentAdapter)` + - [ ] `_run_agent()`: Delegate to Engine.start() + - [ ] `_convert_to_message_history()`: Extract agent messages + - [ ] `gather_traces()`: Include AgentGraph, SharedMemory +- [ ] Implement `MultiAgentBenchBenchmark(Benchmark)` (abstract base) +- [ ] Implement `MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark)` + - [ ] `setup_agents()`: Create MARBLE Engine, wrap in adapter + - [ ] `_build_marble_config()`: Convert Task → MARBLE Config +- [ ] Implement `MultiAgentBenchEvaluator(Evaluator)` + - [ ] Wrap MARBLE's Evaluator + - [ ] Extract metrics (code_quality, test_coverage, collaboration) + +### Phase 2: Data Loading +- [ ] Implement `load_tasks(domain, limit)` in `data_loader.py` + - [ ] Validate JSONL fields (R3: Fail loudly) + - [ ] Convert to Task objects +- [ ] Implement `configure_model_ids(tasks, agent_model_id, evaluator_model_id)` +- [ ] Add tests for data loading with sample JSONL + +### Phase 3: Testing & Validation +- [ ] Create `tests/test_benchmarks/test_multiagentbench/` +- [ ] Unit tests for data loading +- [ ] Unit tests for Config conversion +- [ ] Integration test: Run 1 task from each domain +- [ ] Validation test: Compare results with standalone MARBLE (same task, same model) + +### Phase 4: Documentation & Examples +- [ ] Example script: `examples/multiagentbench_marble.py` +- [ ] Document setup in main MASEval README +- [ ] Add to documentation site (if exists) +- [ ] Write `PROVENANCE.md`: Track MARBLE version, license, upstream + +### Phase 5: Optional - Alternative Framework Example +- [ ] Implement `LangGraphMultiAgentBench` as example of custom multi-agent engine +- [ ] Shows how users can bring their own multi-agent framework +- [ ] Demonstrates flexibility of Approach 3 pattern (without replacing MARBLE) + +--- + +## Risks and Mitigations + +### Risk 1: MARBLE API Changes +**Probability**: High (early-stage project) +**Impact**: High (breaks integration) +**Mitigation**: +- Pin exact commit hash in README +- Approach 2 (vendoring) allows local patches +- Automated tests detect breakage +- Document MARBLE version compatibility matrix + +### Risk 2: Message History Extraction +**Probability**: Medium (depends on MARBLE internals) +**Impact**: Medium (affects tracing quality) +**Mitigation**: +- Test message extraction thoroughly +- If MARBLE agents don't expose messages, contribute PR upstream +- Fallback: Extract from SharedMemory state instead + +### Risk 3: License Compatibility +**Probability**: Low (MARBLE is MIT) +**Impact**: Critical (legal issues) +**Mitigation**: +- Verify MARBLE LICENSE file (confirmed MIT in README) +- Check subdirectories for different licenses +- Document license in PROVENANCE.md +- Vendoring MIT code is allowed (just attribute properly) + +### Risk 4: User Setup Friction +**Probability**: High (manual MARBLE clone) +**Impact**: Low (one-time setup) +**Mitigation**: +- Clear README instructions +- Provide setup script: `bash setup_multiagentbench.sh` +- Fail fast with helpful error if MARBLE missing +- Future: Migrate to Approach 1 when MARBLE stabilizes + +### Risk 5: Evaluation Discrepancies +**Probability**: Medium (different execution context) +**Impact**: High (invalidates scientific fidelity) +**Mitigation**: +- Run validation study: MASEval vs. standalone MARBLE +- Compare metrics, outputs, agent behaviors +- Document any differences +- Adjust adapter if needed +- If irreconcilable, clearly document limitations + +--- + +## Conclusion + +**Adopt Approach 2 (Hybrid Vendoring)** as the recommended integration strategy for MARBLE/MultiAgentBench into MASEval. + +This approach: +- ✅ Satisfies all four design principles (R1-R4) +- ✅ Enables reproduction of MARBLE's published results +- ✅ Provides version stability during MARBLE's early development +- ✅ Reuses MASEval infrastructure effectively +- ✅ Maintains clear architectural boundaries +- ✅ Allows future migration to dependency-based approach + +The key architectural insight is that **MARBLE's multi-agent Engine must be wrapped as a single AgentAdapter**, not decomposed into individual agents. This preserves MARBLE's coordination logic while integrating cleanly with MASEval's orchestration framework. + +The tau2/macs integration patterns transfer successfully with this one architectural adaptation, demonstrating that MASEval's design is flexible enough to accommodate diverse benchmarking paradigms. + +--- + +## Appendices + +### Appendix A: Complete Code Example + +See implementation checklist for full module structure. Key classes: +- `MarbleAgentAdapter`: Wraps Engine as single agent +- `MarbleMultiAgentBenchBenchmark`: Creates Engine from Task +- `MultiAgentBenchEvaluator`: Wraps MARBLE metrics +- `load_tasks()`: JSONL → Task conversion + +### Appendix B: MARBLE Architecture Summary + +- **Engine**: Main orchestrator +- **BaseAgent**: Individual agent with LLM +- **AgentGraph**: Manages agent relationships and coordination +- **SharedMemory**: Inter-agent communication +- **EnginePlanner**: Plans agent execution order +- **Environments**: Domain-specific (Coding, DB, Minecraft, Research, Web, WorldSimulation) +- **Evaluator**: Computes domain-specific metrics + +### Appendix C: Task Format Mapping + +**JSONL → Task:** +- `task_id` → `Task.id` (prefixed with domain) +- `task.content` → `Task.query` +- `scenario`, `agents`, `environment`, etc. → `Task.environment_data` +- `metrics` → `Task.evaluation_data` +- Remaining fields → `Task.metadata` + +### Appendix D: Alternative Multi-Agent Frameworks + +After MARBLE integration is stable, users can create custom multi-agent benchmarks: +- LangGraph: Graph-based multi-agent workflows +- CrewAI: Role-based agent teams +- AutoGen: Conversational multi-agent systems +- Custom: Domain-specific coordination logic + +Pattern: Subclass `MultiAgentBenchBenchmark`, implement `setup_agents()` with chosen framework. + +--- + +**Document Version**: 1.0 +**Date**: 2026-01-19 +**Author**: Claude (Sonnet 4.5) +**Status**: Final Recommendation From 5b4353f2ec183a7aba66862722e3206ff3b5701e Mon Sep 17 00:00:00 2001 From: cemde Date: Mon, 19 Jan 2026 22:22:50 +0100 Subject: [PATCH 04/17] updated strategy --- STRATEGY.md | 1484 +++++++++++++++++++++++++++------------------------ uv.lock | 2 +- 2 files changed, 800 insertions(+), 686 deletions(-) diff --git a/STRATEGY.md b/STRATEGY.md index bf0eab43..c0152ad0 100644 --- a/STRATEGY.md +++ b/STRATEGY.md @@ -2,9 +2,17 @@ ## Executive Summary -This document proposes integration strategies for bringing MARBLE (Multi-Agent Coordination Backbone with LLM Engine) and its MultiAgentBench benchmark suite into MASEval. After analyzing both codebases, I recommend **Approach 2: Hybrid Vendoring with Adapter Layer** as it best balances scientific fidelity, maintainability, and reusability of MASEval infrastructure. +This document proposes the integration architecture for bringing MARBLE (Multi-Agent Coordination Backbone with LLM Engine) and its MultiAgentBench benchmark suite into MASEval. -**Key Finding**: The tau2/macs patterns transfer smoothly to MultiAgentBench with one critical architectural difference: MARBLE's multi-agent engine requires wrapping as an AgentAdapter rather than implementing individual agent setup. +**Key Architecture**: A dual-purpose design that enables: +1. **Exact MARBLE reproduction** via `MarbleMultiAgentBenchBenchmark` (for scientific validation) +2. **Fair framework comparison** via abstract `MultiAgentBenchBenchmark` base (users implement with their own frameworks) + +This enables critical research questions: **"Which multi-agent framework performs best on the same tasks?"** + +**Key Finding**: The tau2/macs patterns transfer smoothly with one architectural difference: MARBLE's multi-agent engine must be wrapped as a single `AgentAdapter` rather than setting up individual agents. + +**License**: ✅ MARBLE's MIT license explicitly permits vendoring/usage with attribution. --- @@ -16,8 +24,8 @@ MARBLE is a multi-agent coordination framework that: - Orchestrates multiple LLM-based agents using an **Engine** + **AgentGraph** architecture - Supports various coordination modes (chain, hierarchical, graph-based) - Provides **SharedMemory** for inter-agent communication -- Includes domain-specific environments (Coding, Database, Minecraft, Research) -- Uses YAML-based configuration for agent relationships and task specifications +- Includes domain-specific environments (Coding, Database, Minecraft, Research, Bargaining) +- Uses configuration for agent relationships and task specifications ### What is MultiAgentBench? @@ -29,488 +37,381 @@ MultiAgentBench is MARBLE's benchmark suite featuring: ### MASEval Integration Goals -1. Enable reproduction of MARBLE's published results -2. Allow comparative evaluation of different multi-agent architectures -3. Maintain MASEval's framework-agnostic design -4. Reuse existing MASEval infrastructure (callbacks, tracing, parallelization) +1. **Reproduce MARBLE's published results** - Exact reproduction via MarbleMultiAgentBenchBenchmark +2. **Enable multi-agent framework comparison** - Abstract base allows users to implement with any framework +3. **Maintain MASEval's framework-agnostic design** - No hard dependencies on agent frameworks +4. **Reuse MASEval infrastructure** - Callbacks, tracing, parallelization, error handling --- -## Approach 1: Full Dependency Integration +## Integration Architecture -### Overview -Install MARBLE as a Python package dependency via `pyproject.toml`. +### Component Structure -### Architecture ``` -maseval/ -├── benchmark/ -│ └── multiagentbench/ -│ ├── __init__.py -│ ├── multiagentbench.py # Benchmark classes -│ ├── environment.py # Environment wrapper -│ ├── evaluator.py # Evaluator wrapper -│ ├── data_loader.py # Task loading utilities -│ └── adapters/ -│ └── marble_adapter.py # MarbleAgentAdapter +maseval/benchmark/multiagentbench/ +├── __init__.py +├── README.md # Setup: clone MARBLE to marble/ +├── .gitignore # Ignore marble/ directory +│ +├── multiagentbench.py # Core classes: +│ ├── MultiAgentBenchBenchmark # - Abstract base (framework-agnostic) +│ ├── MarbleMultiAgentBenchBenchmark # - MARBLE reproduction +│ └── MultiAgentBenchEvaluator # - Wraps MARBLE metrics +│ +├── environment.py # MultiAgentBenchEnvironment +├── data_loader.py # load_tasks(), configure_model_ids() +├── adapters/ +│ └── marble_adapter.py # MarbleAgentAdapter +│ +├── marble/ # ← Vendored MARBLE (gitignored) +│ ├── LICENSE # MIT license preserved +│ ├── agent/ +│ ├── engine/ +│ ├── environments/ +│ ├── evaluator/ +│ └── ... # Full MARBLE source +│ +└── data/ # Symlink to marble/multiagentbench/ + ├── coding/ + ├── database/ + ├── minecraft/ + ├── research/ + └── bargaining/ ``` -**MARBLE installed as dependency:** -```toml -[project.optional-dependencies] -multiagentbench = ["marble-bench>=0.1.0"] -``` +### Class Hierarchy -### Implementation Pattern +**In `maseval/benchmark/multiagentbench/`:** -#### 1. Environment Wrapper ```python -class MultiAgentBenchEnvironment(Environment): - """Wraps MARBLE environment instances.""" - - def __init__(self, task_data: Dict[str, Any], callbacks=None): - self.domain = task_data["scenario"] - self.env_config = task_data["environment"] - super().__init__(task_data, callbacks) +# Abstract base - provides task/eval infrastructure for ANY framework +class MultiAgentBenchBenchmark(Benchmark): + """Framework-agnostic base for MultiAgentBench tasks. - def setup_state(self, task_data: Dict[str, Any]) -> Any: - # Convert MASEval task data to MARBLE environment config - return { - "domain": task_data["scenario"], - "workspace_dir": task_data["environment"].get("workspace_dir", "workspace"), - "max_iterations": task_data["environment"].get("max_iterations", 10), - } + Users implement setup_agents() with their chosen framework. + """ + def setup_environment(...) → MultiAgentBenchEnvironment # Shared + def setup_evaluators(...) → MultiAgentBenchEvaluator # Shared + def setup_agents(...) → ABSTRACT # User implements - def create_tools(self) -> Dict[str, Any]: - # MARBLE environments manage tools internally - # Return empty dict as tools are accessed through MARBLE Engine - return {} +# MARBLE reproduction - the only concrete implementation in main library +class MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark): + """Exact MARBLE reproduction for scientific validation.""" + def setup_agents(...): + # Convert Task → MARBLE Config + # Create MARBLE Engine (contains multiple agents internally) + # Wrap as single MarbleAgentAdapter + # Return to MASEval ``` -#### 2. Agent Adapter (Critical Component) -```python -class MarbleAgentAdapter(AgentAdapter): - """Wraps MARBLE's multi-agent Engine as a single agent.""" +**User implementations (examples/ directory only):** - def __init__( - self, - engine: "marble.engine.Engine", - name: str = "marble_engine", - callbacks: Optional[List[AgentCallback]] = None - ): - """ - Args: - engine: MARBLE Engine instance (pre-configured with agents, graph, memory) - name: Adapter name for tracing - callbacks: Optional callbacks - """ - self.engine = engine - super().__init__(engine, name, callbacks) +```python +# Example: LangGraph implementation (in examples/, NOT in main library) +class LangGraphMultiAgentBench(MultiAgentBenchBenchmark): + def setup_agents(...): + # Build LangGraph from task.environment_data["relationships"] + # User's custom implementation + +# Example: Smolagents implementation (in examples/, NOT in main library) +class SmolagentsMultiAgentBench(MultiAgentBenchBenchmark): + def setup_agents(...): + # User's custom implementation +``` - def _run_agent(self, query: str) -> Any: - """Execute MARBLE's multi-agent coordination.""" - # MARBLE's Engine.start() runs the full multi-agent workflow - self.engine.start() +--- - # Extract final output from SharedMemory or designated output agent - final_output = self.engine.memory.get("final_answer") +## How The Integration Works - # Convert MARBLE's execution trace to MessageHistory - messages = self._convert_to_message_history(self.engine) - self.messages = messages +### Execution Flow - return final_output +Let me trace a complete execution from user code to MARBLE coordination: - def _convert_to_message_history(self, engine) -> MessageHistory: - """Convert MARBLE agent traces to MASEval MessageHistory format.""" - messages = [] +#### 1. User Loads Tasks - # Extract messages from all agents in the engine - for agent in engine.agents: - agent_messages = agent.get_messages() # Assume MARBLE agents track messages - for msg in agent_messages: - messages.append({ - "role": f"agent_{agent.agent_id}", - "content": msg.content, - "timestamp": msg.timestamp, - "agent_id": agent.agent_id - }) +```python +from maseval.benchmark.multiagentbench import load_tasks, configure_model_ids - return MessageHistory(messages) +# Load tasks from JSONL +tasks = load_tasks("coding", limit=5) +# Reads marble/multiagentbench/coding/coding_main.jsonl +# Each JSONL line → MASEval Task object - def gather_traces(self) -> Dict[str, Any]: - """Gather MARBLE-specific traces.""" - return { - **super().gather_traces(), - "coordination_mode": self.engine.coordinate_mode, - "agent_graph": self.engine.graph.to_dict(), - "shared_memory": self.engine.memory.to_dict(), - "iterations": self.engine.current_iteration, - "max_iterations": self.engine.max_iterations, - } +# Configure model for all agents +configure_model_ids(tasks, agent_model_id="gpt-4") ``` -#### 3. Benchmark Harness +**Task structure after loading:** ```python -class MultiAgentBenchBenchmark(Benchmark): - """Framework-agnostic MultiAgentBench benchmark. - - Users must implement get_model_adapter() for their LLM provider. - For MARBLE reproduction, use MarbleMultiAgentBenchBenchmark. - """ - - def setup_environment( - self, - agent_data: Dict[str, Any], - task: Task - ) -> MultiAgentBenchEnvironment: - return MultiAgentBenchEnvironment( - task_data=task.environment_data, - ) - - def setup_user( - self, - agent_data: Dict[str, Any], - environment: MultiAgentBenchEnvironment, - task: Task - ) -> Optional[User]: - # MultiAgentBench doesn't use external user simulation - return None - - @abstractmethod - def setup_agents( - self, - agent_data: Dict[str, Any], - environment: MultiAgentBenchEnvironment, - task: Task, - user: Optional[User], - ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: - """Must be implemented by subclass for specific multi-agent framework.""" - pass - - def setup_evaluators( - self, - environment: MultiAgentBenchEnvironment, - task: Task, - agents: Sequence[AgentAdapter], - user: Optional[User], - ) -> Sequence[Evaluator]: - return [ - MultiAgentBenchEvaluator( - task=task, - environment=environment, - ) - ] - - def run_agents( - self, - agents: Sequence[AgentAdapter], - task: Task, - environment: MultiAgentBenchEnvironment, - query: str = "", - ) -> Any: - # For MultiAgentBench, run single "agent" (which is the multi-agent engine) - return agents[0].run(query) - - -class MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark): - """MARBLE-specific implementation for reproduction.""" - - def setup_agents( - self, - agent_data: Dict[str, Any], - environment: MultiAgentBenchEnvironment, - task: Task, - user: Optional[User], - ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: - """Create MARBLE Engine and wrap as single agent.""" - from marble.configs.config import Config - from marble.engine.engine import Engine - - # Convert MASEval task to MARBLE config - marble_config = self._build_marble_config(agent_data, task) - - # Create MARBLE Engine (contains multiple agents internally) - engine = Engine(marble_config) - - # Wrap engine as single AgentAdapter - engine_adapter = MarbleAgentAdapter( - engine=engine, - name="marble_engine", - ) +Task( + id="coding_1", + query="Software Development Task: Please write a system called...", + environment_data={ + "scenario": "coding", + "coordinate_mode": "chain", # or "hierarchical", "graph" + "relationships": [ + ["agent1", "agent2", "reports_to"], + ["agent2", "agent3", "collaborates_with"], + ], + "agents": [ + {"agent_id": "agent1", "profile": "Senior Developer...", "type": "CodingAgent"}, + {"agent_id": "agent2", "profile": "Code Reviewer...", "type": "CodingAgent"}, + {"agent_id": "agent3", "profile": "QA Engineer...", "type": "CodingAgent"}, + ], + "environment": {"type": "Coding", "workspace_dir": "workspace", "max_iterations": 10}, + "llm": "gpt-4", + }, + evaluation_data={ + "metrics": {"code_quality": true, "test_coverage": true, "collaboration_effectiveness": true} + }, +) +``` - # Return as single agent (MASEval runs this one adapter) - return [engine_adapter], {"marble_engine": engine_adapter} +#### 2. User Chooses Implementation - def _build_marble_config( - self, - agent_data: Dict[str, Any], - task: Task - ) -> "Config": - """Convert MASEval Task to MARBLE Config.""" - from marble.configs.config import Config - - config_dict = { - "coordinate_mode": task.environment_data.get("coordinate_mode", "chain"), - "relationships": task.environment_data.get("relationships", []), - "llm": agent_data.get("model_id", "gpt-4"), - "environment": task.environment_data.get("environment", {}), - "task": task.environment_data.get("task", {}), - "agents": task.environment_data.get("agents", []), - "memory": task.environment_data.get("memory", {"type": "SharedMemory"}), - "metrics": task.evaluation_data.get("metrics", {}), - "engine_planner": task.environment_data.get("engine_planner", {}), - } +**Option A: MARBLE Reproduction (built-in)** +```python +from maseval.benchmark.multiagentbench import MarbleMultiAgentBenchBenchmark - return Config.from_dict(config_dict) +benchmark = MarbleMultiAgentBenchBenchmark() +results = benchmark.run(tasks) # Exact MARBLE behavior ``` -#### 4. Data Loading +**Option B: Custom Framework Implementation (user-provided)** ```python -def load_tasks( - domain: str, - data_dir: Optional[Path] = None, - limit: Optional[int] = None, -) -> TaskQueue: - """Load MultiAgentBench tasks from JSONL. +# User implements their own benchmark (e.g., in examples/ or custom code) +# Example shown in examples/multiagentbench_langgraph.py - Args: - domain: One of "coding", "database", "minecraft", "research", "bargaining" - data_dir: Base data directory (default: multiagentbench package data) - limit: Maximum number of tasks to load +from examples.multiagentbench_langgraph import LangGraphMultiAgentBench - Returns: - TaskQueue containing Task objects - """ - data_dir = Path(data_dir) if data_dir else DEFAULT_DATA_DIR - jsonl_path = data_dir / f"{domain}_main.jsonl" - - tasks = [] - with jsonl_path.open() as f: - for idx, line in enumerate(f): - if limit and idx >= limit: - break - - entry = json.loads(line) - - # Convert JSONL entry to MASEval Task - task = Task( - id=f"{domain}_{entry['task_id']}", - query=entry["task"]["content"], - environment_data={ - "scenario": entry["scenario"], - "coordinate_mode": entry["coordinate_mode"], - "relationships": entry["relationships"], - "environment": entry["environment"], - "task": entry["task"], - "agents": entry["agents"], - "memory": entry["memory"], - "engine_planner": entry.get("engine_planner", {}), - }, - evaluation_data={ - "metrics": entry.get("metrics", {}), - }, - metadata={ - "domain": domain, - "task_id": entry["task_id"], - "llm": entry.get("llm", ""), - }, - ) - tasks.append(task) - - return TaskQueue(tasks) - - -def configure_model_ids( - tasks: Union[TaskQueue, List[Task]], - *, - agent_model_id: str, - evaluator_model_id: Optional[str] = None, -) -> Union[TaskQueue, List[Task]]: - """Configure model IDs for MARBLE agents and evaluator.""" - for task in tasks: - # Set model for MARBLE agents - task.environment_data["llm"] = agent_model_id - - # Set evaluator model if LLM-based evaluation needed - if evaluator_model_id: - task.evaluation_data["evaluate_llm"] = evaluator_model_id - - return tasks +benchmark = LangGraphMultiAgentBench() +results = benchmark.run(tasks) # Same tasks, user's framework ``` -### Pros -✅ **Clean separation**: MARBLE remains an independent package -✅ **Standard Python packaging**: Uses established dependency management -✅ **Easy updates**: Can upgrade MARBLE version via `uv add --optional multiagentbench marble-bench@latest` -✅ **R4 (Maintainability)**: Clear boundary between MASEval and MARBLE code - -### Cons -❌ **MARBLE not on PyPI**: Would need to install from Git or build local wheel -❌ **Version pinning challenges**: MARBLE development may break compatibility -❌ **Dependency bloat**: MARBLE's dependencies become transitive dependencies -❌ **Limited control**: Can't patch MARBLE bugs without forking +#### 3. MASEval Orchestration Loop -### Design Principle Assessment +For each task, `benchmark.run()` calls: -- **R1 (Reuse MASEval)**: ✅ Excellent - uses all MASEval patterns -- **R2 (Scientific fidelity)**: ✅ Good - depends on MARBLE version stability -- **R3 (Fail loudly)**: ✅ Good - validation at config conversion boundary -- **R4 (Maintainability)**: ⚠️ Moderate - depends on MARBLE API stability +```python +# Phase 1: Setup components +environment = benchmark.setup_environment(agent_data={}, task=task) +# → MultiAgentBenchEnvironment (wraps MARBLE CodingEnvironment) + +user = benchmark.setup_user(agent_data={}, environment, task) +# → None (MultiAgentBench doesn't use external user simulation) + +agents, agents_dict = benchmark.setup_agents(agent_data={}, environment, task, user) +# → This is where MARBLE vs LangGraph differ! + +evaluators = benchmark.setup_evaluators(environment, task, agents, user) +# → MultiAgentBenchEvaluator (wraps MARBLE's Evaluator) + +# Phase 2: Execute agents +final_answer = benchmark.run_agents(agents, task, environment, query=task.query) +# → Calls agent.run(query) - delegates to MARBLE or LangGraph + +# Phase 3: Evaluate +traces = benchmark.collect_all_traces() +# → Gathers agent messages, tool calls, memory state + +eval_results = benchmark.evaluate(evaluators, agents_dict, final_answer, traces) +# → Computes code_quality, test_coverage, collaboration metrics + +# Store result +report = { + "task_id": task.id, + "final_answer": final_answer, + "eval": eval_results, + "traces": traces, + "status": "success" +} +``` ---- +#### 4. Inside `MarbleMultiAgentBenchBenchmark.setup_agents()` -## Approach 2: Hybrid Vendoring with Adapter Layer (RECOMMENDED) +This is where MARBLE integration happens: -### Overview -Clone MARBLE source into `maseval/benchmark/multiagentbench/marble/` (gitignored), create thin adapter layer in MASEval. +```python +def setup_agents( + self, + agent_data: Dict[str, Any], + environment: MultiAgentBenchEnvironment, + task: Task, + user: Optional[User], +) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: + """Create MARBLE Engine and wrap as single AgentAdapter.""" + + from .marble.configs.config import Config + from .marble.engine.engine import Engine + + # Step 1: Convert MASEval Task → MARBLE Config + marble_config = Config.from_dict({ + "coordinate_mode": task.environment_data["coordinate_mode"], + "relationships": task.environment_data["relationships"], + "agents": task.environment_data["agents"], + "environment": task.environment_data["environment"], + "task": {"content": task.query}, + "llm": task.environment_data["llm"], + "memory": {"type": "SharedMemory"}, + "metrics": task.evaluation_data["metrics"], + "engine_planner": {"initial_progress": "Starting task"}, + }) + + # Step 2: Create MARBLE Engine + # This internally creates: + # - 3 BaseAgent instances (agent1, agent2, agent3) + # - AgentGraph managing relationships + # - SharedMemory for inter-agent communication + # - EnginePlanner for coordination strategy + # - Domain environment (CodingEnvironment) + engine = Engine(marble_config) + + # Step 3: Wrap entire engine as single AgentAdapter + # MASEval sees one "agent" but it contains 3+ agents internally + engine_adapter = MarbleAgentAdapter( + engine=engine, + name="marble_engine", + ) -### Architecture -``` -maseval/ -├── benchmark/ -│ └── multiagentbench/ -│ ├── __init__.py -│ ├── multiagentbench.py # Benchmark + Evaluator -│ ├── environment.py # Environment wrapper -│ ├── data_loader.py # load_tasks(), configure_model_ids() -│ ├── .gitignore # Ignore marble/ directory -│ ├── marble/ # ← Vendored MARBLE source (gitignored) -│ │ ├── __init__.py -│ │ ├── agent/ -│ │ ├── engine/ -│ │ ├── environments/ -│ │ ├── evaluator/ -│ │ ├── graph/ -│ │ ├── llms/ -│ │ ├── memory/ -│ │ └── utils/ -│ └── README.md # Setup instructions + # Step 4: Return to MASEval + # agents_to_run: [engine_adapter] - MASEval will call .run() on this + # agents_dict: {"marble_engine": engine_adapter} - for tracing + return [engine_adapter], {"marble_engine": engine_adapter} ``` -**`.gitignore` in `multiagentbench/`:** -``` -marble/ -``` +**Key Insight**: MASEval never sees MARBLE's internal agents (agent1, agent2, agent3). It only sees the wrapper. This preserves MARBLE's coordination logic while integrating cleanly. -**`README.md` in `multiagentbench/`:** -```markdown -# MultiAgentBench Integration +#### 5. Inside `MarbleAgentAdapter._run_agent()` -## Setup +When MASEval calls `agent.run(query)`: -1. Clone MARBLE into this directory: - ```bash - cd maseval/benchmark/multiagentbench - git clone https://github.com/ulab-uiuc/MARBLE.git marble - cd marble - git checkout # Pin to tested version - ``` +```python +class MarbleAgentAdapter(AgentAdapter): + """Wraps MARBLE's multi-agent Engine as a single agent.""" -2. Install MARBLE dependencies: - ```bash - uv pip install -r marble/pyproject.toml - ``` + def __init__(self, engine: "Engine", name: str = "marble_engine", callbacks=None): + self.engine = engine # Contains multiple agents + coordination + super().__init__(engine, name, callbacks) + + def _run_agent(self, query: str) -> Any: + """Execute MARBLE's multi-agent coordination.""" -## Data + # Delegate to MARBLE Engine + self.engine.start() -MultiAgentBench tasks are located in `marble/multiagentbench/`. -``` + # Engine internally does: + # 1. EnginePlanner decides execution order based on coordinate_mode + # 2. agent1.run() → writes to SharedMemory + # 3. agent2.run() → reads SharedMemory, writes response + # 4. agent3.run() → reads SharedMemory, final output + # 5. Agents use AgentGraph to check who they can communicate with + # 6. Runs until task complete or max_iterations reached -### Implementation Pattern + # Extract final answer from SharedMemory or designated output + final_answer = self.engine.memory.get("final_answer") + if not final_answer: + # Fallback: get from last agent's output + final_answer = self.engine.agents[-1].last_output -Same as Approach 1, but: -- Import from vendored source: `from .marble.engine.engine import Engine` -- Version control via documented commit hash in README -- Can patch MARBLE locally if needed (document patches in `MARBLE_PATCHES.md`) + # Convert MARBLE's execution traces → MASEval MessageHistory + self.messages = self._convert_to_message_history(self.engine) -### Pros -✅ **Full control**: Can patch bugs, optimize, or modify MARBLE as needed -✅ **Reproducibility**: Pin exact MARBLE version via commit hash -✅ **No external dependency**: Works offline, no PyPI/Git availability issues -✅ **Flexible testing**: Can test against multiple MARBLE versions easily -✅ **R2 (Scientific fidelity)**: Guaranteed exact MARBLE behavior + return final_answer -### Cons -❌ **Setup friction**: Users must manually clone MARBLE (documented in README) -❌ **Disk space**: Each MASEval checkout includes full MARBLE source -❌ **Update overhead**: Must manually update vendored copy -❌ **Licensing**: Must verify MARBLE's MIT license allows vendoring + def _convert_to_message_history(self, engine) -> MessageHistory: + """Extract messages from all MARBLE agents.""" + messages = [] -### Design Principle Assessment + # Gather from each internal agent + for agent in engine.agents: + agent_msgs = getattr(agent, "messages", []) + for msg in agent_msgs: + messages.append({ + "role": f"agent_{agent.agent_id}", + "content": str(msg), + "agent_id": agent.agent_id, + "timestamp": getattr(msg, "timestamp", None), + }) -- **R1 (Reuse MASEval)**: ✅ Excellent - identical adapter code to Approach 1 -- **R2 (Scientific fidelity)**: ✅ Excellent - exact version pinning -- **R3 (Fail loudly)**: ✅ Good - clear setup instructions, missing vendor errors -- **R4 (Maintainability)**: ✅ Good - documented update process, local patches possible + # Include SharedMemory state + messages.append({ + "role": "system", + "content": f"SharedMemory: {engine.memory.to_dict()}", + }) -### Why This is Recommended + return MessageHistory(messages) -1. **Scientific Reproducibility**: Pinning exact MARBLE commit ensures bit-for-bit reproduction of results -2. **Development Velocity**: Can iterate on integration without waiting for MARBLE releases -3. **Risk Mitigation**: MARBLE is early-stage (v0.1.0) and may have breaking changes -4. **Pragmatic**: MASEval is a research tool, not production software - vendoring is acceptable + def gather_traces(self) -> Dict[str, Any]: + """Gather MARBLE-specific execution data.""" + return { + **super().gather_traces(), + "coordination_mode": self.engine.coordinate_mode, + "agent_graph": self.engine.graph.to_dict(), + "shared_memory": self.engine.memory.to_dict(), + "iterations": self.engine.current_iteration, + "max_iterations": self.engine.max_iterations, + "internal_agents": [ + {"agent_id": a.agent_id, "type": a.agent_type} + for a in self.engine.agents + ], + } +``` ---- +#### 6. Alternative: User Framework Implementation Example -## Approach 3: Adapter-Only (No MARBLE Engine) +**Note**: This is a USER implementation (in examples/), NOT part of main library. -### Overview -Implement only MultiAgentBench task loading without using MARBLE's multi-agent engine. Users bring their own multi-agent framework. +For comparison with a different framework, users implement their own benchmark: -### Architecture ```python -class MultiAgentBenchBenchmark(Benchmark): - """Framework-agnostic MultiAgentBench harness. +# In examples/multiagentbench_langgraph.py (NOT in maseval/benchmark/) - Does NOT include MARBLE's agent engine. - Users implement setup_agents() with their own multi-agent system. - """ +class LangGraphMultiAgentBench(MultiAgentBenchBenchmark): + """Example user implementation using LangGraph.""" - @abstractmethod - def setup_agents(self, agent_data, environment, task, user): - """User must implement their own multi-agent coordination.""" - pass -``` + def setup_agents( + self, + agent_data: Dict[str, Any], + environment: MultiAgentBenchEnvironment, + task: Task, + user: Optional[User], + ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: + """Create LangGraph-based multi-agent system.""" -**Example usage:** -```python -class LangGraphMultiAgentBench(MultiAgentBenchBenchmark): - def setup_agents(self, agent_data, environment, task, user): - # Use LangGraph for multi-agent coordination from langgraph.graph import StateGraph - graph = StateGraph(...) + # Same task data, different implementation + graph = StateGraph(AgentState) + + # Create agents from same specs for agent_spec in task.environment_data["agents"]: - agent = create_langgraph_agent(agent_spec) + agent = self._create_langgraph_agent( + agent_id=agent_spec["agent_id"], + profile=agent_spec["profile"], + tools=environment.tools, # Same environment! + model=task.environment_data["llm"], # Same model! + ) graph.add_node(agent_spec["agent_id"], agent) - # Add edges based on task.environment_data["relationships"] - ... - - compiled_graph = graph.compile() - adapter = LangGraphAdapter(compiled_graph, "multi_agent_graph") - return [adapter], {"multi_agent_graph": adapter} -``` - -### Pros -✅ **Minimal dependencies**: No MARBLE code required -✅ **Maximum flexibility**: Users can test any multi-agent architecture -✅ **R1 (Reuse MASEval)**: Excellent - pure MASEval patterns - -### Cons -❌ **Cannot reproduce MARBLE results**: Different agent implementation = different results -❌ **Duplicated effort**: Users must re-implement multi-agent coordination -❌ **R2 (Scientific fidelity)**: ❌ FAILED - defeats purpose of benchmark integration + # Interpret relationships as edges + for src, dst, rel_type in task.environment_data["relationships"]: + if rel_type in ["reports_to", "collaborates_with"]: + graph.add_edge(src, dst) -### Design Principle Assessment + # Compile and wrap + compiled = graph.compile() + adapter = LangGraphAdapter(compiled, "langgraph_system") -- **R1 (Reuse MASEval)**: ✅ Perfect -- **R2 (Scientific fidelity)**: ❌ FAILS - cannot reproduce original results -- **R3 (Fail loudly)**: ✅ N/A -- **R4 (Maintainability)**: ✅ Excellent - minimal code + return [adapter], {"langgraph_system": adapter} +``` -**Verdict**: This approach is suitable for creating a *new* multi-agent benchmark inspired by MultiAgentBench, but NOT for integrating MARBLE's existing benchmark with reproduction capabilities. +**Result**: Both MARBLE and user frameworks run on: +- ✅ Same tasks (coding task #1) +- ✅ Same agent specs (agent1, agent2, agent3) +- ✅ Same environment (CodingEnvironment) +- ✅ Same evaluation (code_quality, collaboration) +- ❌ **Different coordination strategy** - this is what enables comparison! --- @@ -518,43 +419,42 @@ class LangGraphMultiAgentBench(MultiAgentBenchBenchmark): ### Summary: YES, with one architectural adaptation -The tau2/macs integration patterns transfer smoothly to MultiAgentBench **with one critical difference**: the multi-agent engine must be wrapped as a single `AgentAdapter` rather than setting up individual agents. +The tau2/macs integration patterns transfer smoothly to MultiAgentBench. ### Pattern Comparison -| Component | MACS/Tau2 Pattern | MultiAgentBench Pattern | Transfer Success | -|-----------|-------------------|-------------------------|------------------| -| **Environment** | Wraps domain-specific environment (tools, database) | Wraps MARBLE domain environment (Coding, DB, Minecraft) | ✅ Direct transfer | -| **Evaluator** | filter_traces() + LLM or deterministic eval | MARBLE evaluator metrics (code quality, collaboration) | ✅ Direct transfer | -| **Data Loading** | load_tasks() from JSON/JSONL | load_tasks() from JSONL | ✅ Direct transfer | -| **Model Config** | configure_model_ids() sets LLM per component | configure_model_ids() sets LLM for all MARBLE agents | ✅ Direct transfer | -| **Benchmark Base** | Abstract base, user implements setup_agents() | Abstract base, user implements setup_agents() | ✅ Direct transfer | -| **Agent Setup** | Create individual AgentAdapters | **DIFFERENT**: Wrap entire MARBLE Engine as single adapter | ⚠️ Requires adaptation | +| Component | MACS/Tau2 Pattern | MultiAgentBench Pattern | Transfer | +|-----------|-------------------|-------------------------|----------| +| **Environment** | Wraps domain-specific environment (tools, database) | Wraps MARBLE domain environment (Coding, DB, Minecraft) | ✅ Direct | +| **Evaluator** | filter_traces() + LLM or deterministic eval | MARBLE evaluator metrics (code quality, collaboration) | ✅ Direct | +| **Data Loading** | load_tasks() from JSON/JSONL | load_tasks() from JSONL | ✅ Direct | +| **Model Config** | configure_model_ids() sets LLM per component | configure_model_ids() sets LLM for all MARBLE agents | ✅ Direct | +| **Benchmark Base** | Abstract base, user implements setup_agents() | Abstract base, user implements setup_agents() | ✅ Direct | +| **Agent Setup** | Create individual AgentAdapters | **DIFFERENT**: Wrap entire MARBLE Engine as single adapter | ⚠️ Adapted | -### Key Architectural Difference: MarbleAgentAdapter +### Key Architectural Difference **MACS/Tau2 Pattern:** ```python def setup_agents(self, agent_data, environment, task, user): # Create individual agent adapters - agent1 = MyAgent("supervisor", tools=environment.tools) - agent2 = MyAgent("worker", tools=environment.tools) + supervisor = MyAgent("supervisor", tools=environment.tools) + worker = MyAgent("worker", tools=environment.tools) - adapter1 = AgentAdapter(agent1, "supervisor") - adapter2 = AgentAdapter(agent2, "worker") + adapter1 = AgentAdapter(supervisor, "supervisor") + adapter2 = AgentAdapter(worker, "worker") return [adapter1], {"supervisor": adapter1, "worker": adapter2} ``` -**MultiAgentBench Pattern (Approach 1 & 2):** +**MultiAgentBench Pattern:** ```python def setup_agents(self, agent_data, environment, task, user): # MARBLE Engine contains multiple agents internally - from marble.engine.engine import Engine - from marble.configs.config import Config + from .marble.engine.engine import Engine - config = self._build_marble_config(task) # Includes agent specs - engine = Engine(config) # Engine manages multiple BaseAgents + config = self._build_marble_config(task) + engine = Engine(config) # Creates agent1, agent2, agent3 internally # Wrap entire engine as single adapter engine_adapter = MarbleAgentAdapter(engine, "marble_engine") @@ -563,106 +463,73 @@ def setup_agents(self, agent_data, environment, task, user): ``` **Why this works:** -- MASEval's `Benchmark.run_agents()` calls `agent.run(query)` on agents in the returned list -- `MarbleAgentAdapter._run_agent()` delegates to `engine.start()`, which coordinates all internal agents -- Individual agent messages are aggregated in `MarbleAgentAdapter.gather_traces()` -- From MASEval's perspective, the multi-agent system appears as a single "black box" agent - -### Benefits of This Pattern - -1. **Preserves MARBLE's coordination logic**: No need to reimplement AgentGraph, SharedMemory, EnginePlanner -2. **Clean abstraction boundary**: MASEval orchestrates benchmark execution, MARBLE handles multi-agent dynamics -3. **Scientific fidelity**: Using MARBLE's Engine exactly as designed ensures reproducible results -4. **Framework comparison**: Easy to swap MARBLE for LangGraph, CrewAI, etc. by implementing different adapters +- MASEval's `run_agents()` calls `agent.run(query)` on returned agents +- `MarbleAgentAdapter._run_agent()` delegates to `engine.start()` +- MARBLE handles all internal coordination (agent-to-agent communication) +- Individual agent messages are aggregated in `gather_traces()` -### Limitations - -- **Less granular tracing**: Individual agent messages require extracting from Engine internals -- **Single invocation**: MASEval's `max_invocations` doesn't apply to MARBLE's internal iterations - - **Solution**: MARBLE's `max_iterations` in config controls internal agent rounds -- **Callback integration**: MARBLE's internal agent callbacks don't automatically trigger MASEval callbacks - - **Solution**: MarbleAgentAdapter can bridge by polling Engine state in `gather_traces()` +**Benefits:** +1. Preserves MARBLE's coordination logic (no reimplementation) +2. Clean separation: MASEval orchestrates outer loop, MARBLE handles inner coordination +3. Scientific fidelity: Using MARBLE exactly as designed +4. Extensible: Users can implement custom frameworks by subclassing `MultiAgentBenchBenchmark` ### Verdict: Strong Pattern Transfer -✅ **4/5 components** transfer directly -⚠️ **1/5 components** (agent setup) requires architectural adaptation that is **clean and well-motivated** +✅ **5/6 components** transfer directly +⚠️ **1/6 components** (agent setup) requires clean, well-motivated adaptation --- ## Critical Implementation Details -### 1. Message History Conversion +### 1. Environment Integration -**Challenge**: MARBLE's agents track messages internally; MASEval expects `MessageHistory` from `AgentAdapter.get_messages()`. +**Challenge**: MARBLE environments manage tools internally; MASEval expects `Environment.create_tools()`. **Solution**: ```python -class MarbleAgentAdapter(AgentAdapter): - def _convert_to_message_history(self, engine) -> MessageHistory: - """Extract and aggregate messages from all MARBLE agents.""" - messages = [] - - for agent in engine.agents: - # Assume MARBLE agents store messages (may need to add this) - agent_messages = getattr(agent, "messages", []) - - for msg in agent_messages: - messages.append({ - "role": f"agent_{agent.agent_id}", - "content": str(msg), - "agent_id": agent.agent_id, - "timestamp": getattr(msg, "timestamp", None), - }) - - # Include shared memory state - memory_state = engine.memory.to_dict() - messages.append({ - "role": "system", - "content": f"Shared Memory State: {json.dumps(memory_state)}", - }) - - return MessageHistory(messages) -``` - -**Risk**: MARBLE agents may not expose message history. -**Mitigation**: Add message tracking to MARBLE BaseAgent if needed (document patch). +class MultiAgentBenchEnvironment(Environment): + """Wraps MARBLE environment instances.""" -### 2. Environment Tool Integration + def __init__(self, task_data: Dict[str, Any], callbacks=None): + self.domain = task_data["scenario"] + self.marble_env_config = task_data["environment"] + super().__init__(task_data, callbacks) -**Challenge**: MARBLE environments manage tools internally; MASEval expects `Environment.create_tools()`. + def setup_state(self, task_data: Dict[str, Any]) -> Any: + return { + "domain": task_data["scenario"], + "marble_config": task_data["environment"], + } -**Solution**: Return empty dict, document that tools are accessed through MARBLE Engine. -```python -class MultiAgentBenchEnvironment(Environment): def create_tools(self) -> Dict[str, Any]: # MARBLE environments manage tools internally - # Tools are available to MARBLE agents through Engine + # Tools are accessed through MARBLE agents, not directly + # Return empty dict to satisfy MASEval interface return {} + def get_tool(self, name: str) -> Optional[Any]: + # R3: Fail loudly if someone tries to access tools directly + raise NotImplementedError( + "MultiAgentBenchEnvironment tools are managed by MARBLE Engine. " + "Access tools through MARBLE agents, not directly from environment." + ) + def gather_traces(self) -> Dict[str, Any]: - """Override to extract tool traces from MARBLE environment.""" + """Extract traces from MARBLE environment if available.""" return { **super().gather_traces(), - "marble_env_type": self.domain, - "marble_tools": self._extract_marble_tool_traces(), + "domain": self.domain, + "marble_env_type": self.marble_env_config.get("type"), } ``` -**R3 (Fail loudly)**: If user calls `environment.get_tool()`, raise: -```python -def get_tool(self, name: str) -> Optional[Any]: - raise NotImplementedError( - "MultiAgentBenchEnvironment tools are managed by MARBLE Engine. " - "Access tools through MARBLE agents, not directly from environment." - ) -``` - -### 3. Evaluator Design +### 2. Evaluator Design -**Challenge**: MultiAgentBench uses MARBLE's Evaluator with domain-specific metrics (code_quality, test_coverage, collaboration_effectiveness). +**Challenge**: MARBLE uses domain-specific metrics (code_quality, collaboration_effectiveness). -**Solution**: Wrap MARBLE evaluator in MASEval Evaluator pattern. +**Solution**: ```python class MultiAgentBenchEvaluator(Evaluator): """Wraps MARBLE's Evaluator for MASEval integration.""" @@ -673,19 +540,23 @@ class MultiAgentBenchEvaluator(Evaluator): self.metrics_config = task.evaluation_data.get("metrics", {}) # Create MARBLE evaluator - from marble.evaluator.evaluator import Evaluator as MarbleEvaluator + from .marble.evaluator.evaluator import Evaluator as MarbleEvaluator self.marble_evaluator = MarbleEvaluator(metrics_config=self.metrics_config) def filter_traces(self, traces: Dict[str, Any]) -> Dict[str, Any]: """Extract MARBLE-specific execution data.""" - # For coding tasks: extract generated code - # For DB tasks: extract query sequences - # For collaboration tasks: extract agent interaction patterns - marble_engine = traces["agents"]["marble_engine"] + # For coding: extract generated code + # For DB: extract query sequences + # For collaboration: extract agent interactions + + marble_engine = traces.get("agents", {}).get("marble_engine", {}) + return { "engine_traces": marble_engine, - "environment_state": traces.get("environment", {}), + "shared_memory": marble_engine.get("shared_memory", {}), + "agent_graph": marble_engine.get("agent_graph", {}), + "iterations": marble_engine.get("iterations", 0), } def __call__( @@ -694,6 +565,7 @@ class MultiAgentBenchEvaluator(Evaluator): final_answer: Optional[str] = None ) -> Dict[str, Any]: """Run MARBLE evaluation metrics.""" + # Extract relevant data engine_traces = traces["engine_traces"] @@ -704,57 +576,121 @@ class MultiAgentBenchEvaluator(Evaluator): traces=engine_traces, ) - return results # Should include code_quality, test_coverage, etc. + # Results should include: + # - code_quality: float (for coding tasks) + # - test_coverage: float (for coding tasks) + # - collaboration_effectiveness: float (for all tasks) + # - task_completion: bool + + return results ``` -### 4. Data Format Validation +### 3. Data Loading -**R3 (Fail loudly)**: Validate JSONL→Task conversion catches missing fields. +**Task format validation (R3: Fail loudly)**: ```python -def load_tasks(domain: str, ...) -> TaskQueue: - REQUIRED_FIELDS = [ - "scenario", "task_id", "task", "agents", - "environment", "relationships" - ] - - for entry in jsonl_entries: - missing = [f for f in REQUIRED_FIELDS if f not in entry] - if missing: - raise ValueError( - f"Task {entry.get('task_id', 'unknown')} missing required fields: {missing}. " - f"Ensure MultiAgentBench JSONL format is correct." - ) +def load_tasks( + domain: str, + data_dir: Optional[Path] = None, + limit: Optional[int] = None, +) -> TaskQueue: + """Load MultiAgentBench tasks from JSONL. + + Args: + domain: One of "coding", "database", "minecraft", "research", "bargaining" + data_dir: Base data directory (default: marble/multiagentbench/) + limit: Maximum number of tasks + + Returns: + TaskQueue containing Task objects + + Raises: + ValueError: If domain invalid or required fields missing + """ + VALID_DOMAINS = ("coding", "database", "minecraft", "research", "bargaining") + if domain not in VALID_DOMAINS: + raise ValueError(f"Invalid domain '{domain}'. Must be one of {VALID_DOMAINS}") + + # Default to vendored MARBLE data + data_dir = Path(data_dir) if data_dir else Path(__file__).parent / "marble" / "multiagentbench" + jsonl_path = data_dir / domain / f"{domain}_main.jsonl" + + if not jsonl_path.exists(): + raise FileNotFoundError( + f"Task data not found: {jsonl_path}\n" + f"Ensure MARBLE is cloned to multiagentbench/marble/\n" + f"See multiagentbench/README.md for setup instructions." + ) - # Validate agent specifications - for agent_spec in entry["agents"]: - if "agent_id" not in agent_spec: + tasks = [] + with jsonl_path.open() as f: + for idx, line in enumerate(f): + if limit and idx >= limit: + break + + entry = json.loads(line) + + # R3: Validate required fields + REQUIRED_FIELDS = ["scenario", "task_id", "task", "agents", "environment", "relationships"] + missing = [f for f in REQUIRED_FIELDS if f not in entry] + if missing: raise ValueError( - f"Agent in task {entry['task_id']} missing 'agent_id'. " - f"Agent spec: {agent_spec}" + f"Task {entry.get('task_id', idx)} missing required fields: {missing}\n" + f"Ensure JSONL format matches MultiAgentBench specification." ) -``` -### 5. Model Configuration + # Validate agent specifications + for agent_spec in entry["agents"]: + if "agent_id" not in agent_spec: + raise ValueError( + f"Agent in task {entry['task_id']} missing 'agent_id'\n" + f"Agent spec: {agent_spec}" + ) + + # Convert to MASEval Task + task = Task( + id=f"{domain}_{entry['task_id']}", + query=entry["task"]["content"], + environment_data={ + "scenario": entry["scenario"], + "coordinate_mode": entry.get("coordinate_mode", "chain"), + "relationships": entry["relationships"], + "environment": entry["environment"], + "task": entry["task"], + "agents": entry["agents"], + "memory": entry.get("memory", {"type": "SharedMemory"}), + "engine_planner": entry.get("engine_planner", {}), + }, + evaluation_data={ + "metrics": entry.get("metrics", {}), + }, + metadata={ + "domain": domain, + "task_id": entry["task_id"], + "llm": entry.get("llm", ""), + }, + ) + tasks.append(task) + + return TaskQueue(tasks) -**Pattern**: Similar to MACS, use `configure_model_ids()` to inject runtime model config. -```python def configure_model_ids( tasks: Union[TaskQueue, List[Task]], *, agent_model_id: str, evaluator_model_id: Optional[str] = None, ) -> Union[TaskQueue, List[Task]]: - """Configure LLM model IDs for MARBLE agents and evaluator. + """Configure model IDs for MARBLE agents and evaluator. Args: tasks: TaskQueue or list of Tasks - agent_model_id: Model ID for all MARBLE agents (e.g., "gpt-4", "claude-sonnet-4.5") - evaluator_model_id: Optional model ID for LLM-based evaluation metrics + agent_model_id: Model for all MARBLE agents (e.g., "gpt-4", "claude-sonnet-4.5") + evaluator_model_id: Optional model for LLM-based evaluation metrics Returns: - Tasks with model IDs configured in environment_data and evaluation_data + Tasks with model IDs configured (mutated in place) """ for task in tasks: # Set global LLM for all MARBLE agents @@ -770,41 +706,111 @@ def configure_model_ids( return tasks ``` ---- +### 4. Message History Extraction -## Recommendations +**Challenge**: MARBLE agents may not expose message history directly. -### Primary Recommendation: Approach 2 (Hybrid Vendoring) +**Fallback strategies**: +```python +def _convert_to_message_history(self, engine) -> MessageHistory: + """Extract messages from MARBLE agents with fallbacks.""" + messages = [] + + # Strategy 1: Direct message access + for agent in engine.agents: + if hasattr(agent, "messages"): + for msg in agent.messages: + messages.append({ + "role": f"agent_{agent.agent_id}", + "content": str(msg), + "agent_id": agent.agent_id, + }) -**Adopt Approach 2** for initial integration due to: + # Strategy 2: Fallback to SharedMemory + if not messages: + memory_state = engine.memory.to_dict() + for key, value in memory_state.items(): + messages.append({ + "role": "system", + "content": f"{key}: {value}", + }) + + # Strategy 3: Fallback to agent outputs + if not messages: + for agent in engine.agents: + if hasattr(agent, "last_output"): + messages.append({ + "role": f"agent_{agent.agent_id}", + "content": str(agent.last_output), + "agent_id": agent.agent_id, + }) -1. **Scientific Fidelity (R2)**: Guarantees exact MARBLE behavior via version pinning -2. **Development Velocity**: Enables rapid iteration without dependency management overhead -3. **Risk Mitigation**: MARBLE is early-stage; vendoring provides stability -4. **Pragmatic**: Research tool priorities favor reproducibility over packaging aesthetics + return MessageHistory(messages) +``` + +--- + +## Setup Instructions + +### Getting MARBLE Source + +Since MARBLE's pip installation has bugs, clone the source directly: -### Migration Path: Approach 2 → Approach 1 +```bash +cd maseval/benchmark/multiagentbench +git clone https://github.com/ulab-uiuc/MARBLE.git marble +cd marble +git checkout # Pin to tested version +``` -Once MARBLE stabilizes (v1.0+ release, stable API), migrate to Approach 1: +**Recommended commit**: `` (tested with MASEval integration) -1. Publish MARBLE to PyPI with semantic versioning -2. Replace vendored source with `[project.optional-dependencies]` entry -3. Keep adapter code identical (no MASEval code changes) -4. Document migration in changelog +**Document in `README.md`**: +```markdown +# MultiAgentBench Integration Setup -**This is a low-risk transition** because adapter code is identical between approaches. +1. Clone MARBLE source: + ```bash + cd maseval/benchmark/multiagentbench + git clone https://github.com/ulab-uiuc/MARBLE.git marble + cd marble + git checkout abc123def # Pinned version + ``` -### Explicitly Reject Approach 3 +2. Install MARBLE dependencies: + ```bash + # From maseval root + uv pip install -e ./maseval/benchmark/multiagentbench/marble + ``` -Do NOT pursue Approach 3 unless the goal shifts from "integrate MARBLE benchmark" to "create new multi-agent benchmark inspired by MultiAgentBench tasks." +3. Run example: + ```bash + python examples/multiagentbench_marble.py + ``` +``` -Approach 3 violates **R2 (Scientific Fidelity)** by making it impossible to reproduce MARBLE's published results. +**Document in `PROVENANCE.md`**: +```markdown +# MARBLE Integration Provenance + +- **Source**: https://github.com/ulab-uiuc/MARBLE +- **Version**: Commit `abc123def` (2025-01-19) +- **License**: MIT (Copyright 2024 Haofei Yu) +- **Vendoring**: Permitted by MIT license with attribution +- **Paper**: arXiv:2503.01935 +- **Last Updated**: 2025-01-19 +- **Integration**: Wrapped via MarbleAgentAdapter + +## Changes from Upstream +- None (vanilla MARBLE source) +- If patches needed, document here +``` --- ## Design Principle Compliance -### R1: Reuse MASEval Infrastructure ✅ +### R1: Reuse MASEval ✅ **Fully compliant.** The integration: - Uses `Benchmark`, `Environment`, `Evaluator`, `AgentAdapter` base classes @@ -815,26 +821,21 @@ Approach 3 violates **R2 (Scientific Fidelity)** by making it impossible to repr ### R2: Scientific Fidelity ✅ -**Compliant with caveats.** +**Compliant with validation.** **Preserved**: - Uses MARBLE's Engine, AgentGraph, SharedMemory exactly as designed -- Version pinning via vendored source (Approach 2) or Git dependency (Approach 1) +- Version pinning via commit hash - Task data preserved from original JSONL format +- Same coordination strategies (chain, hierarchical, graph) -**Trade-offs**: -- **Different execution environment**: MASEval orchestration vs. standalone MARBLE - - *Mitigation*: MarbleAgentAdapter delegates directly to Engine.start() - - *Impact*: Minimal - same agent coordination logic runs -- **Aggregated agent messages**: Individual agent traces must be extracted - - *Mitigation*: Implement comprehensive trace extraction in MarbleAgentAdapter - - *Impact*: Low - agent interactions are preserved, just formatted differently - -**Validation strategy**: +**Validation Strategy**: 1. Run subset of tasks with both standalone MARBLE and MASEval integration -2. Compare outputs, metrics, and agent behaviors +2. Compare final outputs, metrics, agent behaviors 3. Document any discrepancies -4. If material differences exist, adjust adapter implementation +4. Adjust adapter if differences are material + +**Expected**: Minor differences in execution traces (formatted differently) but identical final outputs and metrics. ### R3: Fail Loudly ✅ @@ -842,111 +843,159 @@ Approach 3 violates **R2 (Scientific Fidelity)** by making it impossible to repr **Validation at boundaries**: ```python -# Data loading +# Missing MARBLE source +if not Path("marble").exists(): + raise FileNotFoundError("MARBLE not found. See README.md for setup.") + +# Invalid domain +if domain not in VALID_DOMAINS: + raise ValueError(f"Invalid domain '{domain}'") + +# Missing required fields if "agent_id" not in agent_spec: - raise ValueError("Agent missing 'agent_id'") + raise ValueError(f"Agent missing 'agent_id': {agent_spec}") -# Environment tools +# Incorrect tool access def get_tool(self, name): raise NotImplementedError("Tools managed by MARBLE Engine") - -# Missing config -if not Path("marble").exists(): - raise FileNotFoundError( - "MARBLE source not found. See multiagentbench/README.md for setup." - ) ``` **No defensive defaults**: -- Missing JSONL fields → crash with clear error -- Invalid agent relationships → propagate MARBLE error -- Missing MARBLE dependency → crash at import - -**Clear error messages**: -- Include context (task ID, agent ID, field name) -- Suggest fixes ("See README.md for setup") -- Link to documentation +- Missing JSONL fields → crash with error message +- Invalid relationships → propagate MARBLE error +- Missing config → fail at Engine initialization ### R4: Maintainability ✅ **Compliant.** The integration: **Clear module boundaries**: -- `data_loader.py`: JSONL → Task conversion (0 dependencies on MARBLE internals) +- `data_loader.py`: JSONL → Task conversion (zero MARBLE dependencies) - `environment.py`: Thin wrapper, delegates to MARBLE -- `multiagentbench.py`: Benchmark + Evaluator, well-documented adapter pattern -- `marble/`: Vendored source, not modified (patches documented separately if needed) +- `multiagentbench.py`: Benchmark + Evaluator, documented adapter pattern +- `adapters/marble_adapter.py`: Single-purpose adapter +- `marble/`: Vendored source, not modified **Documentation**: -- `README.md`: Setup instructions, MARBLE version pinning -- `PROVENANCE.md`: Track MARBLE commit hash, upstream changes -- Docstrings: Explain adapter rationale, MARBLE delegation +- `README.md`: Setup instructions, version pinning +- `PROVENANCE.md`: Track upstream, document changes +- Docstrings: Explain rationale, MARBLE delegation +- Code comments: Clarify adapter pattern decisions -**Update process**: +**Update Process**: ```bash # Update vendored MARBLE cd maseval/benchmark/multiagentbench/marble git pull origin main -git checkout +git checkout # Test integration cd ../../../.. pytest tests/test_benchmarks/test_multiagentbench/ -v # Document update -echo "" > multiagentbench/MARBLE_VERSION.txt +# Update PROVENANCE.md with new commit hash and date ``` -**Long-term maintenance**: -- **Low coupling**: Adapter layer isolates MASEval from MARBLE internals -- **Testable**: Can mock MARBLE Engine for unit tests -- **Upgradable**: Switching to Approach 1 (dependency) requires zero adapter code changes +**Long-term**: +- Low coupling: Adapter isolates MASEval from MARBLE internals +- Testable: Can mock MARBLE Engine for unit tests +- Extensible: Users can add custom framework implementations via examples/ --- ## Implementation Checklist -### Phase 1: Core Integration (Approach 2) +### Phase 1: Core Infrastructure - [ ] Create `maseval/benchmark/multiagentbench/` directory -- [ ] Add `.gitignore` to exclude `marble/` +- [ ] Add `.gitignore` excluding `marble/` - [ ] Write `README.md` with MARBLE setup instructions +- [ ] Write `PROVENANCE.md` documenting MARBLE version and license +- [ ] Clone MARBLE to `marble/` directory (manual step for users) +- [ ] Create symlink: `data/` → `marble/multiagentbench/` + +### Phase 2: Core Classes - [ ] Implement `MultiAgentBenchEnvironment(Environment)` + - [ ] `setup_state()`: Store domain and config + - [ ] `create_tools()`: Return empty dict (tools managed by MARBLE) + - [ ] `get_tool()`: Raise NotImplementedError with clear message (R3) + - [ ] `gather_traces()`: Extract domain info - [ ] Implement `MarbleAgentAdapter(AgentAdapter)` - - [ ] `_run_agent()`: Delegate to Engine.start() - - [ ] `_convert_to_message_history()`: Extract agent messages - - [ ] `gather_traces()`: Include AgentGraph, SharedMemory + - [ ] `__init__()`: Store MARBLE Engine + - [ ] `_run_agent()`: Delegate to `engine.start()` + - [ ] `_convert_to_message_history()`: Extract from agents + SharedMemory + - [ ] `gather_traces()`: Include AgentGraph, SharedMemory, iterations +- [ ] Implement `MultiAgentBenchEvaluator(Evaluator)` + - [ ] Wrap MARBLE's Evaluator + - [ ] `filter_traces()`: Extract engine traces + - [ ] `__call__()`: Compute metrics (code_quality, collaboration, etc.) + +### Phase 3: Benchmark Classes - [ ] Implement `MultiAgentBenchBenchmark(Benchmark)` (abstract base) + - [ ] `setup_environment()`: Create MultiAgentBenchEnvironment + - [ ] `setup_user()`: Return None + - [ ] `setup_agents()`: ABSTRACT + - [ ] `setup_evaluators()`: Create MultiAgentBenchEvaluator + - [ ] `run_agents()`: Call agent.run(query) - [ ] Implement `MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark)` - [ ] `setup_agents()`: Create MARBLE Engine, wrap in adapter - [ ] `_build_marble_config()`: Convert Task → MARBLE Config -- [ ] Implement `MultiAgentBenchEvaluator(Evaluator)` - - [ ] Wrap MARBLE's Evaluator - - [ ] Extract metrics (code_quality, test_coverage, collaboration) + - [ ] `get_model_adapter()`: Create/register model adapters -### Phase 2: Data Loading -- [ ] Implement `load_tasks(domain, limit)` in `data_loader.py` - - [ ] Validate JSONL fields (R3: Fail loudly) +### Phase 4: Data Loading +- [ ] Implement `load_tasks()` in `data_loader.py` + - [ ] Validate domain (R3) + - [ ] Validate JSONL fields (R3) - [ ] Convert to Task objects -- [ ] Implement `configure_model_ids(tasks, agent_model_id, evaluator_model_id)` -- [ ] Add tests for data loading with sample JSONL + - [ ] Clear error messages for missing data +- [ ] Implement `configure_model_ids()` + - [ ] Set agent model in environment_data + - [ ] Set evaluator model in evaluation_data +- [ ] Add unit tests for data loading -### Phase 3: Testing & Validation +### Phase 5: Testing & Validation - [ ] Create `tests/test_benchmarks/test_multiagentbench/` -- [ ] Unit tests for data loading -- [ ] Unit tests for Config conversion -- [ ] Integration test: Run 1 task from each domain -- [ ] Validation test: Compare results with standalone MARBLE (same task, same model) - -### Phase 4: Documentation & Examples -- [ ] Example script: `examples/multiagentbench_marble.py` -- [ ] Document setup in main MASEval README +- [ ] Unit tests: + - [ ] `test_load_tasks()`: Validates JSONL parsing + - [ ] `test_configure_model_ids()`: Validates model config + - [ ] `test_marble_config_conversion()`: Task → Config + - [ ] `test_environment()`: MultiAgentBenchEnvironment + - [ ] `test_evaluator()`: MultiAgentBenchEvaluator +- [ ] Integration tests: + - [ ] Run 1 coding task with MARBLE + - [ ] Run 1 database task with MARBLE + - [ ] Verify metrics computed + - [ ] Verify traces collected +- [ ] Validation test: + - [ ] Run same task standalone vs MASEval + - [ ] Compare outputs and metrics + - [ ] Document any discrepancies + +### Phase 6: Documentation & Examples +- [ ] Example: `examples/multiagentbench_marble.py` + - [ ] Load coding tasks + - [ ] Run with MarbleMultiAgentBenchBenchmark + - [ ] Print results and metrics +- [ ] Update main MASEval README + - [ ] Document MultiAgentBench integration + - [ ] Setup instructions for MARBLE - [ ] Add to documentation site (if exists) -- [ ] Write `PROVENANCE.md`: Track MARBLE version, license, upstream -### Phase 5: Optional - Alternative Framework Example -- [ ] Implement `LangGraphMultiAgentBench` as example of custom multi-agent engine -- [ ] Shows how users can bring their own multi-agent framework -- [ ] Demonstrates flexibility of Approach 3 pattern (without replacing MARBLE) +### Phase 7: Optional User Framework Examples (in examples/ only) +- [ ] Example: `examples/multiagentbench_langgraph.py` + - [ ] Implement `LangGraphMultiAgentBench(MultiAgentBenchBenchmark)` + - [ ] Parse relationships → LangGraph structure + - [ ] Demonstrate how users can implement custom frameworks + - [ ] Document in example README +- [ ] Example: `examples/multiagentbench_smolagents.py` + - [ ] Implement `SmolagentsMultiAgentBench(MultiAgentBenchBenchmark)` + - [ ] Show second framework example + - [ ] Demonstrate comparison methodology +- [ ] Document in examples README: + - [ ] How to implement custom multi-agent framework + - [ ] How to compare metrics across frameworks + - [ ] What differences are expected vs problematic + - [ ] **Important**: These are EXAMPLES, not part of main library --- @@ -956,109 +1005,174 @@ echo "" > multiagentbench/MARBLE_VERSION.txt **Probability**: High (early-stage project) **Impact**: High (breaks integration) **Mitigation**: -- Pin exact commit hash in README -- Approach 2 (vendoring) allows local patches -- Automated tests detect breakage +- Pin exact commit hash in README and PROVENANCE.md +- Vendored source allows local patches if needed +- Automated tests detect breakage quickly - Document MARBLE version compatibility matrix ### Risk 2: Message History Extraction **Probability**: Medium (depends on MARBLE internals) **Impact**: Medium (affects tracing quality) **Mitigation**: -- Test message extraction thoroughly -- If MARBLE agents don't expose messages, contribute PR upstream -- Fallback: Extract from SharedMemory state instead +- Implement fallback strategies (SharedMemory, agent outputs) +- Test extraction thoroughly +- If MARBLE doesn't expose messages, contribute PR upstream +- Document limitations in traces ### Risk 3: License Compatibility -**Probability**: Low (MARBLE is MIT) -**Impact**: Critical (legal issues) -**Mitigation**: -- Verify MARBLE LICENSE file (confirmed MIT in README) -- Check subdirectories for different licenses -- Document license in PROVENANCE.md -- Vendoring MIT code is allowed (just attribute properly) +**Probability**: None (VERIFIED ✅) +**Impact**: N/A +**Verification**: +- ✅ MARBLE uses MIT License (Copyright 2024 Haofei Yu) +- ✅ MIT explicitly allows: "use, copy, modify, merge, publish, distribute" without restriction +- ✅ Only requirement: Include copyright notice and LICENSE file +**Implementation**: +- Keep `LICENSE` file in vendored `marble/` directory +- Document in `PROVENANCE.md`: "MARBLE is MIT licensed, vendoring permitted with attribution" ### Risk 4: User Setup Friction **Probability**: High (manual MARBLE clone) -**Impact**: Low (one-time setup) +**Impact**: Low (one-time setup, well-documented) **Mitigation**: -- Clear README instructions +- Clear README with step-by-step instructions - Provide setup script: `bash setup_multiagentbench.sh` - Fail fast with helpful error if MARBLE missing -- Future: Migrate to Approach 1 when MARBLE stabilizes +- Consider setup script that automates cloning ### Risk 5: Evaluation Discrepancies **Probability**: Medium (different execution context) **Impact**: High (invalidates scientific fidelity) **Mitigation**: -- Run validation study: MASEval vs. standalone MARBLE -- Compare metrics, outputs, agent behaviors -- Document any differences -- Adjust adapter if needed -- If irreconcilable, clearly document limitations +- Run validation study: MASEval vs standalone MARBLE +- Compare outputs, metrics, agent behaviors on subset of tasks +- Document any differences in PROVENANCE.md +- If material differences exist, adjust adapter +- Acceptable: Different trace formatting, same final results --- ## Conclusion -**Adopt Approach 2 (Hybrid Vendoring)** as the recommended integration strategy for MARBLE/MultiAgentBench into MASEval. +The integration architecture provides: + +✅ **Exact MARBLE reproduction** via `MarbleMultiAgentBenchBenchmark` +✅ **Fair framework comparison** via abstract `MultiAgentBenchBenchmark` base +✅ **Reuse of MASEval infrastructure** (callbacks, tracing, parallelization) +✅ **Clean architectural boundaries** (adapter pattern) +✅ **Scientific fidelity** (version pinning, validation) +✅ **Maintainability** (clear modules, documented patterns) + +**Key Architectural Insights**: + +1. **MARBLE's multi-agent Engine wraps as single AgentAdapter** - Preserves coordination logic while integrating with MASEval orchestration + +2. **Dual-purpose design enables comparative research** - Abstract base allows users to implement any framework, enabling "which framework is better?" questions -This approach: -- ✅ Satisfies all four design principles (R1-R4) -- ✅ Enables reproduction of MARBLE's published results -- ✅ Provides version stability during MARBLE's early development -- ✅ Reuses MASEval infrastructure effectively -- ✅ Maintains clear architectural boundaries -- ✅ Allows future migration to dependency-based approach +3. **tau2/macs patterns transfer with one adaptation** - Demonstrates MASEval's flexibility for diverse benchmarking paradigms -The key architectural insight is that **MARBLE's multi-agent Engine must be wrapped as a single AgentAdapter**, not decomposed into individual agents. This preserves MARBLE's coordination logic while integrating cleanly with MASEval's orchestration framework. +4. **License verified** - MIT permits vendoring with attribution -The tau2/macs integration patterns transfer successfully with this one architectural adaptation, demonstrating that MASEval's design is flexible enough to accommodate diverse benchmarking paradigms. +**Next Steps**: Begin implementation with Phase 1 (infrastructure setup). --- ## Appendices -### Appendix A: Complete Code Example +### Appendix A: Complete File Structure -See implementation checklist for full module structure. Key classes: -- `MarbleAgentAdapter`: Wraps Engine as single agent -- `MarbleMultiAgentBenchBenchmark`: Creates Engine from Task -- `MultiAgentBenchEvaluator`: Wraps MARBLE metrics -- `load_tasks()`: JSONL → Task conversion +``` +maseval/benchmark/multiagentbench/ +├── __init__.py +├── README.md +├── PROVENANCE.md +├── .gitignore +├── multiagentbench.py +├── environment.py +├── data_loader.py +├── adapters/ +│ ├── __init__.py +│ └── marble_adapter.py +├── marble/ # Gitignored, user clones +│ ├── LICENSE +│ ├── multiagentbench/ +│ │ ├── coding/ +│ │ ├── database/ +│ │ ├── minecraft/ +│ │ ├── research/ +│ │ └── bargaining/ +│ └── ... +└── data/ # Symlink to marble/multiagentbench/ +``` ### Appendix B: MARBLE Architecture Summary -- **Engine**: Main orchestrator -- **BaseAgent**: Individual agent with LLM -- **AgentGraph**: Manages agent relationships and coordination -- **SharedMemory**: Inter-agent communication -- **EnginePlanner**: Plans agent execution order -- **Environments**: Domain-specific (Coding, DB, Minecraft, Research, Web, WorldSimulation) -- **Evaluator**: Computes domain-specific metrics +- **Engine**: Main orchestrator, coordinates agents and environment +- **BaseAgent**: Individual agent with LLM, tools, profile +- **AgentGraph**: Manages agent relationships and communication rules +- **SharedMemory**: Inter-agent communication and state sharing +- **EnginePlanner**: Plans agent execution order based on coordination mode +- **Environments**: Domain-specific (Coding, DB, Minecraft, Research, Bargaining, Web, WorldSimulation) +- **Evaluator**: Computes domain-specific metrics (code_quality, collaboration_effectiveness) -### Appendix C: Task Format Mapping +### Appendix C: Example Usage -**JSONL → Task:** -- `task_id` → `Task.id` (prefixed with domain) -- `task.content` → `Task.query` -- `scenario`, `agents`, `environment`, etc. → `Task.environment_data` -- `metrics` → `Task.evaluation_data` -- Remaining fields → `Task.metadata` +**Basic usage with MARBLE reproduction:** -### Appendix D: Alternative Multi-Agent Frameworks +```python +from maseval.benchmark.multiagentbench import ( + load_tasks, + configure_model_ids, + MarbleMultiAgentBenchBenchmark, +) + +# Load tasks +tasks = load_tasks("coding", limit=10) +configure_model_ids(tasks, agent_model_id="gpt-4") + +# Run with MARBLE (exact reproduction) +marble_bench = MarbleMultiAgentBenchBenchmark() +marble_results = marble_bench.run(tasks) + +# Print results +for result in marble_results: + print(f"Task: {result['task_id']}") + print(f"Code quality: {result['eval']['code_quality']}") + print(f"Collaboration: {result['eval']['collaboration_effectiveness']}") +``` -After MARBLE integration is stable, users can create custom multi-agent benchmarks: -- LangGraph: Graph-based multi-agent workflows -- CrewAI: Role-based agent teams -- AutoGen: Conversational multi-agent systems -- Custom: Domain-specific coordination logic +**Advanced: Framework comparison (user implementation):** -Pattern: Subclass `MultiAgentBenchBenchmark`, implement `setup_agents()` with chosen framework. +```python +from maseval.benchmark.multiagentbench import ( + load_tasks, + configure_model_ids, + MarbleMultiAgentBenchBenchmark, +) + +# User's custom implementation (from examples/ directory) +from examples.multiagentbench_langgraph import LangGraphMultiAgentBench + +tasks = load_tasks("coding", limit=10) +configure_model_ids(tasks, agent_model_id="gpt-4") + +# Run with MARBLE +marble_bench = MarbleMultiAgentBenchBenchmark() +marble_results = marble_bench.run(tasks) + +# Run with user's LangGraph implementation +langgraph_bench = LangGraphMultiAgentBench() +langgraph_results = langgraph_bench.run(tasks) + +# Compare results +for marble_res, lg_res in zip(marble_results, langgraph_results): + print(f"Task: {marble_res['task_id']}") + print(f" MARBLE code_quality: {marble_res['eval']['code_quality']}") + print(f" LangGraph code_quality: {lg_res['eval']['code_quality']}") +``` --- -**Document Version**: 1.0 +**Document Version**: 2.0 **Date**: 2026-01-19 **Author**: Claude (Sonnet 4.5) **Status**: Final Recommendation diff --git a/uv.lock b/uv.lock index 09c06f6c..7b02b069 100644 --- a/uv.lock +++ b/uv.lock @@ -2424,7 +2424,7 @@ wheels = [ [[package]] name = "maseval" -version = "0.2.0" +version = "0.3.0" source = { editable = "." } dependencies = [ { name = "gitpython" }, From 9330f55db07d97589d078764f77315c6e54c02c3 Mon Sep 17 00:00:00 2001 From: cemde Date: Mon, 19 Jan 2026 23:14:39 +0100 Subject: [PATCH 05/17] updated strategy --- STRATEGY.md | 1594 +++++++++++++++++++----------------------- STRATEGY_BACKUP.md | 1120 +++++++++++++++++++++++++++++ STRATEGY_CRITIQUE.md | 517 ++++++++++++++ 3 files changed, 2361 insertions(+), 870 deletions(-) create mode 100644 STRATEGY_BACKUP.md create mode 100644 STRATEGY_CRITIQUE.md diff --git a/STRATEGY.md b/STRATEGY.md index c0152ad0..d4da4928 100644 --- a/STRATEGY.md +++ b/STRATEGY.md @@ -10,7 +10,7 @@ This document proposes the integration architecture for bringing MARBLE (Multi-A This enables critical research questions: **"Which multi-agent framework performs best on the same tasks?"** -**Key Finding**: The tau2/macs patterns transfer smoothly with one architectural difference: MARBLE's multi-agent engine must be wrapped as a single `AgentAdapter` rather than setting up individual agents. +**Key Finding**: Individual MARBLE agents CAN be extracted and wrapped in separate AgentAdapters, providing per-agent trace visibility while preserving MARBLE's coordination logic. **License**: ✅ MARBLE's MIT license explicitly permits vendoring/usage with attribution. @@ -22,15 +22,15 @@ This enables critical research questions: **"Which multi-agent framework perform MARBLE is a multi-agent coordination framework that: - Orchestrates multiple LLM-based agents using an **Engine** + **AgentGraph** architecture -- Supports various coordination modes (chain, hierarchical, graph-based) -- Provides **SharedMemory** for inter-agent communication -- Includes domain-specific environments (Coding, Database, Minecraft, Research, Bargaining) +- Supports various coordination modes (star, chain, tree, graph) +- Provides **inter-agent communication** via direct message passing (stored in `agent.msg_box`) +- Includes domain-specific environments (Coding, Database, Minecraft, Research, Bargaining, Web, WorldSimulation) - Uses configuration for agent relationships and task specifications ### What is MultiAgentBench? MultiAgentBench is MARBLE's benchmark suite featuring: -- **5 domains**: Coding, Database, Minecraft, Research, Bargaining +- **7 domains**: Coding, Database, Minecraft, Research, Bargaining, Web, WorldSimulation - **JSONL task format** with rich metadata (agent relationships, coordination modes, evaluation metrics) - **500+ tasks** testing collaboration and competition scenarios - Original paper: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents" (arXiv:2503.01935) @@ -41,6 +41,7 @@ MultiAgentBench is MARBLE's benchmark suite featuring: 2. **Enable multi-agent framework comparison** - Abstract base allows users to implement with any framework 3. **Maintain MASEval's framework-agnostic design** - No hard dependencies on agent frameworks 4. **Reuse MASEval infrastructure** - Callbacks, tracing, parallelization, error handling +5. **Per-agent trace visibility** - Wrap individual agents, not just the engine --- @@ -51,7 +52,8 @@ MultiAgentBench is MARBLE's benchmark suite featuring: ``` maseval/benchmark/multiagentbench/ ├── __init__.py -├── README.md # Setup: clone MARBLE to marble/ +├── README.md # Setup instructions +├── PROVENANCE.md # Track upstream version, license ├── .gitignore # Ignore marble/ directory │ ├── multiagentbench.py # Core classes: @@ -62,28 +64,19 @@ maseval/benchmark/multiagentbench/ ├── environment.py # MultiAgentBenchEnvironment ├── data_loader.py # load_tasks(), configure_model_ids() ├── adapters/ -│ └── marble_adapter.py # MarbleAgentAdapter +│ └── marble_adapter.py # MarbleAgentAdapter (per-agent) │ -├── marble/ # ← Vendored MARBLE (gitignored) -│ ├── LICENSE # MIT license preserved -│ ├── agent/ -│ ├── engine/ -│ ├── environments/ -│ ├── evaluator/ -│ └── ... # Full MARBLE source -│ -└── data/ # Symlink to marble/multiagentbench/ - ├── coding/ - ├── database/ - ├── minecraft/ - ├── research/ - └── bargaining/ +└── marble/ # ← Vendored MARBLE (gitignored) + ├── LICENSE # MIT license preserved + ├── agent/ + ├── engine/ + ├── environments/ + ├── evaluator/ + └── ... # Full MARBLE source ``` ### Class Hierarchy -**In `maseval/benchmark/multiagentbench/`:** - ```python # Abstract base - provides task/eval infrastructure for ANY framework class MultiAgentBenchBenchmark(Benchmark): @@ -95,525 +88,609 @@ class MultiAgentBenchBenchmark(Benchmark): def setup_evaluators(...) → MultiAgentBenchEvaluator # Shared def setup_agents(...) → ABSTRACT # User implements -# MARBLE reproduction - the only concrete implementation in main library +# MARBLE reproduction - wraps individual agents for trace visibility class MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark): """Exact MARBLE reproduction for scientific validation.""" def setup_agents(...): - # Convert Task → MARBLE Config - # Create MARBLE Engine (contains multiple agents internally) - # Wrap as single MarbleAgentAdapter - # Return to MASEval -``` - -**User implementations (examples/ directory only):** - -```python -# Example: LangGraph implementation (in examples/, NOT in main library) -class LangGraphMultiAgentBench(MultiAgentBenchBenchmark): - def setup_agents(...): - # Build LangGraph from task.environment_data["relationships"] - # User's custom implementation - -# Example: Smolagents implementation (in examples/, NOT in main library) -class SmolagentsMultiAgentBench(MultiAgentBenchBenchmark): - def setup_agents(...): - # User's custom implementation + # Create MARBLE agents from task config + # Wrap EACH agent in MarbleAgentAdapter + # Return list of adapters to MASEval ``` --- -## How The Integration Works - -### Execution Flow - -Let me trace a complete execution from user code to MARBLE coordination: - -#### 1. User Loads Tasks - -```python -from maseval.benchmark.multiagentbench import load_tasks, configure_model_ids - -# Load tasks from JSONL -tasks = load_tasks("coding", limit=5) -# Reads marble/multiagentbench/coding/coding_main.jsonl -# Each JSONL line → MASEval Task object - -# Configure model for all agents -configure_model_ids(tasks, agent_model_id="gpt-4") -``` - -**Task structure after loading:** -```python -Task( - id="coding_1", - query="Software Development Task: Please write a system called...", - environment_data={ - "scenario": "coding", - "coordinate_mode": "chain", # or "hierarchical", "graph" - "relationships": [ - ["agent1", "agent2", "reports_to"], - ["agent2", "agent3", "collaborates_with"], - ], - "agents": [ - {"agent_id": "agent1", "profile": "Senior Developer...", "type": "CodingAgent"}, - {"agent_id": "agent2", "profile": "Code Reviewer...", "type": "CodingAgent"}, - {"agent_id": "agent3", "profile": "QA Engineer...", "type": "CodingAgent"}, - ], - "environment": {"type": "Coding", "workspace_dir": "workspace", "max_iterations": 10}, - "llm": "gpt-4", - }, - evaluation_data={ - "metrics": {"code_quality": true, "test_coverage": true, "collaboration_effectiveness": true} - }, -) -``` - -#### 2. User Chooses Implementation - -**Option A: MARBLE Reproduction (built-in)** -```python -from maseval.benchmark.multiagentbench import MarbleMultiAgentBenchBenchmark +## Agent Wrapping Strategy -benchmark = MarbleMultiAgentBenchBenchmark() -results = benchmark.run(tasks) # Exact MARBLE behavior -``` - -**Option B: Custom Framework Implementation (user-provided)** -```python -# User implements their own benchmark (e.g., in examples/ or custom code) -# Example shown in examples/multiagentbench_langgraph.py - -from examples.multiagentbench_langgraph import LangGraphMultiAgentBench +### Per-Agent Adapters (Preferred Pattern) -benchmark = LangGraphMultiAgentBench() -results = benchmark.run(tasks) # Same tasks, user's framework -``` +MARBLE agents CAN be extracted and wrapped individually. This provides: +- Per-agent trace visibility (matches tau2/macs pattern) +- Better error attribution +- Callback hooks fire per-agent +- Cleaner debugging -#### 3. MASEval Orchestration Loop +**Requirements for independent agent execution:** +1. Each agent needs an `AgentGraph` reference (even if minimal) +2. Communicating agents must share the same `AgentGraph` +3. Each agent needs an environment reference +4. Call `agent.act(task)` directly -For each task, `benchmark.run()` calls: - -```python -# Phase 1: Setup components -environment = benchmark.setup_environment(agent_data={}, task=task) -# → MultiAgentBenchEnvironment (wraps MARBLE CodingEnvironment) - -user = benchmark.setup_user(agent_data={}, environment, task) -# → None (MultiAgentBench doesn't use external user simulation) - -agents, agents_dict = benchmark.setup_agents(agent_data={}, environment, task, user) -# → This is where MARBLE vs LangGraph differ! - -evaluators = benchmark.setup_evaluators(environment, task, agents, user) -# → MultiAgentBenchEvaluator (wraps MARBLE's Evaluator) - -# Phase 2: Execute agents -final_answer = benchmark.run_agents(agents, task, environment, query=task.query) -# → Calls agent.run(query) - delegates to MARBLE or LangGraph - -# Phase 3: Evaluate -traces = benchmark.collect_all_traces() -# → Gathers agent messages, tool calls, memory state - -eval_results = benchmark.evaluate(evaluators, agents_dict, final_answer, traces) -# → Computes code_quality, test_coverage, collaboration metrics - -# Store result -report = { - "task_id": task.id, - "final_answer": final_answer, - "eval": eval_results, - "traces": traces, - "status": "success" -} -``` - -#### 4. Inside `MarbleMultiAgentBenchBenchmark.setup_agents()` - -This is where MARBLE integration happens: - -```python -def setup_agents( - self, - agent_data: Dict[str, Any], - environment: MultiAgentBenchEnvironment, - task: Task, - user: Optional[User], -) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: - """Create MARBLE Engine and wrap as single AgentAdapter.""" - - from .marble.configs.config import Config - from .marble.engine.engine import Engine - - # Step 1: Convert MASEval Task → MARBLE Config - marble_config = Config.from_dict({ - "coordinate_mode": task.environment_data["coordinate_mode"], - "relationships": task.environment_data["relationships"], - "agents": task.environment_data["agents"], - "environment": task.environment_data["environment"], - "task": {"content": task.query}, - "llm": task.environment_data["llm"], - "memory": {"type": "SharedMemory"}, - "metrics": task.evaluation_data["metrics"], - "engine_planner": {"initial_progress": "Starting task"}, - }) - - # Step 2: Create MARBLE Engine - # This internally creates: - # - 3 BaseAgent instances (agent1, agent2, agent3) - # - AgentGraph managing relationships - # - SharedMemory for inter-agent communication - # - EnginePlanner for coordination strategy - # - Domain environment (CodingEnvironment) - engine = Engine(marble_config) - - # Step 3: Wrap entire engine as single AgentAdapter - # MASEval sees one "agent" but it contains 3+ agents internally - engine_adapter = MarbleAgentAdapter( - engine=engine, - name="marble_engine", - ) - - # Step 4: Return to MASEval - # agents_to_run: [engine_adapter] - MASEval will call .run() on this - # agents_dict: {"marble_engine": engine_adapter} - for tracing - return [engine_adapter], {"marble_engine": engine_adapter} -``` - -**Key Insight**: MASEval never sees MARBLE's internal agents (agent1, agent2, agent3). It only sees the wrapper. This preserves MARBLE's coordination logic while integrating cleanly. - -#### 5. Inside `MarbleAgentAdapter._run_agent()` - -When MASEval calls `agent.run(query)`: +### How It Works ```python class MarbleAgentAdapter(AgentAdapter): - """Wraps MARBLE's multi-agent Engine as a single agent.""" - - def __init__(self, engine: "Engine", name: str = "marble_engine", callbacks=None): - self.engine = engine # Contains multiple agents + coordination - super().__init__(engine, name, callbacks) - - def _run_agent(self, query: str) -> Any: - """Execute MARBLE's multi-agent coordination.""" - - # Delegate to MARBLE Engine - self.engine.start() + """Wraps a single MARBLE BaseAgent.""" - # Engine internally does: - # 1. EnginePlanner decides execution order based on coordinate_mode - # 2. agent1.run() → writes to SharedMemory - # 3. agent2.run() → reads SharedMemory, writes response - # 4. agent3.run() → reads SharedMemory, final output - # 5. Agents use AgentGraph to check who they can communicate with - # 6. Runs until task complete or max_iterations reached - - # Extract final answer from SharedMemory or designated output - final_answer = self.engine.memory.get("final_answer") - if not final_answer: - # Fallback: get from last agent's output - final_answer = self.engine.agents[-1].last_output - - # Convert MARBLE's execution traces → MASEval MessageHistory - self.messages = self._convert_to_message_history(self.engine) - - return final_answer - - def _convert_to_message_history(self, engine) -> MessageHistory: - """Extract messages from all MARBLE agents.""" + def __init__( + self, + agent: "BaseAgent", + name: str, + callbacks: Optional[List[Any]] = None, + ): + super().__init__(agent, name, callbacks) + self._agent = agent + + def _run_agent(self, query: str) -> str: + """Execute MARBLE agent's act() method.""" + result, communication = self._agent.act(query) + return result + + def get_messages(self) -> MessageHistory: + """Extract messages from agent's msg_box.""" + return self._extract_message_history() + + def _extract_message_history(self) -> MessageHistory: + """Extract messages from MARBLE's msg_box structure. + + MARBLE stores messages in agent.msg_box: + msg_box[session_id][other_agent_id] = List[(direction, message)] + direction: 0 = FORWARD_TO (sent), 1 = RECV_FROM (received) + """ messages = [] - - # Gather from each internal agent - for agent in engine.agents: - agent_msgs = getattr(agent, "messages", []) - for msg in agent_msgs: - messages.append({ - "role": f"agent_{agent.agent_id}", - "content": str(msg), - "agent_id": agent.agent_id, - "timestamp": getattr(msg, "timestamp", None), - }) - - # Include SharedMemory state - messages.append({ - "role": "system", - "content": f"SharedMemory: {engine.memory.to_dict()}", - }) - + for session_id, conversations in self._agent.msg_box.items(): + for other_agent_id, msg_list in conversations.items(): + for direction, content in msg_list: + messages.append({ + "role": "agent" if direction == 0 else "other", + "content": str(content), + "agent_id": self._agent.agent_id, + "other_agent_id": other_agent_id, + "direction": "sent" if direction == 0 else "received", + "session_id": session_id, + }) return MessageHistory(messages) def gather_traces(self) -> Dict[str, Any]: """Gather MARBLE-specific execution data.""" return { **super().gather_traces(), - "coordination_mode": self.engine.coordinate_mode, - "agent_graph": self.engine.graph.to_dict(), - "shared_memory": self.engine.memory.to_dict(), - "iterations": self.engine.current_iteration, - "max_iterations": self.engine.max_iterations, - "internal_agents": [ - {"agent_id": a.agent_id, "type": a.agent_type} - for a in self.engine.agents - ], + "agent_id": self._agent.agent_id, + "agent_type": getattr(self._agent, "agent_type", "BaseAgent"), + "profile": getattr(self._agent, "profile", ""), + "task_history": getattr(self._agent, "task_history", []), + "relationships": self._extract_relationships(), } -``` -#### 6. Alternative: User Framework Implementation Example + def _extract_relationships(self) -> List[Dict[str, str]]: + """Extract this agent's relationships from the graph.""" + relationships = [] + if self._agent.agent_graph: + for src, dst, rel_type in self._agent.agent_graph.relationships: + if src == self._agent.agent_id or dst == self._agent.agent_id: + relationships.append({ + "source": src, + "target": dst, + "type": rel_type, + }) + return relationships +``` -**Note**: This is a USER implementation (in examples/), NOT part of main library. +### Coordination Without Engine.start() -For comparison with a different framework, users implement their own benchmark: +Instead of calling `engine.start()`, we implement coordination in MASEval's `run_agents()`: ```python -# In examples/multiagentbench_langgraph.py (NOT in maseval/benchmark/) +def run_agents( + self, + agents: Sequence[AgentAdapter], + task: Task, + environment: MultiAgentBenchEnvironment, + query: str = "", +) -> Any: + """Execute agents according to coordination mode.""" + coordinate_mode = task.environment_data.get("coordinate_mode", "star") + + if coordinate_mode == "star": + return self._star_coordinate(agents, task, query) + elif coordinate_mode == "chain": + return self._chain_coordinate(agents, task, query) + elif coordinate_mode == "tree": + return self._tree_coordinate(agents, task, query) + elif coordinate_mode == "graph": + return self._graph_coordinate(agents, task, query) + else: + raise ValueError(f"Unknown coordination mode: {coordinate_mode}") + +def _star_coordinate( + self, + agents: Sequence[AgentAdapter], + task: Task, + query: str, +) -> Any: + """Star coordination: central planner assigns tasks to agents.""" + max_iterations = task.environment_data.get("max_iterations", 10) + results = {} -class LangGraphMultiAgentBench(MultiAgentBenchBenchmark): - """Example user implementation using LangGraph.""" + for iteration in range(max_iterations): + # Assign tasks (simplified - full impl uses EnginePlanner) + task_assignments = self._assign_tasks(agents, task, results) - def setup_agents( - self, - agent_data: Dict[str, Any], - environment: MultiAgentBenchEnvironment, - task: Task, - user: Optional[User], - ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: - """Create LangGraph-based multi-agent system.""" - - from langgraph.graph import StateGraph - - # Same task data, different implementation - graph = StateGraph(AgentState) - - # Create agents from same specs - for agent_spec in task.environment_data["agents"]: - agent = self._create_langgraph_agent( - agent_id=agent_spec["agent_id"], - profile=agent_spec["profile"], - tools=environment.tools, # Same environment! - model=task.environment_data["llm"], # Same model! - ) - graph.add_node(agent_spec["agent_id"], agent) + if not task_assignments: + break - # Interpret relationships as edges - for src, dst, rel_type in task.environment_data["relationships"]: - if rel_type in ["reports_to", "collaborates_with"]: - graph.add_edge(src, dst) + # Execute each agent with its assigned task + for agent, agent_task in task_assignments.items(): + result = agent.run(agent_task) + results[agent.name] = result - # Compile and wrap - compiled = graph.compile() - adapter = LangGraphAdapter(compiled, "langgraph_system") + # Check termination + if self._should_terminate(results, task): + break - return [adapter], {"langgraph_system": adapter} + return self._aggregate_results(results) ``` -**Result**: Both MARBLE and user frameworks run on: -- ✅ Same tasks (coding task #1) -- ✅ Same agent specs (agent1, agent2, agent3) -- ✅ Same environment (CodingEnvironment) -- ✅ Same evaluation (code_quality, collaboration) -- ❌ **Different coordination strategy** - this is what enables comparison! - --- -## Pattern Transfer Analysis: Do tau2/macs Patterns Apply? +## Critical MARBLE API Details -### Summary: YES, with one architectural adaptation +### Message History Storage -The tau2/macs integration patterns transfer smoothly to MultiAgentBench. +**MARBLE stores messages in `agent.msg_box`, NOT SharedMemory:** -### Pattern Comparison +```python +# BaseAgent structure (base_agent.py) +self.msg_box: DefaultDict[str, DefaultDict[str, List[Tuple[int, str]]]] +# Structure: msg_box[session_id][other_agent_id] = [(direction, message), ...] +# Direction: 0 = FORWARD_TO (sent), 1 = RECV_FROM (received) + +# Serialize via (note: typo in MARBLE) +serialized = agent.seralize_message(session_id="") +``` -| Component | MACS/Tau2 Pattern | MultiAgentBench Pattern | Transfer | -|-----------|-------------------|-------------------------|----------| -| **Environment** | Wraps domain-specific environment (tools, database) | Wraps MARBLE domain environment (Coding, DB, Minecraft) | ✅ Direct | -| **Evaluator** | filter_traces() + LLM or deterministic eval | MARBLE evaluator metrics (code quality, collaboration) | ✅ Direct | -| **Data Loading** | load_tasks() from JSON/JSONL | load_tasks() from JSONL | ✅ Direct | -| **Model Config** | configure_model_ids() sets LLM per component | configure_model_ids() sets LLM for all MARBLE agents | ✅ Direct | -| **Benchmark Base** | Abstract base, user implements setup_agents() | Abstract base, user implements setup_agents() | ✅ Direct | -| **Agent Setup** | Create individual AgentAdapters | **DIFFERENT**: Wrap entire MARBLE Engine as single adapter | ⚠️ Adapted | +### SharedMemory is NOT Shared -### Key Architectural Difference +Despite the name, each MARBLE agent creates its own `SharedMemory` instance: -**MACS/Tau2 Pattern:** ```python -def setup_agents(self, agent_data, environment, task, user): - # Create individual agent adapters - supervisor = MyAgent("supervisor", tools=environment.tools) - worker = MyAgent("worker", tools=environment.tools) +# In BaseAgent.__init__() (base_agent.py:72-73) +self.memory = BaseMemory() # Per-agent memory +self.shared_memory = SharedMemory() # Also per-agent, NOT shared! +``` + +**Do NOT rely on SharedMemory for inter-agent state.** Use `msg_box` or environment state instead. - adapter1 = AgentAdapter(supervisor, "supervisor") - adapter2 = AgentAdapter(worker, "worker") +### AgentGraph is Required - return [adapter1], {"supervisor": adapter1, "worker": adapter2} +`agent.act()` requires an AgentGraph reference: + +```python +# In BaseAgent.act() (base_agent.py:145-147) +assert self.agent_graph is not None, \ + "Agent graph is not set. Please set the agent graph using set_agent_graph method first." ``` -**MultiAgentBench Pattern:** +**Solution:** Create AgentGraph during setup and set it on all agents: + ```python def setup_agents(self, agent_data, environment, task, user): - # MARBLE Engine contains multiple agents internally - from .marble.engine.engine import Engine + # Create agents + agents = self._create_agents(task) + + # Create and configure AgentGraph + from .marble.graph.agent_graph import AgentGraph + graph = AgentGraph(agents, self._build_graph_config(task)) - config = self._build_marble_config(task) - engine = Engine(config) # Creates agent1, agent2, agent3 internally + # Set graph reference on each agent (REQUIRED) + for agent in agents: + agent.set_agent_graph(graph) - # Wrap entire engine as single adapter - engine_adapter = MarbleAgentAdapter(engine, "marble_engine") + # Wrap in adapters + adapters = [MarbleAgentAdapter(agent, agent.agent_id) for agent in agents] + agents_dict = {a.name: a for a in adapters} - return [engine_adapter], {"marble_engine": engine_adapter} + return adapters, agents_dict ``` -**Why this works:** -- MASEval's `run_agents()` calls `agent.run(query)` on returned agents -- `MarbleAgentAdapter._run_agent()` delegates to `engine.start()` -- MARBLE handles all internal coordination (agent-to-agent communication) -- Individual agent messages are aggregated in `gather_traces()` +### Known MARBLE Bug: chain_coordinate() -**Benefits:** -1. Preserves MARBLE's coordination logic (no reimplementation) -2. Clean separation: MASEval orchestrates outer loop, MARBLE handles inner coordination -3. Scientific fidelity: Using MARBLE exactly as designed -4. Extensible: Users can implement custom frameworks by subclassing `MultiAgentBenchBenchmark` +**Bug:** `engine.py:702` calls `self.graph.get_agent_profiles_linked()` which does not exist in AgentGraph. -### Verdict: Strong Pattern Transfer +**Impact:** Chain coordination mode will crash. -✅ **5/6 components** transfer directly -⚠️ **1/6 components** (agent setup) requires clean, well-motivated adaptation +**Workaround:** Either: +1. Avoid chain mode tasks initially +2. Patch MARBLE locally (add the missing method) +3. Implement chain coordination in MASEval's `run_agents()` --- -## Critical Implementation Details +## Environment Integration + +### Domain-Specific Requirements + +| Domain | External Dependencies | Initial Support | +|--------|----------------------|-----------------| +| Research | None | ✅ Yes | +| Bargaining | None | ✅ Yes | +| Coding | Filesystem access | ✅ Yes | +| Web | Network access | ✅ Yes | +| WorldSimulation | None | ✅ Yes | +| Database | Docker + PostgreSQL | ⚠️ Optional | +| Minecraft | External game server | ❌ Deferred | -### 1. Environment Integration +**Strategy:** Start with domains that don't require external services. Add infrastructure-heavy domains as optional extras with clear skip logic. -**Challenge**: MARBLE environments manage tools internally; MASEval expects `Environment.create_tools()`. +### MultiAgentBenchEnvironment -**Solution**: ```python class MultiAgentBenchEnvironment(Environment): """Wraps MARBLE environment instances.""" - def __init__(self, task_data: Dict[str, Any], callbacks=None): - self.domain = task_data["scenario"] - self.marble_env_config = task_data["environment"] + # Domains that require external infrastructure + INFRASTRUCTURE_DOMAINS = {"database", "minecraft"} + + def __init__( + self, + task_data: Dict[str, Any], + callbacks: Optional[List[Any]] = None, + ): + self.domain = task_data.get("scenario", "") + self._marble_env: Optional["BaseEnvironment"] = None super().__init__(task_data, callbacks) - def setup_state(self, task_data: Dict[str, Any]) -> Any: + def setup_state(self, task_data: Dict[str, Any]) -> Dict[str, Any]: + """Initialize state and create MARBLE environment.""" + domain = task_data.get("scenario", "") + env_config = task_data.get("environment", {}) + + # Check infrastructure requirements + if domain in self.INFRASTRUCTURE_DOMAINS: + if not self._check_infrastructure(domain): + raise EnvironmentError( + f"Domain '{domain}' requires external infrastructure. " + f"See README.md for setup instructions.", + component="MultiAgentBenchEnvironment", + ) + + # Create MARBLE environment + self._marble_env = self._create_marble_environment(domain, env_config) + return { - "domain": task_data["scenario"], - "marble_config": task_data["environment"], + "domain": domain, + "env_config": env_config, + "marble_env_type": type(self._marble_env).__name__, + } + + def _check_infrastructure(self, domain: str) -> bool: + """Check if required infrastructure is available.""" + if domain == "database": + # Check Docker availability + import shutil + return shutil.which("docker") is not None + elif domain == "minecraft": + # Minecraft requires external server - always fail for now + return False + return True + + def _create_marble_environment( + self, + domain: str, + env_config: Dict[str, Any], + ) -> "BaseEnvironment": + """Create the appropriate MARBLE environment.""" + from .marble.environments.base_env import BaseEnvironment + + # Import domain-specific environments + env_classes = { + "coding": "marble.environments.coding_env.CodingEnvironment", + "database": "marble.environments.db_env.DBEnvironment", + "research": "marble.environments.research_env.ResearchEnvironment", + "bargaining": "marble.environments.bargaining_env.BargainingEnvironment", + "web": "marble.environments.web_env.WebEnvironment", + "worldsimulation": "marble.environments.world_env.WorldSimulationEnvironment", } + env_class_path = env_classes.get(domain.lower()) + if not env_class_path: + # Fallback to base environment + return BaseEnvironment(env_config) + + # Dynamic import + module_path, class_name = env_class_path.rsplit(".", 1) + module = __import__(module_path, fromlist=[class_name]) + env_class = getattr(module, class_name) + return env_class(env_config) + def create_tools(self) -> Dict[str, Any]: - # MARBLE environments manage tools internally - # Tools are accessed through MARBLE agents, not directly - # Return empty dict to satisfy MASEval interface - return {} + """Extract tools from MARBLE environment for tracing. + + MARBLE environments expose tools via action_handler_descriptions. + We wrap these for MASEval tracing. + """ + if not self._marble_env: + return {} + + tools = {} + for action_name in self._marble_env._action_handlers: + handler = self._marble_env._action_handlers[action_name] + # Wrap handler for tracing + tools[action_name] = self._wrap_tool_for_tracing(action_name, handler) + return tools + + def _wrap_tool_for_tracing( + self, + name: str, + handler: Callable, + ) -> Callable: + """Wrap a MARBLE action handler for MASEval tracing.""" + tool_history = ToolInvocationHistory() + + def traced_handler(**kwargs) -> Any: + try: + result = handler(**kwargs) + tool_history.add_invocation( + inputs=kwargs, + outputs=result, + status="success", + ) + return result + except Exception as e: + tool_history.add_invocation( + inputs=kwargs, + outputs=str(e), + status="error", + ) + raise + + # Attach history for trace collection + traced_handler._history = tool_history + traced_handler._original_name = name + return traced_handler def get_tool(self, name: str) -> Optional[Any]: - # R3: Fail loudly if someone tries to access tools directly - raise NotImplementedError( - "MultiAgentBenchEnvironment tools are managed by MARBLE Engine. " - "Access tools through MARBLE agents, not directly from environment." - ) + """Get a specific tool by name.""" + return self.tools.get(name) def gather_traces(self) -> Dict[str, Any]: - """Extract traces from MARBLE environment if available.""" - return { - **super().gather_traces(), - "domain": self.domain, - "marble_env_type": self.marble_env_config.get("type"), - } + """Gather traces including tool invocations.""" + traces = super().gather_traces() + traces["domain"] = self.domain + + # Collect tool invocation histories + tool_traces = {} + for name, tool in self.tools.items(): + if hasattr(tool, "_history"): + tool_traces[name] = { + "invocations": tool._history.to_list(), + } + traces["tool_invocations"] = tool_traces + + return traces ``` -### 2. Evaluator Design +--- + +## Evaluator Design + +### Metrics Mapping + +MARBLE Evaluator produces metrics that must be mapped to MASEval format: + +| MARBLE Metric | Type | MASEval Mapping | +|---------------|------|-----------------| +| `task_completion` | List[0/1] | `passed` (last value) | +| `token_consumption` | List[int] | `total_tokens` (sum) | +| `planning_score` | List[1-5] | `planning_score` (mean) | +| `communication_score` | List[1-5] | `communication_score` (mean) | +| `code_quality` | Dict | `code_quality` (pass-through) | +| `agent_kpis` | Dict[agent_id, int] | `agent_kpis` (pass-through) | -**Challenge**: MARBLE uses domain-specific metrics (code_quality, collaboration_effectiveness). +### MultiAgentBenchEvaluator -**Solution**: ```python class MultiAgentBenchEvaluator(Evaluator): """Wraps MARBLE's Evaluator for MASEval integration.""" - def __init__(self, task: Task, environment: MultiAgentBenchEnvironment): + def __init__( + self, + task: Task, + environment: MultiAgentBenchEnvironment, + agents: Sequence[AgentAdapter], + ): self.task = task self.environment = environment + self.agents = agents self.metrics_config = task.evaluation_data.get("metrics", {}) # Create MARBLE evaluator from .marble.evaluator.evaluator import Evaluator as MarbleEvaluator - self.marble_evaluator = MarbleEvaluator(metrics_config=self.metrics_config) + self._marble_evaluator = MarbleEvaluator(metrics_config=self.metrics_config) def filter_traces(self, traces: Dict[str, Any]) -> Dict[str, Any]: - """Extract MARBLE-specific execution data.""" + """Extract data needed for MARBLE evaluation.""" + # Get agent traces + agent_traces = traces.get("agents", {}) - # For coding: extract generated code - # For DB: extract query sequences - # For collaboration: extract agent interactions + # Get environment state + env_traces = traces.get("environment", {}) - marble_engine = traces.get("agents", {}).get("marble_engine", {}) + # Get tool invocations + tool_traces = env_traces.get("tool_invocations", {}) return { - "engine_traces": marble_engine, - "shared_memory": marble_engine.get("shared_memory", {}), - "agent_graph": marble_engine.get("agent_graph", {}), - "iterations": marble_engine.get("iterations", 0), + "agent_traces": agent_traces, + "environment": env_traces, + "tool_invocations": tool_traces, + "task_id": self.task.id, } def __call__( self, traces: Dict[str, Any], - final_answer: Optional[str] = None + final_answer: Optional[str] = None, ) -> Dict[str, Any]: - """Run MARBLE evaluation metrics.""" + """Run MARBLE evaluation and convert to MASEval format.""" + # Extract MARBLE agents from adapters + marble_agents = [ + adapter._agent for adapter in self.agents + if hasattr(adapter, "_agent") + ] + + # Call MARBLE evaluator update (REQUIRED - not automatic) + if self.environment._marble_env: + self._marble_evaluator.update( + environment=self.environment._marble_env, + agents=marble_agents, + ) - # Extract relevant data - engine_traces = traces["engine_traces"] + # Finalize metrics + self._marble_evaluator.finalize() - # Call MARBLE evaluator - results = self.marble_evaluator.evaluate( - output=final_answer, - task_spec=self.task.evaluation_data, - traces=engine_traces, - ) + # Get raw metrics + raw_metrics = self._marble_evaluator.get_metrics() + + # Convert to MASEval format + return self._convert_metrics(raw_metrics, final_answer) - # Results should include: - # - code_quality: float (for coding tasks) - # - test_coverage: float (for coding tasks) - # - collaboration_effectiveness: float (for all tasks) - # - task_completion: bool + def _convert_metrics( + self, + raw_metrics: Dict[str, Any], + final_answer: Optional[str], + ) -> Dict[str, Any]: + """Convert MARBLE metrics to MASEval format.""" + task_completion = raw_metrics.get("task_completion", []) + planning_scores = raw_metrics.get("planning_score", []) + comm_scores = raw_metrics.get("communication_score", []) + token_counts = raw_metrics.get("token_consumption", []) + + # Compute aggregates + passed = task_completion[-1] == 1 if task_completion else False + planning_score = sum(planning_scores) / len(planning_scores) if planning_scores else 0.0 + communication_score = sum(comm_scores) / len(comm_scores) if comm_scores else 0.0 + total_tokens = sum(token_counts) - return results + return { + "passed": passed, + "task_completion": task_completion[-1] if task_completion else 0, + "planning_score": planning_score, + "communication_score": communication_score, + "total_tokens": total_tokens, + "code_quality": raw_metrics.get("code_quality", {}), + "agent_kpis": raw_metrics.get("agent_kpis", {}), + "total_milestones": raw_metrics.get("total_milestones", 0), + "raw_metrics": raw_metrics, # Include raw for debugging + } ``` -### 3. Data Loading +--- + +## Error Classification -**Task format validation (R3: Fail loudly)**: +Map MARBLE errors to MASEval's `TaskExecutionStatus`: ```python +from maseval import TaskExecutionStatus, AgentError, EnvironmentError + +def classify_marble_error(error: Exception) -> TaskExecutionStatus: + """Classify MARBLE errors into MASEval status codes.""" + error_str = str(error).lower() + + # Infrastructure failures (not agent's fault) + if any(x in error_str for x in ["docker", "connection", "timeout", "rate limit"]): + return TaskExecutionStatus.ENVIRONMENT_ERROR + + # MARBLE internal bugs (not agent's fault) + if "marble" in error_str or "agentgraph" in error_str: + return TaskExecutionStatus.ENVIRONMENT_ERROR + + # Missing dependencies + if "import" in error_str or "module" in error_str: + return TaskExecutionStatus.SETUP_FAILED + + # LLM/API failures + if any(x in error_str for x in ["api", "openai", "anthropic", "quota"]): + return TaskExecutionStatus.ENVIRONMENT_ERROR + + # Agent assertion failures, invalid actions + if any(x in error_str for x in ["assert", "invalid", "not found"]): + return TaskExecutionStatus.AGENT_ERROR + + # Default to agent error (conservative - holds agent accountable) + return TaskExecutionStatus.AGENT_ERROR + + +def wrap_marble_error(error: Exception) -> Exception: + """Wrap MARBLE exceptions in MASEval exception types.""" + status = classify_marble_error(error) + + if status == TaskExecutionStatus.ENVIRONMENT_ERROR: + return EnvironmentError( + str(error), + component="MARBLE", + ) + elif status == TaskExecutionStatus.AGENT_ERROR: + return AgentError( + str(error), + component="MarbleAgent", + ) + else: + return error +``` + +--- + +## Data Loading + +### Task Loading with Validation + +```python +from pathlib import Path +from typing import List, Optional, Union +import json + +VALID_DOMAINS = frozenset({ + "coding", "database", "minecraft", "research", + "bargaining", "web", "worldsimulation", +}) + def load_tasks( domain: str, data_dir: Optional[Path] = None, limit: Optional[int] = None, -) -> TaskQueue: +) -> List[Task]: """Load MultiAgentBench tasks from JSONL. Args: - domain: One of "coding", "database", "minecraft", "research", "bargaining" - data_dir: Base data directory (default: marble/multiagentbench/) + domain: One of the valid domains (see VALID_DOMAINS) + data_dir: Base data directory (default: auto-detect) limit: Maximum number of tasks Returns: - TaskQueue containing Task objects + List of Task objects Raises: - ValueError: If domain invalid or required fields missing + ValueError: If domain is invalid or required fields missing + FileNotFoundError: If data files not found """ - VALID_DOMAINS = ("coding", "database", "minecraft", "research", "bargaining") - if domain not in VALID_DOMAINS: - raise ValueError(f"Invalid domain '{domain}'. Must be one of {VALID_DOMAINS}") + # Validate domain + if domain.lower() not in VALID_DOMAINS: + raise ValueError( + f"Invalid domain '{domain}'. Must be one of: {sorted(VALID_DOMAINS)}" + ) - # Default to vendored MARBLE data - data_dir = Path(data_dir) if data_dir else Path(__file__).parent / "marble" / "multiagentbench" + # Find data directory + data_dir = _resolve_data_dir(data_dir) jsonl_path = data_dir / domain / f"{domain}_main.jsonl" if not jsonl_path.exists(): @@ -630,493 +707,279 @@ def load_tasks( break entry = json.loads(line) + task = _parse_task_entry(entry, domain, idx) + tasks.append(task) - # R3: Validate required fields - REQUIRED_FIELDS = ["scenario", "task_id", "task", "agents", "environment", "relationships"] - missing = [f for f in REQUIRED_FIELDS if f not in entry] - if missing: - raise ValueError( - f"Task {entry.get('task_id', idx)} missing required fields: {missing}\n" - f"Ensure JSONL format matches MultiAgentBench specification." - ) + return tasks - # Validate agent specifications - for agent_spec in entry["agents"]: - if "agent_id" not in agent_spec: - raise ValueError( - f"Agent in task {entry['task_id']} missing 'agent_id'\n" - f"Agent spec: {agent_spec}" - ) - - # Convert to MASEval Task - task = Task( - id=f"{domain}_{entry['task_id']}", - query=entry["task"]["content"], - environment_data={ - "scenario": entry["scenario"], - "coordinate_mode": entry.get("coordinate_mode", "chain"), - "relationships": entry["relationships"], - "environment": entry["environment"], - "task": entry["task"], - "agents": entry["agents"], - "memory": entry.get("memory", {"type": "SharedMemory"}), - "engine_planner": entry.get("engine_planner", {}), - }, - evaluation_data={ - "metrics": entry.get("metrics", {}), - }, - metadata={ - "domain": domain, - "task_id": entry["task_id"], - "llm": entry.get("llm", ""), - }, + +def _resolve_data_dir(data_dir: Optional[Path]) -> Path: + """Resolve the MARBLE data directory.""" + if data_dir: + return Path(data_dir) + + # Check standard locations + candidates = [ + Path(__file__).parent / "marble" / "multiagentbench", + Path.cwd() / "marble" / "multiagentbench", + ] + + # Check environment variable + import os + env_dir = os.environ.get("MARBLE_DATA_DIR") + if env_dir: + candidates.insert(0, Path(env_dir)) + + for candidate in candidates: + if candidate.exists(): + return candidate + + raise FileNotFoundError( + "MARBLE data directory not found. Either:\n" + "1. Clone MARBLE to maseval/benchmark/multiagentbench/marble/\n" + "2. Set MARBLE_DATA_DIR environment variable\n" + "See README.md for setup instructions." + ) + + +def _parse_task_entry( + entry: Dict[str, Any], + domain: str, + idx: int, +) -> Task: + """Parse a JSONL entry into a MASEval Task. + + Raises: + ValueError: If required fields are missing (fail loudly, no defaults) + """ + # Required fields - fail if missing + REQUIRED_FIELDS = ["scenario", "task_id", "task", "agents", "environment", "relationships"] + missing = [f for f in REQUIRED_FIELDS if f not in entry] + if missing: + raise ValueError( + f"Task entry {idx} missing required fields: {missing}\n" + f"Entry keys: {list(entry.keys())}" + ) + + # Validate agent specifications + for i, agent_spec in enumerate(entry["agents"]): + if "agent_id" not in agent_spec: + raise ValueError( + f"Agent {i} in task {entry['task_id']} missing 'agent_id'\n" + f"Agent spec: {agent_spec}" ) - tasks.append(task) - return TaskQueue(tasks) + # Extract task content + task_content = entry["task"] + if isinstance(task_content, dict): + query = task_content.get("content", "") + else: + query = str(task_content) + + if not query: + raise ValueError( + f"Task {entry['task_id']} has empty query/content" + ) + + return Task( + id=f"{domain}_{entry['task_id']}", + query=query, + environment_data={ + "scenario": entry["scenario"], + "coordinate_mode": entry.get("coordinate_mode", "star"), + "relationships": entry["relationships"], + "environment": entry["environment"], + "task": entry["task"], + "agents": entry["agents"], + "max_iterations": entry.get("max_iterations", 10), + # Store raw entry for MARBLE compatibility + "raw_marble_config": entry, + }, + evaluation_data={ + "metrics": entry.get("metrics", {}), + }, + metadata={ + "domain": domain, + "task_id": entry["task_id"], + }, + ) def configure_model_ids( - tasks: Union[TaskQueue, List[Task]], + tasks: List[Task], *, agent_model_id: str, evaluator_model_id: Optional[str] = None, -) -> Union[TaskQueue, List[Task]]: +) -> List[Task]: """Configure model IDs for MARBLE agents and evaluator. Args: - tasks: TaskQueue or list of Tasks - agent_model_id: Model for all MARBLE agents (e.g., "gpt-4", "claude-sonnet-4.5") - evaluator_model_id: Optional model for LLM-based evaluation metrics + tasks: List of Tasks + agent_model_id: Model for all MARBLE agents + evaluator_model_id: Optional model for LLM-based evaluation Returns: Tasks with model IDs configured (mutated in place) """ for task in tasks: - # Set global LLM for all MARBLE agents + # Set agent model task.environment_data["llm"] = agent_model_id - # Individual agents can override via agent_config["llm"] - # This is preserved in task.environment_data["agents"] - - # Set evaluator model if LLM-based metrics enabled - if evaluator_model_id and task.evaluation_data.get("metrics", {}).get("use_llm_eval"): - task.evaluation_data["evaluate_llm"] = evaluator_model_id + # Set evaluator model + if evaluator_model_id: + task.evaluation_data["model_id"] = evaluator_model_id return tasks ``` -### 4. Message History Extraction - -**Challenge**: MARBLE agents may not expose message history directly. - -**Fallback strategies**: -```python -def _convert_to_message_history(self, engine) -> MessageHistory: - """Extract messages from MARBLE agents with fallbacks.""" - messages = [] - - # Strategy 1: Direct message access - for agent in engine.agents: - if hasattr(agent, "messages"): - for msg in agent.messages: - messages.append({ - "role": f"agent_{agent.agent_id}", - "content": str(msg), - "agent_id": agent.agent_id, - }) - - # Strategy 2: Fallback to SharedMemory - if not messages: - memory_state = engine.memory.to_dict() - for key, value in memory_state.items(): - messages.append({ - "role": "system", - "content": f"{key}: {value}", - }) - - # Strategy 3: Fallback to agent outputs - if not messages: - for agent in engine.agents: - if hasattr(agent, "last_output"): - messages.append({ - "role": f"agent_{agent.agent_id}", - "content": str(agent.last_output), - "agent_id": agent.agent_id, - }) - - return MessageHistory(messages) -``` - ---- - -## Setup Instructions - -### Getting MARBLE Source - -Since MARBLE's pip installation has bugs, clone the source directly: - -```bash -cd maseval/benchmark/multiagentbench -git clone https://github.com/ulab-uiuc/MARBLE.git marble -cd marble -git checkout # Pin to tested version -``` - -**Recommended commit**: `` (tested with MASEval integration) - -**Document in `README.md`**: -```markdown -# MultiAgentBench Integration Setup - -1. Clone MARBLE source: - ```bash - cd maseval/benchmark/multiagentbench - git clone https://github.com/ulab-uiuc/MARBLE.git marble - cd marble - git checkout abc123def # Pinned version - ``` - -2. Install MARBLE dependencies: - ```bash - # From maseval root - uv pip install -e ./maseval/benchmark/multiagentbench/marble - ``` - -3. Run example: - ```bash - python examples/multiagentbench_marble.py - ``` -``` - -**Document in `PROVENANCE.md`**: -```markdown -# MARBLE Integration Provenance - -- **Source**: https://github.com/ulab-uiuc/MARBLE -- **Version**: Commit `abc123def` (2025-01-19) -- **License**: MIT (Copyright 2024 Haofei Yu) -- **Vendoring**: Permitted by MIT license with attribution -- **Paper**: arXiv:2503.01935 -- **Last Updated**: 2025-01-19 -- **Integration**: Wrapped via MarbleAgentAdapter - -## Changes from Upstream -- None (vanilla MARBLE source) -- If patches needed, document here -``` - --- ## Design Principle Compliance ### R1: Reuse MASEval ✅ -**Fully compliant.** The integration: - Uses `Benchmark`, `Environment`, `Evaluator`, `AgentAdapter` base classes - Leverages callback system for tracing -- Uses `TaskQueue` and `Task` data structures +- Uses `Task` data structures - Benefits from parallel execution, progress bars, error handling -- No reimplementation of MASEval features ### R2: Scientific Fidelity ✅ -**Compliant with validation.** - -**Preserved**: -- Uses MARBLE's Engine, AgentGraph, SharedMemory exactly as designed - Version pinning via commit hash - Task data preserved from original JSONL format -- Same coordination strategies (chain, hierarchical, graph) - -**Validation Strategy**: -1. Run subset of tasks with both standalone MARBLE and MASEval integration -2. Compare final outputs, metrics, agent behaviors -3. Document any discrepancies -4. Adjust adapter if differences are material - -**Expected**: Minor differences in execution traces (formatted differently) but identical final outputs and metrics. +- Same coordination strategies available +- MARBLE's agent logic preserved in adapters ### R3: Fail Loudly ✅ -**Fully compliant.** The integration includes: - -**Validation at boundaries**: -```python -# Missing MARBLE source -if not Path("marble").exists(): - raise FileNotFoundError("MARBLE not found. See README.md for setup.") - -# Invalid domain -if domain not in VALID_DOMAINS: - raise ValueError(f"Invalid domain '{domain}'") - -# Missing required fields -if "agent_id" not in agent_spec: - raise ValueError(f"Agent missing 'agent_id': {agent_spec}") - -# Incorrect tool access -def get_tool(self, name): - raise NotImplementedError("Tools managed by MARBLE Engine") -``` - -**No defensive defaults**: -- Missing JSONL fields → crash with error message -- Invalid relationships → propagate MARBLE error -- Missing config → fail at Engine initialization +- No silent fallbacks that change benchmark results +- Explicit validation with clear error messages +- Infrastructure requirements checked upfront +- Missing fields cause immediate failure ### R4: Maintainability ✅ -**Compliant.** The integration: - -**Clear module boundaries**: -- `data_loader.py`: JSONL → Task conversion (zero MARBLE dependencies) -- `environment.py`: Thin wrapper, delegates to MARBLE -- `multiagentbench.py`: Benchmark + Evaluator, documented adapter pattern -- `adapters/marble_adapter.py`: Single-purpose adapter -- `marble/`: Vendored source, not modified - -**Documentation**: -- `README.md`: Setup instructions, version pinning -- `PROVENANCE.md`: Track upstream, document changes -- Docstrings: Explain rationale, MARBLE delegation -- Code comments: Clarify adapter pattern decisions - -**Update Process**: -```bash -# Update vendored MARBLE -cd maseval/benchmark/multiagentbench/marble -git pull origin main -git checkout - -# Test integration -cd ../../../.. -pytest tests/test_benchmarks/test_multiagentbench/ -v - -# Document update -# Update PROVENANCE.md with new commit hash and date -``` - -**Long-term**: -- Low coupling: Adapter isolates MASEval from MARBLE internals -- Testable: Can mock MARBLE Engine for unit tests -- Extensible: Users can add custom framework implementations via examples/ +- Clear module boundaries +- Per-agent adapters match tau2/macs pattern +- Documentation of MARBLE quirks +- Update process documented --- ## Implementation Checklist ### Phase 1: Core Infrastructure -- [ ] Create `maseval/benchmark/multiagentbench/` directory +- [ ] Create `maseval/benchmark/multiagentbench/` directory structure - [ ] Add `.gitignore` excluding `marble/` - [ ] Write `README.md` with MARBLE setup instructions - [ ] Write `PROVENANCE.md` documenting MARBLE version and license -- [ ] Clone MARBLE to `marble/` directory (manual step for users) -- [ ] Create symlink: `data/` → `marble/multiagentbench/` - -### Phase 2: Core Classes -- [ ] Implement `MultiAgentBenchEnvironment(Environment)` - - [ ] `setup_state()`: Store domain and config - - [ ] `create_tools()`: Return empty dict (tools managed by MARBLE) - - [ ] `get_tool()`: Raise NotImplementedError with clear message (R3) - - [ ] `gather_traces()`: Extract domain info -- [ ] Implement `MarbleAgentAdapter(AgentAdapter)` - - [ ] `__init__()`: Store MARBLE Engine - - [ ] `_run_agent()`: Delegate to `engine.start()` - - [ ] `_convert_to_message_history()`: Extract from agents + SharedMemory - - [ ] `gather_traces()`: Include AgentGraph, SharedMemory, iterations -- [ ] Implement `MultiAgentBenchEvaluator(Evaluator)` - - [ ] Wrap MARBLE's Evaluator - - [ ] `filter_traces()`: Extract engine traces - - [ ] `__call__()`: Compute metrics (code_quality, collaboration, etc.) - -### Phase 3: Benchmark Classes -- [ ] Implement `MultiAgentBenchBenchmark(Benchmark)` (abstract base) - - [ ] `setup_environment()`: Create MultiAgentBenchEnvironment - - [ ] `setup_user()`: Return None - - [ ] `setup_agents()`: ABSTRACT - - [ ] `setup_evaluators()`: Create MultiAgentBenchEvaluator - - [ ] `run_agents()`: Call agent.run(query) -- [ ] Implement `MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark)` - - [ ] `setup_agents()`: Create MARBLE Engine, wrap in adapter - - [ ] `_build_marble_config()`: Convert Task → MARBLE Config - - [ ] `get_model_adapter()`: Create/register model adapters - -### Phase 4: Data Loading -- [ ] Implement `load_tasks()` in `data_loader.py` - - [ ] Validate domain (R3) - - [ ] Validate JSONL fields (R3) - - [ ] Convert to Task objects - - [ ] Clear error messages for missing data -- [ ] Implement `configure_model_ids()` - - [ ] Set agent model in environment_data - - [ ] Set evaluator model in evaluation_data -- [ ] Add unit tests for data loading +- [ ] Implement `_resolve_data_dir()` for flexible data location -### Phase 5: Testing & Validation +### Phase 2: Data Loading +- [ ] Implement `load_tasks()` with strict validation +- [ ] Implement `_parse_task_entry()` with required field checks +- [ ] Implement `configure_model_ids()` +- [ ] Unit tests for data loading (valid/invalid inputs) + +### Phase 3: Environment +- [ ] Implement `MultiAgentBenchEnvironment` +- [ ] Implement `_create_marble_environment()` for each supported domain +- [ ] Implement `_check_infrastructure()` for Docker-dependent domains +- [ ] Implement `create_tools()` with tracing wrappers +- [ ] Implement `gather_traces()` with tool invocation histories +- [ ] Unit tests for environment setup + +### Phase 4: Agent Adapter +- [ ] Implement `MarbleAgentAdapter` +- [ ] Implement `_extract_message_history()` using `msg_box` +- [ ] Implement `gather_traces()` with relationships +- [ ] Implement `_extract_relationships()` from AgentGraph +- [ ] Unit tests for adapter + +### Phase 5: Benchmark Class +- [ ] Implement `MultiAgentBenchBenchmark` (abstract base) + - [ ] `setup_environment()` - create MultiAgentBenchEnvironment + - [ ] `setup_user()` - return None (no user simulation) + - [ ] `setup_agents()` - ABSTRACT + - [ ] `setup_evaluators()` - create MultiAgentBenchEvaluator +- [ ] Implement `MarbleMultiAgentBenchBenchmark` + - [ ] `setup_agents()` - create agents, AgentGraph, wrap in adapters + - [ ] `_create_agents()` - instantiate MARBLE BaseAgent instances + - [ ] `_build_graph_config()` - create AgentGraph config from task + - [ ] `run_agents()` - implement coordination modes +- [ ] Implement coordination methods: + - [ ] `_star_coordinate()` + - [ ] `_graph_coordinate()` + - [ ] `_tree_coordinate()` (optional - defer if complex) + - [ ] `_chain_coordinate()` (document MARBLE bug, implement workaround) + +### Phase 6: Evaluator +- [ ] Implement `MultiAgentBenchEvaluator` +- [ ] Implement `filter_traces()` - extract agent/env/tool data +- [ ] Implement `__call__()` with explicit `evaluator.update()` call +- [ ] Implement `_convert_metrics()` - MARBLE → MASEval format +- [ ] Unit tests for evaluator + +### Phase 7: Error Handling +- [ ] Implement `classify_marble_error()` +- [ ] Implement `wrap_marble_error()` +- [ ] Add try/except wrappers in adapter and benchmark +- [ ] Test error classification + +### Phase 8: Integration Testing - [ ] Create `tests/test_benchmarks/test_multiagentbench/` -- [ ] Unit tests: - - [ ] `test_load_tasks()`: Validates JSONL parsing - - [ ] `test_configure_model_ids()`: Validates model config - - [ ] `test_marble_config_conversion()`: Task → Config - - [ ] `test_environment()`: MultiAgentBenchEnvironment - - [ ] `test_evaluator()`: MultiAgentBenchEvaluator -- [ ] Integration tests: - - [ ] Run 1 coding task with MARBLE - - [ ] Run 1 database task with MARBLE - - [ ] Verify metrics computed - - [ ] Verify traces collected -- [ ] Validation test: - - [ ] Run same task standalone vs MASEval - - [ ] Compare outputs and metrics - - [ ] Document any discrepancies - -### Phase 6: Documentation & Examples -- [ ] Example: `examples/multiagentbench_marble.py` - - [ ] Load coding tasks - - [ ] Run with MarbleMultiAgentBenchBenchmark - - [ ] Print results and metrics -- [ ] Update main MASEval README - - [ ] Document MultiAgentBench integration - - [ ] Setup instructions for MARBLE -- [ ] Add to documentation site (if exists) - -### Phase 7: Optional User Framework Examples (in examples/ only) -- [ ] Example: `examples/multiagentbench_langgraph.py` - - [ ] Implement `LangGraphMultiAgentBench(MultiAgentBenchBenchmark)` - - [ ] Parse relationships → LangGraph structure - - [ ] Demonstrate how users can implement custom frameworks - - [ ] Document in example README -- [ ] Example: `examples/multiagentbench_smolagents.py` - - [ ] Implement `SmolagentsMultiAgentBench(MultiAgentBenchBenchmark)` - - [ ] Show second framework example - - [ ] Demonstrate comparison methodology -- [ ] Document in examples README: - - [ ] How to implement custom multi-agent framework - - [ ] How to compare metrics across frameworks - - [ ] What differences are expected vs problematic - - [ ] **Important**: These are EXAMPLES, not part of main library +- [ ] Integration test: Run 1 Research task +- [ ] Integration test: Run 1 Bargaining task +- [ ] Integration test: Run 1 Coding task +- [ ] Verify traces collected correctly +- [ ] Verify metrics computed correctly +- [ ] Verify error handling works + +### Phase 9: Example & Documentation +- [ ] Create `examples/multiagentbench_marble.py` +- [ ] Update main MASEval README with MultiAgentBench section +- [ ] Document known limitations (chain mode bug, Minecraft not supported) --- -## Risks and Mitigations - -### Risk 1: MARBLE API Changes -**Probability**: High (early-stage project) -**Impact**: High (breaks integration) -**Mitigation**: -- Pin exact commit hash in README and PROVENANCE.md -- Vendored source allows local patches if needed -- Automated tests detect breakage quickly -- Document MARBLE version compatibility matrix - -### Risk 2: Message History Extraction -**Probability**: Medium (depends on MARBLE internals) -**Impact**: Medium (affects tracing quality) -**Mitigation**: -- Implement fallback strategies (SharedMemory, agent outputs) -- Test extraction thoroughly -- If MARBLE doesn't expose messages, contribute PR upstream -- Document limitations in traces - -### Risk 3: License Compatibility -**Probability**: None (VERIFIED ✅) -**Impact**: N/A -**Verification**: -- ✅ MARBLE uses MIT License (Copyright 2024 Haofei Yu) -- ✅ MIT explicitly allows: "use, copy, modify, merge, publish, distribute" without restriction -- ✅ Only requirement: Include copyright notice and LICENSE file -**Implementation**: -- Keep `LICENSE` file in vendored `marble/` directory -- Document in `PROVENANCE.md`: "MARBLE is MIT licensed, vendoring permitted with attribution" - -### Risk 4: User Setup Friction -**Probability**: High (manual MARBLE clone) -**Impact**: Low (one-time setup, well-documented) -**Mitigation**: -- Clear README with step-by-step instructions -- Provide setup script: `bash setup_multiagentbench.sh` -- Fail fast with helpful error if MARBLE missing -- Consider setup script that automates cloning - -### Risk 5: Evaluation Discrepancies -**Probability**: Medium (different execution context) -**Impact**: High (invalidates scientific fidelity) -**Mitigation**: -- Run validation study: MASEval vs standalone MARBLE -- Compare outputs, metrics, agent behaviors on subset of tasks -- Document any differences in PROVENANCE.md -- If material differences exist, adjust adapter -- Acceptable: Different trace formatting, same final results - ---- - -## Conclusion - -The integration architecture provides: - -✅ **Exact MARBLE reproduction** via `MarbleMultiAgentBenchBenchmark` -✅ **Fair framework comparison** via abstract `MultiAgentBenchBenchmark` base -✅ **Reuse of MASEval infrastructure** (callbacks, tracing, parallelization) -✅ **Clean architectural boundaries** (adapter pattern) -✅ **Scientific fidelity** (version pinning, validation) -✅ **Maintainability** (clear modules, documented patterns) +## Setup Instructions -**Key Architectural Insights**: +### Getting MARBLE Source -1. **MARBLE's multi-agent Engine wraps as single AgentAdapter** - Preserves coordination logic while integrating with MASEval orchestration +```bash +cd maseval/benchmark/multiagentbench +git clone https://github.com/ulab-uiuc/MARBLE.git marble +cd marble +git checkout # Pin to tested version +``` -2. **Dual-purpose design enables comparative research** - Abstract base allows users to implement any framework, enabling "which framework is better?" questions +**Recommended:** Pin to a specific commit after testing. -3. **tau2/macs patterns transfer with one adaptation** - Demonstrates MASEval's flexibility for diverse benchmarking paradigms +### PROVENANCE.md Template -4. **License verified** - MIT permits vendoring with attribution +```markdown +# MARBLE Integration Provenance -**Next Steps**: Begin implementation with Phase 1 (infrastructure setup). +- **Source**: https://github.com/ulab-uiuc/MARBLE +- **Version**: Commit `` (YYYY-MM-DD) +- **License**: MIT (Copyright 2024 Haofei Yu) +- **Vendoring**: Permitted by MIT license with attribution +- **Paper**: arXiv:2503.01935 +- **Last Updated**: YYYY-MM-DD ---- +## Known Issues in MARBLE -## Appendices +1. `AgentGraph.get_agent_profiles_linked()` does not exist - breaks chain coordination +2. SharedMemory is per-agent, not actually shared -### Appendix A: Complete File Structure +## Local Patches Applied +- None (document any patches here) ``` -maseval/benchmark/multiagentbench/ -├── __init__.py -├── README.md -├── PROVENANCE.md -├── .gitignore -├── multiagentbench.py -├── environment.py -├── data_loader.py -├── adapters/ -│ ├── __init__.py -│ └── marble_adapter.py -├── marble/ # Gitignored, user clones -│ ├── LICENSE -│ ├── multiagentbench/ -│ │ ├── coding/ -│ │ ├── database/ -│ │ ├── minecraft/ -│ │ ├── research/ -│ │ └── bargaining/ -│ └── ... -└── data/ # Symlink to marble/multiagentbench/ -``` - -### Appendix B: MARBLE Architecture Summary - -- **Engine**: Main orchestrator, coordinates agents and environment -- **BaseAgent**: Individual agent with LLM, tools, profile -- **AgentGraph**: Manages agent relationships and communication rules -- **SharedMemory**: Inter-agent communication and state sharing -- **EnginePlanner**: Plans agent execution order based on coordination mode -- **Environments**: Domain-specific (Coding, DB, Minecraft, Research, Bargaining, Web, WorldSimulation) -- **Evaluator**: Computes domain-specific metrics (code_quality, collaboration_effectiveness) -### Appendix C: Example Usage +--- -**Basic usage with MARBLE reproduction:** +## Example Usage ```python from maseval.benchmark.multiagentbench import ( @@ -1125,54 +988,45 @@ from maseval.benchmark.multiagentbench import ( MarbleMultiAgentBenchBenchmark, ) -# Load tasks -tasks = load_tasks("coding", limit=10) -configure_model_ids(tasks, agent_model_id="gpt-4") +# Load tasks from a simple domain (no Docker required) +tasks = load_tasks("research", limit=5) +configure_model_ids(tasks, agent_model_id="gpt-4o") + +# Create benchmark +class MyMarbleBenchmark(MarbleMultiAgentBenchBenchmark): + def get_model_adapter(self, model_id, **kwargs): + from maseval.interface.openai import OpenAIModelAdapter + adapter = OpenAIModelAdapter(model_id) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter -# Run with MARBLE (exact reproduction) -marble_bench = MarbleMultiAgentBenchBenchmark() -marble_results = marble_bench.run(tasks) +# Run +benchmark = MyMarbleBenchmark() +results = benchmark.run(tasks) # Print results -for result in marble_results: +for result in results: print(f"Task: {result['task_id']}") - print(f"Code quality: {result['eval']['code_quality']}") - print(f"Collaboration: {result['eval']['collaboration_effectiveness']}") + print(f"Status: {result['status']}") + if result['eval']: + print(f"Passed: {result['eval'][0]['passed']}") + print(f"Planning Score: {result['eval'][0]['planning_score']:.2f}") ``` -**Advanced: Framework comparison (user implementation):** - -```python -from maseval.benchmark.multiagentbench import ( - load_tasks, - configure_model_ids, - MarbleMultiAgentBenchBenchmark, -) - -# User's custom implementation (from examples/ directory) -from examples.multiagentbench_langgraph import LangGraphMultiAgentBench - -tasks = load_tasks("coding", limit=10) -configure_model_ids(tasks, agent_model_id="gpt-4") - -# Run with MARBLE -marble_bench = MarbleMultiAgentBenchBenchmark() -marble_results = marble_bench.run(tasks) +--- -# Run with user's LangGraph implementation -langgraph_bench = LangGraphMultiAgentBench() -langgraph_results = langgraph_bench.run(tasks) +## Risks and Mitigations -# Compare results -for marble_res, lg_res in zip(marble_results, langgraph_results): - print(f"Task: {marble_res['task_id']}") - print(f" MARBLE code_quality: {marble_res['eval']['code_quality']}") - print(f" LangGraph code_quality: {lg_res['eval']['code_quality']}") -``` +| Risk | Probability | Impact | Mitigation | +|------|-------------|--------|------------| +| MARBLE API changes | High | High | Pin commit hash, vendored source allows local patches | +| Chain coordination bug | Confirmed | Medium | Document, implement MASEval workaround | +| Docker-dependent domains | Medium | Low | Clear error messages, skip logic, optional support | +| Evaluation discrepancies | Medium | High | Validation study comparing standalone vs MASEval | --- -**Document Version**: 2.0 +**Document Version**: 3.0 **Date**: 2026-01-19 -**Author**: Claude (Sonnet 4.5) -**Status**: Final Recommendation +**Status**: Ready for Implementation diff --git a/STRATEGY_BACKUP.md b/STRATEGY_BACKUP.md new file mode 100644 index 00000000..32979671 --- /dev/null +++ b/STRATEGY_BACKUP.md @@ -0,0 +1,1120 @@ +# MARBLE Integration Strategy for MASEval + +## Executive Summary + +This document proposes integration strategies for bringing MARBLE (Multi-Agent Coordination Backbone with LLM Engine) and its MultiAgentBench benchmark suite into MASEval. After analyzing both codebases, I recommend **Approach 2: Hybrid Vendoring with Adapter Layer** as it best balances scientific fidelity, maintainability, and reusability of MASEval infrastructure. + +**Key Findings**: +1. The tau2/macs patterns transfer smoothly to MultiAgentBench with one critical architectural difference: MARBLE's multi-agent engine requires wrapping as an AgentAdapter rather than implementing individual agent setup. +2. The architecture must support **two complementary purposes**: + - **Exact MARBLE reproduction** via `MarbleMultiAgentBenchBenchmark` (for scientific validation) + - **Fair framework comparison** via abstract `MultiAgentBenchBenchmark` base (for LangGraph, smolagents, CrewAI implementations) +3. This dual-purpose design enables critical research questions: "Which multi-agent framework performs best on the same tasks?" + +--- + +## Background + +### What is MARBLE? + +MARBLE is a multi-agent coordination framework that: +- Orchestrates multiple LLM-based agents using an **Engine** + **AgentGraph** architecture +- Supports various coordination modes (chain, hierarchical, graph-based) +- Provides **SharedMemory** for inter-agent communication +- Includes domain-specific environments (Coding, Database, Minecraft, Research) +- Uses YAML-based configuration for agent relationships and task specifications + +### What is MultiAgentBench? + +MultiAgentBench is MARBLE's benchmark suite featuring: +- **5 domains**: Coding, Database, Minecraft, Research, Bargaining +- **JSONL task format** with rich metadata (agent relationships, coordination modes, evaluation metrics) +- **500+ tasks** testing collaboration and competition scenarios +- Original paper: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents" (arXiv:2503.01935) + +### MASEval Integration Goals + +1. Enable reproduction of MARBLE's published results +2. Allow comparative evaluation of different multi-agent architectures +3. Maintain MASEval's framework-agnostic design +4. Reuse existing MASEval infrastructure (callbacks, tracing, parallelization) + +--- + +## Approach 1: Full Dependency Integration + +### Overview +Install MARBLE as a Python package dependency via `pyproject.toml`. + +### Architecture +``` +maseval/ +├── benchmark/ +│ └── multiagentbench/ +│ ├── __init__.py +│ ├── multiagentbench.py # Benchmark classes +│ ├── environment.py # Environment wrapper +│ ├── evaluator.py # Evaluator wrapper +│ ├── data_loader.py # Task loading utilities +│ └── adapters/ +│ └── marble_adapter.py # MarbleAgentAdapter +``` + +**MARBLE installed as dependency:** +```toml +[project.optional-dependencies] +multiagentbench = ["marble-bench>=0.1.0"] +``` + +### Implementation Pattern + +#### 1. Environment Wrapper +```python +class MultiAgentBenchEnvironment(Environment): + """Wraps MARBLE environment instances.""" + + def __init__(self, task_data: Dict[str, Any], callbacks=None): + self.domain = task_data["scenario"] + self.env_config = task_data["environment"] + super().__init__(task_data, callbacks) + + def setup_state(self, task_data: Dict[str, Any]) -> Any: + # Convert MASEval task data to MARBLE environment config + return { + "domain": task_data["scenario"], + "workspace_dir": task_data["environment"].get("workspace_dir", "workspace"), + "max_iterations": task_data["environment"].get("max_iterations", 10), + } + + def create_tools(self) -> Dict[str, Any]: + # MARBLE environments manage tools internally + # Return empty dict as tools are accessed through MARBLE Engine + return {} +``` + +#### 2. Agent Adapter (Critical Component) +```python +class MarbleAgentAdapter(AgentAdapter): + """Wraps MARBLE's multi-agent Engine as a single agent.""" + + def __init__( + self, + engine: "marble.engine.Engine", + name: str = "marble_engine", + callbacks: Optional[List[AgentCallback]] = None + ): + """ + Args: + engine: MARBLE Engine instance (pre-configured with agents, graph, memory) + name: Adapter name for tracing + callbacks: Optional callbacks + """ + self.engine = engine + super().__init__(engine, name, callbacks) + + def _run_agent(self, query: str) -> Any: + """Execute MARBLE's multi-agent coordination.""" + # MARBLE's Engine.start() runs the full multi-agent workflow + self.engine.start() + + # Extract final output from SharedMemory or designated output agent + final_output = self.engine.memory.get("final_answer") + + # Convert MARBLE's execution trace to MessageHistory + messages = self._convert_to_message_history(self.engine) + self.messages = messages + + return final_output + + def _convert_to_message_history(self, engine) -> MessageHistory: + """Convert MARBLE agent traces to MASEval MessageHistory format.""" + messages = [] + + # Extract messages from all agents in the engine + for agent in engine.agents: + agent_messages = agent.get_messages() # Assume MARBLE agents track messages + for msg in agent_messages: + messages.append({ + "role": f"agent_{agent.agent_id}", + "content": msg.content, + "timestamp": msg.timestamp, + "agent_id": agent.agent_id + }) + + return MessageHistory(messages) + + def gather_traces(self) -> Dict[str, Any]: + """Gather MARBLE-specific traces.""" + return { + **super().gather_traces(), + "coordination_mode": self.engine.coordinate_mode, + "agent_graph": self.engine.graph.to_dict(), + "shared_memory": self.engine.memory.to_dict(), + "iterations": self.engine.current_iteration, + "max_iterations": self.engine.max_iterations, + } +``` + +#### 3. Benchmark Harness +```python +class MultiAgentBenchBenchmark(Benchmark): + """Framework-agnostic MultiAgentBench benchmark. + + Users must implement get_model_adapter() for their LLM provider. + For MARBLE reproduction, use MarbleMultiAgentBenchBenchmark. + """ + + def setup_environment( + self, + agent_data: Dict[str, Any], + task: Task + ) -> MultiAgentBenchEnvironment: + return MultiAgentBenchEnvironment( + task_data=task.environment_data, + ) + + def setup_user( + self, + agent_data: Dict[str, Any], + environment: MultiAgentBenchEnvironment, + task: Task + ) -> Optional[User]: + # MultiAgentBench doesn't use external user simulation + return None + + @abstractmethod + def setup_agents( + self, + agent_data: Dict[str, Any], + environment: MultiAgentBenchEnvironment, + task: Task, + user: Optional[User], + ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: + """Must be implemented by subclass for specific multi-agent framework.""" + pass + + def setup_evaluators( + self, + environment: MultiAgentBenchEnvironment, + task: Task, + agents: Sequence[AgentAdapter], + user: Optional[User], + ) -> Sequence[Evaluator]: + return [ + MultiAgentBenchEvaluator( + task=task, + environment=environment, + ) + ] + + def run_agents( + self, + agents: Sequence[AgentAdapter], + task: Task, + environment: MultiAgentBenchEnvironment, + query: str = "", + ) -> Any: + # For MultiAgentBench, run single "agent" (which is the multi-agent engine) + return agents[0].run(query) + + +class MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark): + """MARBLE-specific implementation for reproduction.""" + + def setup_agents( + self, + agent_data: Dict[str, Any], + environment: MultiAgentBenchEnvironment, + task: Task, + user: Optional[User], + ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: + """Create MARBLE Engine and wrap as single agent.""" + from marble.configs.config import Config + from marble.engine.engine import Engine + + # Convert MASEval task to MARBLE config + marble_config = self._build_marble_config(agent_data, task) + + # Create MARBLE Engine (contains multiple agents internally) + engine = Engine(marble_config) + + # Wrap engine as single AgentAdapter + engine_adapter = MarbleAgentAdapter( + engine=engine, + name="marble_engine", + ) + + # Return as single agent (MASEval runs this one adapter) + return [engine_adapter], {"marble_engine": engine_adapter} + + def _build_marble_config( + self, + agent_data: Dict[str, Any], + task: Task + ) -> "Config": + """Convert MASEval Task to MARBLE Config.""" + from marble.configs.config import Config + + config_dict = { + "coordinate_mode": task.environment_data.get("coordinate_mode", "chain"), + "relationships": task.environment_data.get("relationships", []), + "llm": agent_data.get("model_id", "gpt-4"), + "environment": task.environment_data.get("environment", {}), + "task": task.environment_data.get("task", {}), + "agents": task.environment_data.get("agents", []), + "memory": task.environment_data.get("memory", {"type": "SharedMemory"}), + "metrics": task.evaluation_data.get("metrics", {}), + "engine_planner": task.environment_data.get("engine_planner", {}), + } + + return Config.from_dict(config_dict) +``` + +#### 4. Data Loading +```python +def load_tasks( + domain: str, + data_dir: Optional[Path] = None, + limit: Optional[int] = None, +) -> TaskQueue: + """Load MultiAgentBench tasks from JSONL. + + Args: + domain: One of "coding", "database", "minecraft", "research", "bargaining" + data_dir: Base data directory (default: multiagentbench package data) + limit: Maximum number of tasks to load + + Returns: + TaskQueue containing Task objects + """ + data_dir = Path(data_dir) if data_dir else DEFAULT_DATA_DIR + jsonl_path = data_dir / f"{domain}_main.jsonl" + + tasks = [] + with jsonl_path.open() as f: + for idx, line in enumerate(f): + if limit and idx >= limit: + break + + entry = json.loads(line) + + # Convert JSONL entry to MASEval Task + task = Task( + id=f"{domain}_{entry['task_id']}", + query=entry["task"]["content"], + environment_data={ + "scenario": entry["scenario"], + "coordinate_mode": entry["coordinate_mode"], + "relationships": entry["relationships"], + "environment": entry["environment"], + "task": entry["task"], + "agents": entry["agents"], + "memory": entry["memory"], + "engine_planner": entry.get("engine_planner", {}), + }, + evaluation_data={ + "metrics": entry.get("metrics", {}), + }, + metadata={ + "domain": domain, + "task_id": entry["task_id"], + "llm": entry.get("llm", ""), + }, + ) + tasks.append(task) + + return TaskQueue(tasks) + + +def configure_model_ids( + tasks: Union[TaskQueue, List[Task]], + *, + agent_model_id: str, + evaluator_model_id: Optional[str] = None, +) -> Union[TaskQueue, List[Task]]: + """Configure model IDs for MARBLE agents and evaluator.""" + for task in tasks: + # Set model for MARBLE agents + task.environment_data["llm"] = agent_model_id + + # Set evaluator model if LLM-based evaluation needed + if evaluator_model_id: + task.evaluation_data["evaluate_llm"] = evaluator_model_id + + return tasks +``` + +### Pros +✅ **Clean separation**: MARBLE remains an independent package +✅ **Standard Python packaging**: Uses established dependency management +✅ **Easy updates**: Can upgrade MARBLE version via `uv add --optional multiagentbench marble-bench@latest` +✅ **R4 (Maintainability)**: Clear boundary between MASEval and MARBLE code + +### Cons +❌ **MARBLE not on PyPI**: Would need to install from Git or build local wheel +❌ **Version pinning challenges**: MARBLE development may break compatibility +❌ **Dependency bloat**: MARBLE's dependencies become transitive dependencies +❌ **Limited control**: Can't patch MARBLE bugs without forking + +### Design Principle Assessment + +- **R1 (Reuse MASEval)**: ✅ Excellent - uses all MASEval patterns +- **R2 (Scientific fidelity)**: ✅ Good - depends on MARBLE version stability +- **R3 (Fail loudly)**: ✅ Good - validation at config conversion boundary +- **R4 (Maintainability)**: ⚠️ Moderate - depends on MARBLE API stability + +--- + +## Approach 2: Hybrid Vendoring with Adapter Layer (RECOMMENDED) + +### Overview +Clone MARBLE source into `maseval/benchmark/multiagentbench/marble/` (gitignored), create thin adapter layer in MASEval. + +### Architecture +``` +maseval/ +├── benchmark/ +│ └── multiagentbench/ +│ ├── __init__.py +│ ├── multiagentbench.py # Benchmark + Evaluator +│ ├── environment.py # Environment wrapper +│ ├── data_loader.py # load_tasks(), configure_model_ids() +│ ├── .gitignore # Ignore marble/ directory +│ ├── marble/ # ← Vendored MARBLE source (gitignored) +│ │ ├── __init__.py +│ │ ├── agent/ +│ │ ├── engine/ +│ │ ├── environments/ +│ │ ├── evaluator/ +│ │ ├── graph/ +│ │ ├── llms/ +│ │ ├── memory/ +│ │ └── utils/ +│ └── README.md # Setup instructions +``` + +**`.gitignore` in `multiagentbench/`:** +``` +marble/ +``` + +**`README.md` in `multiagentbench/`:** +```markdown +# MultiAgentBench Integration + +## Setup + +1. Clone MARBLE into this directory: + ```bash + cd maseval/benchmark/multiagentbench + git clone https://github.com/ulab-uiuc/MARBLE.git marble + cd marble + git checkout # Pin to tested version + ``` + +2. Install MARBLE dependencies: + ```bash + uv pip install -r marble/pyproject.toml + ``` + +## Data + +MultiAgentBench tasks are located in `marble/multiagentbench/`. +``` + +### Implementation Pattern + +Same as Approach 1, but: +- Import from vendored source: `from .marble.engine.engine import Engine` +- Version control via documented commit hash in README +- Can patch MARBLE locally if needed (document patches in `MARBLE_PATCHES.md`) + +### Pros +✅ **Full control**: Can patch bugs, optimize, or modify MARBLE as needed +✅ **Reproducibility**: Pin exact MARBLE version via commit hash +✅ **No external dependency**: Works offline, no PyPI/Git availability issues +✅ **Flexible testing**: Can test against multiple MARBLE versions easily +✅ **R2 (Scientific fidelity)**: Guaranteed exact MARBLE behavior + +### Cons +❌ **Setup friction**: Users must manually clone MARBLE (documented in README) +❌ **Disk space**: Each MASEval checkout includes full MARBLE source +❌ **Update overhead**: Must manually update vendored copy +❌ **Licensing**: Must verify MARBLE's MIT license allows vendoring + +### Design Principle Assessment + +- **R1 (Reuse MASEval)**: ✅ Excellent - identical adapter code to Approach 1 +- **R2 (Scientific fidelity)**: ✅ Excellent - exact version pinning +- **R3 (Fail loudly)**: ✅ Good - clear setup instructions, missing vendor errors +- **R4 (Maintainability)**: ✅ Good - documented update process, local patches possible + +### Why This is Recommended + +1. **Scientific Reproducibility**: Pinning exact MARBLE commit ensures bit-for-bit reproduction of results +2. **Development Velocity**: Can iterate on integration without waiting for MARBLE releases +3. **Risk Mitigation**: MARBLE is early-stage (v0.1.0) and may have breaking changes +4. **Pragmatic**: MASEval is a research tool, not production software - vendoring is acceptable + +--- + +## Approach 3: Multi-Framework Comparison Architecture + +### Overview +**This is NOT a replacement for MARBLE - it's the comparison mechanism.** The architecture enables: +1. **Exact MARBLE reproduction** via `MarbleMultiAgentBenchBenchmark` +2. **Alternative framework implementations** via custom subclasses (LangGraph, smolagents, CrewAI) +3. **Fair comparison** - all frameworks run on the same tasks, environment, and evaluation + +### Architecture +```python +class MultiAgentBenchBenchmark(Benchmark): + """Abstract base providing task/eval infrastructure for ANY multi-agent framework. + + Subclasses: + - MarbleMultiAgentBenchBenchmark: Exact MARBLE reproduction + - LangGraphMultiAgentBenchBenchmark: LangGraph implementation + - SmolagentsMultiAgentBenchBenchmark: Smolagents implementation + - CrewAIMultiAgentBenchBenchmark: CrewAI implementation + """ + + @abstractmethod + def setup_agents(self, agent_data, environment, task, user): + """Implement multi-agent coordination using your chosen framework.""" + pass +``` + +**Example: LangGraph implementation for comparison** +```python +class LangGraphMultiAgentBench(MultiAgentBenchBenchmark): + """LangGraph implementation of MultiAgentBench tasks. + + Uses same tasks, environment, and evaluation as MARBLE, + but implements multi-agent coordination with LangGraph. + """ + + def setup_agents(self, agent_data, environment, task, user): + from langgraph.graph import StateGraph + + # Create graph-based multi-agent system + graph = StateGraph(...) + + # Add agents from task specification + for agent_spec in task.environment_data["agents"]: + agent = self._create_langgraph_agent(agent_spec) + graph.add_node(agent_spec["agent_id"], agent) + + # Add edges based on relationships + for src, dst, rel_type in task.environment_data["relationships"]: + graph.add_edge(src, dst) + + compiled_graph = graph.compile() + adapter = LangGraphAdapter(compiled_graph, "langgraph_system") + return [adapter], {"langgraph_system": adapter} +``` + +### Purpose: Enable Fair Multi-Agent Framework Comparison + +This architecture enables researchers to ask: +- **"How does MARBLE compare to LangGraph on the same tasks?"** +- **"Which coordination strategy works best for coding tasks?"** +- **"Does hierarchical vs. graph-based coordination matter?"** + +All comparisons use: +- ✅ Same task specifications (from JSONL) +- ✅ Same evaluation metrics (code quality, collaboration) +- ✅ Same environment (Coding, DB, Minecraft, etc.) +- ✅ Only difference: multi-agent coordination implementation + +### Pros +✅ **Enables scientific comparison**: Test multiple multi-agent architectures fairly +✅ **Preserves MARBLE reproduction**: MarbleMultiAgentBenchBenchmark still exists +✅ **Framework flexibility**: Users can implement with any multi-agent library +✅ **R1 (Reuse MASEval)**: Perfect - pure MASEval patterns +✅ **Research value**: Answers "which multi-agent approach is better?" + +### Cons +⚠️ **Implementation effort**: Each framework requires custom adapter +⚠️ **Results divergence**: Different frameworks produce different results (expected) +⚠️ **Coordination semantics**: Not all frameworks can express all coordination patterns + +### Design Principle Assessment + +- **R1 (Reuse MASEval)**: ✅ Perfect - clean abstraction +- **R2 (Scientific fidelity)**: ✅ **FOR MARBLE** (via MarbleMultiAgentBenchBenchmark), ✅ **FOR COMPARISONS** (same tasks/eval) +- **R3 (Fail loudly)**: ✅ Clear separation between reproduction and comparison +- **R4 (Maintainability)**: ✅ Excellent - each framework is independent + +**Verdict**: This is NOT an alternative to MARBLE integration - it's the **comparison layer** that makes MultiAgentBench valuable for multi-agent systems research. The architecture must support BOTH exact MARBLE reproduction AND alternative framework implementations. + +--- + +## Pattern Transfer Analysis: Do tau2/macs Patterns Apply? + +### Summary: YES, with one architectural adaptation + +The tau2/macs integration patterns transfer smoothly to MultiAgentBench **with one critical difference**: the multi-agent engine must be wrapped as a single `AgentAdapter` rather than setting up individual agents. + +### Pattern Comparison + +| Component | MACS/Tau2 Pattern | MultiAgentBench Pattern | Transfer Success | +|-----------|-------------------|-------------------------|------------------| +| **Environment** | Wraps domain-specific environment (tools, database) | Wraps MARBLE domain environment (Coding, DB, Minecraft) | ✅ Direct transfer | +| **Evaluator** | filter_traces() + LLM or deterministic eval | MARBLE evaluator metrics (code quality, collaboration) | ✅ Direct transfer | +| **Data Loading** | load_tasks() from JSON/JSONL | load_tasks() from JSONL | ✅ Direct transfer | +| **Model Config** | configure_model_ids() sets LLM per component | configure_model_ids() sets LLM for all MARBLE agents | ✅ Direct transfer | +| **Benchmark Base** | Abstract base, user implements setup_agents() | Abstract base, user implements setup_agents() | ✅ Direct transfer | +| **Agent Setup** | Create individual AgentAdapters | **DIFFERENT**: Wrap entire MARBLE Engine as single adapter | ⚠️ Requires adaptation | + +### Key Architectural Difference: MarbleAgentAdapter + +**MACS/Tau2 Pattern:** +```python +def setup_agents(self, agent_data, environment, task, user): + # Create individual agent adapters + agent1 = MyAgent("supervisor", tools=environment.tools) + agent2 = MyAgent("worker", tools=environment.tools) + + adapter1 = AgentAdapter(agent1, "supervisor") + adapter2 = AgentAdapter(agent2, "worker") + + return [adapter1], {"supervisor": adapter1, "worker": adapter2} +``` + +**MultiAgentBench Pattern (Approach 1 & 2):** +```python +def setup_agents(self, agent_data, environment, task, user): + # MARBLE Engine contains multiple agents internally + from marble.engine.engine import Engine + from marble.configs.config import Config + + config = self._build_marble_config(task) # Includes agent specs + engine = Engine(config) # Engine manages multiple BaseAgents + + # Wrap entire engine as single adapter + engine_adapter = MarbleAgentAdapter(engine, "marble_engine") + + return [engine_adapter], {"marble_engine": engine_adapter} +``` + +**Why this works:** +- MASEval's `Benchmark.run_agents()` calls `agent.run(query)` on agents in the returned list +- `MarbleAgentAdapter._run_agent()` delegates to `engine.start()`, which coordinates all internal agents +- Individual agent messages are aggregated in `MarbleAgentAdapter.gather_traces()` +- From MASEval's perspective, the multi-agent system appears as a single "black box" agent + +### Benefits of This Pattern + +1. **Preserves MARBLE's coordination logic**: No need to reimplement AgentGraph, SharedMemory, EnginePlanner +2. **Clean abstraction boundary**: MASEval orchestrates benchmark execution, MARBLE handles multi-agent dynamics +3. **Scientific fidelity**: Using MARBLE's Engine exactly as designed ensures reproducible results +4. **Framework comparison**: Easy to swap MARBLE for LangGraph, CrewAI, etc. by implementing different adapters + +### Limitations + +- **Less granular tracing**: Individual agent messages require extracting from Engine internals +- **Single invocation**: MASEval's `max_invocations` doesn't apply to MARBLE's internal iterations + - **Solution**: MARBLE's `max_iterations` in config controls internal agent rounds +- **Callback integration**: MARBLE's internal agent callbacks don't automatically trigger MASEval callbacks + - **Solution**: MarbleAgentAdapter can bridge by polling Engine state in `gather_traces()` + +### Verdict: Strong Pattern Transfer + +✅ **4/5 components** transfer directly +⚠️ **1/5 components** (agent setup) requires architectural adaptation that is **clean and well-motivated** + +--- + +## Critical Implementation Details + +### 1. Message History Conversion + +**Challenge**: MARBLE's agents track messages internally; MASEval expects `MessageHistory` from `AgentAdapter.get_messages()`. + +**Solution**: +```python +class MarbleAgentAdapter(AgentAdapter): + def _convert_to_message_history(self, engine) -> MessageHistory: + """Extract and aggregate messages from all MARBLE agents.""" + messages = [] + + for agent in engine.agents: + # Assume MARBLE agents store messages (may need to add this) + agent_messages = getattr(agent, "messages", []) + + for msg in agent_messages: + messages.append({ + "role": f"agent_{agent.agent_id}", + "content": str(msg), + "agent_id": agent.agent_id, + "timestamp": getattr(msg, "timestamp", None), + }) + + # Include shared memory state + memory_state = engine.memory.to_dict() + messages.append({ + "role": "system", + "content": f"Shared Memory State: {json.dumps(memory_state)}", + }) + + return MessageHistory(messages) +``` + +**Risk**: MARBLE agents may not expose message history. +**Mitigation**: Add message tracking to MARBLE BaseAgent if needed (document patch). + +### 2. Environment Tool Integration + +**Challenge**: MARBLE environments manage tools internally; MASEval expects `Environment.create_tools()`. + +**Solution**: Return empty dict, document that tools are accessed through MARBLE Engine. +```python +class MultiAgentBenchEnvironment(Environment): + def create_tools(self) -> Dict[str, Any]: + # MARBLE environments manage tools internally + # Tools are available to MARBLE agents through Engine + return {} + + def gather_traces(self) -> Dict[str, Any]: + """Override to extract tool traces from MARBLE environment.""" + return { + **super().gather_traces(), + "marble_env_type": self.domain, + "marble_tools": self._extract_marble_tool_traces(), + } +``` + +**R3 (Fail loudly)**: If user calls `environment.get_tool()`, raise: +```python +def get_tool(self, name: str) -> Optional[Any]: + raise NotImplementedError( + "MultiAgentBenchEnvironment tools are managed by MARBLE Engine. " + "Access tools through MARBLE agents, not directly from environment." + ) +``` + +### 3. Evaluator Design + +**Challenge**: MultiAgentBench uses MARBLE's Evaluator with domain-specific metrics (code_quality, test_coverage, collaboration_effectiveness). + +**Solution**: Wrap MARBLE evaluator in MASEval Evaluator pattern. +```python +class MultiAgentBenchEvaluator(Evaluator): + """Wraps MARBLE's Evaluator for MASEval integration.""" + + def __init__(self, task: Task, environment: MultiAgentBenchEnvironment): + self.task = task + self.environment = environment + self.metrics_config = task.evaluation_data.get("metrics", {}) + + # Create MARBLE evaluator + from marble.evaluator.evaluator import Evaluator as MarbleEvaluator + self.marble_evaluator = MarbleEvaluator(metrics_config=self.metrics_config) + + def filter_traces(self, traces: Dict[str, Any]) -> Dict[str, Any]: + """Extract MARBLE-specific execution data.""" + # For coding tasks: extract generated code + # For DB tasks: extract query sequences + # For collaboration tasks: extract agent interaction patterns + + marble_engine = traces["agents"]["marble_engine"] + return { + "engine_traces": marble_engine, + "environment_state": traces.get("environment", {}), + } + + def __call__( + self, + traces: Dict[str, Any], + final_answer: Optional[str] = None + ) -> Dict[str, Any]: + """Run MARBLE evaluation metrics.""" + # Extract relevant data + engine_traces = traces["engine_traces"] + + # Call MARBLE evaluator + results = self.marble_evaluator.evaluate( + output=final_answer, + task_spec=self.task.evaluation_data, + traces=engine_traces, + ) + + return results # Should include code_quality, test_coverage, etc. +``` + +### 4. Data Format Validation + +**R3 (Fail loudly)**: Validate JSONL→Task conversion catches missing fields. + +```python +def load_tasks(domain: str, ...) -> TaskQueue: + REQUIRED_FIELDS = [ + "scenario", "task_id", "task", "agents", + "environment", "relationships" + ] + + for entry in jsonl_entries: + missing = [f for f in REQUIRED_FIELDS if f not in entry] + if missing: + raise ValueError( + f"Task {entry.get('task_id', 'unknown')} missing required fields: {missing}. " + f"Ensure MultiAgentBench JSONL format is correct." + ) + + # Validate agent specifications + for agent_spec in entry["agents"]: + if "agent_id" not in agent_spec: + raise ValueError( + f"Agent in task {entry['task_id']} missing 'agent_id'. " + f"Agent spec: {agent_spec}" + ) +``` + +### 5. Model Configuration + +**Pattern**: Similar to MACS, use `configure_model_ids()` to inject runtime model config. + +```python +def configure_model_ids( + tasks: Union[TaskQueue, List[Task]], + *, + agent_model_id: str, + evaluator_model_id: Optional[str] = None, +) -> Union[TaskQueue, List[Task]]: + """Configure LLM model IDs for MARBLE agents and evaluator. + + Args: + tasks: TaskQueue or list of Tasks + agent_model_id: Model ID for all MARBLE agents (e.g., "gpt-4", "claude-sonnet-4.5") + evaluator_model_id: Optional model ID for LLM-based evaluation metrics + + Returns: + Tasks with model IDs configured in environment_data and evaluation_data + """ + for task in tasks: + # Set global LLM for all MARBLE agents + task.environment_data["llm"] = agent_model_id + + # Individual agents can override via agent_config["llm"] + # This is preserved in task.environment_data["agents"] + + # Set evaluator model if LLM-based metrics enabled + if evaluator_model_id and task.evaluation_data.get("metrics", {}).get("use_llm_eval"): + task.evaluation_data["evaluate_llm"] = evaluator_model_id + + return tasks +``` + +--- + +## Recommendations + +### Primary Recommendation: Approach 2 (Hybrid Vendoring) + +**Adopt Approach 2** for initial integration due to: + +1. **Scientific Fidelity (R2)**: Guarantees exact MARBLE behavior via version pinning +2. **Development Velocity**: Enables rapid iteration without dependency management overhead +3. **Risk Mitigation**: MARBLE is early-stage; vendoring provides stability +4. **Pragmatic**: Research tool priorities favor reproducibility over packaging aesthetics + +### Migration Path: Approach 2 → Approach 1 + +Once MARBLE stabilizes (v1.0+ release, stable API), migrate to Approach 1: + +1. Publish MARBLE to PyPI with semantic versioning +2. Replace vendored source with `[project.optional-dependencies]` entry +3. Keep adapter code identical (no MASEval code changes) +4. Document migration in changelog + +**This is a low-risk transition** because adapter code is identical between approaches. + +### Approach 3 is Essential, Not Optional + +**IMPORTANT**: Approach 3 is NOT a replacement for MARBLE reproduction - it's the **reason MultiAgentBench is valuable for research**. + +The complete integration must support: +1. ✅ **Exact MARBLE reproduction** via `MarbleMultiAgentBenchBenchmark` (Approach 1 or 2) +2. ✅ **Fair framework comparison** via abstract `MultiAgentBenchBenchmark` base (Approach 3) + +**Research value**: Enables questions like: +- "Does MARBLE's coordination outperform LangGraph?" +- "Which multi-agent framework is best for coding tasks?" +- "How important is the coordination strategy vs. the LLM?" + +**Implementation**: Approach 3 is simply an abstract base class - it requires minimal code since the infrastructure (Environment, Evaluator, data loading) is shared with MARBLE reproduction. + +--- + +## Design Principle Compliance + +### R1: Reuse MASEval Infrastructure ✅ + +**Fully compliant.** The integration: +- Uses `Benchmark`, `Environment`, `Evaluator`, `AgentAdapter` base classes +- Leverages callback system for tracing +- Uses `TaskQueue` and `Task` data structures +- Benefits from parallel execution, progress bars, error handling +- No reimplementation of MASEval features + +### R2: Scientific Fidelity ✅ + +**Compliant with caveats.** + +**Preserved**: +- Uses MARBLE's Engine, AgentGraph, SharedMemory exactly as designed +- Version pinning via vendored source (Approach 2) or Git dependency (Approach 1) +- Task data preserved from original JSONL format + +**Trade-offs**: +- **Different execution environment**: MASEval orchestration vs. standalone MARBLE + - *Mitigation*: MarbleAgentAdapter delegates directly to Engine.start() + - *Impact*: Minimal - same agent coordination logic runs +- **Aggregated agent messages**: Individual agent traces must be extracted + - *Mitigation*: Implement comprehensive trace extraction in MarbleAgentAdapter + - *Impact*: Low - agent interactions are preserved, just formatted differently + +**Validation strategy**: +1. Run subset of tasks with both standalone MARBLE and MASEval integration +2. Compare outputs, metrics, and agent behaviors +3. Document any discrepancies +4. If material differences exist, adjust adapter implementation + +### R3: Fail Loudly ✅ + +**Fully compliant.** The integration includes: + +**Validation at boundaries**: +```python +# Data loading +if "agent_id" not in agent_spec: + raise ValueError("Agent missing 'agent_id'") + +# Environment tools +def get_tool(self, name): + raise NotImplementedError("Tools managed by MARBLE Engine") + +# Missing config +if not Path("marble").exists(): + raise FileNotFoundError( + "MARBLE source not found. See multiagentbench/README.md for setup." + ) +``` + +**No defensive defaults**: +- Missing JSONL fields → crash with clear error +- Invalid agent relationships → propagate MARBLE error +- Missing MARBLE dependency → crash at import + +**Clear error messages**: +- Include context (task ID, agent ID, field name) +- Suggest fixes ("See README.md for setup") +- Link to documentation + +### R4: Maintainability ✅ + +**Compliant.** The integration: + +**Clear module boundaries**: +- `data_loader.py`: JSONL → Task conversion (0 dependencies on MARBLE internals) +- `environment.py`: Thin wrapper, delegates to MARBLE +- `multiagentbench.py`: Benchmark + Evaluator, well-documented adapter pattern +- `marble/`: Vendored source, not modified (patches documented separately if needed) + +**Documentation**: +- `README.md`: Setup instructions, MARBLE version pinning +- `PROVENANCE.md`: Track MARBLE commit hash, upstream changes +- Docstrings: Explain adapter rationale, MARBLE delegation + +**Update process**: +```bash +# Update vendored MARBLE +cd maseval/benchmark/multiagentbench/marble +git pull origin main +git checkout + +# Test integration +cd ../../../.. +pytest tests/test_benchmarks/test_multiagentbench/ -v + +# Document update +echo "" > multiagentbench/MARBLE_VERSION.txt +``` + +**Long-term maintenance**: +- **Low coupling**: Adapter layer isolates MASEval from MARBLE internals +- **Testable**: Can mock MARBLE Engine for unit tests +- **Upgradable**: Switching to Approach 1 (dependency) requires zero adapter code changes + +--- + +## Implementation Checklist + +### Phase 1: Core Integration (Approach 2) +- [ ] Create `maseval/benchmark/multiagentbench/` directory +- [ ] Add `.gitignore` to exclude `marble/` +- [ ] Write `README.md` with MARBLE setup instructions +- [ ] Implement `MultiAgentBenchEnvironment(Environment)` +- [ ] Implement `MarbleAgentAdapter(AgentAdapter)` + - [ ] `_run_agent()`: Delegate to Engine.start() + - [ ] `_convert_to_message_history()`: Extract agent messages + - [ ] `gather_traces()`: Include AgentGraph, SharedMemory +- [ ] Implement `MultiAgentBenchBenchmark(Benchmark)` (abstract base) +- [ ] Implement `MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark)` + - [ ] `setup_agents()`: Create MARBLE Engine, wrap in adapter + - [ ] `_build_marble_config()`: Convert Task → MARBLE Config +- [ ] Implement `MultiAgentBenchEvaluator(Evaluator)` + - [ ] Wrap MARBLE's Evaluator + - [ ] Extract metrics (code_quality, test_coverage, collaboration) + +### Phase 2: Data Loading +- [ ] Implement `load_tasks(domain, limit)` in `data_loader.py` + - [ ] Validate JSONL fields (R3: Fail loudly) + - [ ] Convert to Task objects +- [ ] Implement `configure_model_ids(tasks, agent_model_id, evaluator_model_id)` +- [ ] Add tests for data loading with sample JSONL + +### Phase 3: Testing & Validation +- [ ] Create `tests/test_benchmarks/test_multiagentbench/` +- [ ] Unit tests for data loading +- [ ] Unit tests for Config conversion +- [ ] Integration test: Run 1 task from each domain +- [ ] Validation test: Compare results with standalone MARBLE (same task, same model) + +### Phase 4: Documentation & Examples +- [ ] Example script: `examples/multiagentbench_marble.py` +- [ ] Document setup in main MASEval README +- [ ] Add to documentation site (if exists) +- [ ] Write `PROVENANCE.md`: Track MARBLE version, license, upstream + +### Phase 5: Alternative Framework Examples (Essential for Research Value) +- [ ] Implement `LangGraphMultiAgentBench` as example of alternative multi-agent framework +- [ ] Implement `SmolagentsMultiAgentBench` as second comparison point +- [ ] Shows how users can bring their own multi-agent framework +- [ ] Enables comparative research: "MARBLE vs LangGraph vs Smolagents on same tasks" +- [ ] Demonstrates that `MultiAgentBenchBenchmark` abstract base works for ANY framework +- [ ] Document performance comparison methodology + +--- + +## Risks and Mitigations + +### Risk 1: MARBLE API Changes +**Probability**: High (early-stage project) +**Impact**: High (breaks integration) +**Mitigation**: +- Pin exact commit hash in README +- Approach 2 (vendoring) allows local patches +- Automated tests detect breakage +- Document MARBLE version compatibility matrix + +### Risk 2: Message History Extraction +**Probability**: Medium (depends on MARBLE internals) +**Impact**: Medium (affects tracing quality) +**Mitigation**: +- Test message extraction thoroughly +- If MARBLE agents don't expose messages, contribute PR upstream +- Fallback: Extract from SharedMemory state instead + +### Risk 3: License Compatibility +**Probability**: None (VERIFIED ✅) +**Impact**: N/A +**Verification**: +- ✅ MARBLE uses MIT License (Copyright 2024 Haofei Yu) +- ✅ MIT explicitly allows: "use, copy, modify, merge, publish, distribute" without restriction +- ✅ Only requirement: Include copyright notice and license file +**Implementation**: +- Keep `LICENSE` file in vendored `marble/` directory +- Document in `PROVENANCE.md`: "MARBLE is MIT licensed, vendoring permitted with attribution" + +### Risk 4: User Setup Friction +**Probability**: High (manual MARBLE clone) +**Impact**: Low (one-time setup) +**Mitigation**: +- Clear README instructions +- Provide setup script: `bash setup_multiagentbench.sh` +- Fail fast with helpful error if MARBLE missing +- Future: Migrate to Approach 1 when MARBLE stabilizes + +### Risk 5: Evaluation Discrepancies +**Probability**: Medium (different execution context) +**Impact**: High (invalidates scientific fidelity) +**Mitigation**: +- Run validation study: MASEval vs. standalone MARBLE +- Compare metrics, outputs, agent behaviors +- Document any differences +- Adjust adapter if needed +- If irreconcilable, clearly document limitations + +--- + +## Conclusion + +**Adopt Approach 2 (Hybrid Vendoring)** as the recommended integration strategy for MARBLE/MultiAgentBench into MASEval. + +This approach: +- ✅ Satisfies all four design principles (R1-R4) +- ✅ Enables reproduction of MARBLE's published results (via `MarbleMultiAgentBenchBenchmark`) +- ✅ Enables fair multi-agent framework comparison (via abstract `MultiAgentBenchBenchmark` base) +- ✅ Provides version stability during MARBLE's early development +- ✅ Reuses MASEval infrastructure effectively +- ✅ Maintains clear architectural boundaries +- ✅ Allows future migration to dependency-based approach + +**Key architectural insights**: + +1. **MARBLE's multi-agent Engine must be wrapped as a single AgentAdapter**, not decomposed into individual agents. This preserves MARBLE's coordination logic while integrating cleanly with MASEval's orchestration framework. + +2. **The architecture must support dual purposes**: Exact MARBLE reproduction (for scientific validation) AND alternative framework implementations (for comparative research). This is what makes MultiAgentBench valuable - researchers can ask "Which multi-agent approach performs better?" on the same task set. + +3. **License verified**: MARBLE's MIT license explicitly permits vendoring with attribution (copyright notice + LICENSE file in vendored directory). + +The tau2/macs integration patterns transfer successfully with this one architectural adaptation, demonstrating that MASEval's design is flexible enough to accommodate diverse benchmarking paradigms - including multi-agent systems research. + +--- + +## Appendices + +### Appendix A: Complete Code Example + +See implementation checklist for full module structure. Key classes: +- `MarbleAgentAdapter`: Wraps Engine as single agent +- `MarbleMultiAgentBenchBenchmark`: Creates Engine from Task +- `MultiAgentBenchEvaluator`: Wraps MARBLE metrics +- `load_tasks()`: JSONL → Task conversion + +### Appendix B: MARBLE Architecture Summary + +- **Engine**: Main orchestrator +- **BaseAgent**: Individual agent with LLM +- **AgentGraph**: Manages agent relationships and coordination +- **SharedMemory**: Inter-agent communication +- **EnginePlanner**: Plans agent execution order +- **Environments**: Domain-specific (Coding, DB, Minecraft, Research, Web, WorldSimulation) +- **Evaluator**: Computes domain-specific metrics + +### Appendix C: Task Format Mapping + +**JSONL → Task:** +- `task_id` → `Task.id` (prefixed with domain) +- `task.content` → `Task.query` +- `scenario`, `agents`, `environment`, etc. → `Task.environment_data` +- `metrics` → `Task.evaluation_data` +- Remaining fields → `Task.metadata` + +### Appendix D: Alternative Multi-Agent Frameworks + +After MARBLE integration is stable, users can create custom multi-agent benchmarks: +- LangGraph: Graph-based multi-agent workflows +- CrewAI: Role-based agent teams +- AutoGen: Conversational multi-agent systems +- Custom: Domain-specific coordination logic + +Pattern: Subclass `MultiAgentBenchBenchmark`, implement `setup_agents()` with chosen framework. + +--- + +**Document Version**: 1.0 +**Date**: 2026-01-19 +**Author**: Claude (Sonnet 4.5) +**Status**: Final Recommendation diff --git a/STRATEGY_CRITIQUE.md b/STRATEGY_CRITIQUE.md new file mode 100644 index 00000000..30937736 --- /dev/null +++ b/STRATEGY_CRITIQUE.md @@ -0,0 +1,517 @@ +# STRATEGY.md Critique: MultiAgentBench Integration + +This document analyzes the proposed STRATEGY.md for integrating MARBLE/MultiAgentBench into MASEval, comparing against established patterns from tau2 and macs benchmarks. + +--- + +## Executive Summary + +The STRATEGY.md proposes a reasonable high-level architecture but contains several issues: + +| Category | Count | Severity | +|----------|-------|----------| +| **Incorrect API Assumptions** | 5 | High | +| **Pattern Violations** | 3 | Medium | +| **Clumsy Implementations** | 4 | Low-Medium | +| **Missing Considerations** | 3 | Medium | + +**Overall Assessment**: The strategy requires significant revision before implementation, particularly around MARBLE API assumptions and the "Engine as single adapter" pattern. + +--- + +## 1. Incorrect MARBLE API Assumptions + +### 1.1 SharedMemory is NOT Actually Shared (HIGH SEVERITY) + +**STRATEGY.md claims:** +```python +# Extract final answer from SharedMemory or designated output +final_answer = self.engine.memory.get("final_answer") +``` + +**Reality**: Each MARBLE agent creates its own `SharedMemory` instance at initialization (`base_agent.py:73`). The memory is NOT shared between agents despite the misleading name. There is no central `engine.memory` that accumulates agent outputs. + +**Evidence from MARBLE source:** +```python +# In BaseAgent.__init__(): +self.memory = BaseMemory() # Per-agent memory +self.shared_memory = SharedMemory() # Also per-agent, NOT shared! +``` + +**Impact**: The proposed `_convert_to_message_history()` and final answer extraction logic will fail silently or return empty data. + +**Recommendation**: Extract final answers from: +- `engine.agents[-1].last_output` (if exists) +- Agent task_history or msg_box +- Environment state after execution + +--- + +### 1.2 Missing `get_agent_profiles_linked()` Method (HIGH SEVERITY) + +**STRATEGY.md implies MARBLE's chain coordination works:** +> "Agents use AgentGraph to check who they can communicate with" + +**Reality**: `engine.py:702` calls `self.graph.get_agent_profiles_linked(current_agent.agent_id)` but this method **does not exist** in `AgentGraph`. This is a bug in MARBLE that will crash chain coordination mode. + +**Impact**: Cannot reliably run tasks with `coordinate_mode: "chain"` using vanilla MARBLE. + +**Recommendation**: +1. Document this limitation +2. Either patch MARBLE locally or avoid chain mode tasks initially +3. File upstream issue + +--- + +### 1.3 Message History Access Pattern is Wrong (HIGH SEVERITY) + +**STRATEGY.md proposes:** +```python +def _convert_to_message_history(self, engine) -> MessageHistory: + for agent in engine.agents: + if hasattr(agent, "messages"): + for msg in agent.messages: + # ... +``` + +**Reality**: MARBLE agents store messages in `msg_box`, not a `messages` attribute: +```python +# BaseAgent structure: +self.msg_box: Dict[session_id, Dict[agent_id, List[Tuple[direction, message]]]] +# Direction: 0 = FORWARD_TO (sent), 1 = RECV_FROM (received) +``` + +Messages are serialized via `agent.seralize_message(session_id)` (note: typo in MARBLE). + +**Recommendation**: Use the actual MARBLE API: +```python +for agent in engine.agents: + serialized = agent.seralize_message("") # Empty session_id for all + # Parse serialized string or access msg_box directly +``` + +--- + +### 1.4 `engine.start()` Return Value Not Specified + +**STRATEGY.md assumes:** +```python +def _run_agent(self, query: str) -> Any: + self.engine.start() + # ... extract results +``` + +**Reality**: `engine.start()` dispatches to coordination methods (`star_coordinate`, `chain_coordinate`, etc.) but the return behavior varies and is not consistently documented. Some modes return results, others mutate state. + +**Recommendation**: Study each coordination mode's return behavior and document clearly. Don't assume uniform behavior. + +--- + +### 1.5 Evaluator.update() Must Be Called Explicitly + +**STRATEGY.md implies:** +> "MARBLE evaluator metrics (code_quality, collaboration)" + +**Reality**: MARBLE's `Evaluator.update(environment, agents)` must be called explicitly after each iteration—the engine does NOT call it automatically. Without explicit calls, metrics remain empty. + +**Impact**: `MultiAgentBenchEvaluator` wrapper may return empty metrics. + +**Recommendation**: Call `marble_evaluator.update()` after engine execution, then `marble_evaluator.finalize()` before reading metrics. + +--- + +## 2. Pattern Violations + +### 2.1 "Engine as Single AgentAdapter" Violates Multi-Agent Visibility (MEDIUM SEVERITY) + +**STRATEGY.md proposes:** +```python +# Wrap entire engine as single adapter +engine_adapter = MarbleAgentAdapter(engine, "marble_engine") +return [engine_adapter], {"marble_engine": engine_adapter} +``` + +**MASEval Pattern (from tau2/macs)**: +- Each agent gets its own `AgentAdapter` +- `agents_dict` maps agent names to individual adapters +- Traces are collected per-agent for analysis + +**Why this matters**: +1. **Trace granularity lost**: Cannot analyze individual agent behavior +2. **Callback hooks incomplete**: `on_agent_step_start/end` fires once for entire system +3. **Error attribution unclear**: Which internal agent failed? +4. **Inconsistent with macs**: MACS explicitly creates adapters per agent in hierarchy + +**Comparison to MACS pattern:** +```python +# MACS approach - each agent is visible +def setup_agents(self, agent_data, environment, task, user): + adapters = [] + agents_dict = {} + for agent_spec in agent_data["agents"]: + agent = create_agent(agent_spec, environment) + adapter = AgentAdapter(agent, agent_spec["id"]) + adapters.append(adapter) + agents_dict[agent_spec["id"]] = adapter + return adapters, agents_dict +``` + +**Justified Reason to Deviate**: MARBLE's coordination logic is tightly coupled in the Engine. Exposing individual agents would require reimplementing coordination, defeating the purpose of scientific fidelity. + +**Recommendation**: Document this as an explicit architectural trade-off. Consider: +1. Keep Engine-as-adapter for `MarbleMultiAgentBenchBenchmark` (reproduction mode) +2. For custom framework implementations, encourage per-agent adapters +3. Enhance `gather_traces()` to expose internal agent details as nested structure + +--- + +### 2.2 Environment.create_tools() Returns Empty Dict (MEDIUM SEVERITY) + +**STRATEGY.md proposes:** +```python +def create_tools(self) -> Dict[str, Any]: + # MARBLE environments manage tools internally + return {} + +def get_tool(self, name: str) -> Optional[Any]: + raise NotImplementedError("Tools managed by MARBLE Engine") +``` + +**MASEval Pattern (from tau2/macs)**: +- `Environment.create_tools()` returns actual tool instances +- Tools are registered and traced via MASEval infrastructure +- Agents receive tools from environment + +**tau2 example:** +```python +def create_tools(self) -> Dict[str, Callable]: + return self.toolkit.get_tools() # Actual callable tools +``` + +**macs example:** +```python +def create_tools(self) -> Dict[str, MACSGenericTool]: + tools = {} + for spec in self.state["tool_specs"]: + tools[spec["name"]] = MACSGenericTool(spec, model) + return tools +``` + +**Why this matters**: +1. **Tool tracing lost**: MASEval won't capture tool invocations +2. **ToolInvocationHistory empty**: No tool usage data in traces +3. **Inconsistent behavior**: Users expect `environment.tools` to work + +**Recommendation**: Either: +1. Extract tool definitions from MARBLE and create MASEval-traceable wrappers +2. Or document clearly that tool tracing requires post-hoc extraction from engine traces + +--- + +### 2.3 User Simulator Returning None (MINOR) + +**STRATEGY.md proposes:** +```python +def setup_user(self, agent_data, environment, task, user): + # → None (MultiAgentBench doesn't use external user simulation) + return None +``` + +**This is acceptable** - tau2 and macs support optional user simulation. However, MARBLE does have some scenarios with user interaction. + +**Recommendation**: Verify which MultiAgentBench domains require user simulation and handle appropriately. + +--- + +## 3. Clumsy Implementations + +### 3.1 Overly Complex Config Conversion + +**STRATEGY.md proposes:** +```python +marble_config = Config.from_dict({ + "coordinate_mode": task.environment_data["coordinate_mode"], + "relationships": task.environment_data["relationships"], + "agents": task.environment_data["agents"], + "environment": task.environment_data["environment"], + "task": {"content": task.query}, + "llm": task.environment_data["llm"], + "memory": {"type": "SharedMemory"}, + "engine_planner": {"initial_progress": "Starting task"}, + "metrics": task.evaluation_data["metrics"], +}) +``` + +**Issue**: This duplicates MARBLE's config loading. MARBLE already has `Config.from_file()` and the JSONL entries ARE MARBLE configs. + +**Cleaner approach:** +```python +# MARBLE configs are stored directly in environment_data +marble_config = Config.from_dict(task.environment_data["raw_marble_config"]) +``` + +Or even simpler - load MARBLE tasks directly and convert to MASEval Task: +```python +def load_tasks(domain): + # Let MARBLE parse its own format + marble_tasks = marble.load_tasks(domain) + return [Task.from_marble(mt) for mt in marble_tasks] +``` + +--- + +### 3.2 Symlink for Data Directory is Fragile + +**STRATEGY.md proposes:** +``` +└── data/ # Symlink to marble/multiagentbench/ +``` + +**Issues**: +1. Symlinks don't work well on Windows +2. Adds complexity to setup +3. Git doesn't track symlinks reliably + +**Cleaner approach (tau2 pattern):** +```python +# In data_loader.py +def _get_data_dir() -> Path: + # Look for MARBLE in known locations + candidates = [ + Path(__file__).parent / "marble" / "multiagentbench", + Path(os.environ.get("MARBLE_DATA_DIR", "")), + ] + for candidate in candidates: + if candidate.exists(): + return candidate + raise FileNotFoundError("MARBLE data not found. See README.md for setup.") +``` + +--- + +### 3.3 Redundant Fallback Strategies for Message History + +**STRATEGY.md proposes three fallback strategies:** +```python +# Strategy 1: Direct message access +# Strategy 2: Fallback to SharedMemory +# Strategy 3: Fallback to agent outputs +``` + +**Issue**: Strategy 2 won't work (SharedMemory isn't shared). This creates dead code and false confidence. + +**Recommendation**: Remove non-functional fallbacks. Be explicit about what actually works: +```python +def _extract_messages(self, engine) -> List[Dict]: + """Extract messages from MARBLE agents. + + Note: MARBLE stores messages in agent.msg_box, not a central location. + """ + messages = [] + for agent in engine.agents: + # Only method that actually works + for session_id, conversations in agent.msg_box.items(): + for other_agent_id, msg_list in conversations.items(): + for direction, content in msg_list: + messages.append({ + "agent_id": agent.agent_id, + "direction": "sent" if direction == 0 else "received", + "to/from": other_agent_id, + "content": content, + }) + return messages +``` + +--- + +### 3.4 Over-Specification in Implementation Checklist + +The checklist has 40+ items spanning 7 phases. Compare to tau2 implementation which is ~500 lines across 4 files. + +**Recommendation**: Simplify to MVP: +1. Phase 1: Data loader + basic environment +2. Phase 2: MarbleAgentAdapter + benchmark class +3. Phase 3: Evaluator integration +4. Phase 4: Tests + example + +Defer user framework examples (LangGraph, smolagents) to later PRs. + +--- + +## 4. Missing Considerations + +### 4.1 No Discussion of Domain-Specific Environments + +**MARBLE has 7 distinct environments:** +- CodingEnvironment (file operations) +- DBEnvironment (Docker + PostgreSQL) +- MinecraftEnvironment (external process) +- ResearchEnvironment +- BargainingEnvironment +- WebEnvironment +- WorldSimulationEnvironment + +**Each has different setup requirements:** +- DBEnvironment requires Docker +- MinecraftEnvironment requires external game server +- CodingEnvironment needs filesystem access + +**STRATEGY.md mentions this but doesn't address:** +- How to handle missing dependencies gracefully +- Which domains to support initially +- How to skip domains without required infrastructure + +**Recommendation**: Start with simpler domains (Research, Bargaining) that don't require external services. Add infrastructure-heavy domains (DB, Minecraft) as optional extras. + +--- + +### 4.2 No Error Classification Strategy + +**MASEval uses structured error attribution:** +```python +class TaskExecutionStatus(Enum): + SUCCESS = "success" + AGENT_ERROR = "agent_error" + ENVIRONMENT_ERROR = "environment_error" + USER_ERROR = "user_error" + TASK_TIMEOUT = "task_timeout" + # ... +``` + +**STRATEGY.md doesn't discuss:** +- How MARBLE errors map to these statuses +- What happens when MARBLE's internal agents fail +- How to distinguish MARBLE bugs from agent failures + +**Recommendation**: Add error classification logic: +```python +def _classify_marble_error(self, error: Exception) -> TaskExecutionStatus: + if "MARBLE" in str(error) or isinstance(error, MarbleInternalError): + return TaskExecutionStatus.ENVIRONMENT_ERROR # MARBLE bug + if "LLM" in str(error) or "rate limit" in str(error).lower(): + return TaskExecutionStatus.ENVIRONMENT_ERROR # External service + return TaskExecutionStatus.AGENT_ERROR # Agent's fault +``` + +--- + +### 4.3 Evaluation Metrics Not Fully Mapped + +**MARBLE Evaluator produces:** +- `task_completion`: List[0/1] +- `token_consumption`: List[int] +- `planning_score`: List[1-5] +- `communication_score`: List[1-5] +- `code_quality`: Dict +- `agent_kpis`: Dict[agent_id, milestone_count] + +**MASEval evaluation format:** +```python +{"metric_name": value, "passed": bool, ...} +``` + +**STRATEGY.md proposes:** +```python +return self.marble_evaluator.evaluate(output, task_spec, traces) +``` + +But doesn't specify how MARBLE's metrics map to MASEval's format. + +**Recommendation**: Define explicit mapping: +```python +def __call__(self, traces, final_answer) -> Dict[str, Any]: + marble_metrics = self.marble_evaluator.get_metrics() + return { + "passed": marble_metrics.get("task_completion", [0])[-1] == 1, + "task_completion": marble_metrics.get("task_completion", [0])[-1], + "planning_score": mean(marble_metrics.get("planning_score", [3])), + "communication_score": mean(marble_metrics.get("communication_score", [3])), + "total_tokens": sum(marble_metrics.get("token_consumption", [0])), + # ... + } +``` + +--- + +## 5. Justified Pattern Deviations + +### 5.1 Vendoring MARBLE Source (JUSTIFIED) + +**Deviation**: Vendoring entire MARBLE repo rather than pip dependency. + +**Justification**: +- MARBLE's pip package has bugs +- Need to pin exact version for reproducibility +- May need local patches +- MIT license explicitly permits this + +**This is the right choice** - tau2 does something similar with its data. + +--- + +### 5.2 Abstract Base + Concrete MARBLE Implementation (JUSTIFIED) + +**Deviation**: Creating `MultiAgentBenchBenchmark` (abstract) and `MarbleMultiAgentBenchBenchmark` (concrete). + +**Justification**: +- Enables framework comparison research +- Matches MASEval's "BYO agent" philosophy +- Same pattern used by tau2 and macs + +**This is well-aligned** with MASEval patterns. + +--- + +### 5.3 Engine-Level Wrapping for Scientific Fidelity (PARTIALLY JUSTIFIED) + +**Deviation**: Wrapping MARBLE Engine as single adapter. + +**Partial Justification**: +- Preserves MARBLE's exact coordination logic +- Required for reproducing published results +- Would require reimplementing MARBLE to do otherwise + +**Concern**: Loses per-agent observability. Should be documented as explicit trade-off with enhanced trace extraction to compensate. + +--- + +## 6. Recommended Changes Before Implementation + +### High Priority (Must Fix) + +1. **Fix SharedMemory assumptions** - It's not shared; update all code that assumes it is +2. **Fix message history extraction** - Use `msg_box` and `seralize_message()` +3. **Document chain mode bug** - `get_agent_profiles_linked()` doesn't exist +4. **Add explicit Evaluator.update() calls** + +### Medium Priority (Should Fix) + +5. **Enhance traces from Engine adapter** - Include internal agent details as nested structure +6. **Simplify config conversion** - Use MARBLE's native loading where possible +7. **Add error classification** - Map MARBLE errors to MASEval statuses +8. **Define metric mapping** - MARBLE metrics → MASEval format + +### Low Priority (Nice to Have) + +9. **Remove symlink approach** - Use path resolution instead +10. **Simplify implementation phases** - Start with MVP +11. **Document domain requirements** - Which need Docker, external services, etc. +12. **Remove non-functional fallbacks** - Dead code creates false confidence + +--- + +## 7. Conclusion + +The STRATEGY.md demonstrates good understanding of MASEval's architecture but makes several incorrect assumptions about MARBLE's API that would cause runtime failures. The "Engine as single adapter" pattern is a reasonable trade-off for scientific fidelity but should be clearly documented with enhanced trace extraction. + +**Recommended next steps:** +1. Revise STRATEGY.md with corrections from this critique +2. Create minimal proof-of-concept with single domain (Research or Bargaining) +3. Validate MARBLE API assumptions with actual execution +4. Expand to other domains incrementally + +--- + +*Document Version: 1.0* +*Date: 2026-01-19* +*Reviewer: Claude (Opus 4.5)* From 9ce4840fe780a874ab8d02f9b0a5da7061c16491 Mon Sep 17 00:00:00 2001 From: cemde Date: Mon, 19 Jan 2026 23:48:46 +0100 Subject: [PATCH 06/17] initial implementation --- maseval/benchmark/multiagentbench/.gitignore | 10 + .../benchmark/multiagentbench/PROVENANCE.md | 74 +++ maseval/benchmark/multiagentbench/README.md | 166 ++++++ maseval/benchmark/multiagentbench/__init__.py | 135 +++++ .../multiagentbench/adapters/__init__.py | 7 + .../adapters/marble_adapter.py | 224 +++++++ .../benchmark/multiagentbench/data_loader.py | 312 ++++++++++ .../benchmark/multiagentbench/environment.py | 363 ++++++++++++ .../benchmark/multiagentbench/evaluator.py | 552 ++++++++++++++++++ .../multiagentbench/multiagentbench.py | 481 +++++++++++++++ .../test_multiagentbench/__init__.py | 1 + .../test_multiagentbench/conftest.py | 280 +++++++++ .../test_multiagentbench/test_benchmark.py | 252 ++++++++ .../test_multiagentbench/test_data_loader.py | 321 ++++++++++ .../test_multiagentbench/test_environment.py | 163 ++++++ .../test_multiagentbench/test_evaluator.py | 319 ++++++++++ 16 files changed, 3660 insertions(+) create mode 100644 maseval/benchmark/multiagentbench/.gitignore create mode 100644 maseval/benchmark/multiagentbench/PROVENANCE.md create mode 100644 maseval/benchmark/multiagentbench/README.md create mode 100644 maseval/benchmark/multiagentbench/__init__.py create mode 100644 maseval/benchmark/multiagentbench/adapters/__init__.py create mode 100644 maseval/benchmark/multiagentbench/adapters/marble_adapter.py create mode 100644 maseval/benchmark/multiagentbench/data_loader.py create mode 100644 maseval/benchmark/multiagentbench/environment.py create mode 100644 maseval/benchmark/multiagentbench/evaluator.py create mode 100644 maseval/benchmark/multiagentbench/multiagentbench.py create mode 100644 tests/test_benchmarks/test_multiagentbench/__init__.py create mode 100644 tests/test_benchmarks/test_multiagentbench/conftest.py create mode 100644 tests/test_benchmarks/test_multiagentbench/test_benchmark.py create mode 100644 tests/test_benchmarks/test_multiagentbench/test_data_loader.py create mode 100644 tests/test_benchmarks/test_multiagentbench/test_environment.py create mode 100644 tests/test_benchmarks/test_multiagentbench/test_evaluator.py diff --git a/maseval/benchmark/multiagentbench/.gitignore b/maseval/benchmark/multiagentbench/.gitignore new file mode 100644 index 00000000..abb9714c --- /dev/null +++ b/maseval/benchmark/multiagentbench/.gitignore @@ -0,0 +1,10 @@ +# Vendored MARBLE source (clone manually) +marble/ + +# Python cache +__pycache__/ +*.pyc +*.pyo + +# Test artifacts +.pytest_cache/ diff --git a/maseval/benchmark/multiagentbench/PROVENANCE.md b/maseval/benchmark/multiagentbench/PROVENANCE.md new file mode 100644 index 00000000..50fe413a --- /dev/null +++ b/maseval/benchmark/multiagentbench/PROVENANCE.md @@ -0,0 +1,74 @@ +# MARBLE Integration Provenance + +## Source Information + +- **Source Repository**: https://github.com/ulab-uiuc/MARBLE +- **Version**: Not yet pinned (clone latest and test) +- **License**: MIT (Copyright 2024 Haofei Yu) +- **Vendoring**: Permitted by MIT license with attribution + +## Reference + +**Paper**: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents" +- arXiv: https://arxiv.org/abs/2503.01935 +- Authors: Haofei Yu et al. +- Publication Date: 2025 + +## License Text (MIT) + +``` +MIT License + +Copyright (c) 2024 Haofei Yu + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +``` + +## Known Issues in MARBLE + +1. **Missing method**: `AgentGraph.get_agent_profiles_linked()` does not exist but is + called in `engine.py:702`. This breaks chain coordination mode. + +2. **SharedMemory naming**: Despite the name, `SharedMemory` is instantiated per-agent + in `BaseAgent.__init__()` and is NOT shared between agents. Use `msg_box` for + inter-agent communication. + +3. **Environment constructor signature**: Some environments expect different constructor + arguments. Check each environment's `__init__` signature before use. + +## Local Patches Applied + +None currently. Document any patches here if applied. + +## Update Process + +To update MARBLE to a newer version: + +1. `cd maseval/benchmark/multiagentbench/marble` +2. `git fetch origin` +3. `git log --oneline origin/main` (review changes) +4. `git checkout ` +5. Run integration tests +6. Update this file with new version info + +## Last Updated + +- **Date**: 2026-01-19 +- **Updated By**: Claude Code +- **Version Tested**: Initial integration (not yet pinned) diff --git a/maseval/benchmark/multiagentbench/README.md b/maseval/benchmark/multiagentbench/README.md new file mode 100644 index 00000000..5a189bfc --- /dev/null +++ b/maseval/benchmark/multiagentbench/README.md @@ -0,0 +1,166 @@ +# MultiAgentBench Integration + +Framework-agnostic implementation of the MultiAgentBench benchmark suite from MARBLE +(Multi-Agent Coordination Backbone with LLM Engine) for evaluating multi-agent collaboration. + +**Original Paper**: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents" +(arXiv:2503.01935) + +**Original Repository**: https://github.com/ulab-uiuc/MARBLE + +## Setup + +This benchmark requires the MARBLE source code to be cloned locally. The benchmark +does NOT install MARBLE as a dependency - instead, it imports directly from a local copy. + +### 1. Clone MARBLE + +```bash +cd maseval/benchmark/multiagentbench +git clone https://github.com/ulab-uiuc/MARBLE.git marble +cd marble +# Pin to tested version (recommended) +git checkout +``` + +### 2. Install MARBLE Dependencies + +MARBLE requires additional dependencies. Add them to your environment: + +```bash +# If using uv (recommended) +uv add litellm ruamel.yaml + +# Or with pip +pip install litellm ruamel.yaml +``` + +### 3. Verify Setup + +```python +from maseval.benchmark.multiagentbench import load_tasks + +# Should load without error +tasks = load_tasks("research", limit=1) +print(f"Loaded {len(tasks)} task(s)") +``` + +## Usage + +### Basic Usage (Abstract Base) + +The abstract `MultiAgentBenchBenchmark` provides task loading, environment setup, +and evaluation infrastructure. You implement `setup_agents()` with your framework: + +```python +from maseval.benchmark.multiagentbench import ( + MultiAgentBenchBenchmark, + MultiAgentBenchEnvironment, + load_tasks, + configure_model_ids, +) + +class MyMultiAgentBenchmark(MultiAgentBenchBenchmark): + def setup_agents(self, agent_data, environment, task, user): + # Your framework-specific agent creation + ... + + def get_model_adapter(self, model_id, **kwargs): + adapter = MyModelAdapter(model_id) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter + +# Load and configure tasks +tasks = load_tasks("research", limit=5) +configure_model_ids(tasks, agent_model_id="gpt-4o") + +# Run +benchmark = MyMultiAgentBenchmark() +results = benchmark.run(tasks) +``` + +### MARBLE Reproduction + +Use `MarbleMultiAgentBenchBenchmark` for exact reproduction of MARBLE's published results: + +```python +from maseval.benchmark.multiagentbench import ( + MarbleMultiAgentBenchBenchmark, + load_tasks, + configure_model_ids, +) + +# Load tasks from a simple domain (no Docker required) +tasks = load_tasks("research", limit=5) +configure_model_ids(tasks, agent_model_id="gpt-4o") + +# Create benchmark with model adapter implementation +class MyMarbleBenchmark(MarbleMultiAgentBenchBenchmark): + def get_model_adapter(self, model_id, **kwargs): + from maseval.interface.openai import OpenAIModelAdapter + adapter = OpenAIModelAdapter(model_id) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter + +benchmark = MyMarbleBenchmark() +results = benchmark.run(tasks) + +# Print results +for result in results: + print(f"Task: {result['task_id']}") + print(f"Status: {result['status']}") + if result['eval']: + print(f"Passed: {result['eval'][0]['passed']}") +``` + +## Domains + +MultiAgentBench includes 7 domains with different requirements: + +| Domain | External Dependencies | Initial Support | +|--------|----------------------|-----------------| +| Research | None | Yes | +| Bargaining | None | Yes | +| Coding | Filesystem access | Yes | +| Web | Network access | Yes | +| WorldSimulation | None | Yes | +| Database | Docker + PostgreSQL | Optional | +| Minecraft | External game server | Deferred | + +### Domain-Specific Notes + +- **Research/Bargaining**: Recommended for initial testing - no infrastructure required +- **Coding**: Creates files in a workspace directory +- **Database**: Requires Docker with PostgreSQL image +- **Minecraft**: Not currently supported (requires external game server) + +## Known Limitations + +1. **Chain coordination mode bug**: MARBLE's `engine.py` references `get_agent_profiles_linked()` + which doesn't exist in `AgentGraph`. Tasks using chain coordination may fail. + +2. **SharedMemory is per-agent**: Despite the name, each MARBLE agent creates its own + `SharedMemory` instance. Use `msg_box` for inter-agent communication. + +3. **Requires manual MARBLE clone**: MARBLE must be cloned manually into the `marble/` + subdirectory (gitignored by default). + +## File Structure + +``` +multiagentbench/ +├── __init__.py # Public API exports +├── README.md # This file +├── PROVENANCE.md # MARBLE version and license info +├── .gitignore # Ignores marble/ directory +├── multiagentbench.py # Benchmark classes +├── environment.py # MultiAgentBenchEnvironment +├── data_loader.py # Task loading utilities +├── adapters/ +│ ├── __init__.py +│ └── marble_adapter.py # MarbleAgentAdapter +└── marble/ # ← Vendored MARBLE (gitignored) + └── ... +``` diff --git a/maseval/benchmark/multiagentbench/__init__.py b/maseval/benchmark/multiagentbench/__init__.py new file mode 100644 index 00000000..19c3d2de --- /dev/null +++ b/maseval/benchmark/multiagentbench/__init__.py @@ -0,0 +1,135 @@ +"""MultiAgentBench - Multi-Agent Coordination Benchmark from MARBLE. + +Framework-agnostic implementation of the MultiAgentBench benchmark suite for +evaluating multi-agent collaboration and competition in LLM-based systems. + +Original Repository: https://github.com/ulab-uiuc/MARBLE +Paper: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents" + (arXiv:2503.01935) + +Domains: + - research: Research idea generation and collaboration + - bargaining: Negotiation and bargaining scenarios + - coding: Software development collaboration + - database: Database manipulation and querying (requires Docker) + - minecraft: Collaborative building (requires external server) + - web: Web-based task completion + - worldsimulation: World simulation and interaction + +Setup: + This benchmark requires MARBLE source code to be cloned locally: + + ```bash + cd maseval/benchmark/multiagentbench + git clone https://github.com/ulab-uiuc/MARBLE.git marble + ``` + + See README.md in this directory for detailed setup instructions. + +Usage: + from maseval.benchmark.multiagentbench import ( + MultiAgentBenchBenchmark, + MarbleMultiAgentBenchBenchmark, + MultiAgentBenchEnvironment, + MultiAgentBenchEvaluator, + MarbleAgentAdapter, + load_tasks, + configure_model_ids, + get_domain_info, + VALID_DOMAINS, + ) + + # Load and configure tasks + tasks = load_tasks("research", limit=5) + configure_model_ids(tasks, agent_model_id="gpt-4o") + + # Create your framework-specific benchmark subclass + class MyMultiAgentBenchmark(MultiAgentBenchBenchmark): + def setup_agents(self, agent_data, environment, task, user): + # Create your agents + ... + + def get_model_adapter(self, model_id, **kwargs): + adapter = MyModelAdapter(model_id) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter + + # Run benchmark + benchmark = MyMultiAgentBenchmark() + results = benchmark.run(tasks, agent_data={}) + +MARBLE Reproduction Mode: + For exact reproduction of MARBLE's published results, use + MarbleMultiAgentBenchBenchmark which wraps MARBLE's native agents: + + ```python + class MyMarbleBenchmark(MarbleMultiAgentBenchBenchmark): + def get_model_adapter(self, model_id, **kwargs): + from maseval.interface.openai import OpenAIModelAdapter + adapter = OpenAIModelAdapter(model_id) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter + + benchmark = MyMarbleBenchmark() + results = benchmark.run(tasks, agent_data={}) + ``` +""" + +# Core benchmark classes +from maseval.benchmark.multiagentbench.multiagentbench import ( + MultiAgentBenchBenchmark, + MarbleMultiAgentBenchBenchmark, +) + +# Environment +from maseval.benchmark.multiagentbench.environment import ( + MultiAgentBenchEnvironment, + INFRASTRUCTURE_DOMAINS, +) + +# Evaluator +from maseval.benchmark.multiagentbench.evaluator import ( + MultiAgentBenchEvaluator, + MultiAgentBenchMetrics, +) + +# Agent adapters +from maseval.benchmark.multiagentbench.adapters import ( + MarbleAgentAdapter, +) +from maseval.benchmark.multiagentbench.adapters.marble_adapter import ( + create_marble_agents, +) + +# Data loading +from maseval.benchmark.multiagentbench.data_loader import ( + load_tasks, + configure_model_ids, + get_domain_info, + VALID_DOMAINS, + INFRASTRUCTURE_DOMAINS as INFRASTRUCTURE_REQUIRED_DOMAINS, +) + + +__all__ = [ + # Core benchmark classes + "MultiAgentBenchBenchmark", + "MarbleMultiAgentBenchBenchmark", + # Environment + "MultiAgentBenchEnvironment", + "INFRASTRUCTURE_DOMAINS", + # Evaluator + "MultiAgentBenchEvaluator", + "MultiAgentBenchMetrics", + # Agent adapters + "MarbleAgentAdapter", + "create_marble_agents", + # Data loading + "load_tasks", + "configure_model_ids", + "get_domain_info", + "VALID_DOMAINS", + "INFRASTRUCTURE_REQUIRED_DOMAINS", +] diff --git a/maseval/benchmark/multiagentbench/adapters/__init__.py b/maseval/benchmark/multiagentbench/adapters/__init__.py new file mode 100644 index 00000000..a020fed0 --- /dev/null +++ b/maseval/benchmark/multiagentbench/adapters/__init__.py @@ -0,0 +1,7 @@ +"""Agent adapters for MultiAgentBench.""" + +from maseval.benchmark.multiagentbench.adapters.marble_adapter import ( + MarbleAgentAdapter, +) + +__all__ = ["MarbleAgentAdapter"] diff --git a/maseval/benchmark/multiagentbench/adapters/marble_adapter.py b/maseval/benchmark/multiagentbench/adapters/marble_adapter.py new file mode 100644 index 00000000..d8fa5b59 --- /dev/null +++ b/maseval/benchmark/multiagentbench/adapters/marble_adapter.py @@ -0,0 +1,224 @@ +"""MARBLE agent adapter for MASEval. + +This module provides an adapter that wraps MARBLE's BaseAgent for use with +MASEval's tracing and evaluation infrastructure. +""" + +from typing import TYPE_CHECKING, Any, Dict, List, Optional, Sequence, Tuple + +from maseval import AgentAdapter, AgentError + +if TYPE_CHECKING: + from maseval.core.callback import BenchmarkCallback + + +class MarbleAgentAdapter(AgentAdapter): + """Adapter wrapping a MARBLE BaseAgent for MASEval tracing. + + This adapter provides a unified interface to MARBLE agents while + capturing all relevant traces for evaluation. + + Attributes: + agent_id: Unique identifier for the agent + marble_agent: The underlying MARBLE BaseAgent instance + profile: Agent's role profile from MARBLE config + """ + + def __init__( + self, + marble_agent: Any, + agent_id: str, + *, + callbacks: Optional[List["BenchmarkCallback"]] = None, + ): + """Initialize the adapter. + + Args: + marble_agent: MARBLE BaseAgent instance + agent_id: Unique identifier for this agent + callbacks: Optional list of callbacks + """ + self._marble_agent = marble_agent + self._agent_id = agent_id + self._profile = getattr(marble_agent, "profile", "") + self._communication_log: List[Dict[str, Any]] = [] + self._action_log: List[Dict[str, Any]] = [] + super().__init__(callbacks=callbacks) + + @property + def agent_id(self) -> str: + """Return the agent's unique identifier.""" + return self._agent_id + + @property + def profile(self) -> str: + """Return the agent's profile.""" + return self._profile + + @property + def marble_agent(self) -> Any: + """Return the underlying MARBLE agent.""" + return self._marble_agent + + def _run_agent(self, query: str) -> str: + """Execute the MARBLE agent's act() method. + + Args: + query: The task/query to pass to the agent + + Returns: + String result from the agent + + Raises: + AgentError: If agent execution fails + """ + try: + # Call MARBLE agent's act method + result, communication = self._marble_agent.act(query) + + # Log action + self._action_log.append( + { + "task": query, + "result": result, + "has_communication": communication is not None, + } + ) + + # Log communication if present + if communication is not None: + self._communication_log.append( + { + "session_id": str(getattr(self._marble_agent, "session_id", "")), + "communication": communication, + } + ) + + # Update message history + self._messages.add_message(role="user", content=query) + self._messages.add_message(role="assistant", content=result) + + return result + + except Exception as e: + raise AgentError( + f"MARBLE agent '{self._agent_id}' failed: {e}", + component=f"MarbleAgentAdapter:{self._agent_id}", + ) from e + + def get_token_usage(self) -> int: + """Get the total token usage from the MARBLE agent. + + Returns: + Total tokens used by the agent + """ + if hasattr(self._marble_agent, "get_token_usage"): + return self._marble_agent.get_token_usage() + return 0 + + def get_memory_str(self) -> str: + """Get the agent's memory as a string. + + Returns: + Serialized memory string + """ + memory = getattr(self._marble_agent, "memory", None) + if memory is not None and hasattr(memory, "get_memory_str"): + return memory.get_memory_str() + return "" + + def get_serialized_messages(self, session_id: str = "") -> str: + """Get serialized inter-agent messages. + + Args: + session_id: Optional session ID filter + + Returns: + Serialized message string + """ + if hasattr(self._marble_agent, "seralize_message"): + return self._marble_agent.seralize_message(session_id) + return "" + + def gather_traces(self) -> Dict[str, Any]: + """Gather traces including agent-specific data. + + Returns: + Dict with all agent traces + """ + traces = super().gather_traces() + + # Add MARBLE-specific traces + traces["agent_id"] = self._agent_id + traces["profile"] = self._profile + traces["token_usage"] = self.get_token_usage() + traces["action_log"] = self._action_log + traces["communication_log"] = self._communication_log + traces["memory"] = self.get_memory_str() + traces["relationships"] = getattr(self._marble_agent, "relationships", {}) + + # Add task history if available + traces["task_history"] = getattr(self._marble_agent, "task_history", []) + + return traces + + def gather_config(self) -> Dict[str, Any]: + """Gather agent configuration. + + Returns: + Dict with agent configuration + """ + config = super().gather_config() + + config["agent_id"] = self._agent_id + config["profile"] = self._profile + config["strategy"] = getattr(self._marble_agent, "strategy", "default") + config["llm"] = getattr(self._marble_agent, "llm", "unknown") + + return config + + +def create_marble_agents( + agent_configs: Sequence[Dict[str, Any]], + marble_env: Any, + model: str, + callbacks: Optional[List["BenchmarkCallback"]] = None, +) -> Tuple[List[MarbleAgentAdapter], Dict[str, MarbleAgentAdapter]]: + """Create MarbleAgentAdapters from agent configurations. + + This is a factory function that creates MARBLE BaseAgent instances + and wraps them in MarbleAgentAdapters. + + Args: + agent_configs: List of agent configuration dicts from task data + marble_env: MARBLE environment instance + model: Model ID to use for agents + callbacks: Optional callbacks to attach + + Returns: + Tuple of (agents_list, agents_dict) + + Raises: + ImportError: If MARBLE is not available + """ + try: + from ..marble.agent.base_agent import BaseAgent + except ImportError as e: + raise ImportError(f"MARBLE is not available. Clone MARBLE to maseval/benchmark/multiagentbench/marble/\nOriginal error: {e}") from e + + agents_list: List[MarbleAgentAdapter] = [] + agents_dict: Dict[str, MarbleAgentAdapter] = {} + + for config in agent_configs: + agent_id = config.get("agent_id", f"agent_{len(agents_list)}") + + # Create MARBLE agent + marble_agent = BaseAgent(config=config, env=marble_env, model=model) + + # Wrap in adapter + adapter = MarbleAgentAdapter(marble_agent=marble_agent, agent_id=agent_id, callbacks=callbacks) + + agents_list.append(adapter) + agents_dict[agent_id] = adapter + + return agents_list, agents_dict diff --git a/maseval/benchmark/multiagentbench/data_loader.py b/maseval/benchmark/multiagentbench/data_loader.py new file mode 100644 index 00000000..dcbf5899 --- /dev/null +++ b/maseval/benchmark/multiagentbench/data_loader.py @@ -0,0 +1,312 @@ +"""Data loading utilities for MultiAgentBench tasks. + +This module provides functions for loading and configuring tasks from +MARBLE's MultiAgentBench JSONL files. +""" + +import json +import os +from pathlib import Path +from typing import Any, Dict, FrozenSet, List, Optional + +from maseval import Task + +# Valid domain names +VALID_DOMAINS: FrozenSet[str] = frozenset( + { + "coding", + "database", + "minecraft", + "research", + "bargaining", + "web", + "worldsimulation", + } +) + +# Domains requiring external infrastructure +INFRASTRUCTURE_DOMAINS: FrozenSet[str] = frozenset({"database", "minecraft"}) + + +def _resolve_data_dir(data_dir: Optional[Path] = None) -> Path: + """Resolve the MARBLE data directory. + + Searches for MARBLE data in the following order: + 1. Provided data_dir argument + 2. MARBLE_DATA_DIR environment variable + 3. marble/multiagentbench/ relative to this module + 4. Current working directory / marble/multiagentbench/ + + Args: + data_dir: Optional explicit data directory + + Returns: + Path to the MARBLE multiagentbench data directory + + Raises: + FileNotFoundError: If no valid data directory is found + """ + if data_dir is not None: + path = Path(data_dir) + if path.exists(): + return path + raise FileNotFoundError(f"Specified data_dir does not exist: {data_dir}") + + # Check environment variable + env_dir = os.environ.get("MARBLE_DATA_DIR") + + candidates = [] + if env_dir: + candidates.append(Path(env_dir)) + + # Relative to this module (vendored MARBLE) + candidates.append(Path(__file__).parent / "marble" / "multiagentbench") + + # Current working directory + candidates.append(Path.cwd() / "marble" / "multiagentbench") + + for candidate in candidates: + if candidate.exists(): + return candidate + + raise FileNotFoundError( + "MARBLE data directory not found. Either:\n" + "1. Clone MARBLE to maseval/benchmark/multiagentbench/marble/\n" + "2. Set MARBLE_DATA_DIR environment variable\n" + "See multiagentbench/README.md for setup instructions." + ) + + +def _parse_task_entry(entry: Dict[str, Any], domain: str, idx: int) -> Task: + """Parse a JSONL entry into a MASEval Task. + + Args: + entry: Raw JSONL entry dict + domain: Domain name + idx: Entry index (for error messages) + + Returns: + MASEval Task object + + Raises: + ValueError: If required fields are missing + """ + # Required fields - fail if missing + REQUIRED_FIELDS = ["scenario", "task_id", "task", "agents", "relationships"] + missing = [f for f in REQUIRED_FIELDS if f not in entry] + if missing: + raise ValueError(f"Task entry {idx} missing required fields: {missing}\nEntry keys: {list(entry.keys())}") + + # Validate agent specifications + for i, agent_spec in enumerate(entry["agents"]): + if "agent_id" not in agent_spec: + raise ValueError(f"Agent {i} in task {entry['task_id']} missing 'agent_id'\nAgent spec: {agent_spec}") + + # Extract task content + task_content = entry["task"] + if isinstance(task_content, dict): + query = task_content.get("content", "") + output_format = task_content.get("output_format", "") + else: + query = str(task_content) + output_format = "" + + if not query: + raise ValueError(f"Task {entry['task_id']} has empty query/content") + + # Extract environment config + env_config = entry.get("environment", {}) + + # Extract coordination mode + coordinate_mode = entry.get("coordinate_mode", "") + if not coordinate_mode: + # Default based on domain if not specified + coordinate_mode = "star" + + # Build Task object + return Task( + id=f"{domain}_{entry['task_id']}", + query=query, + environment_data={ + "scenario": entry["scenario"], + "coordinate_mode": coordinate_mode, + "relationships": entry["relationships"], + "environment": env_config, + "task": entry["task"], + "agents": entry["agents"], + "max_iterations": env_config.get("max_iterations") or 10, + "engine_planner": entry.get("engine_planner", {}), + "memory": entry.get("memory", {}), + "output": entry.get("output", {}), + # Store raw entry for MARBLE compatibility + "raw_marble_config": entry, + }, + evaluation_data={ + "metrics": entry.get("metrics", {}), + "output_format": output_format, + }, + metadata={ + "domain": domain, + "task_id": entry["task_id"], + "scenario": entry["scenario"], + }, + ) + + +def load_tasks( + domain: str, + data_dir: Optional[Path] = None, + limit: Optional[int] = None, +) -> List[Task]: + """Load MultiAgentBench tasks from JSONL files. + + Args: + domain: Domain name (one of: coding, database, minecraft, research, + bargaining, web, worldsimulation) + data_dir: Optional path to MARBLE data directory + limit: Maximum number of tasks to load (None for all) + + Returns: + List of Task objects + + Raises: + ValueError: If domain is invalid + FileNotFoundError: If data files not found + + Example: + >>> tasks = load_tasks("research", limit=5) + >>> len(tasks) + 5 + >>> tasks[0].metadata["domain"] + 'research' + """ + # Normalize and validate domain + domain_lower = domain.lower() + if domain_lower not in VALID_DOMAINS: + raise ValueError(f"Invalid domain '{domain}'. Must be one of: {sorted(VALID_DOMAINS)}") + + # Find data directory + resolved_data_dir = _resolve_data_dir(data_dir) + jsonl_path = resolved_data_dir / domain_lower / f"{domain_lower}_main.jsonl" + + if not jsonl_path.exists(): + raise FileNotFoundError( + f"Task data not found: {jsonl_path}\n" + f"Ensure MARBLE is cloned to multiagentbench/marble/\n" + f"See multiagentbench/README.md for setup instructions." + ) + + tasks = [] + with jsonl_path.open(encoding="utf-8") as f: + for idx, line in enumerate(f): + if limit is not None and idx >= limit: + break + + line = line.strip() + if not line: + continue + + entry = json.loads(line) + task = _parse_task_entry(entry, domain_lower, idx) + tasks.append(task) + + return tasks + + +def configure_model_ids( + tasks: List[Task], + *, + agent_model_id: str, + evaluator_model_id: Optional[str] = None, +) -> List[Task]: + """Configure model IDs for MARBLE agents and evaluator. + + Modifies tasks in-place to set the LLM model IDs used by agents + and optionally the evaluator. + + Args: + tasks: List of Tasks to configure + agent_model_id: Model ID for all MARBLE agents (e.g., "gpt-4o") + evaluator_model_id: Optional model ID for LLM-based evaluation + + Returns: + The input tasks (modified in-place) + + Example: + >>> tasks = load_tasks("research", limit=5) + >>> configure_model_ids(tasks, agent_model_id="gpt-4o") + >>> tasks[0].environment_data["llm"] + 'gpt-4o' + """ + for task in tasks: + # Set agent model + task.environment_data["llm"] = agent_model_id + + # Set evaluator model if provided + if evaluator_model_id: + task.evaluation_data["model_id"] = evaluator_model_id + else: + # Default to agent model for evaluation + task.evaluation_data["model_id"] = agent_model_id + + return tasks + + +def get_domain_info(domain: str) -> Dict[str, Any]: + """Get information about a domain. + + Args: + domain: Domain name + + Returns: + Dict with domain information including: + - requires_infrastructure: Whether external services needed + - description: Brief domain description + - coordination_mode: Default coordination mode + + Raises: + ValueError: If domain is invalid + """ + domain_lower = domain.lower() + if domain_lower not in VALID_DOMAINS: + raise ValueError(f"Invalid domain '{domain}'. Must be one of: {sorted(VALID_DOMAINS)}") + + domain_info = { + "coding": { + "requires_infrastructure": False, + "description": "Software development collaboration tasks", + "coordination_mode": "tree", + }, + "database": { + "requires_infrastructure": True, + "description": "Database manipulation and querying tasks (requires Docker)", + "coordination_mode": "star", + }, + "minecraft": { + "requires_infrastructure": True, + "description": "Collaborative building in Minecraft (requires game server)", + "coordination_mode": "cooperative", + }, + "research": { + "requires_infrastructure": False, + "description": "Research idea generation and collaboration", + "coordination_mode": "cooperative", + }, + "bargaining": { + "requires_infrastructure": False, + "description": "Negotiation and bargaining scenarios", + "coordination_mode": "cooperative", + }, + "web": { + "requires_infrastructure": False, + "description": "Web-based task completion", + "coordination_mode": "star", + }, + "worldsimulation": { + "requires_infrastructure": False, + "description": "World simulation and interaction tasks", + "coordination_mode": "cooperative", + }, + } + + return domain_info[domain_lower] diff --git a/maseval/benchmark/multiagentbench/environment.py b/maseval/benchmark/multiagentbench/environment.py new file mode 100644 index 00000000..aec52ec7 --- /dev/null +++ b/maseval/benchmark/multiagentbench/environment.py @@ -0,0 +1,363 @@ +"""MultiAgentBench Environment implementation. + +This module provides the MASEval Environment wrapper for MARBLE environments. +""" + +import shutil +from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional + +from maseval import Environment, EnvironmentError, ToolInvocationHistory + +if TYPE_CHECKING: + from maseval.core.callback import BenchmarkCallback + + +# Domains requiring external infrastructure +INFRASTRUCTURE_DOMAINS = frozenset({"database", "minecraft"}) + + +class MultiAgentBenchEnvironment(Environment): + """MASEval Environment wrapper for MARBLE environments. + + This environment wraps MARBLE's domain-specific environments (Research, + Bargaining, Coding, etc.) and exposes their tools through MASEval's + tracing infrastructure. + + Attributes: + domain: The domain name (e.g., "research", "bargaining") + marble_env: The underlying MARBLE environment instance + """ + + def __init__( + self, + task_data: Dict[str, Any], + callbacks: Optional[List["BenchmarkCallback"]] = None, + ): + """Initialize the environment. + + Args: + task_data: Task data containing environment configuration + callbacks: Optional list of callbacks + + Raises: + EnvironmentError: If required infrastructure is unavailable + """ + self.domain = task_data.get("scenario", "") + self._marble_env: Optional[Any] = None + self._tool_histories: Dict[str, ToolInvocationHistory] = {} + super().__init__(task_data, callbacks) + + def setup_state(self, task_data: Dict[str, Any]) -> Dict[str, Any]: + """Initialize state and optionally create MARBLE environment. + + Args: + task_data: Task data containing environment configuration + + Returns: + Initial state dictionary + + Raises: + EnvironmentError: If required infrastructure is unavailable + """ + domain = task_data.get("scenario", "") + env_config = task_data.get("environment", {}) + task_config = task_data.get("task", {}) + + # Check infrastructure requirements + if domain.lower() in INFRASTRUCTURE_DOMAINS: + if not self._check_infrastructure(domain): + raise EnvironmentError( + f"Domain '{domain}' requires external infrastructure. See README.md for setup instructions.", + component="MultiAgentBenchEnvironment", + ) + + # Build config for MARBLE environment + marble_config = { + "description": env_config.get("description", f"{domain} environment"), + "task_description": task_config.get("content", "") if isinstance(task_config, dict) else str(task_config), + "ground_truth": env_config.get("ground_truth", ""), + "max_iterations": env_config.get("max_iterations") or task_data.get("max_iterations", 10), + } + + # Try to create MARBLE environment (may fail if MARBLE not available) + try: + self._marble_env = self._create_marble_environment(domain, marble_config) + except ImportError: + # MARBLE not available - store config for later use + self._marble_env = None + + return { + "domain": domain, + "env_config": env_config, + "task_config": task_config, + "marble_env_type": type(self._marble_env).__name__ if self._marble_env else "None", + "max_iterations": marble_config["max_iterations"], + } + + def _check_infrastructure(self, domain: str) -> bool: + """Check if required infrastructure is available. + + Args: + domain: Domain name + + Returns: + True if infrastructure is available, False otherwise + """ + domain_lower = domain.lower() + + if domain_lower == "database": + # Check Docker availability + return shutil.which("docker") is not None + + if domain_lower == "minecraft": + # Minecraft requires external server - always fail for now + return False + + return True + + def _create_marble_environment( + self, + domain: str, + config: Dict[str, Any], + ) -> Any: + """Create the appropriate MARBLE environment. + + Args: + domain: Domain name + config: Environment configuration + + Returns: + MARBLE environment instance + + Raises: + ImportError: If MARBLE is not available + """ + domain_lower = domain.lower() + env_name = config.get("name", domain_lower) + + # Import MARBLE environments + try: + from .marble.environments.base_env import BaseEnvironment + except ImportError as e: + raise ImportError(f"MARBLE is not available. Clone MARBLE to maseval/benchmark/multiagentbench/marble/\nOriginal error: {e}") from e + + # Map domains to environment classes + env_mapping: Dict[str, str] = { + "coding": "marble.environments.coding_env.CodingEnvironment", + "database": "marble.environments.db_env.DBEnvironment", + "research": "marble.environments.research_env.ResearchEnvironment", + "bargaining": "marble.environments.bargaining_env.BargainingEnvironment", + "web": "marble.environments.web_env.WebEnvironment", + "worldsimulation": "marble.environments.world_env.WorldSimulationEnvironment", + "minecraft": "marble.environments.minecraft_env.MinecraftEnvironment", + } + + env_class_path = env_mapping.get(domain_lower) + + if env_class_path is None: + # Use base environment for unknown domains + return BaseEnvironment(env_name, config) + + try: + # Dynamic import of domain-specific environment + module_path, class_name = env_class_path.rsplit(".", 1) + # Adjust import path for vendored MARBLE + module_path = module_path.replace("marble.", ".marble.", 1) + module = __import__(module_path, globals(), locals(), [class_name], 1) + env_class = getattr(module, class_name) + return env_class(env_name, config) + except (ImportError, AttributeError): + # Fall back to base environment + return BaseEnvironment(env_name, config) + + def create_tools(self) -> Dict[str, Callable]: + """Create tools from MARBLE environment for MASEval tracing. + + MARBLE environments expose tools via action_handler_descriptions. + This method wraps them for MASEval's tracing infrastructure. + + Returns: + Dict mapping tool names to wrapped callables + """ + if self._marble_env is None: + return {} + + tools: Dict[str, Callable] = {} + + # Get action handlers from MARBLE environment + action_handlers = getattr(self._marble_env, "_action_handlers", {}) + + for action_name, handler in action_handlers.items(): + # Create tool history for this action + history = ToolInvocationHistory() + self._tool_histories[action_name] = history + + # Wrap handler for tracing + wrapped = self._wrap_tool_for_tracing(action_name, handler, history) + tools[action_name] = wrapped + + return tools + + def _wrap_tool_for_tracing( + self, + name: str, + handler: Callable, + history: ToolInvocationHistory, + ) -> Callable: + """Wrap a MARBLE action handler for MASEval tracing. + + Args: + name: Tool name + handler: Original handler callable + history: ToolInvocationHistory to record invocations + + Returns: + Wrapped callable that records invocations + """ + + def traced_handler(**kwargs: Any) -> Any: + try: + result = handler(**kwargs) + history.add_invocation( + inputs=kwargs, + outputs=result, + status="success", + ) + return result + except Exception as e: + history.add_invocation( + inputs=kwargs, + outputs=str(e), + status="error", + ) + raise + + # Attach metadata for inspection + traced_handler._original_name = name # type: ignore[attr-defined] + traced_handler._history = history # type: ignore[attr-defined] + + return traced_handler + + def get_tool(self, name: str) -> Optional[Callable]: + """Get a specific tool by name. + + Args: + name: Tool name + + Returns: + Tool callable if found, None otherwise + """ + return self.tools.get(name) + + def get_tool_descriptions(self) -> Dict[str, Any]: + """Get tool descriptions in OpenAI function format. + + Returns: + Dict mapping tool names to their OpenAI-format descriptions + """ + if self._marble_env is None: + return {} + + return getattr(self._marble_env, "action_handler_descriptions", {}) + + def apply_action( + self, + agent_id: Optional[str], + action_name: str, + arguments: Dict[str, Any], + ) -> Dict[str, Any]: + """Execute an action in the MARBLE environment. + + Args: + agent_id: ID of the agent performing the action + action_name: Name of the action to execute + arguments: Arguments for the action + + Returns: + Action result dictionary + + Raises: + EnvironmentError: If MARBLE environment is not available + """ + if self._marble_env is None: + raise EnvironmentError( + "MARBLE environment not available. Cannot execute actions.", + component="MultiAgentBenchEnvironment", + ) + + return self._marble_env.apply_action(agent_id, action_name, arguments) + + def is_done(self) -> bool: + """Check if the environment has reached a terminal state. + + Returns: + True if done, False otherwise + """ + if self._marble_env is None: + return False + return self._marble_env.is_done() + + def is_task_completed(self) -> bool: + """Check if the task has been completed successfully. + + Returns: + True if task completed, False otherwise + """ + if self._marble_env is None: + return False + return self._marble_env.is_task_completed() + + def get_marble_state(self) -> Dict[str, Any]: + """Get the current MARBLE environment state. + + Returns: + State dictionary from MARBLE environment + """ + if self._marble_env is None: + return {} + return self._marble_env.get_state() + + def gather_traces(self) -> Dict[str, Any]: + """Gather traces including tool invocations. + + Returns: + Dict with environment traces + """ + traces = super().gather_traces() + + # Add domain-specific info + traces["domain"] = self.domain + traces["marble_env_type"] = type(self._marble_env).__name__ if self._marble_env else "None" + + # Add MARBLE state if available + if self._marble_env is not None: + traces["marble_state"] = self.get_marble_state() + traces["is_done"] = self.is_done() + traces["is_task_completed"] = self.is_task_completed() + + # Collect tool invocation histories + tool_traces = {} + for name, history in self._tool_histories.items(): + tool_traces[name] = { + "invocations": history.to_list(), + "invocation_count": len(history), + } + traces["tool_invocations"] = tool_traces + + return traces + + def gather_config(self) -> Dict[str, Any]: + """Gather environment configuration. + + Returns: + Dict with environment configuration + """ + config = super().gather_config() + + config["domain"] = self.domain + config["marble_env_type"] = type(self._marble_env).__name__ if self._marble_env else "None" + + # Add tool descriptions + config["tool_descriptions"] = self.get_tool_descriptions() + + return config diff --git a/maseval/benchmark/multiagentbench/evaluator.py b/maseval/benchmark/multiagentbench/evaluator.py new file mode 100644 index 00000000..61dd6039 --- /dev/null +++ b/maseval/benchmark/multiagentbench/evaluator.py @@ -0,0 +1,552 @@ +"""MultiAgentBench evaluator implementation. + +This module provides evaluation metrics matching MARBLE's evaluation methodology. +""" + +import json +import re +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional + +from maseval import Evaluator, ModelAdapter + + +@dataclass +class MultiAgentBenchMetrics: + """Metrics collected during MultiAgentBench evaluation. + + Attributes: + task_completion: Whether the task was completed + token_consumption: Total tokens used + planning_score: Score for planning/coordination (1-5) + communication_score: Score for inter-agent communication (1-5) + task_evaluation: Domain-specific evaluation results + agent_kpis: Per-agent key performance indicators + total_milestones: Number of milestones achieved + """ + + task_completion: bool = False + token_consumption: int = 0 + planning_score: float = -1.0 + communication_score: float = -1.0 + task_evaluation: Dict[str, Any] = field(default_factory=dict) + agent_kpis: Dict[str, int] = field(default_factory=dict) + total_milestones: int = 0 + + def to_dict(self) -> Dict[str, Any]: + """Convert metrics to dictionary.""" + return { + "task_completion": self.task_completion, + "token_consumption": self.token_consumption, + "planning_score": self.planning_score, + "communication_score": self.communication_score, + "task_evaluation": self.task_evaluation, + "agent_kpis": self.agent_kpis, + "total_milestones": self.total_milestones, + } + + +class MultiAgentBenchEvaluator(Evaluator): + """Evaluator for MultiAgentBench tasks matching MARBLE's methodology. + + This evaluator implements MARBLE's LLM-based evaluation metrics: + - Task completion assessment + - Communication quality scoring + - Planning/coordination scoring + - Domain-specific task evaluation (research, bargaining, etc.) + + Attributes: + domain: The benchmark domain (research, bargaining, etc.) + model_adapter: Model adapter for LLM-based evaluation + metrics_config: Configuration for metrics to evaluate + """ + + def __init__( + self, + domain: str, + model_adapter: ModelAdapter, + metrics_config: Optional[Dict[str, Any]] = None, + output_format: str = "", + ): + """Initialize the evaluator. + + Args: + domain: Benchmark domain (research, bargaining, etc.) + model_adapter: Model adapter for LLM evaluation + metrics_config: Configuration for evaluation metrics + output_format: Expected output format for task evaluation + """ + self.domain = domain.lower() + self.model_adapter = model_adapter + self.metrics_config = metrics_config or {} + self.output_format = output_format + self._evaluation_prompts = self._load_evaluation_prompts() + + def _load_evaluation_prompts(self) -> Dict[str, Any]: + """Load evaluation prompts (matching MARBLE's evaluator_prompts.json).""" + # These prompts mirror MARBLE's evaluation methodology + return { + "communication": { + "prompt": """Evaluate the communication between agents for the following task. + +Task: {task} + +Communication Log: +{communications} + +Rate the communication quality on a scale of 1-5: +1 - Poor: Irrelevant or confusing communication +2 - Below Average: Some relevant communication but lacks clarity +3 - Average: Adequate communication that addresses the task +4 - Good: Clear and relevant communication with good coordination +5 - Excellent: Highly effective communication with perfect coordination + +Respond with a JSON object: {{"rating": }}""" + }, + "planning": { + "prompt": """Evaluate the planning and coordination quality for the following task. + +Task Summary: {summary} +Agent Profiles: {agent_profiles} +Task Assignments: {agent_tasks} +Results: {results} + +Rate the planning quality on a scale of 1-5: +1 - Poor: No clear plan or coordination +2 - Below Average: Some planning but poorly coordinated +3 - Average: Adequate planning with basic coordination +4 - Good: Well-planned with effective coordination +5 - Excellent: Optimal planning and seamless coordination + +Respond with a JSON object: {{"rating": }}""" + }, + "research": { + "task_evaluation": { + "prompt": """Evaluate the following research idea based on innovation, safety, and feasibility. + +Task: {task} + +Research Result: +{result} + +Rate each dimension on a scale of 1-5: +- Innovation: How novel and creative is the research idea? +- Safety: Does the research consider ethical implications and safety? +- Feasibility: How practical and achievable is the proposed research? + +Respond with a JSON object: +{{"innovation": , "safety": , "feasibility": }}""" + } + }, + "bargaining": { + "task_evaluation": { + "buyer_prompt": """Evaluate the buyer's negotiation performance. + +Task: {task} + +Negotiation Result: +{result} + +Rate each dimension on a scale of 1-5: +- Effectiveness of Strategies: How well did the buyer negotiate? +- Progress and Outcome: Did the buyer achieve a favorable outcome? +- Interaction Dynamics: How well did the buyer engage with the seller? + +Respond with a JSON object: +{{"effectiveness_of_strategies": , "progress_and_outcome": , "interaction_dynamics": }}""", + "seller_prompt": """Evaluate the seller's negotiation performance. + +Task: {task} + +Negotiation Result: +{result} + +Rate each dimension on a scale of 1-5: +- Effectiveness of Strategies: How well did the seller negotiate? +- Progress and Outcome: Did the seller achieve a favorable outcome? +- Interaction Dynamics: How well did the seller engage with the buyer? + +Respond with a JSON object: +{{"effectiveness_of_strategies": , "progress_and_outcome": , "interaction_dynamics": }}""", + } + }, + "coding": { + "task_evaluation": { + "prompt": """Evaluate the code quality based on the following criteria. + +Task Description: {task_description} +Implementation Requirements: {requirements} + +Solution: +{solution} + +Rate each dimension on a scale of 1-5: +- Instruction Following: Does the code fulfill all requirements? +- Executability: Is the code syntactically correct and executable? +- Consistency: Is the code consistent in naming, formatting, and logic? +- Quality: Is the code well-documented, clear, and modular? + +Respond with a JSON object: +{{"instruction_following": , "executability": , "consistency": , "quality": }}""" + } + }, + } + + def filter_traces(self, traces: Dict[str, Any]) -> Dict[str, Any]: + """Filter traces for evaluation. + + Args: + traces: All collected traces + + Returns: + Filtered traces relevant for evaluation + """ + return { + "agents": traces.get("agents", {}), + "environment": traces.get("environment", {}), + "communications": self._extract_communications(traces), + "results": self._extract_results(traces), + } + + def _extract_communications(self, traces: Dict[str, Any]) -> str: + """Extract communication logs from traces. + + Args: + traces: Execution traces + + Returns: + Formatted communication string + """ + communications: List[str] = [] + + # Extract from agent traces + agent_traces = traces.get("agents", {}) + for agent_id, agent_trace in agent_traces.items(): + comm_log = agent_trace.get("communication_log", []) + for entry in comm_log: + comm = entry.get("communication", "") + if comm: + communications.append(f"[{agent_id}]: {comm}") + + return "\n".join(communications) if communications else "No communications recorded." + + def _extract_results(self, traces: Dict[str, Any]) -> str: + """Extract agent results from traces. + + Args: + traces: Execution traces + + Returns: + Formatted results string + """ + results: List[str] = [] + + agent_traces = traces.get("agents", {}) + for agent_id, agent_trace in agent_traces.items(): + action_log = agent_trace.get("action_log", []) + for entry in action_log: + result = entry.get("result", "") + if result: + results.append(f"[{agent_id}]: {result}") + + return "\n".join(results) if results else "No results recorded." + + def __call__(self, traces: Dict[str, Any], final_answer: Any) -> Dict[str, Any]: + """Evaluate the task execution. + + Args: + traces: Filtered execution traces + final_answer: Final output from agents + + Returns: + Evaluation results dictionary + """ + metrics = MultiAgentBenchMetrics() + + # Extract filtered data + filtered = self.filter_traces(traces) + communications = filtered["communications"] + # Note: results are available in filtered["results"] but not used directly + # since we get task_desc and final_result separately + + # Calculate token consumption + metrics.token_consumption = self._calculate_token_consumption(traces) + + # Evaluate communication if present + if communications != "No communications recorded.": + metrics.communication_score = self._evaluate_communication(self._get_task_description(traces), communications) + + # Domain-specific evaluation + task_desc = self._get_task_description(traces) + final_result = self._format_final_answer(final_answer) + + if self.domain == "research": + metrics.task_evaluation = self._evaluate_research(task_desc, final_result) + elif self.domain == "bargaining" or self.domain == "worldsimulation": + metrics.task_evaluation = self._evaluate_bargaining(task_desc, final_result) + elif self.domain == "coding": + metrics.task_evaluation = self._evaluate_coding(task_desc, final_result) + elif self.domain == "database": + metrics.task_evaluation = self._evaluate_database(task_desc, final_result) + else: + # Default: check if task has a completion marker + metrics.task_completion = bool(final_result) + + # Set task completion based on evaluation + metrics.task_completion = self._determine_completion(metrics) + + return { + "passed": metrics.task_completion, + "metrics": metrics.to_dict(), + "domain": self.domain, + } + + def _get_task_description(self, traces: Dict[str, Any]) -> str: + """Get task description from traces.""" + env_traces = traces.get("environment", {}) + state = env_traces.get("marble_state", {}) + return state.get("task_description", "") + + def _format_final_answer(self, final_answer: Any) -> str: + """Format final answer for evaluation.""" + if isinstance(final_answer, dict): + # Handle structured output from run_agents + results = final_answer.get("agent_results", []) + if results: + return "\n".join(f"[{r.get('agent_id', 'unknown')}]: {r.get('result', '')}" for r in results) + return json.dumps(final_answer) + elif isinstance(final_answer, list): + return "\n".join(f"[{r.get('agent_id', 'unknown')}]: {r.get('result', '')}" for r in final_answer if isinstance(r, dict)) + return str(final_answer) if final_answer else "" + + def _calculate_token_consumption(self, traces: Dict[str, Any]) -> int: + """Calculate total token consumption.""" + total = 0 + + agent_traces = traces.get("agents", {}) + for agent_trace in agent_traces.values(): + token_usage = agent_trace.get("token_usage", 0) + if isinstance(token_usage, int): + total += token_usage + + return total + + def _evaluate_communication(self, task: str, communications: str) -> float: + """Evaluate communication quality using LLM.""" + prompt_template = self._evaluation_prompts["communication"]["prompt"] + prompt = prompt_template.format(task=task, communications=communications) + + try: + response = self.model_adapter.generate(prompt) + return self._parse_score(response) + except Exception: + return -1.0 + + def _evaluate_research(self, task: str, result: str) -> Dict[str, Any]: + """Evaluate research task output.""" + prompt_template = self._evaluation_prompts["research"]["task_evaluation"]["prompt"] + prompt = prompt_template.format(task=task, result=result) + + try: + response = self.model_adapter.generate(prompt) + return self._parse_research_ratings(response) + except Exception: + return {"innovation": -1, "safety": -1, "feasibility": -1} + + def _evaluate_bargaining(self, task: str, result: str) -> Dict[str, Any]: + """Evaluate bargaining/world simulation task output.""" + # Evaluate both buyer and seller perspectives + buyer_prompt = self._evaluation_prompts["bargaining"]["task_evaluation"]["buyer_prompt"] + seller_prompt = self._evaluation_prompts["bargaining"]["task_evaluation"]["seller_prompt"] + + ratings = {"buyer": {}, "seller": {}} + + try: + buyer_response = self.model_adapter.generate(buyer_prompt.format(task=task, result=result)) + ratings["buyer"] = self._parse_bargaining_ratings(buyer_response) + except Exception: + ratings["buyer"] = { + "effectiveness_of_strategies": -1, + "progress_and_outcome": -1, + "interaction_dynamics": -1, + } + + try: + seller_response = self.model_adapter.generate(seller_prompt.format(task=task, result=result)) + ratings["seller"] = self._parse_bargaining_ratings(seller_response) + except Exception: + ratings["seller"] = { + "effectiveness_of_strategies": -1, + "progress_and_outcome": -1, + "interaction_dynamics": -1, + } + + return ratings + + def _evaluate_coding(self, task: str, result: str) -> Dict[str, Any]: + """Evaluate coding task output.""" + prompt_template = self._evaluation_prompts["coding"]["task_evaluation"]["prompt"] + + # For coding, we need requirements and solution separately + # If not available, use task as description and result as solution + prompt = prompt_template.format( + task_description=task, + requirements="See task description", + solution=result, + ) + + try: + response = self.model_adapter.generate(prompt) + return self._parse_coding_ratings(response) + except Exception: + return { + "instruction_following": -1, + "executability": -1, + "consistency": -1, + "quality": -1, + } + + def _evaluate_database(self, task: str, result: str) -> Dict[str, Any]: + """Evaluate database task output. + + Database tasks have ground truth labels that would be compared + separately. Here we just store the prediction. + """ + return { + "predicted": result, + "root_cause": [], # Would be filled from task data + } + + def _parse_score(self, response: str) -> float: + """Parse a single score from LLM response.""" + try: + content = response.strip() + + # Remove markdown code block markers + if content.startswith("```json"): + content = content[7:] + if content.startswith("```"): + content = content[3:] + if content.endswith("```"): + content = content[:-3] + content = content.strip() + + # Find JSON object + json_start = content.find("{") + json_end = content.rfind("}") + 1 + + if json_start >= 0 and json_end > json_start: + json_str = content[json_start:json_end] + rating_data = json.loads(json_str) + if isinstance(rating_data, dict) and "rating" in rating_data: + score = int(rating_data["rating"]) + if 1 <= score <= 5: + return float(score) + + # Fallback: find a single digit 1-5 + numbers = re.findall(r"\b[1-5]\b", content) + if numbers: + return float(int(numbers[0])) + + return 3.0 # Default middle score + + except Exception: + return 3.0 + + def _parse_research_ratings(self, response: str) -> Dict[str, int]: + """Parse research evaluation ratings.""" + try: + content = response.strip() + json_start = content.find("{") + json_end = content.rfind("}") + 1 + + if json_start >= 0 and json_end > json_start: + json_str = content[json_start:json_end] + ratings = json.loads(json_str) + return {k: int(v) for k, v in ratings.items()} + except Exception: + pass + + return {"innovation": -1, "safety": -1, "feasibility": -1} + + def _parse_bargaining_ratings(self, response: str) -> Dict[str, int]: + """Parse bargaining evaluation ratings.""" + try: + content = response.strip() + json_start = content.find("{") + json_end = content.rfind("}") + 1 + + if json_start >= 0 and json_end > json_start: + json_str = content[json_start:json_end] + ratings = json.loads(json_str) + return { + "effectiveness_of_strategies": int(ratings.get("effectiveness_of_strategies", -1)), + "progress_and_outcome": int(ratings.get("progress_and_outcome", -1)), + "interaction_dynamics": int(ratings.get("interaction_dynamics", -1)), + } + except Exception: + pass + + return { + "effectiveness_of_strategies": -1, + "progress_and_outcome": -1, + "interaction_dynamics": -1, + } + + def _parse_coding_ratings(self, response: str) -> Dict[str, int]: + """Parse coding evaluation ratings.""" + try: + content = response.strip() + json_start = content.find("{") + json_end = content.rfind("}") + 1 + + if json_start >= 0 and json_end > json_start: + json_str = content[json_start:json_end] + ratings = json.loads(json_str) + return { + "instruction_following": int(ratings.get("instruction_following", -1)), + "executability": int(ratings.get("executability", -1)), + "consistency": int(ratings.get("consistency", -1)), + "quality": int(ratings.get("quality", -1)), + } + except Exception: + pass + + return { + "instruction_following": -1, + "executability": -1, + "consistency": -1, + "quality": -1, + } + + def _determine_completion(self, metrics: MultiAgentBenchMetrics) -> bool: + """Determine if task was completed based on metrics.""" + eval_data = metrics.task_evaluation + + if not eval_data: + return False + + if self.domain == "research": + # Consider completed if all scores are positive + scores = [eval_data.get(k, -1) for k in ["innovation", "safety", "feasibility"]] + return all(s > 0 for s in scores) + + elif self.domain in ("bargaining", "worldsimulation"): + # Check both buyer and seller have positive scores + buyer = eval_data.get("buyer", {}) + seller = eval_data.get("seller", {}) + buyer_scores = [buyer.get(k, -1) for k in ["effectiveness_of_strategies", "progress_and_outcome", "interaction_dynamics"]] + seller_scores = [seller.get(k, -1) for k in ["effectiveness_of_strategies", "progress_and_outcome", "interaction_dynamics"]] + return all(s > 0 for s in buyer_scores) and all(s > 0 for s in seller_scores) + + elif self.domain == "coding": + # All coding metrics should be positive + scores = [eval_data.get(k, -1) for k in ["instruction_following", "executability", "consistency", "quality"]] + return all(s > 0 for s in scores) + + elif self.domain == "database": + # Database completion is determined by comparing prediction to labels + # This requires external validation + return bool(eval_data.get("predicted")) + + return False diff --git a/maseval/benchmark/multiagentbench/multiagentbench.py b/maseval/benchmark/multiagentbench/multiagentbench.py new file mode 100644 index 00000000..c15a11c9 --- /dev/null +++ b/maseval/benchmark/multiagentbench/multiagentbench.py @@ -0,0 +1,481 @@ +"""MultiAgentBench benchmark implementations. + +This module provides benchmark classes for the MARBLE MultiAgentBench suite: +- MultiAgentBenchBenchmark: Abstract base for framework-agnostic evaluation +- MarbleMultiAgentBenchBenchmark: Exact MARBLE reproduction mode +""" + +from abc import abstractmethod +from typing import Any, Dict, List, Optional, Sequence, Tuple + +from maseval import ( + AgentAdapter, + Benchmark, + Environment, + Evaluator, + ModelAdapter, + Task, + User, +) +from maseval.core.callback import BenchmarkCallback + +from maseval.benchmark.multiagentbench.environment import MultiAgentBenchEnvironment +from maseval.benchmark.multiagentbench.evaluator import ( + MultiAgentBenchEvaluator, +) + + +class MultiAgentBenchBenchmark(Benchmark): + """Abstract base class for framework-agnostic MultiAgentBench evaluation. + + This benchmark provides the infrastructure for evaluating multi-agent systems + on MARBLE's MultiAgentBench tasks. Subclasses implement `setup_agents()` with + their specific agent framework. + + The benchmark supports: + - Multiple coordination modes (star, cooperative, tree, hierarchical) + - Multiple domains (research, bargaining, coding, database, etc.) + - LLM-based evaluation matching MARBLE's metrics + - Comprehensive tracing of agent interactions + + Example: + ```python + class MyMultiAgentBenchmark(MultiAgentBenchBenchmark): + def setup_agents(self, agent_data, environment, task, user): + # Create agents using your framework + ... + + def get_model_adapter(self, model_id, **kwargs): + adapter = MyModelAdapter(model_id) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter + + benchmark = MyMultiAgentBenchmark() + results = benchmark.run(tasks, agent_data={}) + ``` + """ + + def __init__( + self, + callbacks: Optional[List[BenchmarkCallback]] = None, + n_task_repeats: int = 1, + max_invocations: int = 10, + num_workers: int = 1, + fail_on_setup_error: bool = False, + fail_on_task_error: bool = False, + fail_on_evaluation_error: bool = False, + progress_bar: bool | str = True, + ): + """Initialize the benchmark. + + Args: + callbacks: Optional list of callbacks + n_task_repeats: Number of times to repeat each task + max_invocations: Maximum agent invocations per task + num_workers: Number of parallel workers + fail_on_setup_error: Raise on setup errors + fail_on_task_error: Raise on task errors + fail_on_evaluation_error: Raise on evaluation errors + progress_bar: Progress bar configuration + """ + super().__init__( + callbacks=callbacks, + n_task_repeats=n_task_repeats, + max_invocations=max_invocations, + num_workers=num_workers, + fail_on_setup_error=fail_on_setup_error, + fail_on_task_error=fail_on_task_error, + fail_on_evaluation_error=fail_on_evaluation_error, + progress_bar=progress_bar, + ) + + def setup_environment(self, agent_data: Dict[str, Any], task: Task) -> Environment: + """Create the MultiAgentBench environment. + + Args: + agent_data: Agent configuration + task: The task to set up + + Returns: + MultiAgentBenchEnvironment instance + """ + return MultiAgentBenchEnvironment( + task_data=task.environment_data, + callbacks=self.callbacks, + ) + + def setup_user( + self, + agent_data: Dict[str, Any], + environment: Environment, + task: Task, + ) -> Optional[User]: + """MultiAgentBench tasks don't use user simulators. + + The multi-agent coordination replaces user interaction. + + Returns: + None + """ + return None + + @abstractmethod + def setup_agents( + self, + agent_data: Dict[str, Any], + environment: Environment, + task: Task, + user: Optional[User], + ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: + """Create agents for the task (implement in subclass). + + Subclasses should: + 1. Read agent specifications from task.environment_data["agents"] + 2. Create agents using their framework + 3. Wrap them in AgentAdapter + 4. Set up relationships from task.environment_data["relationships"] + + Args: + agent_data: Agent configuration (model IDs, etc.) + environment: The environment instance + task: The task containing agent specs + user: User simulator (None for MultiAgentBench) + + Returns: + Tuple of (agents_to_run, agents_dict) + """ + pass + + def setup_evaluators( + self, + environment: Environment, + task: Task, + agents: Sequence[AgentAdapter], + user: Optional[User], + ) -> Sequence[Evaluator]: + """Create evaluators for the task. + + Args: + environment: The environment + task: The task with evaluation data + agents: The agents + user: User simulator (None for MultiAgentBench) + + Returns: + List of evaluators + """ + # Get evaluation model ID from task or default + eval_model_id = task.evaluation_data.get("model_id", "gpt-4o-mini") + + # Create model adapter for evaluation + model_adapter = self.get_model_adapter( + eval_model_id, + register_name="evaluator_model", + ) + + # Get domain-specific evaluation configuration + domain = task.environment_data.get("scenario", "") + metrics_config = task.evaluation_data.get("metrics", {}) + output_format = task.evaluation_data.get("output_format", "") + + return [ + MultiAgentBenchEvaluator( + domain=domain, + model_adapter=model_adapter, + metrics_config=metrics_config, + output_format=output_format, + ) + ] + + @abstractmethod + def get_model_adapter(self, model_id: str, **kwargs: Any) -> ModelAdapter: + """Provide a model adapter (implement in subclass). + + Args: + model_id: Model identifier + **kwargs: Additional arguments including register_name + + Returns: + ModelAdapter instance + """ + pass + + def run_agents( + self, + agents: Sequence[AgentAdapter], + task: Task, + environment: Environment, + query: str, + ) -> Any: + """Execute the multi-agent system. + + For MultiAgentBench, this runs all agents on the task and + collects their outputs. + + Args: + agents: Agents to run + task: The task + environment: The environment + query: The query/task content + + Returns: + Combined agent outputs + """ + results: List[Dict[str, Any]] = [] + + for agent in agents: + result = agent.run(query) + results.append( + { + "agent_id": getattr(agent, "agent_id", str(agent)), + "result": result, + } + ) + + return results + + def evaluate( + self, + evaluators: Sequence[Evaluator], + agents: Dict[str, AgentAdapter], + final_answer: Any, + traces: Dict[str, Any], + ) -> List[Dict[str, Any]]: + """Execute evaluators on the results. + + Args: + evaluators: The evaluators + agents: Dict of all agents + final_answer: The combined agent outputs + traces: Execution traces + + Returns: + List of evaluation results + """ + results = [] + + for evaluator in evaluators: + # MultiAgentBenchEvaluator expects traces in a specific format + result = evaluator(traces, final_answer) + results.append(result) + + return results + + +class MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark): + """MARBLE reproduction mode for MultiAgentBench. + + This benchmark uses MARBLE's native agents and engine for exact + reproduction of published results. It wraps MARBLE components + in MASEval adapters for unified tracing. + + Example: + ```python + from maseval.benchmark.multiagentbench import ( + MarbleMultiAgentBenchBenchmark, + load_tasks, + configure_model_ids, + ) + + class MyMarbleBenchmark(MarbleMultiAgentBenchBenchmark): + def get_model_adapter(self, model_id, **kwargs): + from maseval.interface.openai import OpenAIModelAdapter + adapter = OpenAIModelAdapter(model_id) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter + + tasks = load_tasks("research", limit=5) + configure_model_ids(tasks, agent_model_id="gpt-4o") + + benchmark = MyMarbleBenchmark() + results = benchmark.run(tasks, agent_data={}) + ``` + """ + + def setup_agents( + self, + agent_data: Dict[str, Any], + environment: Environment, + task: Task, + user: Optional[User], + ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: + """Create MARBLE agents wrapped in MASEval adapters. + + Args: + agent_data: Agent configuration + environment: The environment + task: The task with agent specifications + user: User simulator (None) + + Returns: + Tuple of (agents_to_run, agents_dict) + """ + from maseval.benchmark.multiagentbench.adapters.marble_adapter import ( + create_marble_agents, + ) + + # Get agent configurations from task + agent_configs = task.environment_data.get("agents", []) + model_id = task.environment_data.get("llm", "gpt-4o-mini") + + # Get MARBLE environment from our wrapper + marble_env = None + if isinstance(environment, MultiAgentBenchEnvironment): + marble_env = environment._marble_env + + # Create MARBLE environment if not available + if marble_env is None: + marble_env = self._create_marble_env(task) + + # Create agents using factory function + agents_list, agents_dict = create_marble_agents( + agent_configs=agent_configs, + marble_env=marble_env, + model=model_id, + callbacks=self.callbacks, + ) + + # Set up agent graph for inter-agent communication + self._setup_agent_graph(agents_dict, task, marble_env) + + # Register agents for tracing + for agent_id, adapter in agents_dict.items(): + self.register("agents", agent_id, adapter) + + return agents_list, agents_dict + + def _create_marble_env(self, task: Task) -> Any: + """Create a MARBLE environment for the task. + + Args: + task: The task with environment configuration + + Returns: + MARBLE environment instance + """ + try: + from .marble.environments.base_env import BaseEnvironment + except ImportError as e: + raise ImportError(f"MARBLE is not available. Clone MARBLE to maseval/benchmark/multiagentbench/marble/\nOriginal error: {e}") from e + + env_config = task.environment_data.get("environment", {}) + task_config = task.environment_data.get("task", {}) + + config = { + "description": f"{task.environment_data.get('scenario', '')} environment", + "task_description": task_config.get("content", "") if isinstance(task_config, dict) else str(task_config), + "max_iterations": env_config.get("max_iterations") or task.environment_data.get("max_iterations", 10), + } + + return BaseEnvironment(name=config["description"], config=config) + + def _setup_agent_graph( + self, + agents_dict: Dict[str, AgentAdapter], + task: Task, + marble_env: Any, + ) -> None: + """Set up MARBLE's AgentGraph for inter-agent communication. + + Args: + agents_dict: Dict of agent adapters + task: Task with relationship data + marble_env: MARBLE environment + """ + try: + from .marble.graph.agent_graph import AgentGraph + except ImportError: + # MARBLE not available, skip graph setup + return + + # Extract MARBLE agents from adapters + marble_agents = [adapter.marble_agent for adapter in agents_dict.values()] # type: ignore[attr-defined] + + # Build config for AgentGraph + relationships = task.environment_data.get("relationships", []) + coordination_mode = task.environment_data.get("coordinate_mode", "cooperative") + + # Create a minimal Config object + class MinimalConfig: + def __init__(self) -> None: + self.coordination_mode = coordination_mode + self.relationships = relationships + + config = MinimalConfig() + + try: + # Create agent graph + graph = AgentGraph(marble_agents, config) # type: ignore + + # Set graph on all agents + for agent in marble_agents: + agent.set_agent_graph(graph) + + except Exception: + # Graph creation failed, agents will work without inter-agent communication + pass + + def run_agents( + self, + agents: Sequence[AgentAdapter], + task: Task, + environment: Environment, + query: str, + ) -> Any: + """Execute agents using MARBLE's coordination patterns. + + Args: + agents: Agents to run (MarbleAgentAdapters) + task: The task + environment: The environment + query: The task query + + Returns: + Combined agent outputs with communication logs + """ + results: List[Dict[str, Any]] = [] + communications: List[str] = [] + + # Get coordination mode + coordination_mode = task.environment_data.get("coordinate_mode", "cooperative") + + # Simple execution: each agent acts on the task + # More complex coordination modes (star, tree, hierarchical) would + # require the MARBLE Engine, which is outside the scope of this adapter + for agent in agents: + result = agent.run(query) + agent_id = getattr(agent, "agent_id", str(agent)) + + results.append( + { + "agent_id": agent_id, + "result": result, + } + ) + + # Collect communication logs if available + if hasattr(agent, "get_serialized_messages"): + comm = agent.get_serialized_messages() # type: ignore + if comm: + communications.append(comm) + + return { + "agent_results": results, + "communications": communications, + "coordination_mode": coordination_mode, + } + + @abstractmethod + def get_model_adapter(self, model_id: str, **kwargs: Any) -> ModelAdapter: + """Provide a model adapter (implement in subclass). + + Args: + model_id: Model identifier + **kwargs: Additional arguments + + Returns: + ModelAdapter instance + """ + pass diff --git a/tests/test_benchmarks/test_multiagentbench/__init__.py b/tests/test_benchmarks/test_multiagentbench/__init__.py new file mode 100644 index 00000000..eaf3f093 --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/__init__.py @@ -0,0 +1 @@ +"""Tests for MultiAgentBench benchmark.""" diff --git a/tests/test_benchmarks/test_multiagentbench/conftest.py b/tests/test_benchmarks/test_multiagentbench/conftest.py new file mode 100644 index 00000000..c3a73968 --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/conftest.py @@ -0,0 +1,280 @@ +"""Shared fixtures for MultiAgentBench tests. + +Fixture Hierarchy +----------------- +- tests/conftest.py: Generic fixtures (DummyModelAdapter, dummy_model, etc.) + These are automatically available via pytest's conftest inheritance. +- tests/test_benchmarks/test_multiagentbench/conftest.py: MultiAgentBench-specific fixtures + +MultiAgentBench-Specific Components +----------------------------------- +- MultiAgentBenchAgentAdapter: Test adapter for multi-agent scenarios +- ConcreteMultiAgentBenchBenchmark: Concrete implementation for testing +- Sample task fixtures for different domains +""" + +import pytest +from typing import Any, Dict, List, Optional, Sequence, Tuple +from unittest.mock import MagicMock + +from conftest import DummyModelAdapter +from maseval import AgentAdapter, Task, MessageHistory + + +# ============================================================================= +# Sample Task Data +# ============================================================================= + + +@pytest.fixture +def sample_research_task_data() -> Dict[str, Any]: + """Sample task data for research domain.""" + return { + "scenario": "research", + "task_id": 1, + "agents": [ + { + "agent_id": "agent1", + "profile": "I am a researcher focused on machine learning.", + "type": "BaseAgent", + }, + { + "agent_id": "agent2", + "profile": "I am a researcher focused on NLP.", + "type": "BaseAgent", + }, + ], + "coordinate_mode": "cooperative", + "relationships": [ + ["agent1", "agent2", "collaborate with"], + ], + "environment": { + "type": "Research", + "max_iterations": 10, + }, + "task": { + "content": "Collaborate to generate a research idea about federated learning.", + "output_format": "Present the idea in 5Q format.", + }, + "llm": "gpt-4o-mini", + "memory": {"type": "SharedMemory"}, + "metrics": { + "diversity_of_perspectives": True, + "engagement_level": True, + "relevance": True, + }, + "max_iterations": 10, + "engine_planner": {"initial_progress": "Starting research collaboration."}, + "output": {"format": "jsonl"}, + "raw_marble_config": {}, + } + + +@pytest.fixture +def sample_bargaining_task_data() -> Dict[str, Any]: + """Sample task data for bargaining domain.""" + return { + "scenario": "bargaining", + "task_id": 1, + "agents": [ + { + "agent_id": "buyer", + "profile": "I am a buyer looking for the best price.", + "type": "BaseAgent", + }, + { + "agent_id": "seller", + "profile": "I am a seller trying to maximize profit.", + "type": "BaseAgent", + }, + ], + "coordinate_mode": "cooperative", + "relationships": [ + ["buyer", "seller", "negotiate with"], + ], + "environment": { + "type": "WorldSimulation", + "max_iterations": 10, + }, + "task": { + "content": "Negotiate the price of a used laptop.", + "output_format": "Final agreed price or disagreement.", + }, + "llm": "gpt-4o-mini", + "memory": {"type": "SharedMemory"}, + "metrics": {}, + "max_iterations": 10, + "engine_planner": {}, + "output": {}, + "raw_marble_config": {}, + } + + +@pytest.fixture +def sample_research_task(sample_research_task_data: Dict[str, Any]) -> Task: + """Create a sample research Task.""" + return Task( + id="research_1", + query=sample_research_task_data["task"]["content"], + environment_data=sample_research_task_data, + evaluation_data={ + "metrics": sample_research_task_data.get("metrics", {}), + "output_format": sample_research_task_data["task"].get("output_format", ""), + "model_id": "gpt-4o-mini", + }, + metadata={ + "domain": "research", + "task_id": 1, + }, + ) + + +@pytest.fixture +def sample_bargaining_task(sample_bargaining_task_data: Dict[str, Any]) -> Task: + """Create a sample bargaining Task.""" + return Task( + id="bargaining_1", + query=sample_bargaining_task_data["task"]["content"], + environment_data=sample_bargaining_task_data, + evaluation_data={ + "metrics": sample_bargaining_task_data.get("metrics", {}), + "output_format": sample_bargaining_task_data["task"].get("output_format", ""), + "model_id": "gpt-4o-mini", + }, + metadata={ + "domain": "bargaining", + "task_id": 1, + }, + ) + + +# ============================================================================= +# Mock Components +# ============================================================================= + + +class MultiAgentBenchAgentAdapter(AgentAdapter): + """Test agent adapter for MultiAgentBench tests. + + Provides controllable responses without needing a real agent implementation. + """ + + def __init__( + self, + agent_id: str = "test_agent", + profile: str = "Test agent profile", + ): + super().__init__(agent_instance=MagicMock(), name=agent_id) + self._agent_id = agent_id + self._profile = profile + self._responses: List[str] = [] + self._call_count = 0 + self.run_calls: List[str] = [] + + @property + def agent_id(self) -> str: + return self._agent_id + + @property + def profile(self) -> str: + return self._profile + + def set_responses(self, responses: List[str]) -> None: + """Set canned responses for the agent.""" + self._responses = responses + + def _run_agent(self, query: str) -> MessageHistory: + self.run_calls.append(query) + if self._responses: + response = self._responses[self._call_count % len(self._responses)] + self._call_count += 1 + else: + response = f"Agent {self._agent_id} response to: {query[:50]}..." + return MessageHistory([{"role": "assistant", "content": response}]) + + def get_token_usage(self) -> int: + return self._call_count * 100 + + def gather_traces(self) -> Dict[str, Any]: + traces = super().gather_traces() + traces["agent_id"] = self._agent_id + traces["profile"] = self._profile + traces["token_usage"] = self.get_token_usage() + traces["action_log"] = [{"task": q, "result": f"Response {i}", "has_communication": False} for i, q in enumerate(self.run_calls)] + traces["communication_log"] = [] + return traces + + +# ============================================================================= +# Concrete Benchmark Implementation +# ============================================================================= + + +@pytest.fixture +def concrete_multiagentbench_benchmark(): + """Create a concrete MultiAgentBenchBenchmark for testing.""" + from maseval.benchmark.multiagentbench import MultiAgentBenchBenchmark + + class ConcreteMultiAgentBenchBenchmark(MultiAgentBenchBenchmark): + """Concrete implementation for testing.""" + + def __init__( + self, + model_factory: Optional[Any] = None, + **kwargs: Any, + ): + if model_factory is None: + self._model_factory = lambda model_name: DummyModelAdapter( + model_id=f"test-model-{model_name}", + responses=['{"rating": 4}'], + ) + elif callable(model_factory): + self._model_factory = model_factory + else: + self._model_factory = lambda model_name: model_factory + super().__init__(**kwargs) + + def get_model_adapter(self, model_id: str, **kwargs): + factory_key = kwargs.get("register_name", model_id) + adapter = self._model_factory(factory_key) + register_name = kwargs.get("register_name") + if register_name: + try: + self.register("models", register_name, adapter) + except ValueError: + pass + return adapter + + def setup_agents( + self, + agent_data: Dict[str, Any], + environment: Any, + task: Task, + user: Optional[Any], + ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: + agent_configs = task.environment_data.get("agents", []) + agents_list: List[AgentAdapter] = [] + agents_dict: Dict[str, AgentAdapter] = {} + + for config in agent_configs: + agent_id = config.get("agent_id", f"agent_{len(agents_list)}") + profile = config.get("profile", "") + adapter = MultiAgentBenchAgentAdapter( + agent_id=agent_id, + profile=profile, + ) + agents_list.append(adapter) + agents_dict[agent_id] = adapter + + return agents_list, agents_dict + + return ConcreteMultiAgentBenchBenchmark + + +@pytest.fixture +def benchmark_instance(concrete_multiagentbench_benchmark): + """Create a benchmark instance with default settings.""" + return concrete_multiagentbench_benchmark( + progress_bar=False, + max_invocations=1, + ) diff --git a/tests/test_benchmarks/test_multiagentbench/test_benchmark.py b/tests/test_benchmarks/test_multiagentbench/test_benchmark.py new file mode 100644 index 00000000..b27209df --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/test_benchmark.py @@ -0,0 +1,252 @@ +"""Tests for MultiAgentBench benchmark classes.""" + + +from maseval import Task +from maseval.benchmark.multiagentbench import ( + MultiAgentBenchEnvironment, + MultiAgentBenchEvaluator, +) + + +class TestMultiAgentBenchBenchmark: + """Tests for MultiAgentBenchBenchmark abstract class.""" + + def test_setup_environment_returns_correct_type( + self, + benchmark_instance, + sample_research_task: Task, + ): + """setup_environment should return MultiAgentBenchEnvironment.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + + assert isinstance(env, MultiAgentBenchEnvironment) + + def test_setup_user_returns_none( + self, + benchmark_instance, + sample_research_task: Task, + ): + """setup_user should return None (no user simulator for multi-agent).""" + env = benchmark_instance.setup_environment({}, sample_research_task) + user = benchmark_instance.setup_user({}, env, sample_research_task) + + assert user is None + + def test_setup_evaluators_returns_evaluator( + self, + benchmark_instance, + sample_research_task: Task, + ): + """setup_evaluators should return MultiAgentBenchEvaluator.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + agents, _ = benchmark_instance.setup_agents({}, env, sample_research_task, None) + evaluators = benchmark_instance.setup_evaluators(env, sample_research_task, agents, None) + + assert len(evaluators) == 1 + assert isinstance(evaluators[0], MultiAgentBenchEvaluator) + + def test_setup_agents_creates_correct_count( + self, + benchmark_instance, + sample_research_task: Task, + ): + """setup_agents should create correct number of agents.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + agents_list, agents_dict = benchmark_instance.setup_agents({}, env, sample_research_task, None) + + # sample_research_task has 2 agents + assert len(agents_list) == 2 + assert len(agents_dict) == 2 + assert "agent1" in agents_dict + assert "agent2" in agents_dict + + def test_run_agents_executes_all_agents( + self, + benchmark_instance, + sample_research_task: Task, + ): + """run_agents should execute all agents and return results.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + agents_list, _ = benchmark_instance.setup_agents({}, env, sample_research_task, None) + + results = benchmark_instance.run_agents( + agents_list, + sample_research_task, + env, + sample_research_task.query, + ) + + assert isinstance(results, list) + assert len(results) == 2 + assert all("agent_id" in r for r in results) + assert all("result" in r for r in results) + + def test_evaluate_calls_evaluators( + self, + benchmark_instance, + sample_research_task: Task, + ): + """evaluate should call all evaluators.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + agents_list, agents_dict = benchmark_instance.setup_agents({}, env, sample_research_task, None) + evaluators = benchmark_instance.setup_evaluators(env, sample_research_task, agents_list, None) + + final_answer = [{"agent_id": "agent1", "result": "Done"}] + traces = { + "agents": {"agent1": {"token_usage": 100, "action_log": [], "communication_log": []}}, + "environment": {}, + } + + results = benchmark_instance.evaluate(evaluators, agents_dict, final_answer, traces) + + assert len(results) == 1 + assert "passed" in results[0] + + +class TestBenchmarkIntegration: + """Integration tests for benchmark execution.""" + + def test_run_single_task( + self, + benchmark_instance, + sample_research_task: Task, + ): + """Benchmark should run a single task end-to-end.""" + results = benchmark_instance.run( + tasks=[sample_research_task], + agent_data={}, + ) + + assert len(results) == 1 + assert results[0]["task_id"] == "research_1" + assert "status" in results[0] + assert "traces" in results[0] + assert "eval" in results[0] + + def test_run_multiple_tasks( + self, + benchmark_instance, + sample_research_task: Task, + sample_bargaining_task: Task, + ): + """Benchmark should run multiple tasks.""" + results = benchmark_instance.run( + tasks=[sample_research_task, sample_bargaining_task], + agent_data={}, + ) + + assert len(results) == 2 + + def test_run_with_task_repeats( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """Benchmark should repeat tasks when n_task_repeats > 1.""" + benchmark = concrete_multiagentbench_benchmark( + n_task_repeats=3, + progress_bar=False, + ) + results = benchmark.run( + tasks=[sample_research_task], + agent_data={}, + ) + + assert len(results) == 3 + assert all(r["task_id"] == "research_1" for r in results) + assert [r["repeat_idx"] for r in results] == [0, 1, 2] + + def test_run_collects_traces( + self, + benchmark_instance, + sample_research_task: Task, + ): + """Benchmark should collect traces from all components.""" + results = benchmark_instance.run( + tasks=[sample_research_task], + agent_data={}, + ) + + traces = results[0]["traces"] + assert "agents" in traces + assert "environment" in traces or traces.get("environment") is None + + def test_run_handles_setup_error( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """Benchmark should handle setup errors gracefully.""" + + class FailingBenchmark(concrete_multiagentbench_benchmark): + def setup_environment(self, agent_data, task): + raise RuntimeError("Setup failed") + + benchmark = FailingBenchmark(progress_bar=False) + results = benchmark.run( + tasks=[sample_research_task], + agent_data={}, + ) + + assert len(results) == 1 + assert results[0]["status"] == "setup_failed" + assert "error" in results[0] + + +class TestAgentCreation: + """Tests for agent creation and configuration.""" + + def test_agents_have_correct_ids( + self, + benchmark_instance, + sample_research_task: Task, + ): + """Created agents should have IDs from task config.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + _, agents_dict = benchmark_instance.setup_agents({}, env, sample_research_task, None) + + assert "agent1" in agents_dict + assert "agent2" in agents_dict + + def test_agents_have_profiles( + self, + benchmark_instance, + sample_research_task: Task, + ): + """Created agents should have profiles from task config.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + _, agents_dict = benchmark_instance.setup_agents({}, env, sample_research_task, None) + + agent1 = agents_dict["agent1"] + assert hasattr(agent1, "profile") + assert "machine learning" in agent1.profile.lower() + + +class TestEvaluatorConfiguration: + """Tests for evaluator configuration.""" + + def test_evaluator_uses_task_model_id( + self, + benchmark_instance, + sample_research_task: Task, + ): + """Evaluator should use model_id from task evaluation_data.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + agents, _ = benchmark_instance.setup_agents({}, env, sample_research_task, None) + evaluators = benchmark_instance.setup_evaluators(env, sample_research_task, agents, None) + + evaluator = evaluators[0] + assert evaluator.domain == "research" + + def test_evaluator_domain_from_task( + self, + benchmark_instance, + sample_bargaining_task: Task, + ): + """Evaluator should get domain from task environment_data.""" + env = benchmark_instance.setup_environment({}, sample_bargaining_task) + agents, _ = benchmark_instance.setup_agents({}, env, sample_bargaining_task, None) + evaluators = benchmark_instance.setup_evaluators(env, sample_bargaining_task, agents, None) + + evaluator = evaluators[0] + assert evaluator.domain == "bargaining" diff --git a/tests/test_benchmarks/test_multiagentbench/test_data_loader.py b/tests/test_benchmarks/test_multiagentbench/test_data_loader.py new file mode 100644 index 00000000..0e8bf0ac --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/test_data_loader.py @@ -0,0 +1,321 @@ +"""Tests for MultiAgentBench data loading functionality.""" + +import pytest +from pathlib import Path +from unittest.mock import patch +import tempfile +import json + +from maseval import Task +from maseval.benchmark.multiagentbench.data_loader import ( + load_tasks, + configure_model_ids, + get_domain_info, + VALID_DOMAINS, + _parse_task_entry, + _resolve_data_dir, +) + + +class TestValidDomains: + """Tests for domain validation.""" + + def test_valid_domains_contains_expected(self): + """VALID_DOMAINS should contain all expected domains.""" + expected = {"coding", "database", "minecraft", "research", "bargaining", "web", "worldsimulation"} + assert expected == VALID_DOMAINS + + def test_valid_domains_is_frozen(self): + """VALID_DOMAINS should be immutable.""" + assert isinstance(VALID_DOMAINS, frozenset) + + +class TestGetDomainInfo: + """Tests for get_domain_info function.""" + + def test_research_domain_info(self): + """get_domain_info should return correct info for research.""" + info = get_domain_info("research") + assert info["requires_infrastructure"] is False + assert info["coordination_mode"] == "cooperative" + assert "description" in info + + def test_database_requires_infrastructure(self): + """Database domain should require infrastructure.""" + info = get_domain_info("database") + assert info["requires_infrastructure"] is True + + def test_minecraft_requires_infrastructure(self): + """Minecraft domain should require infrastructure.""" + info = get_domain_info("minecraft") + assert info["requires_infrastructure"] is True + + def test_invalid_domain_raises(self): + """get_domain_info should raise for invalid domain.""" + with pytest.raises(ValueError, match="Invalid domain"): + get_domain_info("invalid_domain") + + def test_case_insensitive(self): + """get_domain_info should be case-insensitive.""" + info_lower = get_domain_info("research") + info_upper = get_domain_info("RESEARCH") + assert info_lower == info_upper + + +class TestParseTaskEntry: + """Tests for _parse_task_entry function.""" + + def test_parse_minimal_entry(self): + """_parse_task_entry should parse a minimal valid entry.""" + entry = { + "scenario": "research", + "task_id": 1, + "task": {"content": "Do research", "output_format": "5Q format"}, + "agents": [{"agent_id": "agent1", "profile": "Researcher"}], + "relationships": [["agent1", "agent1", "self"]], + } + task = _parse_task_entry(entry, "research", 0) + + assert isinstance(task, Task) + assert task.id == "research_1" + assert task.query == "Do research" + assert task.environment_data["scenario"] == "research" + assert len(task.environment_data["agents"]) == 1 + + def test_parse_entry_missing_required_field(self): + """_parse_task_entry should raise for missing required fields.""" + entry = { + "scenario": "research", + "task_id": 1, + # Missing "task", "agents", "relationships" + } + with pytest.raises(ValueError, match="missing required fields"): + _parse_task_entry(entry, "research", 0) + + def test_parse_entry_missing_agent_id(self): + """_parse_task_entry should raise if agent missing agent_id.""" + entry = { + "scenario": "research", + "task_id": 1, + "task": {"content": "Do research"}, + "agents": [{"profile": "Researcher"}], # Missing agent_id + "relationships": [], + } + with pytest.raises(ValueError, match="missing 'agent_id'"): + _parse_task_entry(entry, "research", 0) + + def test_parse_entry_empty_query(self): + """_parse_task_entry should raise for empty query.""" + entry = { + "scenario": "research", + "task_id": 1, + "task": {"content": "", "output_format": "5Q format"}, + "agents": [{"agent_id": "agent1"}], + "relationships": [], + } + with pytest.raises(ValueError, match="empty query"): + _parse_task_entry(entry, "research", 0) + + def test_parse_entry_with_string_task(self): + """_parse_task_entry should handle task as string.""" + entry = { + "scenario": "research", + "task_id": 1, + "task": "Do research task", + "agents": [{"agent_id": "agent1"}], + "relationships": [], + } + task = _parse_task_entry(entry, "research", 0) + assert task.query == "Do research task" + + def test_parse_entry_preserves_metadata(self): + """_parse_task_entry should preserve metadata correctly.""" + entry = { + "scenario": "bargaining", + "task_id": 42, + "task": {"content": "Negotiate"}, + "agents": [{"agent_id": "buyer"}], + "relationships": [], + "coordinate_mode": "star", + "environment": {"max_iterations": 20}, + } + task = _parse_task_entry(entry, "bargaining", 0) + + assert task.metadata["domain"] == "bargaining" + assert task.metadata["task_id"] == 42 + assert task.environment_data["coordinate_mode"] == "star" + assert task.environment_data["max_iterations"] == 20 + + +class TestLoadTasks: + """Tests for load_tasks function.""" + + def test_load_tasks_invalid_domain(self): + """load_tasks should raise for invalid domain.""" + with pytest.raises(ValueError, match="Invalid domain"): + load_tasks("invalid_domain") + + def test_load_tasks_missing_data_dir(self): + """load_tasks should raise if data directory not found.""" + with patch.dict("os.environ", {}, clear=True): + with pytest.raises(FileNotFoundError, match="does not exist"): + load_tasks("research", data_dir=Path("/nonexistent/path")) + + def test_load_tasks_with_mock_data(self): + """load_tasks should load tasks from JSONL file.""" + # Create temporary JSONL file + with tempfile.TemporaryDirectory() as tmpdir: + research_dir = Path(tmpdir) / "research" + research_dir.mkdir() + jsonl_path = research_dir / "research_main.jsonl" + + # Write sample task + task_data = { + "scenario": "research", + "task_id": 1, + "task": {"content": "Research task", "output_format": "5Q"}, + "agents": [{"agent_id": "agent1", "profile": "Researcher"}], + "relationships": [], + } + with jsonl_path.open("w") as f: + f.write(json.dumps(task_data) + "\n") + + tasks = load_tasks("research", data_dir=Path(tmpdir)) + + assert len(tasks) == 1 + assert tasks[0].query == "Research task" + + def test_load_tasks_with_limit(self): + """load_tasks should respect limit parameter.""" + with tempfile.TemporaryDirectory() as tmpdir: + research_dir = Path(tmpdir) / "research" + research_dir.mkdir() + jsonl_path = research_dir / "research_main.jsonl" + + # Write multiple tasks + with jsonl_path.open("w") as f: + for i in range(5): + task_data = { + "scenario": "research", + "task_id": i + 1, + "task": {"content": f"Research task {i + 1}"}, + "agents": [{"agent_id": f"agent{i + 1}"}], + "relationships": [], + } + f.write(json.dumps(task_data) + "\n") + + tasks = load_tasks("research", data_dir=Path(tmpdir), limit=2) + + assert len(tasks) == 2 + + def test_load_tasks_case_insensitive_domain(self): + """load_tasks should handle domain case-insensitively.""" + with tempfile.TemporaryDirectory() as tmpdir: + research_dir = Path(tmpdir) / "research" + research_dir.mkdir() + jsonl_path = research_dir / "research_main.jsonl" + + task_data = { + "scenario": "research", + "task_id": 1, + "task": {"content": "Test"}, + "agents": [{"agent_id": "agent1"}], + "relationships": [], + } + with jsonl_path.open("w") as f: + f.write(json.dumps(task_data) + "\n") + + tasks_lower = load_tasks("research", data_dir=Path(tmpdir)) + tasks_upper = load_tasks("RESEARCH", data_dir=Path(tmpdir)) + + assert len(tasks_lower) == len(tasks_upper) == 1 + + +class TestConfigureModelIds: + """Tests for configure_model_ids function.""" + + def test_configure_model_ids_sets_llm(self): + """configure_model_ids should set llm in environment_data.""" + task = Task( + id="test_1", + query="Test query", + environment_data={"scenario": "research"}, + evaluation_data={}, + metadata={}, + ) + tasks = [task] + + configure_model_ids(tasks, agent_model_id="gpt-4o") + + assert tasks[0].environment_data["llm"] == "gpt-4o" + + def test_configure_model_ids_sets_evaluator_model(self): + """configure_model_ids should set evaluator model_id.""" + task = Task( + id="test_1", + query="Test query", + environment_data={}, + evaluation_data={}, + metadata={}, + ) + tasks = [task] + + configure_model_ids( + tasks, + agent_model_id="gpt-4o", + evaluator_model_id="gpt-4o-mini", + ) + + assert tasks[0].evaluation_data["model_id"] == "gpt-4o-mini" + + def test_configure_model_ids_defaults_evaluator_to_agent(self): + """configure_model_ids should default evaluator model to agent model.""" + task = Task( + id="test_1", + query="Test query", + environment_data={}, + evaluation_data={}, + metadata={}, + ) + tasks = [task] + + configure_model_ids(tasks, agent_model_id="gpt-4o") + + assert tasks[0].evaluation_data["model_id"] == "gpt-4o" + + def test_configure_model_ids_returns_tasks(self): + """configure_model_ids should return the input tasks.""" + task = Task( + id="test_1", + query="Test query", + environment_data={}, + evaluation_data={}, + metadata={}, + ) + tasks = [task] + + result = configure_model_ids(tasks, agent_model_id="gpt-4o") + + assert result is tasks + + +class TestResolveDataDir: + """Tests for _resolve_data_dir function.""" + + def test_resolve_explicit_path(self): + """_resolve_data_dir should use explicit path if provided.""" + with tempfile.TemporaryDirectory() as tmpdir: + result = _resolve_data_dir(Path(tmpdir)) + assert result == Path(tmpdir) + + def test_resolve_nonexistent_explicit_path(self): + """_resolve_data_dir should raise for nonexistent explicit path.""" + with pytest.raises(FileNotFoundError): + _resolve_data_dir(Path("/nonexistent/path")) + + def test_resolve_from_env_var(self): + """_resolve_data_dir should use MARBLE_DATA_DIR env var.""" + with tempfile.TemporaryDirectory() as tmpdir: + with patch.dict("os.environ", {"MARBLE_DATA_DIR": tmpdir}): + result = _resolve_data_dir() + assert result == Path(tmpdir) diff --git a/tests/test_benchmarks/test_multiagentbench/test_environment.py b/tests/test_benchmarks/test_multiagentbench/test_environment.py new file mode 100644 index 00000000..b18082fd --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/test_environment.py @@ -0,0 +1,163 @@ +"""Tests for MultiAgentBench environment.""" + +import pytest +from typing import Any, Dict +from unittest.mock import patch + +from maseval.benchmark.multiagentbench.environment import ( + MultiAgentBenchEnvironment, + INFRASTRUCTURE_DOMAINS, +) +from maseval import EnvironmentError + + +class TestInfrastructureDomains: + """Tests for infrastructure domain constants.""" + + def test_infrastructure_domains_contains_expected(self): + """INFRASTRUCTURE_DOMAINS should contain expected domains.""" + assert "database" in INFRASTRUCTURE_DOMAINS + assert "minecraft" in INFRASTRUCTURE_DOMAINS + + def test_infrastructure_domains_excludes_simple(self): + """INFRASTRUCTURE_DOMAINS should not include simple domains.""" + assert "research" not in INFRASTRUCTURE_DOMAINS + assert "bargaining" not in INFRASTRUCTURE_DOMAINS + + +class TestMultiAgentBenchEnvironment: + """Tests for MultiAgentBenchEnvironment class.""" + + def test_init_with_research_task(self, sample_research_task_data: Dict[str, Any]): + """Environment should initialize for research domain.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + assert env.domain == "research" + assert env.state is not None + + def test_init_with_bargaining_task(self, sample_bargaining_task_data: Dict[str, Any]): + """Environment should initialize for bargaining domain.""" + env = MultiAgentBenchEnvironment(task_data=sample_bargaining_task_data) + + assert env.domain == "bargaining" + + def test_setup_state_extracts_domain(self, sample_research_task_data: Dict[str, Any]): + """setup_state should extract domain from task data.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + state = env.state + + assert state["domain"] == "research" + + def test_setup_state_extracts_max_iterations(self, sample_research_task_data: Dict[str, Any]): + """setup_state should extract max_iterations.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + state = env.state + + assert state["max_iterations"] == 10 + + def test_is_done_initially_false(self, sample_research_task_data: Dict[str, Any]): + """is_done should return False initially.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + # Without MARBLE env, always returns False + assert env.is_done() is False + + def test_is_task_completed_initially_false(self, sample_research_task_data: Dict[str, Any]): + """is_task_completed should return False initially.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + # Without MARBLE env, always returns False + assert env.is_task_completed() is False + + def test_get_marble_state_empty_without_marble(self, sample_research_task_data: Dict[str, Any]): + """get_marble_state should return empty dict without MARBLE.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + assert env.get_marble_state() == {} + + def test_get_tool_descriptions_empty_without_marble(self, sample_research_task_data: Dict[str, Any]): + """get_tool_descriptions should return empty dict without MARBLE.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + assert env.get_tool_descriptions() == {} + + def test_create_tools_empty_without_marble(self, sample_research_task_data: Dict[str, Any]): + """create_tools should return empty dict without MARBLE.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + tools = env.create_tools() + + assert tools == {} + + def test_gather_traces_includes_domain(self, sample_research_task_data: Dict[str, Any]): + """gather_traces should include domain information.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + traces = env.gather_traces() + + assert traces["domain"] == "research" + assert "tool_invocations" in traces + + def test_gather_config_includes_domain(self, sample_research_task_data: Dict[str, Any]): + """gather_config should include domain information.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + config = env.gather_config() + + assert config["domain"] == "research" + assert "tool_descriptions" in config + + +class TestInfrastructureCheck: + """Tests for infrastructure checking.""" + + def test_database_without_docker_raises(self): + """Environment should raise for database without Docker.""" + task_data = { + "scenario": "database", + "environment": {"type": "DB"}, + "task": {"content": "Query database"}, + "agents": [{"agent_id": "agent1"}], + } + + with patch("shutil.which", return_value=None): + with pytest.raises(EnvironmentError, match="requires external infrastructure"): + MultiAgentBenchEnvironment(task_data=task_data) + + def test_database_with_docker_succeeds(self): + """Environment should succeed for database with Docker.""" + task_data = { + "scenario": "database", + "environment": {"type": "DB"}, + "task": {"content": "Query database"}, + "agents": [{"agent_id": "agent1"}], + } + + with patch("shutil.which", return_value="/usr/bin/docker"): + # Should not raise, but MARBLE env creation may still fail + try: + env = MultiAgentBenchEnvironment(task_data=task_data) + assert env.domain == "database" + except ImportError: + # Expected if MARBLE not available + pass + + def test_minecraft_always_raises(self): + """Environment should raise for minecraft (not supported).""" + task_data = { + "scenario": "minecraft", + "environment": {"type": "Minecraft"}, + "task": {"content": "Build something"}, + "agents": [{"agent_id": "agent1"}], + } + + with pytest.raises(EnvironmentError, match="requires external infrastructure"): + MultiAgentBenchEnvironment(task_data=task_data) + + +class TestApplyAction: + """Tests for apply_action method.""" + + def test_apply_action_without_marble_raises(self, sample_research_task_data: Dict[str, Any]): + """apply_action should raise without MARBLE environment.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + with pytest.raises(EnvironmentError, match="not available"): + env.apply_action("agent1", "some_action", {"arg": "value"}) diff --git a/tests/test_benchmarks/test_multiagentbench/test_evaluator.py b/tests/test_benchmarks/test_multiagentbench/test_evaluator.py new file mode 100644 index 00000000..834e7f4b --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/test_evaluator.py @@ -0,0 +1,319 @@ +"""Tests for MultiAgentBench evaluator.""" + +import pytest + +from conftest import DummyModelAdapter +from maseval.benchmark.multiagentbench.evaluator import ( + MultiAgentBenchEvaluator, + MultiAgentBenchMetrics, +) + + +class TestMultiAgentBenchMetrics: + """Tests for MultiAgentBenchMetrics dataclass.""" + + def test_default_values(self): + """Metrics should have sensible defaults.""" + metrics = MultiAgentBenchMetrics() + + assert metrics.task_completion is False + assert metrics.token_consumption == 0 + assert metrics.planning_score == -1.0 + assert metrics.communication_score == -1.0 + assert metrics.task_evaluation == {} + assert metrics.agent_kpis == {} + assert metrics.total_milestones == 0 + + def test_to_dict(self): + """to_dict should serialize all fields.""" + metrics = MultiAgentBenchMetrics( + task_completion=True, + token_consumption=1000, + planning_score=4.0, + communication_score=5.0, + ) + d = metrics.to_dict() + + assert d["task_completion"] is True + assert d["token_consumption"] == 1000 + assert d["planning_score"] == 4.0 + assert d["communication_score"] == 5.0 + + +class TestMultiAgentBenchEvaluator: + """Tests for MultiAgentBenchEvaluator class.""" + + @pytest.fixture + def mock_model_adapter(self): + """Create a mock model adapter.""" + return DummyModelAdapter( + model_id="test-model", + responses=['{"rating": 4}'], + ) + + @pytest.fixture + def research_evaluator(self, mock_model_adapter): + """Create a research domain evaluator.""" + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=mock_model_adapter, + ) + + @pytest.fixture + def bargaining_evaluator(self, mock_model_adapter): + """Create a bargaining domain evaluator.""" + return MultiAgentBenchEvaluator( + domain="bargaining", + model_adapter=mock_model_adapter, + ) + + def test_init_normalizes_domain(self, mock_model_adapter): + """Evaluator should normalize domain to lowercase.""" + evaluator = MultiAgentBenchEvaluator( + domain="RESEARCH", + model_adapter=mock_model_adapter, + ) + assert evaluator.domain == "research" + + def test_filter_traces_extracts_agents(self, research_evaluator): + """filter_traces should extract agent traces.""" + traces = { + "agents": {"agent1": {"action_log": [{"task": "test", "result": "done"}]}}, + "environment": {}, + } + filtered = research_evaluator.filter_traces(traces) + + assert "agents" in filtered + assert "agent1" in filtered["agents"] + + def test_extract_communications_formats_correctly(self, research_evaluator): + """_extract_communications should format communication logs.""" + traces = { + "agents": { + "agent1": {"communication_log": [{"communication": "Hello from agent1"}]}, + "agent2": {"communication_log": [{"communication": "Hello from agent2"}]}, + } + } + comms = research_evaluator._extract_communications(traces) + + assert "[agent1]: Hello from agent1" in comms + assert "[agent2]: Hello from agent2" in comms + + def test_extract_communications_empty(self, research_evaluator): + """_extract_communications should handle empty logs.""" + traces = {"agents": {}} + comms = research_evaluator._extract_communications(traces) + + assert comms == "No communications recorded." + + def test_extract_results_formats_correctly(self, research_evaluator): + """_extract_results should format action logs.""" + traces = {"agents": {"agent1": {"action_log": [{"result": "Completed task A"}]}}} + results = research_evaluator._extract_results(traces) + + assert "[agent1]: Completed task A" in results + + def test_calculate_token_consumption(self, research_evaluator): + """_calculate_token_consumption should sum agent token usage.""" + traces = { + "agents": { + "agent1": {"token_usage": 500}, + "agent2": {"token_usage": 300}, + } + } + total = research_evaluator._calculate_token_consumption(traces) + + assert total == 800 + + def test_parse_score_valid_json(self, research_evaluator): + """_parse_score should parse valid JSON response.""" + response = '{"rating": 4}' + score = research_evaluator._parse_score(response) + + assert score == 4.0 + + def test_parse_score_with_markdown(self, research_evaluator): + """_parse_score should handle markdown code blocks.""" + response = '```json\n{"rating": 5}\n```' + score = research_evaluator._parse_score(response) + + assert score == 5.0 + + def test_parse_score_fallback(self, research_evaluator): + """_parse_score should fallback to finding digit.""" + response = "The rating is 4 out of 5" + score = research_evaluator._parse_score(response) + + assert score == 4.0 + + def test_parse_score_default(self, research_evaluator): + """_parse_score should return 3 as default.""" + response = "No score here" + score = research_evaluator._parse_score(response) + + assert score == 3.0 + + def test_parse_research_ratings_valid(self, research_evaluator): + """_parse_research_ratings should parse valid ratings.""" + response = '{"innovation": 4, "safety": 3, "feasibility": 5}' + ratings = research_evaluator._parse_research_ratings(response) + + assert ratings["innovation"] == 4 + assert ratings["safety"] == 3 + assert ratings["feasibility"] == 5 + + def test_parse_research_ratings_invalid(self, research_evaluator): + """_parse_research_ratings should return -1 for invalid.""" + response = "Invalid response" + ratings = research_evaluator._parse_research_ratings(response) + + assert ratings["innovation"] == -1 + assert ratings["safety"] == -1 + assert ratings["feasibility"] == -1 + + def test_determine_completion_research_positive(self, research_evaluator): + """_determine_completion should return True for positive research scores.""" + metrics = MultiAgentBenchMetrics(task_evaluation={"innovation": 4, "safety": 3, "feasibility": 5}) + assert research_evaluator._determine_completion(metrics) is True + + def test_determine_completion_research_negative(self, research_evaluator): + """_determine_completion should return False for negative scores.""" + metrics = MultiAgentBenchMetrics(task_evaluation={"innovation": -1, "safety": 3, "feasibility": 5}) + assert research_evaluator._determine_completion(metrics) is False + + def test_call_returns_expected_structure(self, research_evaluator): + """__call__ should return expected result structure.""" + traces = { + "agents": {"agent1": {"token_usage": 100, "action_log": [], "communication_log": []}}, + "environment": {}, + } + final_answer = [{"agent_id": "agent1", "result": "Done"}] + + # Mock model adapter to return research ratings + research_evaluator.model_adapter = DummyModelAdapter( + model_id="test", + responses=['{"innovation": 4, "safety": 4, "feasibility": 4}'], + ) + + result = research_evaluator(traces, final_answer) + + assert "passed" in result + assert "metrics" in result + assert "domain" in result + assert result["domain"] == "research" + + def test_format_final_answer_dict(self, research_evaluator): + """_format_final_answer should format dict input.""" + final_answer = { + "agent_results": [ + {"agent_id": "agent1", "result": "Result 1"}, + {"agent_id": "agent2", "result": "Result 2"}, + ] + } + formatted = research_evaluator._format_final_answer(final_answer) + + assert "[agent1]: Result 1" in formatted + assert "[agent2]: Result 2" in formatted + + def test_format_final_answer_list(self, research_evaluator): + """_format_final_answer should format list input.""" + final_answer = [ + {"agent_id": "agent1", "result": "Result 1"}, + ] + formatted = research_evaluator._format_final_answer(final_answer) + + assert "[agent1]: Result 1" in formatted + + def test_format_final_answer_string(self, research_evaluator): + """_format_final_answer should pass through string.""" + formatted = research_evaluator._format_final_answer("Simple result") + assert formatted == "Simple result" + + def test_format_final_answer_none(self, research_evaluator): + """_format_final_answer should handle None.""" + formatted = research_evaluator._format_final_answer(None) + assert formatted == "" + + +class TestBargainingEvaluation: + """Tests for bargaining-specific evaluation.""" + + @pytest.fixture + def bargaining_evaluator(self): + """Create evaluator with bargaining-specific responses.""" + adapter = DummyModelAdapter( + model_id="test", + responses=[ + '{"effectiveness_of_strategies": 4, "progress_and_outcome": 3, "interaction_dynamics": 5}', + '{"effectiveness_of_strategies": 3, "progress_and_outcome": 4, "interaction_dynamics": 4}', + ], + ) + return MultiAgentBenchEvaluator( + domain="bargaining", + model_adapter=adapter, + ) + + def test_evaluate_bargaining_returns_buyer_seller(self, bargaining_evaluator): + """_evaluate_bargaining should return buyer and seller ratings.""" + ratings = bargaining_evaluator._evaluate_bargaining("Task", "Result") + + assert "buyer" in ratings + assert "seller" in ratings + assert "effectiveness_of_strategies" in ratings["buyer"] + + def test_determine_completion_bargaining_positive(self, bargaining_evaluator): + """_determine_completion should work for bargaining domain.""" + metrics = MultiAgentBenchMetrics( + task_evaluation={ + "buyer": { + "effectiveness_of_strategies": 4, + "progress_and_outcome": 3, + "interaction_dynamics": 5, + }, + "seller": { + "effectiveness_of_strategies": 3, + "progress_and_outcome": 4, + "interaction_dynamics": 4, + }, + } + ) + assert bargaining_evaluator._determine_completion(metrics) is True + + +class TestCodingEvaluation: + """Tests for coding-specific evaluation.""" + + @pytest.fixture + def coding_evaluator(self): + """Create evaluator with coding-specific responses.""" + adapter = DummyModelAdapter( + model_id="test", + responses=[ + '{"instruction_following": 5, "executability": 4, "consistency": 4, "quality": 3}', + ], + ) + return MultiAgentBenchEvaluator( + domain="coding", + model_adapter=adapter, + ) + + def test_evaluate_coding_returns_all_metrics(self, coding_evaluator): + """_evaluate_coding should return all coding metrics.""" + ratings = coding_evaluator._evaluate_coding("Task", "Solution") + + assert "instruction_following" in ratings + assert "executability" in ratings + assert "consistency" in ratings + assert "quality" in ratings + + def test_determine_completion_coding_positive(self, coding_evaluator): + """_determine_completion should work for coding domain.""" + metrics = MultiAgentBenchMetrics( + task_evaluation={ + "instruction_following": 5, + "executability": 4, + "consistency": 4, + "quality": 3, + } + ) + assert coding_evaluator._determine_completion(metrics) is True From 0667cc07ede3f74771051dea1e58eaa6829a137d Mon Sep 17 00:00:00 2001 From: cemde Date: Mon, 19 Jan 2026 23:56:42 +0100 Subject: [PATCH 07/17] added to docs --- BENCHMARKS.md | 17 ++- docs/benchmark/multiagentbench.md | 120 +++++++++++++++++ maseval/benchmark/multiagentbench/README.md | 26 +++- maseval/benchmark/multiagentbench/__init__.py | 26 +++- .../benchmark/multiagentbench/data_loader.py | 123 ++++++++++++++++++ mkdocs.yml | 1 + 6 files changed, 300 insertions(+), 13 deletions(-) create mode 100644 docs/benchmark/multiagentbench.md diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 38647cc6..38187a3a 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -27,9 +27,22 @@ $\tau^2$-bench is a benchmark for evaluating agentic systems in realistic, multi --- -## 3. [Name of Next Benchmark] +## 3. MultiAgentBench (MARBLE) -(Description for the third benchmark...) +MultiAgentBench is a comprehensive benchmark suite for evaluating multi-agent collaboration and competition in LLM-based systems. It includes diverse scenarios across multiple domains including research collaboration, negotiation, coding tasks, and more. + +### Source and License + +- **Original Repository:** [https://github.com/ulab-uiuc/MARBLE](https://github.com/ulab-uiuc/MARBLE) +- **Paper:** [MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents](https://arxiv.org/abs/2503.01935) +- **Code License:** MIT +- **Data License:** MIT + +--- + +## 4. [Name of Next Benchmark] + +(Description for the next benchmark...) ### Source and License diff --git a/docs/benchmark/multiagentbench.md b/docs/benchmark/multiagentbench.md new file mode 100644 index 00000000..55911cc2 --- /dev/null +++ b/docs/benchmark/multiagentbench.md @@ -0,0 +1,120 @@ +# MultiAgentBench: Multi-Agent Collaboration Benchmark + +The **MultiAgentBench** benchmark evaluates multi-agent collaboration and competition in LLM-based systems across diverse scenarios including research, negotiation, coding, and more. + +## Overview + +[MultiAgentBench](https://github.com/ulab-uiuc/MARBLE) (from the MARBLE framework) is designed to evaluate how multiple LLM-based agents collaborate and compete to solve complex tasks. The benchmark features: + +- **7 diverse domains**: research, bargaining, coding, database, web, worldsimulation, minecraft +- **Multiple coordination modes**: cooperative, star, tree, hierarchical +- **LLM-based evaluation**: Matches MARBLE's evaluation methodology +- **Framework-agnostic**: Use with any agent framework or MARBLE's native agents + +Reference Paper: [MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents](https://arxiv.org/abs/2503.01935) + +Check out the [BENCHMARKS.md](https://github.com/parameterlab/MASEval/blob/main/BENCHMARKS.md) file for more information including licenses. + +## Quick Start + +```python +from maseval.benchmark.multiagentbench import ( + MultiAgentBenchBenchmark, + MultiAgentBenchEnvironment, + MultiAgentBenchEvaluator, + load_tasks, + configure_model_ids, + ensure_marble_exists, +) + +# Ensure MARBLE is installed (auto-downloads if needed) +ensure_marble_exists() + +# Load and configure tasks +tasks = load_tasks("research", limit=5) +configure_model_ids(tasks, agent_model_id="gpt-4o") + +# Create your framework-specific benchmark subclass +class MyMultiAgentBenchmark(MultiAgentBenchBenchmark): + def setup_agents(self, agent_data, environment, task, user): + # Your framework-specific agent creation + agent_configs = task.environment_data.get("agents", []) + # Create agents based on configs... + ... + + def get_model_adapter(self, model_id, **kwargs): + adapter = MyModelAdapter(model_id) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter + +# Run benchmark +benchmark = MyMultiAgentBenchmark() +results = benchmark.run(tasks, agent_data={}) +``` + +## MARBLE Reproduction Mode + +For exact reproduction of MARBLE's published results, use `MarbleMultiAgentBenchBenchmark` which wraps MARBLE's native agents: + +```python +from maseval.benchmark.multiagentbench import ( + MarbleMultiAgentBenchBenchmark, + load_tasks, + configure_model_ids, + ensure_marble_exists, +) + +# Ensure MARBLE is installed +ensure_marble_exists() + +# Load tasks +tasks = load_tasks("research", limit=5) +configure_model_ids(tasks, agent_model_id="gpt-4o") + +# Create benchmark with model adapter +class MyMarbleBenchmark(MarbleMultiAgentBenchBenchmark): + def get_model_adapter(self, model_id, **kwargs): + from maseval.interface.openai import OpenAIModelAdapter + adapter = OpenAIModelAdapter(model_id) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter + +benchmark = MyMarbleBenchmark() +results = benchmark.run(tasks, agent_data={}) +``` + +## Available Domains + +| Domain | Description | Infrastructure | +|--------|-------------|----------------| +| `research` | Research idea generation and collaboration | None | +| `bargaining` | Negotiation scenarios (buyer/seller) | None | +| `coding` | Software development collaboration | Filesystem | +| `database` | Database manipulation and querying | Docker + PostgreSQL | +| `web` | Web-based task completion | Network | +| `worldsimulation` | World simulation and interaction | None | +| `minecraft` | Collaborative building | External server | + +## API Reference + +::: maseval.benchmark.multiagentbench.MultiAgentBenchBenchmark + +::: maseval.benchmark.multiagentbench.MarbleMultiAgentBenchBenchmark + +::: maseval.benchmark.multiagentbench.MultiAgentBenchEnvironment + +::: maseval.benchmark.multiagentbench.MultiAgentBenchEvaluator + +::: maseval.benchmark.multiagentbench.MarbleAgentAdapter + +::: maseval.benchmark.multiagentbench.load_tasks + +::: maseval.benchmark.multiagentbench.configure_model_ids + +::: maseval.benchmark.multiagentbench.ensure_marble_exists + +::: maseval.benchmark.multiagentbench.download_marble + +::: maseval.benchmark.multiagentbench.get_domain_info diff --git a/maseval/benchmark/multiagentbench/README.md b/maseval/benchmark/multiagentbench/README.md index 5a189bfc..ccceda3e 100644 --- a/maseval/benchmark/multiagentbench/README.md +++ b/maseval/benchmark/multiagentbench/README.md @@ -10,10 +10,26 @@ Framework-agnostic implementation of the MultiAgentBench benchmark suite from MA ## Setup -This benchmark requires the MARBLE source code to be cloned locally. The benchmark -does NOT install MARBLE as a dependency - instead, it imports directly from a local copy. +This benchmark requires the MARBLE source code. You can set it up automatically or manually. -### 1. Clone MARBLE +### Option 1: Automatic Setup (Recommended) + +MARBLE will be automatically downloaded when you first use it: + +```python +from maseval.benchmark.multiagentbench import ensure_marble_exists, load_tasks + +# This downloads MARBLE if not present (about 50MB) +ensure_marble_exists() + +# Now load tasks +tasks = load_tasks("research", limit=1) +print(f"Loaded {len(tasks)} task(s)") +``` + +### Option 2: Manual Clone + +If you prefer to clone manually: ```bash cd maseval/benchmark/multiagentbench @@ -23,7 +39,7 @@ cd marble git checkout ``` -### 2. Install MARBLE Dependencies +### Install MARBLE Dependencies MARBLE requires additional dependencies. Add them to your environment: @@ -35,7 +51,7 @@ uv add litellm ruamel.yaml pip install litellm ruamel.yaml ``` -### 3. Verify Setup +### Verify Setup ```python from maseval.benchmark.multiagentbench import load_tasks diff --git a/maseval/benchmark/multiagentbench/__init__.py b/maseval/benchmark/multiagentbench/__init__.py index 19c3d2de..cc7eb688 100644 --- a/maseval/benchmark/multiagentbench/__init__.py +++ b/maseval/benchmark/multiagentbench/__init__.py @@ -17,11 +17,17 @@ - worldsimulation: World simulation and interaction Setup: - This benchmark requires MARBLE source code to be cloned locally: + This benchmark requires MARBLE source code. It will be automatically + downloaded when you first use `load_tasks()` or you can set it up manually: - ```bash - cd maseval/benchmark/multiagentbench - git clone https://github.com/ulab-uiuc/MARBLE.git marble + ```python + # Option 1: Automatic download (recommended) + from maseval.benchmark.multiagentbench import ensure_marble_exists + ensure_marble_exists() # Downloads MARBLE if not present + + # Option 2: Manual clone + # cd maseval/benchmark/multiagentbench + # git clone https://github.com/ulab-uiuc/MARBLE.git marble ``` See README.md in this directory for detailed setup instructions. @@ -35,10 +41,14 @@ MarbleAgentAdapter, load_tasks, configure_model_ids, + ensure_marble_exists, get_domain_info, VALID_DOMAINS, ) + # Ensure MARBLE is installed (auto-downloads if needed) + ensure_marble_exists() + # Load and configure tasks tasks = load_tasks("research", limit=5) configure_model_ids(tasks, agent_model_id="gpt-4o") @@ -103,11 +113,13 @@ def get_model_adapter(self, model_id, **kwargs): create_marble_agents, ) -# Data loading +# Data loading and setup from maseval.benchmark.multiagentbench.data_loader import ( load_tasks, configure_model_ids, get_domain_info, + ensure_marble_exists, + download_marble, VALID_DOMAINS, INFRASTRUCTURE_DOMAINS as INFRASTRUCTURE_REQUIRED_DOMAINS, ) @@ -126,10 +138,12 @@ def get_model_adapter(self, model_id, **kwargs): # Agent adapters "MarbleAgentAdapter", "create_marble_agents", - # Data loading + # Data loading and setup "load_tasks", "configure_model_ids", "get_domain_info", + "ensure_marble_exists", + "download_marble", "VALID_DOMAINS", "INFRASTRUCTURE_REQUIRED_DOMAINS", ] diff --git a/maseval/benchmark/multiagentbench/data_loader.py b/maseval/benchmark/multiagentbench/data_loader.py index dcbf5899..6c4410f7 100644 --- a/maseval/benchmark/multiagentbench/data_loader.py +++ b/maseval/benchmark/multiagentbench/data_loader.py @@ -5,12 +5,20 @@ """ import json +import logging import os +import subprocess from pathlib import Path from typing import Any, Dict, FrozenSet, List, Optional from maseval import Task +logger = logging.getLogger(__name__) + +# MARBLE repository configuration +MARBLE_REPO_URL = "https://github.com/ulab-uiuc/MARBLE.git" +MARBLE_DEFAULT_COMMIT = None # Set to a specific commit hash for reproducibility, or None for latest + # Valid domain names VALID_DOMAINS: FrozenSet[str] = frozenset( { @@ -28,6 +36,121 @@ INFRASTRUCTURE_DOMAINS: FrozenSet[str] = frozenset({"database", "minecraft"}) +def _get_marble_dir() -> Path: + """Get the default MARBLE installation directory. + + Returns: + Path to marble/ directory relative to this module + """ + return Path(__file__).parent / "marble" + + +def download_marble( + target_dir: Optional[Path] = None, + commit: Optional[str] = None, + force: bool = False, +) -> Path: + """Clone MARBLE repository to the specified directory. + + Args: + target_dir: Directory to clone into. Defaults to marble/ relative to this module. + commit: Specific commit hash to checkout. Defaults to MARBLE_DEFAULT_COMMIT or latest. + force: If True, remove existing directory and re-clone. + + Returns: + Path to the cloned MARBLE directory + + Raises: + RuntimeError: If git clone fails + FileExistsError: If directory exists and force=False + """ + if target_dir is None: + target_dir = _get_marble_dir() + + target_dir = Path(target_dir) + + # Check if already exists + if target_dir.exists(): + if not force: + logger.info(f"MARBLE already exists at {target_dir}") + return target_dir + + # Remove existing directory + import shutil + + logger.info(f"Removing existing MARBLE directory: {target_dir}") + shutil.rmtree(target_dir) + + # Clone repository + logger.info(f"Cloning MARBLE from {MARBLE_REPO_URL} to {target_dir}") + + try: + subprocess.run( + ["git", "clone", MARBLE_REPO_URL, str(target_dir)], + check=True, + capture_output=True, + text=True, + ) + except subprocess.CalledProcessError as e: + raise RuntimeError(f"Failed to clone MARBLE: {e.stderr}") from e + except FileNotFoundError: + raise RuntimeError("git is not installed or not in PATH. Please install git and try again.") + + # Checkout specific commit if requested + checkout_commit = commit or MARBLE_DEFAULT_COMMIT + if checkout_commit: + logger.info(f"Checking out commit: {checkout_commit}") + try: + subprocess.run( + ["git", "checkout", checkout_commit], + cwd=target_dir, + check=True, + capture_output=True, + text=True, + ) + except subprocess.CalledProcessError as e: + raise RuntimeError(f"Failed to checkout commit {checkout_commit}: {e.stderr}") from e + + logger.info(f"MARBLE successfully installed at {target_dir}") + return target_dir + + +def ensure_marble_exists(auto_download: bool = True) -> Path: + """Ensure MARBLE is available, optionally downloading it. + + This function checks if MARBLE is installed and optionally downloads it + if not present. + + Args: + auto_download: If True, automatically download MARBLE if not found. + If False, raise an error if MARBLE is not found. + + Returns: + Path to the MARBLE directory + + Raises: + FileNotFoundError: If MARBLE is not found and auto_download=False + + Example: + >>> marble_dir = ensure_marble_exists() + >>> # MARBLE is now available at marble_dir + """ + marble_dir = _get_marble_dir() + + # Check if MARBLE exists and has the expected structure + if marble_dir.exists() and (marble_dir / "multiagentbench").exists(): + return marble_dir + + if not auto_download: + raise FileNotFoundError( + f"MARBLE not found at {marble_dir}.\n" + "Run `ensure_marble_exists(auto_download=True)` to download automatically,\n" + "or manually clone: git clone https://github.com/ulab-uiuc/MARBLE.git marble" + ) + + return download_marble(marble_dir) + + def _resolve_data_dir(data_dir: Optional[Path] = None) -> Path: """Resolve the MARBLE data directory. diff --git a/mkdocs.yml b/mkdocs.yml index da54520c..4a7e8e76 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -125,4 +125,5 @@ nav: - OpenAI: interface/inference/openai.md - Benchmarks: - MACS: benchmark/macs.md + - MultiAgentBench: benchmark/multiagentbench.md - Tau2: benchmark/tau2.md From ebc09610dae64351648ecdd4d817da0b648bd0cb Mon Sep 17 00:00:00 2001 From: cemde Date: Tue, 20 Jan 2026 10:05:03 +0100 Subject: [PATCH 08/17] improved testing --- .../adapters/marble_adapter.py | 10 +- .../test_multiagentbench/test_benchmark.py | 223 +++++++ .../test_multiagentbench/test_data_loader.py | 246 +++++++- .../test_multiagentbench/test_environment.py | 217 ++++++- .../test_multiagentbench/test_evaluator.py | 548 ++++++++++++++++++ .../test_marble_adapter.py | 208 +++++++ 6 files changed, 1447 insertions(+), 5 deletions(-) create mode 100644 tests/test_benchmarks/test_multiagentbench/test_marble_adapter.py diff --git a/maseval/benchmark/multiagentbench/adapters/marble_adapter.py b/maseval/benchmark/multiagentbench/adapters/marble_adapter.py index d8fa5b59..ee3d8c96 100644 --- a/maseval/benchmark/multiagentbench/adapters/marble_adapter.py +++ b/maseval/benchmark/multiagentbench/adapters/marble_adapter.py @@ -43,7 +43,11 @@ def __init__( self._profile = getattr(marble_agent, "profile", "") self._communication_log: List[Dict[str, Any]] = [] self._action_log: List[Dict[str, Any]] = [] - super().__init__(callbacks=callbacks) + super().__init__(agent_instance=marble_agent, name=agent_id, callbacks=callbacks) + # Initialize message history + from maseval import MessageHistory + + self.messages = MessageHistory() @property def agent_id(self) -> str: @@ -95,8 +99,8 @@ def _run_agent(self, query: str) -> str: ) # Update message history - self._messages.add_message(role="user", content=query) - self._messages.add_message(role="assistant", content=result) + self.messages.add_message(role="user", content=query) + self.messages.add_message(role="assistant", content=result) return result diff --git a/tests/test_benchmarks/test_multiagentbench/test_benchmark.py b/tests/test_benchmarks/test_multiagentbench/test_benchmark.py index b27209df..9b7d1e83 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_benchmark.py +++ b/tests/test_benchmarks/test_multiagentbench/test_benchmark.py @@ -1,8 +1,13 @@ """Tests for MultiAgentBench benchmark classes.""" +import pytest +from typing import Any, Dict +from unittest.mock import MagicMock, patch from maseval import Task from maseval.benchmark.multiagentbench import ( + MultiAgentBenchBenchmark, + MarbleMultiAgentBenchBenchmark, MultiAgentBenchEnvironment, MultiAgentBenchEvaluator, ) @@ -250,3 +255,221 @@ def test_evaluator_domain_from_task( evaluator = evaluators[0] assert evaluator.domain == "bargaining" + + +class TestMarbleMultiAgentBenchBenchmark: + """Tests for MarbleMultiAgentBenchBenchmark class.""" + + @pytest.fixture + def marble_benchmark_class(self): + """Create a concrete MarbleMultiAgentBenchBenchmark class.""" + from conftest import DummyModelAdapter + + class ConcreteMarbleBenchmark(MarbleMultiAgentBenchBenchmark): + def get_model_adapter(self, model_id, **kwargs): + adapter = DummyModelAdapter( + model_id=model_id, + responses=['{"rating": 4}'], + ) + register_name = kwargs.get("register_name") + if register_name: + try: + self.register("models", register_name, adapter) + except ValueError: + pass + return adapter + + return ConcreteMarbleBenchmark + + def test_setup_agents_raises_import_error( + self, + marble_benchmark_class, + sample_research_task: Task, + ): + """setup_agents should raise ImportError when MARBLE not available.""" + benchmark = marble_benchmark_class(progress_bar=False) + env = benchmark.setup_environment({}, sample_research_task) + + with pytest.raises(ImportError, match="MARBLE is not available"): + benchmark.setup_agents({}, env, sample_research_task, None) + + def test_create_marble_env_raises_import_error( + self, + marble_benchmark_class, + sample_research_task: Task, + ): + """_create_marble_env should raise ImportError when MARBLE not available.""" + benchmark = marble_benchmark_class(progress_bar=False) + + with pytest.raises(ImportError, match="MARBLE is not available"): + benchmark._create_marble_env(sample_research_task) + + def test_setup_agent_graph_silently_fails( + self, + marble_benchmark_class, + sample_research_task: Task, + ): + """_setup_agent_graph should not raise when MARBLE not available.""" + benchmark = marble_benchmark_class(progress_bar=False) + + # Should not raise, just return silently + benchmark._setup_agent_graph({}, sample_research_task, None) + + def test_run_agents_returns_structured_output( + self, + marble_benchmark_class, + sample_research_task: Task, + ): + """run_agents should return structured output with agent_results.""" + from conftest import DummyAgentAdapter + + benchmark = marble_benchmark_class(progress_bar=False) + env = benchmark.setup_environment({}, sample_research_task) + + # Create mock agents + mock_agent1 = MagicMock() + mock_agent1.run.return_value = "Result from agent1" + mock_agent1.agent_id = "agent1" + + mock_agent2 = MagicMock() + mock_agent2.run.return_value = "Result from agent2" + mock_agent2.agent_id = "agent2" + mock_agent2.get_serialized_messages.return_value = "Communication log" + + result = benchmark.run_agents( + [mock_agent1, mock_agent2], + sample_research_task, + env, + sample_research_task.query, + ) + + assert "agent_results" in result + assert "communications" in result + assert "coordination_mode" in result + assert len(result["agent_results"]) == 2 + assert result["agent_results"][0]["agent_id"] == "agent1" + assert result["agent_results"][1]["agent_id"] == "agent2" + + def test_run_agents_collects_communications( + self, + marble_benchmark_class, + sample_research_task: Task, + ): + """run_agents should collect communications from agents.""" + benchmark = marble_benchmark_class(progress_bar=False) + env = benchmark.setup_environment({}, sample_research_task) + + # Create mock agent with get_serialized_messages + mock_agent = MagicMock() + mock_agent.run.return_value = "Result" + mock_agent.agent_id = "agent1" + mock_agent.get_serialized_messages.return_value = "Hello from agent1" + + result = benchmark.run_agents( + [mock_agent], + sample_research_task, + env, + sample_research_task.query, + ) + + assert "Hello from agent1" in result["communications"] + + +class TestBenchmarkWithDifferentCoordinationModes: + """Tests for different coordination modes.""" + + def test_run_agents_with_cooperative_mode( + self, + benchmark_instance, + sample_research_task: Task, + ): + """run_agents should work with cooperative coordination.""" + # sample_research_task uses cooperative mode by default + env = benchmark_instance.setup_environment({}, sample_research_task) + agents_list, _ = benchmark_instance.setup_agents({}, env, sample_research_task, None) + + results = benchmark_instance.run_agents( + agents_list, + sample_research_task, + env, + sample_research_task.query, + ) + + assert len(results) == 2 + + def test_run_agents_with_star_mode(self, benchmark_instance): + """run_agents should work with star coordination.""" + task_data = { + "scenario": "research", + "task_id": 1, + "agents": [ + {"agent_id": "central", "profile": "Central coordinator"}, + {"agent_id": "worker1", "profile": "Worker 1"}, + ], + "coordinate_mode": "star", + "relationships": [["central", "worker1", "coordinates"]], + "environment": {"max_iterations": 10}, + "task": {"content": "Research task", "output_format": "5Q"}, + "max_iterations": 10, + } + task = Task( + id="test_star", + query="Research task", + environment_data=task_data, + evaluation_data={"model_id": "gpt-4o-mini"}, + metadata={"domain": "research"}, + ) + + env = benchmark_instance.setup_environment({}, task) + agents_list, _ = benchmark_instance.setup_agents({}, env, task, None) + + results = benchmark_instance.run_agents(agents_list, task, env, task.query) + + assert len(results) == 2 + + +class TestBenchmarkWithEmptyAgents: + """Tests for edge cases with agents.""" + + def test_run_agents_with_empty_list( + self, + benchmark_instance, + sample_research_task: Task, + ): + """run_agents should handle empty agent list.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + + results = benchmark_instance.run_agents( + [], + sample_research_task, + env, + sample_research_task.query, + ) + + assert results == [] + + def test_setup_agents_with_no_agents_in_task(self, benchmark_instance): + """setup_agents should handle task with no agents.""" + task_data = { + "scenario": "research", + "task_id": 1, + "agents": [], # No agents + "coordinate_mode": "cooperative", + "relationships": [], + "environment": {"max_iterations": 10}, + "task": {"content": "Research task"}, + "max_iterations": 10, + } + task = Task( + id="test_no_agents", + query="Research task", + environment_data=task_data, + evaluation_data={"model_id": "gpt-4o-mini"}, + metadata={"domain": "research"}, + ) + + env = benchmark_instance.setup_environment({}, task) + agents_list, agents_dict = benchmark_instance.setup_agents({}, env, task, None) + + assert len(agents_list) == 0 + assert len(agents_dict) == 0 diff --git a/tests/test_benchmarks/test_multiagentbench/test_data_loader.py b/tests/test_benchmarks/test_multiagentbench/test_data_loader.py index 0e8bf0ac..67eab09b 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_data_loader.py +++ b/tests/test_benchmarks/test_multiagentbench/test_data_loader.py @@ -2,7 +2,8 @@ import pytest from pathlib import Path -from unittest.mock import patch +from unittest.mock import patch, MagicMock +import subprocess import tempfile import json @@ -11,6 +12,9 @@ load_tasks, configure_model_ids, get_domain_info, + download_marble, + ensure_marble_exists, + _get_marble_dir, VALID_DOMAINS, _parse_task_entry, _resolve_data_dir, @@ -319,3 +323,243 @@ def test_resolve_from_env_var(self): with patch.dict("os.environ", {"MARBLE_DATA_DIR": tmpdir}): result = _resolve_data_dir() assert result == Path(tmpdir) + + def test_resolve_not_found(self): + """_resolve_data_dir should raise when no directory found.""" + with patch.dict("os.environ", {}, clear=True): + with patch( + "maseval.benchmark.multiagentbench.data_loader._get_marble_dir", + return_value=Path("/nonexistent/marble"), + ): + with patch("pathlib.Path.cwd", return_value=Path("/nonexistent/cwd")): + with pytest.raises(FileNotFoundError, match="MARBLE data directory not found"): + _resolve_data_dir() + + +class TestGetMarbleDir: + """Tests for _get_marble_dir function.""" + + def test_returns_path_relative_to_module(self): + """_get_marble_dir should return path relative to module.""" + result = _get_marble_dir() + assert result.name == "marble" + assert "multiagentbench" in str(result.parent) + + +class TestDownloadMarble: + """Tests for download_marble function.""" + + def test_download_marble_already_exists(self): + """download_marble should return existing path if not force.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + marble_dir.mkdir() + + result = download_marble(target_dir=marble_dir, force=False) + + assert result == marble_dir + + def test_download_marble_force_removes_existing(self): + """download_marble should remove existing dir when force=True.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + marble_dir.mkdir() + (marble_dir / "test_file.txt").write_text("test") + + with patch("subprocess.run") as mock_run: + mock_run.return_value = MagicMock(returncode=0) + download_marble(target_dir=marble_dir, force=True) + + # Directory should have been removed and git clone called + mock_run.assert_called() + + def test_download_marble_git_clone_called(self): + """download_marble should call git clone.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + + with patch("subprocess.run") as mock_run: + mock_run.return_value = MagicMock(returncode=0) + download_marble(target_dir=marble_dir) + + # Verify git clone was called + calls = mock_run.call_args_list + assert len(calls) >= 1 + clone_call = calls[0] + assert "git" in clone_call[0][0] + assert "clone" in clone_call[0][0] + + def test_download_marble_with_commit(self): + """download_marble should checkout specific commit if provided.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + + with patch("subprocess.run") as mock_run: + mock_run.return_value = MagicMock(returncode=0) + download_marble(target_dir=marble_dir, commit="abc123") + + # Verify git checkout was called + calls = mock_run.call_args_list + assert len(calls) >= 2 + checkout_call = calls[1] + assert "checkout" in checkout_call[0][0] + assert "abc123" in checkout_call[0][0] + + def test_download_marble_clone_fails(self): + """download_marble should raise RuntimeError on clone failure.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + + with patch("subprocess.run") as mock_run: + mock_run.side_effect = subprocess.CalledProcessError( + 1, "git clone", stderr="Clone failed" + ) + + with pytest.raises(RuntimeError, match="Failed to clone MARBLE"): + download_marble(target_dir=marble_dir) + + def test_download_marble_git_not_found(self): + """download_marble should raise RuntimeError if git not installed.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + + with patch("subprocess.run") as mock_run: + mock_run.side_effect = FileNotFoundError() + + with pytest.raises(RuntimeError, match="git is not installed"): + download_marble(target_dir=marble_dir) + + def test_download_marble_checkout_fails(self): + """download_marble should raise RuntimeError on checkout failure.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + + def mock_run_side_effect(*args, **kwargs): + cmd = args[0] + if "checkout" in cmd: + raise subprocess.CalledProcessError(1, "git checkout", stderr="Checkout failed") + return MagicMock(returncode=0) + + with patch("subprocess.run", side_effect=mock_run_side_effect): + with pytest.raises(RuntimeError, match="Failed to checkout commit"): + download_marble(target_dir=marble_dir, commit="invalid") + + +class TestEnsureMarbleExists: + """Tests for ensure_marble_exists function.""" + + def test_ensure_marble_exists_already_present(self): + """ensure_marble_exists should return path if MARBLE exists.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + marble_dir.mkdir() + (marble_dir / "multiagentbench").mkdir() + + with patch( + "maseval.benchmark.multiagentbench.data_loader._get_marble_dir", + return_value=marble_dir, + ): + result = ensure_marble_exists(auto_download=False) + + assert result == marble_dir + + def test_ensure_marble_exists_not_present_no_download(self): + """ensure_marble_exists should raise if not present and auto_download=False.""" + with patch( + "maseval.benchmark.multiagentbench.data_loader._get_marble_dir", + return_value=Path("/nonexistent/marble"), + ): + with pytest.raises(FileNotFoundError, match="MARBLE not found"): + ensure_marble_exists(auto_download=False) + + def test_ensure_marble_exists_auto_download(self): + """ensure_marble_exists should download if not present and auto_download=True.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + + with patch( + "maseval.benchmark.multiagentbench.data_loader._get_marble_dir", + return_value=marble_dir, + ): + with patch( + "maseval.benchmark.multiagentbench.data_loader.download_marble", + return_value=marble_dir, + ) as mock_download: + result = ensure_marble_exists(auto_download=True) + + mock_download.assert_called_once_with(marble_dir) + assert result == marble_dir + + +class TestLoadTasksEdgeCases: + """Edge case tests for load_tasks.""" + + def test_load_tasks_empty_lines(self): + """load_tasks should skip empty lines in JSONL.""" + with tempfile.TemporaryDirectory() as tmpdir: + research_dir = Path(tmpdir) / "research" + research_dir.mkdir() + jsonl_path = research_dir / "research_main.jsonl" + + # Write tasks with empty lines + with jsonl_path.open("w") as f: + task_data = { + "scenario": "research", + "task_id": 1, + "task": {"content": "Task 1"}, + "agents": [{"agent_id": "agent1"}], + "relationships": [], + } + f.write(json.dumps(task_data) + "\n") + f.write("\n") # Empty line + f.write(" \n") # Whitespace-only line + task_data["task_id"] = 2 + task_data["task"]["content"] = "Task 2" + f.write(json.dumps(task_data) + "\n") + + tasks = load_tasks("research", data_dir=Path(tmpdir)) + + assert len(tasks) == 2 + + def test_load_tasks_file_not_found(self): + """load_tasks should raise FileNotFoundError if JSONL file missing.""" + with tempfile.TemporaryDirectory() as tmpdir: + # Create directory but not the JSONL file + research_dir = Path(tmpdir) / "research" + research_dir.mkdir() + + with pytest.raises(FileNotFoundError, match="Task data not found"): + load_tasks("research", data_dir=Path(tmpdir)) + + +class TestGetDomainInfoAllDomains: + """Test get_domain_info for all domains.""" + + @pytest.mark.parametrize( + "domain", + ["coding", "database", "minecraft", "research", "bargaining", "web", "worldsimulation"], + ) + def test_all_domains_have_info(self, domain): + """All valid domains should return info.""" + info = get_domain_info(domain) + assert "requires_infrastructure" in info + assert "description" in info + assert "coordination_mode" in info + + def test_coding_domain_info(self): + """Coding domain should have tree coordination.""" + info = get_domain_info("coding") + assert info["coordination_mode"] == "tree" + assert info["requires_infrastructure"] is False + + def test_web_domain_info(self): + """Web domain should have star coordination.""" + info = get_domain_info("web") + assert info["coordination_mode"] == "star" + assert info["requires_infrastructure"] is False + + def test_worldsimulation_domain_info(self): + """WorldSimulation domain should have cooperative coordination.""" + info = get_domain_info("worldsimulation") + assert info["coordination_mode"] == "cooperative" + assert info["requires_infrastructure"] is False diff --git a/tests/test_benchmarks/test_multiagentbench/test_environment.py b/tests/test_benchmarks/test_multiagentbench/test_environment.py index b18082fd..8cddce60 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_environment.py +++ b/tests/test_benchmarks/test_multiagentbench/test_environment.py @@ -2,7 +2,7 @@ import pytest from typing import Any, Dict -from unittest.mock import patch +from unittest.mock import patch, MagicMock from maseval.benchmark.multiagentbench.environment import ( MultiAgentBenchEnvironment, @@ -161,3 +161,218 @@ def test_apply_action_without_marble_raises(self, sample_research_task_data: Dic with pytest.raises(EnvironmentError, match="not available"): env.apply_action("agent1", "some_action", {"arg": "value"}) + + +class TestWithMockedMarbleEnv: + """Tests with mocked MARBLE environment.""" + + def test_is_done_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """is_done should delegate to MARBLE env when available.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + # Mock MARBLE env + mock_marble_env = MagicMock() + mock_marble_env.is_done.return_value = True + env._marble_env = mock_marble_env + + assert env.is_done() is True + mock_marble_env.is_done.assert_called_once() + + def test_is_task_completed_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """is_task_completed should delegate to MARBLE env when available.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_marble_env = MagicMock() + mock_marble_env.is_task_completed.return_value = True + env._marble_env = mock_marble_env + + assert env.is_task_completed() is True + mock_marble_env.is_task_completed.assert_called_once() + + def test_get_marble_state_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """get_marble_state should return state from MARBLE env.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_marble_env = MagicMock() + mock_marble_env.get_state.return_value = {"key": "value"} + env._marble_env = mock_marble_env + + assert env.get_marble_state() == {"key": "value"} + mock_marble_env.get_state.assert_called_once() + + def test_get_tool_descriptions_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """get_tool_descriptions should return descriptions from MARBLE env.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_marble_env = MagicMock() + mock_marble_env.action_handler_descriptions = {"tool1": {"desc": "Tool 1"}} + env._marble_env = mock_marble_env + + assert env.get_tool_descriptions() == {"tool1": {"desc": "Tool 1"}} + + def test_apply_action_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """apply_action should delegate to MARBLE env when available.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_marble_env = MagicMock() + mock_marble_env.apply_action.return_value = {"result": "success"} + env._marble_env = mock_marble_env + + result = env.apply_action("agent1", "action1", {"arg": "value"}) + + assert result == {"result": "success"} + mock_marble_env.apply_action.assert_called_once_with("agent1", "action1", {"arg": "value"}) + + def test_gather_traces_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """gather_traces should include MARBLE state when available.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_marble_env = MagicMock() + mock_marble_env.get_state.return_value = {"state_key": "state_value"} + mock_marble_env.is_done.return_value = False + mock_marble_env.is_task_completed.return_value = False + env._marble_env = mock_marble_env + + traces = env.gather_traces() + + assert traces["marble_state"] == {"state_key": "state_value"} + assert traces["is_done"] is False + assert traces["is_task_completed"] is False + assert traces["marble_env_type"] == "MagicMock" + + def test_create_tools_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """create_tools should wrap MARBLE action handlers.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + # Create mock MARBLE env with action handlers + mock_handler = MagicMock(return_value="handler_result") + mock_marble_env = MagicMock() + mock_marble_env._action_handlers = {"test_action": mock_handler} + env._marble_env = mock_marble_env + + tools = env.create_tools() + + assert "test_action" in tools + # Test the wrapped handler + result = tools["test_action"](arg1="value1") + assert result == "handler_result" + mock_handler.assert_called_once_with(arg1="value1") + + def test_create_tools_traces_invocations(self, sample_research_task_data: Dict[str, Any]): + """create_tools should trace tool invocations.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_handler = MagicMock(return_value="result") + mock_marble_env = MagicMock() + mock_marble_env._action_handlers = {"test_action": mock_handler} + env._marble_env = mock_marble_env + + tools = env.create_tools() + tools["test_action"](input="test") + + # Check that invocation was traced + assert "test_action" in env._tool_histories + history = env._tool_histories["test_action"] + invocations = history.to_list() + assert len(invocations) == 1 + + def test_create_tools_traces_errors(self, sample_research_task_data: Dict[str, Any]): + """create_tools should trace tool errors.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_handler = MagicMock(side_effect=RuntimeError("Handler error")) + mock_marble_env = MagicMock() + mock_marble_env._action_handlers = {"test_action": mock_handler} + env._marble_env = mock_marble_env + + tools = env.create_tools() + + with pytest.raises(RuntimeError): + tools["test_action"](input="test") + + # Check that error was traced + history = env._tool_histories["test_action"] + invocations = history.to_list() + assert len(invocations) == 1 + assert invocations[0]["status"] == "error" + + +class TestGetTool: + """Tests for get_tool method.""" + + def test_get_tool_returns_none_if_not_found(self, sample_research_task_data: Dict[str, Any]): + """get_tool should return None for unknown tool.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + assert env.get_tool("unknown_tool") is None + + def test_get_tool_returns_tool_if_found(self, sample_research_task_data: Dict[str, Any]): + """get_tool should return tool if it exists.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_handler = MagicMock() + mock_marble_env = MagicMock() + mock_marble_env._action_handlers = {"test_tool": mock_handler} + env._marble_env = mock_marble_env + + # Create tools and update env.tools (which is normally set during __init__) + env.tools = env.create_tools() + + tool = env.get_tool("test_tool") + assert tool is not None + + +class TestGatherConfig: + """Tests for gather_config method.""" + + def test_gather_config_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """gather_config should include tool descriptions.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_marble_env = MagicMock() + mock_marble_env.action_handler_descriptions = {"tool1": {"name": "Tool 1"}} + env._marble_env = mock_marble_env + + config = env.gather_config() + + assert config["tool_descriptions"] == {"tool1": {"name": "Tool 1"}} + assert config["marble_env_type"] == "MagicMock" + + +class TestSetupStateEdgeCases: + """Tests for setup_state edge cases.""" + + def test_setup_state_with_string_task(self): + """setup_state should handle task as string.""" + task_data = { + "scenario": "research", + "environment": {}, + "task": "String task content", + "agents": [{"agent_id": "agent1"}], + } + + env = MultiAgentBenchEnvironment(task_data=task_data) + assert env.state is not None + + def test_setup_state_default_max_iterations(self): + """setup_state should default max_iterations to 10.""" + task_data = { + "scenario": "research", + "environment": {}, + "task": {"content": "Task"}, + "agents": [{"agent_id": "agent1"}], + } + + env = MultiAgentBenchEnvironment(task_data=task_data) + assert env.state["max_iterations"] == 10 + + def test_setup_state_uses_env_max_iterations(self): + """setup_state should use environment max_iterations if provided.""" + task_data = { + "scenario": "research", + "environment": {"max_iterations": 25}, + "task": {"content": "Task"}, + "agents": [{"agent_id": "agent1"}], + } + + env = MultiAgentBenchEnvironment(task_data=task_data) + assert env.state["max_iterations"] == 25 diff --git a/tests/test_benchmarks/test_multiagentbench/test_evaluator.py b/tests/test_benchmarks/test_multiagentbench/test_evaluator.py index 834e7f4b..31432552 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_evaluator.py +++ b/tests/test_benchmarks/test_multiagentbench/test_evaluator.py @@ -317,3 +317,551 @@ def test_determine_completion_coding_positive(self, coding_evaluator): } ) assert coding_evaluator._determine_completion(metrics) is True + + def test_determine_completion_coding_negative(self, coding_evaluator): + """_determine_completion should return False for negative coding scores.""" + metrics = MultiAgentBenchMetrics( + task_evaluation={ + "instruction_following": 5, + "executability": -1, + "consistency": 4, + "quality": 3, + } + ) + assert coding_evaluator._determine_completion(metrics) is False + + def test_parse_coding_ratings_invalid(self, coding_evaluator): + """_parse_coding_ratings should return -1 for invalid response.""" + ratings = coding_evaluator._parse_coding_ratings("Invalid JSON") + assert ratings["instruction_following"] == -1 + assert ratings["executability"] == -1 + assert ratings["consistency"] == -1 + assert ratings["quality"] == -1 + + +class TestDatabaseEvaluation: + """Tests for database-specific evaluation.""" + + @pytest.fixture + def database_evaluator(self): + """Create evaluator for database domain.""" + adapter = DummyModelAdapter( + model_id="test", + responses=['{"rating": 4}'], + ) + return MultiAgentBenchEvaluator( + domain="database", + model_adapter=adapter, + ) + + def test_evaluate_database_returns_structure(self, database_evaluator): + """_evaluate_database should return predicted result.""" + result = database_evaluator._evaluate_database("Query task", "SELECT * FROM users") + + assert "predicted" in result + assert result["predicted"] == "SELECT * FROM users" + assert "root_cause" in result + assert result["root_cause"] == [] + + def test_determine_completion_database_with_prediction(self, database_evaluator): + """_determine_completion should return True for database with prediction.""" + metrics = MultiAgentBenchMetrics( + task_evaluation={"predicted": "SELECT * FROM users", "root_cause": []} + ) + assert database_evaluator._determine_completion(metrics) is True + + def test_determine_completion_database_empty_prediction(self, database_evaluator): + """_determine_completion should return False for empty prediction.""" + metrics = MultiAgentBenchMetrics( + task_evaluation={"predicted": "", "root_cause": []} + ) + assert database_evaluator._determine_completion(metrics) is False + + def test_call_database_domain(self, database_evaluator): + """__call__ should work for database domain.""" + traces = { + "agents": {"agent1": {"token_usage": 100, "action_log": [], "communication_log": []}}, + "environment": {"marble_state": {"task_description": "Find query issue"}}, + } + final_answer = [{"agent_id": "agent1", "result": "SELECT * FROM orders"}] + + result = database_evaluator(traces, final_answer) + + assert result["domain"] == "database" + assert "passed" in result + + +class TestWorldSimulationEvaluation: + """Tests for worldsimulation domain (alias for bargaining).""" + + @pytest.fixture + def worldsim_evaluator(self): + """Create evaluator for worldsimulation domain.""" + adapter = DummyModelAdapter( + model_id="test", + responses=[ + '{"effectiveness_of_strategies": 4, "progress_and_outcome": 4, "interaction_dynamics": 4}', + '{"effectiveness_of_strategies": 4, "progress_and_outcome": 4, "interaction_dynamics": 4}', + ], + ) + return MultiAgentBenchEvaluator( + domain="worldsimulation", + model_adapter=adapter, + ) + + def test_worldsimulation_uses_bargaining_eval(self, worldsim_evaluator): + """worldsimulation domain should use bargaining evaluation.""" + traces = { + "agents": {"agent1": {"token_usage": 100, "action_log": [], "communication_log": []}}, + "environment": {}, + } + final_answer = "Simulation result" + + result = worldsim_evaluator(traces, final_answer) + + assert result["domain"] == "worldsimulation" + assert "buyer" in result["metrics"]["task_evaluation"] + assert "seller" in result["metrics"]["task_evaluation"] + + def test_determine_completion_worldsimulation(self, worldsim_evaluator): + """_determine_completion should work for worldsimulation domain.""" + metrics = MultiAgentBenchMetrics( + task_evaluation={ + "buyer": { + "effectiveness_of_strategies": 4, + "progress_and_outcome": 4, + "interaction_dynamics": 4, + }, + "seller": { + "effectiveness_of_strategies": 4, + "progress_and_outcome": 4, + "interaction_dynamics": 4, + }, + } + ) + assert worldsim_evaluator._determine_completion(metrics) is True + + +class TestUnknownDomainEvaluation: + """Tests for unknown domain handling.""" + + @pytest.fixture + def unknown_evaluator(self): + """Create evaluator for unknown domain.""" + adapter = DummyModelAdapter( + model_id="test", + responses=['{"rating": 4}'], + ) + return MultiAgentBenchEvaluator( + domain="unknown_domain", + model_adapter=adapter, + ) + + def test_unknown_domain_sets_completion_from_result(self, unknown_evaluator): + """Unknown domain should set completion based on final result presence.""" + traces = { + "agents": {}, + "environment": {}, + } + final_answer = "Some result" + + result = unknown_evaluator(traces, final_answer) + + assert result["domain"] == "unknown_domain" + # Should still have metrics + assert "metrics" in result + + def test_determine_completion_unknown_domain(self, unknown_evaluator): + """_determine_completion should return False for unknown domain.""" + metrics = MultiAgentBenchMetrics(task_evaluation={}) + assert unknown_evaluator._determine_completion(metrics) is False + + +class TestCommunicationEvaluation: + """Tests for communication evaluation.""" + + @pytest.fixture + def evaluator_with_comm_response(self): + """Create evaluator with communication rating response.""" + adapter = DummyModelAdapter( + model_id="test", + responses=['{"rating": 4}', '{"innovation": 4, "safety": 4, "feasibility": 4}'], + ) + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + + def test_evaluate_communication_returns_score(self, evaluator_with_comm_response): + """_evaluate_communication should return score from LLM.""" + score = evaluator_with_comm_response._evaluate_communication( + "Research task", "[agent1]: Hello\n[agent2]: Hi" + ) + assert score == 4.0 + + def test_evaluate_communication_on_error(self): + """_evaluate_communication should return -1 on error.""" + from unittest.mock import MagicMock + + # Create mock that raises exception on generate + mock_adapter = MagicMock() + mock_adapter.generate.side_effect = RuntimeError("Model failed") + + evaluator = MultiAgentBenchEvaluator( + domain="research", + model_adapter=mock_adapter, + ) + + score = evaluator._evaluate_communication("Task", "Comms") + assert score == -1.0 + + def test_call_with_communications(self, evaluator_with_comm_response): + """__call__ should evaluate communications when present.""" + traces = { + "agents": { + "agent1": { + "token_usage": 100, + "action_log": [], + "communication_log": [{"communication": "Hello from agent1"}], + } + }, + "environment": {}, + } + final_answer = "Result" + + result = evaluator_with_comm_response(traces, final_answer) + + # Communication score should be evaluated (not -1) + assert result["metrics"]["communication_score"] == 4.0 + + +class TestExceptionHandling: + """Tests for exception handling in evaluation methods.""" + + @pytest.fixture + def failing_evaluator(self): + """Create evaluator with adapter that fails.""" + from unittest.mock import MagicMock + + # Create mock that raises exception on generate + mock_adapter = MagicMock() + mock_adapter.generate.side_effect = RuntimeError("Model failed") + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=mock_adapter, + ) + + def test_evaluate_research_on_error(self, failing_evaluator): + """_evaluate_research should return default values on error.""" + result = failing_evaluator._evaluate_research("Task", "Result") + assert result["innovation"] == -1 + assert result["safety"] == -1 + assert result["feasibility"] == -1 + + def test_evaluate_bargaining_on_error(self, failing_evaluator): + """_evaluate_bargaining should return default values on error.""" + failing_evaluator.domain = "bargaining" + result = failing_evaluator._evaluate_bargaining("Task", "Result") + assert result["buyer"]["effectiveness_of_strategies"] == -1 + assert result["seller"]["effectiveness_of_strategies"] == -1 + + def test_evaluate_coding_on_error(self, failing_evaluator): + """_evaluate_coding should return default values on error.""" + failing_evaluator.domain = "coding" + result = failing_evaluator._evaluate_coding("Task", "Result") + assert result["instruction_following"] == -1 + assert result["executability"] == -1 + + +class TestParsingEdgeCases: + """Tests for parsing edge cases.""" + + @pytest.fixture + def evaluator(self): + """Create evaluator for parsing tests.""" + adapter = DummyModelAdapter(model_id="test", responses=['{"rating": 4}']) + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + + def test_parse_score_out_of_range(self, evaluator): + """_parse_score should reject scores outside 1-5 range.""" + response = '{"rating": 10}' + score = evaluator._parse_score(response) + # Should fall back to finding digit or default + assert score == 3.0 # Default + + def test_parse_score_with_just_code_block(self, evaluator): + """_parse_score should handle code block without json marker.""" + response = '```\n{"rating": 3}\n```' + score = evaluator._parse_score(response) + assert score == 3.0 + + def test_parse_score_with_text_before_json(self, evaluator): + """_parse_score should find JSON in text.""" + response = 'Here is my rating: {"rating": 5}' + score = evaluator._parse_score(response) + assert score == 5.0 + + def test_parse_bargaining_ratings_partial(self, evaluator): + """_parse_bargaining_ratings should handle partial ratings.""" + response = '{"effectiveness_of_strategies": 4}' # Missing other fields + ratings = evaluator._parse_bargaining_ratings(response) + assert ratings["effectiveness_of_strategies"] == 4 + assert ratings["progress_and_outcome"] == -1 + + def test_parse_bargaining_ratings_invalid(self, evaluator): + """_parse_bargaining_ratings should return -1 for invalid.""" + ratings = evaluator._parse_bargaining_ratings("Invalid") + assert ratings["effectiveness_of_strategies"] == -1 + + +class TestFormatFinalAnswerEdgeCases: + """Tests for _format_final_answer edge cases.""" + + @pytest.fixture + def evaluator(self): + """Create evaluator for format tests.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + + def test_format_dict_without_agent_results(self, evaluator): + """_format_final_answer should JSON dump dict without agent_results.""" + final_answer = {"status": "completed", "data": "some data"} + formatted = evaluator._format_final_answer(final_answer) + assert "status" in formatted + assert "completed" in formatted + + def test_format_dict_with_empty_agent_results(self, evaluator): + """_format_final_answer should handle empty agent_results.""" + final_answer = {"agent_results": []} + formatted = evaluator._format_final_answer(final_answer) + # Empty results should JSON dump the dict + assert formatted == '{"agent_results": []}' + + def test_format_list_with_non_dict_items(self, evaluator): + """_format_final_answer should skip non-dict items in list.""" + final_answer = [ + {"agent_id": "agent1", "result": "Result 1"}, + "not a dict", + {"agent_id": "agent2", "result": "Result 2"}, + ] + formatted = evaluator._format_final_answer(final_answer) + assert "[agent1]: Result 1" in formatted + assert "[agent2]: Result 2" in formatted + assert "not a dict" not in formatted + + +class TestTokenConsumptionEdgeCases: + """Tests for token consumption calculation edge cases.""" + + @pytest.fixture + def evaluator(self): + """Create evaluator for token tests.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + + def test_token_consumption_with_non_int(self, evaluator): + """_calculate_token_consumption should skip non-int values.""" + traces = { + "agents": { + "agent1": {"token_usage": 500}, + "agent2": {"token_usage": "invalid"}, # Non-int + "agent3": {"token_usage": 300}, + } + } + total = evaluator._calculate_token_consumption(traces) + assert total == 800 # Only int values counted + + def test_token_consumption_empty_agents(self, evaluator): + """_calculate_token_consumption should return 0 for no agents.""" + traces = {"agents": {}} + total = evaluator._calculate_token_consumption(traces) + assert total == 0 + + def test_token_consumption_missing_key(self, evaluator): + """_calculate_token_consumption should handle missing token_usage.""" + traces = { + "agents": { + "agent1": {"action_log": []}, # No token_usage + } + } + total = evaluator._calculate_token_consumption(traces) + assert total == 0 + + +class TestGetTaskDescription: + """Tests for _get_task_description method.""" + + @pytest.fixture + def evaluator(self): + """Create evaluator for task description tests.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + + def test_get_task_description_from_traces(self, evaluator): + """_get_task_description should extract from traces.""" + traces = { + "environment": { + "marble_state": {"task_description": "Research AI safety"} + } + } + desc = evaluator._get_task_description(traces) + assert desc == "Research AI safety" + + def test_get_task_description_missing_env(self, evaluator): + """_get_task_description should return empty for missing env.""" + traces = {} + desc = evaluator._get_task_description(traces) + assert desc == "" + + def test_get_task_description_missing_state(self, evaluator): + """_get_task_description should return empty for missing marble_state.""" + traces = {"environment": {}} + desc = evaluator._get_task_description(traces) + assert desc == "" + + +class TestExtractResultsEdgeCases: + """Tests for _extract_results edge cases.""" + + @pytest.fixture + def evaluator(self): + """Create evaluator for extract results tests.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + + def test_extract_results_empty_result(self, evaluator): + """_extract_results should skip empty results.""" + traces = { + "agents": { + "agent1": {"action_log": [{"result": ""}]}, + } + } + results = evaluator._extract_results(traces) + assert results == "No results recorded." + + def test_extract_results_multiple_actions(self, evaluator): + """_extract_results should include all actions.""" + traces = { + "agents": { + "agent1": { + "action_log": [ + {"result": "Step 1 done"}, + {"result": "Step 2 done"}, + ] + } + } + } + results = evaluator._extract_results(traces) + assert "Step 1 done" in results + assert "Step 2 done" in results + + +class TestFilterTracesEdgeCases: + """Tests for filter_traces edge cases.""" + + @pytest.fixture + def evaluator(self): + """Create evaluator for filter traces tests.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + + def test_filter_traces_empty(self, evaluator): + """filter_traces should handle empty traces.""" + filtered = evaluator.filter_traces({}) + assert filtered["agents"] == {} + assert filtered["environment"] == {} + assert "communications" in filtered + assert "results" in filtered + + def test_filter_traces_extracts_all(self, evaluator): + """filter_traces should extract all relevant data.""" + traces = { + "agents": { + "agent1": { + "action_log": [{"result": "Done"}], + "communication_log": [{"communication": "Hello"}], + } + }, + "environment": {"domain": "research"}, + } + filtered = evaluator.filter_traces(traces) + assert "[agent1]: Hello" in filtered["communications"] + assert "[agent1]: Done" in filtered["results"] + + +class TestDetermineCompletionEdgeCases: + """Tests for _determine_completion edge cases.""" + + def test_completion_empty_eval_data(self): + """_determine_completion should return False for empty eval data.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + evaluator = MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + metrics = MultiAgentBenchMetrics(task_evaluation={}) + assert evaluator._determine_completion(metrics) is False + + def test_completion_bargaining_partial_buyer(self): + """_determine_completion should return False if buyer has negative scores.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + evaluator = MultiAgentBenchEvaluator( + domain="bargaining", + model_adapter=adapter, + ) + metrics = MultiAgentBenchMetrics( + task_evaluation={ + "buyer": { + "effectiveness_of_strategies": -1, + "progress_and_outcome": 4, + "interaction_dynamics": 4, + }, + "seller": { + "effectiveness_of_strategies": 4, + "progress_and_outcome": 4, + "interaction_dynamics": 4, + }, + } + ) + assert evaluator._determine_completion(metrics) is False + + def test_completion_bargaining_partial_seller(self): + """_determine_completion should return False if seller has negative scores.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + evaluator = MultiAgentBenchEvaluator( + domain="bargaining", + model_adapter=adapter, + ) + metrics = MultiAgentBenchMetrics( + task_evaluation={ + "buyer": { + "effectiveness_of_strategies": 4, + "progress_and_outcome": 4, + "interaction_dynamics": 4, + }, + "seller": { + "effectiveness_of_strategies": -1, + "progress_and_outcome": 4, + "interaction_dynamics": 4, + }, + } + ) + assert evaluator._determine_completion(metrics) is False diff --git a/tests/test_benchmarks/test_multiagentbench/test_marble_adapter.py b/tests/test_benchmarks/test_multiagentbench/test_marble_adapter.py new file mode 100644 index 00000000..d5661c84 --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/test_marble_adapter.py @@ -0,0 +1,208 @@ +"""Tests for MarbleAgentAdapter.""" + +import pytest +from typing import Any, Dict +from unittest.mock import MagicMock, PropertyMock + +from maseval.benchmark.multiagentbench.adapters.marble_adapter import ( + MarbleAgentAdapter, +) +from maseval import AgentError + + +class TestMarbleAgentAdapter: + """Tests for MarbleAgentAdapter class.""" + + @pytest.fixture + def mock_marble_agent(self): + """Create a mock MARBLE agent.""" + agent = MagicMock() + agent.profile = "Test researcher profile" + agent.strategy = "cot" + agent.llm = "gpt-4o" + agent.relationships = {"agent2": "collaborator"} + agent.task_history = ["task1", "task2"] + agent.session_id = "session_123" + agent.act.return_value = ("Test result", "Communication message") + agent.get_token_usage.return_value = 500 + agent.seralize_message.return_value = "Serialized messages" + return agent + + @pytest.fixture + def adapter(self, mock_marble_agent): + """Create a MarbleAgentAdapter.""" + return MarbleAgentAdapter( + marble_agent=mock_marble_agent, + agent_id="test_agent", + ) + + def test_init(self, adapter, mock_marble_agent): + """Adapter should initialize correctly.""" + assert adapter.agent_id == "test_agent" + assert adapter.profile == "Test researcher profile" + assert adapter.marble_agent is mock_marble_agent + + def test_agent_id_property(self, adapter): + """agent_id property should return correct value.""" + assert adapter.agent_id == "test_agent" + + def test_profile_property(self, adapter): + """profile property should return correct value.""" + assert adapter.profile == "Test researcher profile" + + def test_marble_agent_property(self, adapter, mock_marble_agent): + """marble_agent property should return the underlying agent.""" + assert adapter.marble_agent is mock_marble_agent + + def test_run_agent_success(self, adapter, mock_marble_agent): + """_run_agent should execute MARBLE agent's act method.""" + result = adapter._run_agent("Test query") + + mock_marble_agent.act.assert_called_once_with("Test query") + assert result == "Test result" + + def test_run_agent_logs_action(self, adapter, mock_marble_agent): + """_run_agent should log actions.""" + adapter._run_agent("Test query") + + assert len(adapter._action_log) == 1 + assert adapter._action_log[0]["task"] == "Test query" + assert adapter._action_log[0]["result"] == "Test result" + assert adapter._action_log[0]["has_communication"] is True + + def test_run_agent_logs_communication(self, adapter, mock_marble_agent): + """_run_agent should log communications when present.""" + adapter._run_agent("Test query") + + assert len(adapter._communication_log) == 1 + assert adapter._communication_log[0]["communication"] == "Communication message" + + def test_run_agent_no_communication(self, adapter, mock_marble_agent): + """_run_agent should handle None communication.""" + mock_marble_agent.act.return_value = ("Result", None) + adapter._run_agent("Test query") + + assert len(adapter._communication_log) == 0 + assert adapter._action_log[0]["has_communication"] is False + + def test_run_agent_updates_messages(self, adapter, mock_marble_agent): + """_run_agent should update message history.""" + adapter._run_agent("Test query") + + messages = adapter.get_messages() + assert len(messages) == 2 + assert messages[0]["role"] == "user" + assert messages[0]["content"] == "Test query" + assert messages[1]["role"] == "assistant" + assert messages[1]["content"] == "Test result" + + def test_run_agent_error(self, adapter, mock_marble_agent): + """_run_agent should wrap exceptions in AgentError.""" + mock_marble_agent.act.side_effect = RuntimeError("Agent failed") + + with pytest.raises(AgentError, match="MARBLE agent 'test_agent' failed"): + adapter._run_agent("Test query") + + def test_get_token_usage(self, adapter, mock_marble_agent): + """get_token_usage should return token count from MARBLE agent.""" + assert adapter.get_token_usage() == 500 + mock_marble_agent.get_token_usage.assert_called_once() + + def test_get_token_usage_no_method(self, adapter, mock_marble_agent): + """get_token_usage should return 0 if method not available.""" + del mock_marble_agent.get_token_usage + assert adapter.get_token_usage() == 0 + + def test_get_memory_str(self, adapter, mock_marble_agent): + """get_memory_str should return memory string.""" + mock_memory = MagicMock() + mock_memory.get_memory_str.return_value = "Memory contents" + mock_marble_agent.memory = mock_memory + + assert adapter.get_memory_str() == "Memory contents" + + def test_get_memory_str_no_memory(self, adapter, mock_marble_agent): + """get_memory_str should return empty string if no memory.""" + mock_marble_agent.memory = None + assert adapter.get_memory_str() == "" + + def test_get_memory_str_no_method(self, adapter, mock_marble_agent): + """get_memory_str should return empty string if no get_memory_str method.""" + mock_marble_agent.memory = MagicMock(spec=[]) # No methods + assert adapter.get_memory_str() == "" + + def test_get_serialized_messages(self, adapter, mock_marble_agent): + """get_serialized_messages should return serialized messages.""" + result = adapter.get_serialized_messages("session_1") + + mock_marble_agent.seralize_message.assert_called_once_with("session_1") + assert result == "Serialized messages" + + def test_get_serialized_messages_no_method(self, adapter, mock_marble_agent): + """get_serialized_messages should return empty string if no method.""" + del mock_marble_agent.seralize_message + assert adapter.get_serialized_messages() == "" + + def test_gather_traces(self, adapter, mock_marble_agent): + """gather_traces should include MARBLE-specific data.""" + # Run agent to generate some data + adapter._run_agent("Test query") + + traces = adapter.gather_traces() + + assert traces["agent_id"] == "test_agent" + assert traces["profile"] == "Test researcher profile" + assert traces["token_usage"] == 500 + assert len(traces["action_log"]) == 1 + assert len(traces["communication_log"]) == 1 + assert traces["relationships"] == {"agent2": "collaborator"} + assert traces["task_history"] == ["task1", "task2"] + + def test_gather_traces_no_relationships(self, adapter, mock_marble_agent): + """gather_traces should handle missing relationships.""" + del mock_marble_agent.relationships + traces = adapter.gather_traces() + assert traces["relationships"] == {} + + def test_gather_traces_no_task_history(self, adapter, mock_marble_agent): + """gather_traces should handle missing task_history.""" + del mock_marble_agent.task_history + traces = adapter.gather_traces() + assert traces["task_history"] == [] + + def test_gather_config(self, adapter, mock_marble_agent): + """gather_config should include MARBLE configuration.""" + config = adapter.gather_config() + + assert config["agent_id"] == "test_agent" + assert config["profile"] == "Test researcher profile" + assert config["strategy"] == "cot" + assert config["llm"] == "gpt-4o" + + def test_gather_config_missing_attrs(self, adapter, mock_marble_agent): + """gather_config should handle missing attributes.""" + del mock_marble_agent.strategy + del mock_marble_agent.llm + + config = adapter.gather_config() + + assert config["strategy"] == "default" + assert config["llm"] == "unknown" + + +class TestCreateMarbleAgentsImportError: + """Tests for create_marble_agents when MARBLE is not available.""" + + def test_create_marble_agents_import_error(self): + """create_marble_agents should raise ImportError when MARBLE not available.""" + from maseval.benchmark.multiagentbench.adapters.marble_adapter import ( + create_marble_agents, + ) + + # This will fail because MARBLE is not installed + with pytest.raises(ImportError, match="MARBLE is not available"): + create_marble_agents( + agent_configs=[{"agent_id": "test"}], + marble_env=MagicMock(), + model="gpt-4o", + ) From 48a733d365cf1ee5317c01f2e900493f2a8d7ac9 Mon Sep 17 00:00:00 2001 From: cemde Date: Wed, 4 Feb 2026 12:15:10 +0100 Subject: [PATCH 09/17] updated seeding --- CHANGELOG.md | 9 + .../multiagentbench/multiagentbench.py | 95 +++++++++-- .../test_multiagentbench/conftest.py | 1 + .../test_multiagentbench/test_benchmark.py | 154 +++++++++++++++++- .../test_multiagentbench/test_data_loader.py | 4 +- .../test_multiagentbench/test_evaluator.py | 18 +- .../test_marble_adapter.py | 3 +- 7 files changed, 250 insertions(+), 34 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f4e9310..f1be553c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Added `SeedingError` exception for providers that don't support seeding (Anthropic models raise this if seed is provided) (PR: #24) - Added seed support to interface adapters: `OpenAIModelAdapter`, `GoogleGenAIModelAdapter`, `LiteLLMModelAdapter`, `HuggingFaceModelAdapter` pass seeds to underlying APIs (PR: #24) +**Benchmarks** + +- MultiAgentBench Benchmark: Integration with MARBLE MultiAgentBench for evaluating multi-agent collaboration across research, bargaining, coding, and database domains (PR: #25) +- `MultiAgentBenchBenchmark` abstract base class for framework-agnostic multi-agent evaluation with seeding support for evaluators and agents (PR: #25) +- `MarbleMultiAgentBenchBenchmark` for exact MARBLE reproduction mode using native MARBLE agents (note: MARBLE's internal LLM calls bypass MASEval seeding) (PR: #25) +- `MultiAgentBenchEnvironment` and `MultiAgentBenchEvaluator` components (PR: #25) +- Data loading utilities: `load_tasks()`, `configure_model_ids()`, `list_scenarios()` (PR: #25) +- MARBLE adapter: `MarbleAgentAdapter` for wrapping MARBLE agents with MASEval tracing (PR: #25) + **Interface** - CAMEL-AI integration: `CamelAgentAdapter` and `CamelLLMUser` for evaluating CAMEL-AI ChatAgent-based systems (PR: #22) diff --git a/maseval/benchmark/multiagentbench/multiagentbench.py b/maseval/benchmark/multiagentbench/multiagentbench.py index c15a11c9..104f0235 100644 --- a/maseval/benchmark/multiagentbench/multiagentbench.py +++ b/maseval/benchmark/multiagentbench/multiagentbench.py @@ -41,18 +41,38 @@ class MultiAgentBenchBenchmark(Benchmark): Example: ```python class MyMultiAgentBenchmark(MultiAgentBenchBenchmark): - def setup_agents(self, agent_data, environment, task, user): - # Create agents using your framework - ... + def setup_agents(self, agent_data, environment, task, user, seed_generator=None): + # Derive seeds for agents if seeding is enabled + agent_seeds = {} + if seed_generator is not None: + agents_gen = seed_generator.child("agents") + for config in task.environment_data.get("agents", []): + agent_id = config.get("agent_id") + agent_seeds[agent_id] = agents_gen.derive_seed(agent_id) + + # Create agents using your framework with seeds + agents_list = [] + agents_dict = {} + for config in task.environment_data.get("agents", []): + agent_id = config.get("agent_id") + model = self.get_model_adapter( + agent_data.get("model_id", "gpt-4o"), + register_name=f"agent_{agent_id}", + seed=agent_seeds.get(agent_id), + ) + # Create your agent with the seeded model... + ... + return agents_list, agents_dict def get_model_adapter(self, model_id, **kwargs): - adapter = MyModelAdapter(model_id) + seed = kwargs.pop("seed", None) + adapter = MyModelAdapter(model_id, seed=seed) if "register_name" in kwargs: self.register("models", kwargs["register_name"], adapter) return adapter - benchmark = MyMultiAgentBenchmark() - results = benchmark.run(tasks, agent_data={}) + benchmark = MyMultiAgentBenchmark(seed=42) # Enable seeding + results = benchmark.run(tasks, agent_data={"model_id": "gpt-4o"}) ``` """ @@ -66,6 +86,8 @@ def __init__( fail_on_task_error: bool = False, fail_on_evaluation_error: bool = False, progress_bar: bool | str = True, + seed: Optional[int] = None, + seed_generator=None, ): """Initialize the benchmark. @@ -78,6 +100,8 @@ def __init__( fail_on_task_error: Raise on task errors fail_on_evaluation_error: Raise on evaluation errors progress_bar: Progress bar configuration + seed: Global seed for reproducible benchmark runs + seed_generator: Custom seed generator (takes precedence over seed) """ super().__init__( callbacks=callbacks, @@ -88,14 +112,22 @@ def __init__( fail_on_task_error=fail_on_task_error, fail_on_evaluation_error=fail_on_evaluation_error, progress_bar=progress_bar, + seed=seed, + seed_generator=seed_generator, ) - def setup_environment(self, agent_data: Dict[str, Any], task: Task) -> Environment: + def setup_environment( + self, + agent_data: Dict[str, Any], + task: Task, + seed_generator=None, + ) -> Environment: """Create the MultiAgentBench environment. Args: agent_data: Agent configuration task: The task to set up + seed_generator: Optional seed generator for reproducibility Returns: MultiAgentBenchEnvironment instance @@ -110,11 +142,18 @@ def setup_user( agent_data: Dict[str, Any], environment: Environment, task: Task, + seed_generator=None, ) -> Optional[User]: """MultiAgentBench tasks don't use user simulators. The multi-agent coordination replaces user interaction. + Args: + agent_data: Agent configuration + environment: The environment instance + task: The task + seed_generator: Optional seed generator (unused) + Returns: None """ @@ -127,23 +166,40 @@ def setup_agents( environment: Environment, task: Task, user: Optional[User], + seed_generator=None, ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: """Create agents for the task (implement in subclass). Subclasses should: 1. Read agent specifications from task.environment_data["agents"] - 2. Create agents using their framework - 3. Wrap them in AgentAdapter - 4. Set up relationships from task.environment_data["relationships"] + 2. Derive seeds from seed_generator for each agent's model + 3. Create agents using their framework with seeded models + 4. Wrap them in AgentAdapter + 5. Set up relationships from task.environment_data["relationships"] Args: agent_data: Agent configuration (model IDs, etc.) environment: The environment instance task: The task containing agent specs user: User simulator (None for MultiAgentBench) + seed_generator: Optional seed generator for deriving deterministic seeds. + Use `seed_generator.child("agents")` to create a namespace, then + `derive_seed(agent_id)` for each agent's model. Returns: Tuple of (agents_to_run, agents_dict) + + Example: + ```python + def setup_agents(self, agent_data, environment, task, user, seed_generator=None): + agents_gen = seed_generator.child("agents") if seed_generator else None + + for config in task.environment_data.get("agents", []): + agent_id = config.get("agent_id") + seed = agents_gen.derive_seed(agent_id) if agents_gen else None + model = self.get_model_adapter(model_id, seed=seed) + # Create agent with seeded model... + ``` """ pass @@ -153,6 +209,7 @@ def setup_evaluators( task: Task, agents: Sequence[AgentAdapter], user: Optional[User], + seed_generator=None, ) -> Sequence[Evaluator]: """Create evaluators for the task. @@ -161,6 +218,7 @@ def setup_evaluators( task: The task with evaluation data agents: The agents user: User simulator (None for MultiAgentBench) + seed_generator: Optional seed generator for reproducibility Returns: List of evaluators @@ -168,10 +226,17 @@ def setup_evaluators( # Get evaluation model ID from task or default eval_model_id = task.evaluation_data.get("model_id", "gpt-4o-mini") + # Derive seed for evaluator model if seeding is enabled + evaluator_seed = None + if seed_generator is not None: + eval_gen = seed_generator.child("evaluators") + evaluator_seed = eval_gen.derive_seed("multiagentbench_evaluator") + # Create model adapter for evaluation model_adapter = self.get_model_adapter( eval_model_id, register_name="evaluator_model", + seed=evaluator_seed, ) # Get domain-specific evaluation configuration @@ -300,14 +365,24 @@ def setup_agents( environment: Environment, task: Task, user: Optional[User], + seed_generator=None, ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: """Create MARBLE agents wrapped in MASEval adapters. + Note: + MARBLE agents use their own internal LLM handling with a model ID string, + not MASEval's ModelAdapter. This means seed_generator cannot be applied + to agent LLM calls in this implementation. For reproducible agent behavior, + use `MultiAgentBenchBenchmark` with a custom `setup_agents` that creates + agents using seeded MASEval ModelAdapters. + Args: agent_data: Agent configuration environment: The environment task: The task with agent specifications user: User simulator (None) + seed_generator: Optional seed generator (not used for MARBLE agents, + but seeding is applied to evaluators) Returns: Tuple of (agents_to_run, agents_dict) diff --git a/tests/test_benchmarks/test_multiagentbench/conftest.py b/tests/test_benchmarks/test_multiagentbench/conftest.py index c3a73968..5d875ef6 100644 --- a/tests/test_benchmarks/test_multiagentbench/conftest.py +++ b/tests/test_benchmarks/test_multiagentbench/conftest.py @@ -251,6 +251,7 @@ def setup_agents( environment: Any, task: Task, user: Optional[Any], + seed_generator=None, ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: agent_configs = task.environment_data.get("agents", []) agents_list: List[AgentAdapter] = [] diff --git a/tests/test_benchmarks/test_multiagentbench/test_benchmark.py b/tests/test_benchmarks/test_multiagentbench/test_benchmark.py index 9b7d1e83..f6ad0290 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_benchmark.py +++ b/tests/test_benchmarks/test_multiagentbench/test_benchmark.py @@ -1,12 +1,10 @@ """Tests for MultiAgentBench benchmark classes.""" import pytest -from typing import Any, Dict -from unittest.mock import MagicMock, patch +from unittest.mock import MagicMock from maseval import Task from maseval.benchmark.multiagentbench import ( - MultiAgentBenchBenchmark, MarbleMultiAgentBenchBenchmark, MultiAgentBenchEnvironment, MultiAgentBenchEvaluator, @@ -50,6 +48,153 @@ def test_setup_evaluators_returns_evaluator( assert len(evaluators) == 1 assert isinstance(evaluators[0], MultiAgentBenchEvaluator) + def test_setup_evaluators_uses_seed_generator( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """setup_evaluators should pass seed to model adapter when seed_generator provided.""" + from unittest.mock import patch + from maseval.core.seeding import DefaultSeedGenerator + + # Create a benchmark and mock get_model_adapter + benchmark = concrete_multiagentbench_benchmark(progress_bar=False) + env = benchmark.setup_environment({}, sample_research_task) + agents, _ = benchmark.setup_agents({}, env, sample_research_task, None) + + # Create a seed generator + seed_gen = DefaultSeedGenerator(global_seed=42) + task_seed_gen = seed_gen.for_task("test_task").for_repetition(0) + + # Patch get_model_adapter to capture the seed argument + with patch.object(benchmark, "get_model_adapter", wraps=benchmark.get_model_adapter) as mock_get_model: + benchmark.setup_evaluators(env, sample_research_task, agents, None, seed_generator=task_seed_gen) + + # Verify get_model_adapter was called with a seed + mock_get_model.assert_called_once() + call_kwargs = mock_get_model.call_args.kwargs + assert "seed" in call_kwargs + assert call_kwargs["seed"] is not None + assert isinstance(call_kwargs["seed"], int) + + def test_setup_evaluators_no_seed_without_generator( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """setup_evaluators should pass None seed when no seed_generator provided.""" + from unittest.mock import patch + + benchmark = concrete_multiagentbench_benchmark(progress_bar=False) + env = benchmark.setup_environment({}, sample_research_task) + agents, _ = benchmark.setup_agents({}, env, sample_research_task, None) + + with patch.object(benchmark, "get_model_adapter", wraps=benchmark.get_model_adapter) as mock_get_model: + benchmark.setup_evaluators(env, sample_research_task, agents, None, seed_generator=None) + + mock_get_model.assert_called_once() + call_kwargs = mock_get_model.call_args.kwargs + assert call_kwargs.get("seed") is None + + def test_setup_evaluators_seed_is_deterministic( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """Same global seed should produce same derived seed.""" + from unittest.mock import patch + from maseval.core.seeding import DefaultSeedGenerator + + seeds_collected = [] + + for _ in range(2): + benchmark = concrete_multiagentbench_benchmark(progress_bar=False) + env = benchmark.setup_environment({}, sample_research_task) + agents, _ = benchmark.setup_agents({}, env, sample_research_task, None) + + # Same global seed each time + seed_gen = DefaultSeedGenerator(global_seed=42) + task_seed_gen = seed_gen.for_task("test_task").for_repetition(0) + + with patch.object(benchmark, "get_model_adapter", wraps=benchmark.get_model_adapter) as mock_get_model: + benchmark.setup_evaluators(env, sample_research_task, agents, None, seed_generator=task_seed_gen) + seeds_collected.append(mock_get_model.call_args.kwargs["seed"]) + + # Same seed both times + assert seeds_collected[0] == seeds_collected[1] + + def test_setup_evaluators_different_global_seeds_produce_different_seeds( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """Different global seeds should produce different derived seeds.""" + from unittest.mock import patch + from maseval.core.seeding import DefaultSeedGenerator + + seeds_collected = [] + + for global_seed in [42, 123]: + benchmark = concrete_multiagentbench_benchmark(progress_bar=False) + env = benchmark.setup_environment({}, sample_research_task) + agents, _ = benchmark.setup_agents({}, env, sample_research_task, None) + + seed_gen = DefaultSeedGenerator(global_seed=global_seed) + task_seed_gen = seed_gen.for_task("test_task").for_repetition(0) + + with patch.object(benchmark, "get_model_adapter", wraps=benchmark.get_model_adapter) as mock_get_model: + benchmark.setup_evaluators(env, sample_research_task, agents, None, seed_generator=task_seed_gen) + seeds_collected.append(mock_get_model.call_args.kwargs["seed"]) + + # Different seeds + assert seeds_collected[0] != seeds_collected[1] + + def test_benchmark_run_with_seed_passes_seed_to_evaluators( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """Full benchmark.run() with seed should pass seed through to evaluators.""" + from unittest.mock import patch + + benchmark = concrete_multiagentbench_benchmark(progress_bar=False, seed=42) + + seeds_passed = [] + original_get_model_adapter = benchmark.get_model_adapter + + def capturing_get_model_adapter(model_id, **kwargs): + seeds_passed.append(kwargs.get("seed")) + return original_get_model_adapter(model_id, **kwargs) + + with patch.object(benchmark, "get_model_adapter", side_effect=capturing_get_model_adapter): + benchmark.run(tasks=[sample_research_task], agent_data={}) + + # At least one call should have a seed (the evaluator) + assert any(s is not None for s in seeds_passed), "Expected at least one seeded model adapter call" + + def test_benchmark_run_without_seed_passes_none( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """Full benchmark.run() without seed should pass None to evaluators.""" + from unittest.mock import patch + + benchmark = concrete_multiagentbench_benchmark(progress_bar=False) # No seed + + seeds_passed = [] + original_get_model_adapter = benchmark.get_model_adapter + + def capturing_get_model_adapter(model_id, **kwargs): + seeds_passed.append(kwargs.get("seed")) + return original_get_model_adapter(model_id, **kwargs) + + with patch.object(benchmark, "get_model_adapter", side_effect=capturing_get_model_adapter): + benchmark.run(tasks=[sample_research_task], agent_data={}) + + # All calls should have None seed + assert all(s is None for s in seeds_passed), "Expected all model adapter calls to have None seed" + def test_setup_agents_creates_correct_count( self, benchmark_instance, @@ -184,7 +329,7 @@ def test_run_handles_setup_error( """Benchmark should handle setup errors gracefully.""" class FailingBenchmark(concrete_multiagentbench_benchmark): - def setup_environment(self, agent_data, task): + def setup_environment(self, agent_data, task, seed_generator=None): raise RuntimeError("Setup failed") benchmark = FailingBenchmark(progress_bar=False) @@ -321,7 +466,6 @@ def test_run_agents_returns_structured_output( sample_research_task: Task, ): """run_agents should return structured output with agent_results.""" - from conftest import DummyAgentAdapter benchmark = marble_benchmark_class(progress_bar=False) env = benchmark.setup_environment({}, sample_research_task) diff --git a/tests/test_benchmarks/test_multiagentbench/test_data_loader.py b/tests/test_benchmarks/test_multiagentbench/test_data_loader.py index 67eab09b..6bf11502 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_data_loader.py +++ b/tests/test_benchmarks/test_multiagentbench/test_data_loader.py @@ -411,9 +411,7 @@ def test_download_marble_clone_fails(self): marble_dir = Path(tmpdir) / "marble" with patch("subprocess.run") as mock_run: - mock_run.side_effect = subprocess.CalledProcessError( - 1, "git clone", stderr="Clone failed" - ) + mock_run.side_effect = subprocess.CalledProcessError(1, "git clone", stderr="Clone failed") with pytest.raises(RuntimeError, match="Failed to clone MARBLE"): download_marble(target_dir=marble_dir) diff --git a/tests/test_benchmarks/test_multiagentbench/test_evaluator.py b/tests/test_benchmarks/test_multiagentbench/test_evaluator.py index 31432552..d5729bf5 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_evaluator.py +++ b/tests/test_benchmarks/test_multiagentbench/test_evaluator.py @@ -365,16 +365,12 @@ def test_evaluate_database_returns_structure(self, database_evaluator): def test_determine_completion_database_with_prediction(self, database_evaluator): """_determine_completion should return True for database with prediction.""" - metrics = MultiAgentBenchMetrics( - task_evaluation={"predicted": "SELECT * FROM users", "root_cause": []} - ) + metrics = MultiAgentBenchMetrics(task_evaluation={"predicted": "SELECT * FROM users", "root_cause": []}) assert database_evaluator._determine_completion(metrics) is True def test_determine_completion_database_empty_prediction(self, database_evaluator): """_determine_completion should return False for empty prediction.""" - metrics = MultiAgentBenchMetrics( - task_evaluation={"predicted": "", "root_cause": []} - ) + metrics = MultiAgentBenchMetrics(task_evaluation={"predicted": "", "root_cause": []}) assert database_evaluator._determine_completion(metrics) is False def test_call_database_domain(self, database_evaluator): @@ -494,9 +490,7 @@ def evaluator_with_comm_response(self): def test_evaluate_communication_returns_score(self, evaluator_with_comm_response): """_evaluate_communication should return score from LLM.""" - score = evaluator_with_comm_response._evaluate_communication( - "Research task", "[agent1]: Hello\n[agent2]: Hi" - ) + score = evaluator_with_comm_response._evaluate_communication("Research task", "[agent1]: Hello\n[agent2]: Hi") assert score == 4.0 def test_evaluate_communication_on_error(self): @@ -711,11 +705,7 @@ def evaluator(self): def test_get_task_description_from_traces(self, evaluator): """_get_task_description should extract from traces.""" - traces = { - "environment": { - "marble_state": {"task_description": "Research AI safety"} - } - } + traces = {"environment": {"marble_state": {"task_description": "Research AI safety"}}} desc = evaluator._get_task_description(traces) assert desc == "Research AI safety" diff --git a/tests/test_benchmarks/test_multiagentbench/test_marble_adapter.py b/tests/test_benchmarks/test_multiagentbench/test_marble_adapter.py index d5661c84..93ef2535 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_marble_adapter.py +++ b/tests/test_benchmarks/test_multiagentbench/test_marble_adapter.py @@ -1,8 +1,7 @@ """Tests for MarbleAgentAdapter.""" import pytest -from typing import Any, Dict -from unittest.mock import MagicMock, PropertyMock +from unittest.mock import MagicMock from maseval.benchmark.multiagentbench.adapters.marble_adapter import ( MarbleAgentAdapter, From 98cfbd25d88b85a400a7c5842081a9dbe5aa06c3 Mon Sep 17 00:00:00 2001 From: cemde Date: Wed, 4 Feb 2026 13:22:38 +0100 Subject: [PATCH 10/17] code quality fixes --- .../benchmark/multiagentbench/_constants.py | 8 + .../adapters/marble_adapter.py | 26 ++- .../benchmark/multiagentbench/environment.py | 19 +-- .../benchmark/multiagentbench/evaluator.py | 149 ++++++++---------- .../multiagentbench/multiagentbench.py | 114 +++++--------- .../test_multiagentbench/test_benchmark.py | 20 ++- .../test_multiagentbench/test_data_loader.py | 14 +- .../test_multiagentbench/test_evaluator.py | 84 +++++----- 8 files changed, 194 insertions(+), 240 deletions(-) create mode 100644 maseval/benchmark/multiagentbench/_constants.py diff --git a/maseval/benchmark/multiagentbench/_constants.py b/maseval/benchmark/multiagentbench/_constants.py new file mode 100644 index 00000000..f852e5a6 --- /dev/null +++ b/maseval/benchmark/multiagentbench/_constants.py @@ -0,0 +1,8 @@ +"""Shared constants for MultiAgentBench module. + +This module exists to avoid circular imports between multiagentbench.py, +environment.py, and adapters/marble_adapter.py. +""" + +# Shared error message for MARBLE import failures +MARBLE_IMPORT_ERROR = "MARBLE is not available. Clone MARBLE to maseval/benchmark/multiagentbench/marble/\nOriginal error: {error}" diff --git a/maseval/benchmark/multiagentbench/adapters/marble_adapter.py b/maseval/benchmark/multiagentbench/adapters/marble_adapter.py index ee3d8c96..5d18ea02 100644 --- a/maseval/benchmark/multiagentbench/adapters/marble_adapter.py +++ b/maseval/benchmark/multiagentbench/adapters/marble_adapter.py @@ -4,12 +4,10 @@ MASEval's tracing and evaluation infrastructure. """ -from typing import TYPE_CHECKING, Any, Dict, List, Optional, Sequence, Tuple +from typing import Any, Dict, List, Sequence, Tuple -from maseval import AgentAdapter, AgentError - -if TYPE_CHECKING: - from maseval.core.callback import BenchmarkCallback +from maseval import AgentAdapter, AgentError, MessageHistory +from maseval.benchmark.multiagentbench._constants import MARBLE_IMPORT_ERROR class MarbleAgentAdapter(AgentAdapter): @@ -28,26 +26,21 @@ def __init__( self, marble_agent: Any, agent_id: str, - *, - callbacks: Optional[List["BenchmarkCallback"]] = None, ): """Initialize the adapter. Args: marble_agent: MARBLE BaseAgent instance agent_id: Unique identifier for this agent - callbacks: Optional list of callbacks """ self._marble_agent = marble_agent self._agent_id = agent_id self._profile = getattr(marble_agent, "profile", "") self._communication_log: List[Dict[str, Any]] = [] self._action_log: List[Dict[str, Any]] = [] - super().__init__(agent_instance=marble_agent, name=agent_id, callbacks=callbacks) + super().__init__(agent_instance=marble_agent, name=agent_id) # Initialize message history - from maseval import MessageHistory - - self.messages = MessageHistory() + self.messages: MessageHistory = MessageHistory() @property def agent_id(self) -> str: @@ -140,6 +133,7 @@ def get_serialized_messages(self, session_id: str = "") -> str: Returns: Serialized message string """ + # Note: "seralize_message" is misspelled in MARBLE's API (should be "serialize") if hasattr(self._marble_agent, "seralize_message"): return self._marble_agent.seralize_message(session_id) return "" @@ -186,7 +180,6 @@ def create_marble_agents( agent_configs: Sequence[Dict[str, Any]], marble_env: Any, model: str, - callbacks: Optional[List["BenchmarkCallback"]] = None, ) -> Tuple[List[MarbleAgentAdapter], Dict[str, MarbleAgentAdapter]]: """Create MarbleAgentAdapters from agent configurations. @@ -197,7 +190,6 @@ def create_marble_agents( agent_configs: List of agent configuration dicts from task data marble_env: MARBLE environment instance model: Model ID to use for agents - callbacks: Optional callbacks to attach Returns: Tuple of (agents_list, agents_dict) @@ -206,9 +198,9 @@ def create_marble_agents( ImportError: If MARBLE is not available """ try: - from ..marble.agent.base_agent import BaseAgent + from ..marble.agent.base_agent import BaseAgent # type: ignore[unresolved-import] except ImportError as e: - raise ImportError(f"MARBLE is not available. Clone MARBLE to maseval/benchmark/multiagentbench/marble/\nOriginal error: {e}") from e + raise ImportError(MARBLE_IMPORT_ERROR.format(error=e)) from e agents_list: List[MarbleAgentAdapter] = [] agents_dict: Dict[str, MarbleAgentAdapter] = {} @@ -220,7 +212,7 @@ def create_marble_agents( marble_agent = BaseAgent(config=config, env=marble_env, model=model) # Wrap in adapter - adapter = MarbleAgentAdapter(marble_agent=marble_agent, agent_id=agent_id, callbacks=callbacks) + adapter = MarbleAgentAdapter(marble_agent=marble_agent, agent_id=agent_id) agents_list.append(adapter) agents_dict[agent_id] = adapter diff --git a/maseval/benchmark/multiagentbench/environment.py b/maseval/benchmark/multiagentbench/environment.py index aec52ec7..5be86308 100644 --- a/maseval/benchmark/multiagentbench/environment.py +++ b/maseval/benchmark/multiagentbench/environment.py @@ -4,12 +4,10 @@ """ import shutil -from typing import TYPE_CHECKING, Any, Callable, Dict, List, Optional +from typing import Any, Callable, Dict, Optional from maseval import Environment, EnvironmentError, ToolInvocationHistory - -if TYPE_CHECKING: - from maseval.core.callback import BenchmarkCallback +from maseval.benchmark.multiagentbench._constants import MARBLE_IMPORT_ERROR # Domains requiring external infrastructure @@ -31,13 +29,11 @@ class MultiAgentBenchEnvironment(Environment): def __init__( self, task_data: Dict[str, Any], - callbacks: Optional[List["BenchmarkCallback"]] = None, ): """Initialize the environment. Args: task_data: Task data containing environment configuration - callbacks: Optional list of callbacks Raises: EnvironmentError: If required infrastructure is unavailable @@ -45,7 +41,7 @@ def __init__( self.domain = task_data.get("scenario", "") self._marble_env: Optional[Any] = None self._tool_histories: Dict[str, ToolInvocationHistory] = {} - super().__init__(task_data, callbacks) + super().__init__(task_data) def setup_state(self, task_data: Dict[str, Any]) -> Dict[str, Any]: """Initialize state and optionally create MARBLE environment. @@ -137,9 +133,9 @@ def _create_marble_environment( # Import MARBLE environments try: - from .marble.environments.base_env import BaseEnvironment + from .marble.environments.base_env import BaseEnvironment # type: ignore[unresolved-import] except ImportError as e: - raise ImportError(f"MARBLE is not available. Clone MARBLE to maseval/benchmark/multiagentbench/marble/\nOriginal error: {e}") from e + raise ImportError(MARBLE_IMPORT_ERROR.format(error=e)) from e # Map domains to environment classes env_mapping: Dict[str, str] = { @@ -338,9 +334,10 @@ def gather_traces(self) -> Dict[str, Any]: # Collect tool invocation histories tool_traces = {} for name, history in self._tool_histories.items(): + invocations = history.to_list() tool_traces[name] = { - "invocations": history.to_list(), - "invocation_count": len(history), + "invocations": invocations, + "invocation_count": len(invocations), } traces["tool_invocations"] = tool_traces diff --git a/maseval/benchmark/multiagentbench/evaluator.py b/maseval/benchmark/multiagentbench/evaluator.py index 61dd6039..0cba653b 100644 --- a/maseval/benchmark/multiagentbench/evaluator.py +++ b/maseval/benchmark/multiagentbench/evaluator.py @@ -4,12 +4,15 @@ """ import json -import re from dataclasses import dataclass, field from typing import Any, Dict, List, Optional from maseval import Evaluator, ModelAdapter +# Sentinel value for "evaluation not performed" or "evaluation failed" +# Using None instead of -1 to avoid confusion with valid scores +SCORE_NOT_EVALUATED: None = None + @dataclass class MultiAgentBenchMetrics: @@ -18,8 +21,7 @@ class MultiAgentBenchMetrics: Attributes: task_completion: Whether the task was completed token_consumption: Total tokens used - planning_score: Score for planning/coordination (1-5) - communication_score: Score for inter-agent communication (1-5) + communication_score: Score for inter-agent communication (1-5), None if not evaluated task_evaluation: Domain-specific evaluation results agent_kpis: Per-agent key performance indicators total_milestones: Number of milestones achieved @@ -27,8 +29,7 @@ class MultiAgentBenchMetrics: task_completion: bool = False token_consumption: int = 0 - planning_score: float = -1.0 - communication_score: float = -1.0 + communication_score: Optional[float] = None task_evaluation: Dict[str, Any] = field(default_factory=dict) agent_kpis: Dict[str, int] = field(default_factory=dict) total_milestones: int = 0 @@ -38,7 +39,6 @@ def to_dict(self) -> Dict[str, Any]: return { "task_completion": self.task_completion, "token_consumption": self.token_consumption, - "planning_score": self.planning_score, "communication_score": self.communication_score, "task_evaluation": self.task_evaluation, "agent_kpis": self.agent_kpis, @@ -101,23 +101,6 @@ def _load_evaluation_prompts(self) -> Dict[str, Any]: 4 - Good: Clear and relevant communication with good coordination 5 - Excellent: Highly effective communication with perfect coordination -Respond with a JSON object: {{"rating": }}""" - }, - "planning": { - "prompt": """Evaluate the planning and coordination quality for the following task. - -Task Summary: {summary} -Agent Profiles: {agent_profiles} -Task Assignments: {agent_tasks} -Results: {results} - -Rate the planning quality on a scale of 1-5: -1 - Poor: No clear plan or coordination -2 - Below Average: Some planning but poorly coordinated -3 - Average: Adequate planning with basic coordination -4 - Good: Well-planned with effective coordination -5 - Excellent: Optimal planning and seamless coordination - Respond with a JSON object: {{"rating": }}""" }, "research": { @@ -205,7 +188,6 @@ def filter_traces(self, traces: Dict[str, Any]) -> Dict[str, Any]: "agents": traces.get("agents", {}), "environment": traces.get("environment", {}), "communications": self._extract_communications(traces), - "results": self._extract_results(traces), } def _extract_communications(self, traces: Dict[str, Any]) -> str: @@ -251,12 +233,14 @@ def _extract_results(self, traces: Dict[str, Any]) -> str: return "\n".join(results) if results else "No results recorded." - def __call__(self, traces: Dict[str, Any], final_answer: Any) -> Dict[str, Any]: + def __call__( # type: ignore[override] + self, traces: Dict[str, Any], final_answer: Any + ) -> Dict[str, Any]: """Evaluate the task execution. Args: traces: Filtered execution traces - final_answer: Final output from agents + final_answer: Final output from agents (dict, list, str, or None) Returns: Evaluation results dictionary @@ -266,8 +250,6 @@ def __call__(self, traces: Dict[str, Any], final_answer: Any) -> Dict[str, Any]: # Extract filtered data filtered = self.filter_traces(traces) communications = filtered["communications"] - # Note: results are available in filtered["results"] but not used directly - # since we get task_desc and final_result separately # Calculate token consumption metrics.token_consumption = self._calculate_token_consumption(traces) @@ -282,7 +264,7 @@ def __call__(self, traces: Dict[str, Any], final_answer: Any) -> Dict[str, Any]: if self.domain == "research": metrics.task_evaluation = self._evaluate_research(task_desc, final_result) - elif self.domain == "bargaining" or self.domain == "worldsimulation": + elif self.domain in ("bargaining", "worldsimulation"): metrics.task_evaluation = self._evaluate_bargaining(task_desc, final_result) elif self.domain == "coding": metrics.task_evaluation = self._evaluate_coding(task_desc, final_result) @@ -331,7 +313,7 @@ def _calculate_token_consumption(self, traces: Dict[str, Any]) -> int: return total - def _evaluate_communication(self, task: str, communications: str) -> float: + def _evaluate_communication(self, task: str, communications: str) -> Optional[float]: """Evaluate communication quality using LLM.""" prompt_template = self._evaluation_prompts["communication"]["prompt"] prompt = prompt_template.format(task=task, communications=communications) @@ -340,7 +322,7 @@ def _evaluate_communication(self, task: str, communications: str) -> float: response = self.model_adapter.generate(prompt) return self._parse_score(response) except Exception: - return -1.0 + return None def _evaluate_research(self, task: str, result: str) -> Dict[str, Any]: """Evaluate research task output.""" @@ -351,7 +333,7 @@ def _evaluate_research(self, task: str, result: str) -> Dict[str, Any]: response = self.model_adapter.generate(prompt) return self._parse_research_ratings(response) except Exception: - return {"innovation": -1, "safety": -1, "feasibility": -1} + return {"innovation": None, "safety": None, "feasibility": None} def _evaluate_bargaining(self, task: str, result: str) -> Dict[str, Any]: """Evaluate bargaining/world simulation task output.""" @@ -366,9 +348,9 @@ def _evaluate_bargaining(self, task: str, result: str) -> Dict[str, Any]: ratings["buyer"] = self._parse_bargaining_ratings(buyer_response) except Exception: ratings["buyer"] = { - "effectiveness_of_strategies": -1, - "progress_and_outcome": -1, - "interaction_dynamics": -1, + "effectiveness_of_strategies": None, + "progress_and_outcome": None, + "interaction_dynamics": None, } try: @@ -376,9 +358,9 @@ def _evaluate_bargaining(self, task: str, result: str) -> Dict[str, Any]: ratings["seller"] = self._parse_bargaining_ratings(seller_response) except Exception: ratings["seller"] = { - "effectiveness_of_strategies": -1, - "progress_and_outcome": -1, - "interaction_dynamics": -1, + "effectiveness_of_strategies": None, + "progress_and_outcome": None, + "interaction_dynamics": None, } return ratings @@ -400,10 +382,10 @@ def _evaluate_coding(self, task: str, result: str) -> Dict[str, Any]: return self._parse_coding_ratings(response) except Exception: return { - "instruction_following": -1, - "executability": -1, - "consistency": -1, - "quality": -1, + "instruction_following": None, + "executability": None, + "consistency": None, + "quality": None, } def _evaluate_database(self, task: str, result: str) -> Dict[str, Any]: @@ -417,8 +399,12 @@ def _evaluate_database(self, task: str, result: str) -> Dict[str, Any]: "root_cause": [], # Would be filled from task data } - def _parse_score(self, response: str) -> float: - """Parse a single score from LLM response.""" + def _parse_score(self, response: str) -> Optional[float]: + """Parse a single score from LLM response. + + Returns: + Score as float (1-5), or None if parsing fails + """ try: content = response.strip() @@ -443,17 +429,12 @@ def _parse_score(self, response: str) -> float: if 1 <= score <= 5: return float(score) - # Fallback: find a single digit 1-5 - numbers = re.findall(r"\b[1-5]\b", content) - if numbers: - return float(int(numbers[0])) - - return 3.0 # Default middle score + return None except Exception: - return 3.0 + return None - def _parse_research_ratings(self, response: str) -> Dict[str, int]: + def _parse_research_ratings(self, response: str) -> Dict[str, Optional[int]]: """Parse research evaluation ratings.""" try: content = response.strip() @@ -467,9 +448,9 @@ def _parse_research_ratings(self, response: str) -> Dict[str, int]: except Exception: pass - return {"innovation": -1, "safety": -1, "feasibility": -1} + return {"innovation": None, "safety": None, "feasibility": None} - def _parse_bargaining_ratings(self, response: str) -> Dict[str, int]: + def _parse_bargaining_ratings(self, response: str) -> Dict[str, Optional[int]]: """Parse bargaining evaluation ratings.""" try: content = response.strip() @@ -480,20 +461,22 @@ def _parse_bargaining_ratings(self, response: str) -> Dict[str, int]: json_str = content[json_start:json_end] ratings = json.loads(json_str) return { - "effectiveness_of_strategies": int(ratings.get("effectiveness_of_strategies", -1)), - "progress_and_outcome": int(ratings.get("progress_and_outcome", -1)), - "interaction_dynamics": int(ratings.get("interaction_dynamics", -1)), + "effectiveness_of_strategies": int(ratings["effectiveness_of_strategies"]) + if "effectiveness_of_strategies" in ratings + else None, + "progress_and_outcome": int(ratings["progress_and_outcome"]) if "progress_and_outcome" in ratings else None, + "interaction_dynamics": int(ratings["interaction_dynamics"]) if "interaction_dynamics" in ratings else None, } except Exception: pass return { - "effectiveness_of_strategies": -1, - "progress_and_outcome": -1, - "interaction_dynamics": -1, + "effectiveness_of_strategies": None, + "progress_and_outcome": None, + "interaction_dynamics": None, } - def _parse_coding_ratings(self, response: str) -> Dict[str, int]: + def _parse_coding_ratings(self, response: str) -> Dict[str, Optional[int]]: """Parse coding evaluation ratings.""" try: content = response.strip() @@ -504,49 +487,53 @@ def _parse_coding_ratings(self, response: str) -> Dict[str, int]: json_str = content[json_start:json_end] ratings = json.loads(json_str) return { - "instruction_following": int(ratings.get("instruction_following", -1)), - "executability": int(ratings.get("executability", -1)), - "consistency": int(ratings.get("consistency", -1)), - "quality": int(ratings.get("quality", -1)), + "instruction_following": int(ratings["instruction_following"]) if "instruction_following" in ratings else None, + "executability": int(ratings["executability"]) if "executability" in ratings else None, + "consistency": int(ratings["consistency"]) if "consistency" in ratings else None, + "quality": int(ratings["quality"]) if "quality" in ratings else None, } except Exception: pass return { - "instruction_following": -1, - "executability": -1, - "consistency": -1, - "quality": -1, + "instruction_following": None, + "executability": None, + "consistency": None, + "quality": None, } def _determine_completion(self, metrics: MultiAgentBenchMetrics) -> bool: - """Determine if task was completed based on metrics.""" + """Determine if task was completed based on metrics. + + A task is considered completed if all required scores are present (not None) + and positive (> 0). + """ eval_data = metrics.task_evaluation if not eval_data: return False + def _all_scores_valid(scores: List[Any]) -> bool: + """Check all scores are present and positive.""" + return all(s is not None and s > 0 for s in scores) + if self.domain == "research": - # Consider completed if all scores are positive - scores = [eval_data.get(k, -1) for k in ["innovation", "safety", "feasibility"]] - return all(s > 0 for s in scores) + scores = [eval_data.get(k) for k in ["innovation", "safety", "feasibility"]] + return _all_scores_valid(scores) elif self.domain in ("bargaining", "worldsimulation"): - # Check both buyer and seller have positive scores buyer = eval_data.get("buyer", {}) seller = eval_data.get("seller", {}) - buyer_scores = [buyer.get(k, -1) for k in ["effectiveness_of_strategies", "progress_and_outcome", "interaction_dynamics"]] - seller_scores = [seller.get(k, -1) for k in ["effectiveness_of_strategies", "progress_and_outcome", "interaction_dynamics"]] - return all(s > 0 for s in buyer_scores) and all(s > 0 for s in seller_scores) + buyer_scores = [buyer.get(k) for k in ["effectiveness_of_strategies", "progress_and_outcome", "interaction_dynamics"]] + seller_scores = [seller.get(k) for k in ["effectiveness_of_strategies", "progress_and_outcome", "interaction_dynamics"]] + return _all_scores_valid(buyer_scores) and _all_scores_valid(seller_scores) elif self.domain == "coding": - # All coding metrics should be positive - scores = [eval_data.get(k, -1) for k in ["instruction_following", "executability", "consistency", "quality"]] - return all(s > 0 for s in scores) + scores = [eval_data.get(k) for k in ["instruction_following", "executability", "consistency", "quality"]] + return _all_scores_valid(scores) elif self.domain == "database": # Database completion is determined by comparing prediction to labels - # This requires external validation return bool(eval_data.get("predicted")) return False diff --git a/maseval/benchmark/multiagentbench/multiagentbench.py b/maseval/benchmark/multiagentbench/multiagentbench.py index 104f0235..e20a0a3d 100644 --- a/maseval/benchmark/multiagentbench/multiagentbench.py +++ b/maseval/benchmark/multiagentbench/multiagentbench.py @@ -5,8 +5,10 @@ - MarbleMultiAgentBenchBenchmark: Exact MARBLE reproduction mode """ +import logging from abc import abstractmethod -from typing import Any, Dict, List, Optional, Sequence, Tuple +from types import SimpleNamespace +from typing import TYPE_CHECKING, Any, Dict, List, Optional, Sequence, Tuple from maseval import ( AgentAdapter, @@ -19,11 +21,17 @@ ) from maseval.core.callback import BenchmarkCallback +from maseval.benchmark.multiagentbench._constants import MARBLE_IMPORT_ERROR from maseval.benchmark.multiagentbench.environment import MultiAgentBenchEnvironment from maseval.benchmark.multiagentbench.evaluator import ( MultiAgentBenchEvaluator, ) +if TYPE_CHECKING: + from maseval.benchmark.multiagentbench.adapters.marble_adapter import MarbleAgentAdapter + +logger = logging.getLogger(__name__) + class MultiAgentBenchBenchmark(Benchmark): """Abstract base class for framework-agnostic MultiAgentBench evaluation. @@ -134,7 +142,6 @@ def setup_environment( """ return MultiAgentBenchEnvironment( task_data=task.environment_data, - callbacks=self.callbacks, ) def setup_user( @@ -272,7 +279,7 @@ def run_agents( task: Task, environment: Environment, query: str, - ) -> Any: + ) -> Dict[str, Any]: """Execute the multi-agent system. For MultiAgentBench, this runs all agents on the task and @@ -285,20 +292,35 @@ def run_agents( query: The query/task content Returns: - Combined agent outputs + Dict with agent_results, communications, and coordination_mode """ results: List[Dict[str, Any]] = [] + communications: List[str] = [] + + coordination_mode = task.environment_data.get("coordinate_mode", "cooperative") for agent in agents: result = agent.run(query) + agent_id = getattr(agent, "agent_id", str(agent)) + results.append( { - "agent_id": getattr(agent, "agent_id", str(agent)), + "agent_id": agent_id, "result": result, } ) - return results + # Collect communication logs if available + if hasattr(agent, "get_serialized_messages"): + comm = agent.get_serialized_messages() # type: ignore[operator] + if comm: + communications.append(comm) + + return { + "agent_results": results, + "communications": communications, + "coordination_mode": coordination_mode, + } def evaluate( self, @@ -409,7 +431,6 @@ def setup_agents( agent_configs=agent_configs, marble_env=marble_env, model=model_id, - callbacks=self.callbacks, ) # Set up agent graph for inter-agent communication @@ -419,7 +440,7 @@ def setup_agents( for agent_id, adapter in agents_dict.items(): self.register("agents", agent_id, adapter) - return agents_list, agents_dict + return agents_list, agents_dict # type: ignore[return-value] def _create_marble_env(self, task: Task) -> Any: """Create a MARBLE environment for the task. @@ -431,9 +452,9 @@ def _create_marble_env(self, task: Task) -> Any: MARBLE environment instance """ try: - from .marble.environments.base_env import BaseEnvironment + from .marble.environments.base_env import BaseEnvironment # type: ignore[unresolved-import] except ImportError as e: - raise ImportError(f"MARBLE is not available. Clone MARBLE to maseval/benchmark/multiagentbench/marble/\nOriginal error: {e}") from e + raise ImportError(MARBLE_IMPORT_ERROR.format(error=e)) from e env_config = task.environment_data.get("environment", {}) task_config = task.environment_data.get("task", {}) @@ -448,37 +469,30 @@ def _create_marble_env(self, task: Task) -> Any: def _setup_agent_graph( self, - agents_dict: Dict[str, AgentAdapter], + agents_dict: Dict[str, "MarbleAgentAdapter"], task: Task, marble_env: Any, ) -> None: """Set up MARBLE's AgentGraph for inter-agent communication. Args: - agents_dict: Dict of agent adapters + agents_dict: Dict of MARBLE agent adapters task: Task with relationship data marble_env: MARBLE environment """ try: - from .marble.graph.agent_graph import AgentGraph + from .marble.graph.agent_graph import AgentGraph # type: ignore[unresolved-import] except ImportError: # MARBLE not available, skip graph setup return # Extract MARBLE agents from adapters - marble_agents = [adapter.marble_agent for adapter in agents_dict.values()] # type: ignore[attr-defined] + marble_agents = [adapter.marble_agent for adapter in agents_dict.values()] - # Build config for AgentGraph + # Build config for AgentGraph (MARBLE expects an object with these attributes) relationships = task.environment_data.get("relationships", []) coordination_mode = task.environment_data.get("coordinate_mode", "cooperative") - - # Create a minimal Config object - class MinimalConfig: - def __init__(self) -> None: - self.coordination_mode = coordination_mode - self.relationships = relationships - - config = MinimalConfig() + config = SimpleNamespace(coordination_mode=coordination_mode, relationships=relationships) try: # Create agent graph @@ -488,59 +502,9 @@ def __init__(self) -> None: for agent in marble_agents: agent.set_agent_graph(graph) - except Exception: + except (ValueError, KeyError, AttributeError) as e: # Graph creation failed, agents will work without inter-agent communication - pass - - def run_agents( - self, - agents: Sequence[AgentAdapter], - task: Task, - environment: Environment, - query: str, - ) -> Any: - """Execute agents using MARBLE's coordination patterns. - - Args: - agents: Agents to run (MarbleAgentAdapters) - task: The task - environment: The environment - query: The task query - - Returns: - Combined agent outputs with communication logs - """ - results: List[Dict[str, Any]] = [] - communications: List[str] = [] - - # Get coordination mode - coordination_mode = task.environment_data.get("coordinate_mode", "cooperative") - - # Simple execution: each agent acts on the task - # More complex coordination modes (star, tree, hierarchical) would - # require the MARBLE Engine, which is outside the scope of this adapter - for agent in agents: - result = agent.run(query) - agent_id = getattr(agent, "agent_id", str(agent)) - - results.append( - { - "agent_id": agent_id, - "result": result, - } - ) - - # Collect communication logs if available - if hasattr(agent, "get_serialized_messages"): - comm = agent.get_serialized_messages() # type: ignore - if comm: - communications.append(comm) - - return { - "agent_results": results, - "communications": communications, - "coordination_mode": coordination_mode, - } + logger.warning("AgentGraph setup failed, agents will run without inter-agent communication: %s", e) @abstractmethod def get_model_adapter(self, model_id: str, **kwargs: Any) -> ModelAdapter: diff --git a/tests/test_benchmarks/test_multiagentbench/test_benchmark.py b/tests/test_benchmarks/test_multiagentbench/test_benchmark.py index f6ad0290..6acf67f1 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_benchmark.py +++ b/tests/test_benchmarks/test_multiagentbench/test_benchmark.py @@ -226,10 +226,13 @@ def test_run_agents_executes_all_agents( sample_research_task.query, ) - assert isinstance(results, list) - assert len(results) == 2 - assert all("agent_id" in r for r in results) - assert all("result" in r for r in results) + assert isinstance(results, dict) + assert "agent_results" in results + assert "communications" in results + assert "coordination_mode" in results + assert len(results["agent_results"]) == 2 + assert all("agent_id" in r for r in results["agent_results"]) + assert all("result" in r for r in results["agent_results"]) def test_evaluate_calls_evaluators( self, @@ -539,7 +542,8 @@ def test_run_agents_with_cooperative_mode( sample_research_task.query, ) - assert len(results) == 2 + assert len(results["agent_results"]) == 2 + assert results["coordination_mode"] == "cooperative" def test_run_agents_with_star_mode(self, benchmark_instance): """run_agents should work with star coordination.""" @@ -569,7 +573,8 @@ def test_run_agents_with_star_mode(self, benchmark_instance): results = benchmark_instance.run_agents(agents_list, task, env, task.query) - assert len(results) == 2 + assert len(results["agent_results"]) == 2 + assert results["coordination_mode"] == "star" class TestBenchmarkWithEmptyAgents: @@ -590,7 +595,8 @@ def test_run_agents_with_empty_list( sample_research_task.query, ) - assert results == [] + assert results["agent_results"] == [] + assert results["communications"] == [] def test_setup_agents_with_no_agents_in_task(self, benchmark_instance): """setup_agents should handle task with no agents.""" diff --git a/tests/test_benchmarks/test_multiagentbench/test_data_loader.py b/tests/test_benchmarks/test_multiagentbench/test_data_loader.py index 6bf11502..7c8ecae7 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_data_loader.py +++ b/tests/test_benchmarks/test_multiagentbench/test_data_loader.py @@ -1,11 +1,13 @@ """Tests for MultiAgentBench data loading functionality.""" -import pytest -from pathlib import Path -from unittest.mock import patch, MagicMock +import json import subprocess import tempfile -import json +from pathlib import Path +from typing import Any, Dict +from unittest.mock import MagicMock, patch + +import pytest from maseval import Task from maseval.benchmark.multiagentbench.data_loader import ( @@ -501,7 +503,7 @@ def test_load_tasks_empty_lines(self): # Write tasks with empty lines with jsonl_path.open("w") as f: - task_data = { + task_data: Dict[str, Any] = { "scenario": "research", "task_id": 1, "task": {"content": "Task 1"}, @@ -512,7 +514,7 @@ def test_load_tasks_empty_lines(self): f.write("\n") # Empty line f.write(" \n") # Whitespace-only line task_data["task_id"] = 2 - task_data["task"]["content"] = "Task 2" + task_data["task"]["content"] = "Task 2" # type: ignore[index] f.write(json.dumps(task_data) + "\n") tasks = load_tasks("research", data_dir=Path(tmpdir)) diff --git a/tests/test_benchmarks/test_multiagentbench/test_evaluator.py b/tests/test_benchmarks/test_multiagentbench/test_evaluator.py index d5729bf5..a6327713 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_evaluator.py +++ b/tests/test_benchmarks/test_multiagentbench/test_evaluator.py @@ -18,8 +18,7 @@ def test_default_values(self): assert metrics.task_completion is False assert metrics.token_consumption == 0 - assert metrics.planning_score == -1.0 - assert metrics.communication_score == -1.0 + assert metrics.communication_score is None assert metrics.task_evaluation == {} assert metrics.agent_kpis == {} assert metrics.total_milestones == 0 @@ -29,14 +28,12 @@ def test_to_dict(self): metrics = MultiAgentBenchMetrics( task_completion=True, token_consumption=1000, - planning_score=4.0, communication_score=5.0, ) d = metrics.to_dict() assert d["task_completion"] is True assert d["token_consumption"] == 1000 - assert d["planning_score"] == 4.0 assert d["communication_score"] == 5.0 @@ -139,19 +136,20 @@ def test_parse_score_with_markdown(self, research_evaluator): assert score == 5.0 - def test_parse_score_fallback(self, research_evaluator): - """_parse_score should fallback to finding digit.""" + def test_parse_score_no_json_returns_none(self, research_evaluator): + """_parse_score should return None when no valid JSON found.""" response = "The rating is 4 out of 5" score = research_evaluator._parse_score(response) - assert score == 4.0 + # No regex fallback - returns None for non-JSON responses + assert score is None def test_parse_score_default(self, research_evaluator): - """_parse_score should return 3 as default.""" + """_parse_score should return None when parsing fails.""" response = "No score here" score = research_evaluator._parse_score(response) - assert score == 3.0 + assert score is None def test_parse_research_ratings_valid(self, research_evaluator): """_parse_research_ratings should parse valid ratings.""" @@ -163,13 +161,13 @@ def test_parse_research_ratings_valid(self, research_evaluator): assert ratings["feasibility"] == 5 def test_parse_research_ratings_invalid(self, research_evaluator): - """_parse_research_ratings should return -1 for invalid.""" + """_parse_research_ratings should return None for invalid.""" response = "Invalid response" ratings = research_evaluator._parse_research_ratings(response) - assert ratings["innovation"] == -1 - assert ratings["safety"] == -1 - assert ratings["feasibility"] == -1 + assert ratings["innovation"] is None + assert ratings["safety"] is None + assert ratings["feasibility"] is None def test_determine_completion_research_positive(self, research_evaluator): """_determine_completion should return True for positive research scores.""" @@ -177,8 +175,8 @@ def test_determine_completion_research_positive(self, research_evaluator): assert research_evaluator._determine_completion(metrics) is True def test_determine_completion_research_negative(self, research_evaluator): - """_determine_completion should return False for negative scores.""" - metrics = MultiAgentBenchMetrics(task_evaluation={"innovation": -1, "safety": 3, "feasibility": 5}) + """_determine_completion should return False for None scores.""" + metrics = MultiAgentBenchMetrics(task_evaluation={"innovation": None, "safety": 3, "feasibility": 5}) assert research_evaluator._determine_completion(metrics) is False def test_call_returns_expected_structure(self, research_evaluator): @@ -319,11 +317,11 @@ def test_determine_completion_coding_positive(self, coding_evaluator): assert coding_evaluator._determine_completion(metrics) is True def test_determine_completion_coding_negative(self, coding_evaluator): - """_determine_completion should return False for negative coding scores.""" + """_determine_completion should return False for None coding scores.""" metrics = MultiAgentBenchMetrics( task_evaluation={ "instruction_following": 5, - "executability": -1, + "executability": None, "consistency": 4, "quality": 3, } @@ -331,12 +329,12 @@ def test_determine_completion_coding_negative(self, coding_evaluator): assert coding_evaluator._determine_completion(metrics) is False def test_parse_coding_ratings_invalid(self, coding_evaluator): - """_parse_coding_ratings should return -1 for invalid response.""" + """_parse_coding_ratings should return None for invalid response.""" ratings = coding_evaluator._parse_coding_ratings("Invalid JSON") - assert ratings["instruction_following"] == -1 - assert ratings["executability"] == -1 - assert ratings["consistency"] == -1 - assert ratings["quality"] == -1 + assert ratings["instruction_following"] is None + assert ratings["executability"] is None + assert ratings["consistency"] is None + assert ratings["quality"] is None class TestDatabaseEvaluation: @@ -494,7 +492,7 @@ def test_evaluate_communication_returns_score(self, evaluator_with_comm_response assert score == 4.0 def test_evaluate_communication_on_error(self): - """_evaluate_communication should return -1 on error.""" + """_evaluate_communication should return None on error.""" from unittest.mock import MagicMock # Create mock that raises exception on generate @@ -507,7 +505,7 @@ def test_evaluate_communication_on_error(self): ) score = evaluator._evaluate_communication("Task", "Comms") - assert score == -1.0 + assert score is None def test_call_with_communications(self, evaluator_with_comm_response): """__call__ should evaluate communications when present.""" @@ -548,23 +546,23 @@ def failing_evaluator(self): def test_evaluate_research_on_error(self, failing_evaluator): """_evaluate_research should return default values on error.""" result = failing_evaluator._evaluate_research("Task", "Result") - assert result["innovation"] == -1 - assert result["safety"] == -1 - assert result["feasibility"] == -1 + assert result["innovation"] is None + assert result["safety"] is None + assert result["feasibility"] is None def test_evaluate_bargaining_on_error(self, failing_evaluator): """_evaluate_bargaining should return default values on error.""" failing_evaluator.domain = "bargaining" result = failing_evaluator._evaluate_bargaining("Task", "Result") - assert result["buyer"]["effectiveness_of_strategies"] == -1 - assert result["seller"]["effectiveness_of_strategies"] == -1 + assert result["buyer"]["effectiveness_of_strategies"] is None + assert result["seller"]["effectiveness_of_strategies"] is None def test_evaluate_coding_on_error(self, failing_evaluator): """_evaluate_coding should return default values on error.""" failing_evaluator.domain = "coding" result = failing_evaluator._evaluate_coding("Task", "Result") - assert result["instruction_following"] == -1 - assert result["executability"] == -1 + assert result["instruction_following"] is None + assert result["executability"] is None class TestParsingEdgeCases: @@ -583,14 +581,14 @@ def test_parse_score_out_of_range(self, evaluator): """_parse_score should reject scores outside 1-5 range.""" response = '{"rating": 10}' score = evaluator._parse_score(response) - # Should fall back to finding digit or default - assert score == 3.0 # Default + # Should return None since score is out of range + assert score is None def test_parse_score_with_just_code_block(self, evaluator): """_parse_score should handle code block without json marker.""" response = '```\n{"rating": 3}\n```' score = evaluator._parse_score(response) - assert score == 3.0 + assert score == 3.0 # Valid score parsed from JSON def test_parse_score_with_text_before_json(self, evaluator): """_parse_score should find JSON in text.""" @@ -603,12 +601,12 @@ def test_parse_bargaining_ratings_partial(self, evaluator): response = '{"effectiveness_of_strategies": 4}' # Missing other fields ratings = evaluator._parse_bargaining_ratings(response) assert ratings["effectiveness_of_strategies"] == 4 - assert ratings["progress_and_outcome"] == -1 + assert ratings["progress_and_outcome"] is None def test_parse_bargaining_ratings_invalid(self, evaluator): - """_parse_bargaining_ratings should return -1 for invalid.""" + """_parse_bargaining_ratings should return None for invalid.""" ratings = evaluator._parse_bargaining_ratings("Invalid") - assert ratings["effectiveness_of_strategies"] == -1 + assert ratings["effectiveness_of_strategies"] is None class TestFormatFinalAnswerEdgeCases: @@ -779,7 +777,6 @@ def test_filter_traces_empty(self, evaluator): assert filtered["agents"] == {} assert filtered["environment"] == {} assert "communications" in filtered - assert "results" in filtered def test_filter_traces_extracts_all(self, evaluator): """filter_traces should extract all relevant data.""" @@ -794,7 +791,8 @@ def test_filter_traces_extracts_all(self, evaluator): } filtered = evaluator.filter_traces(traces) assert "[agent1]: Hello" in filtered["communications"] - assert "[agent1]: Done" in filtered["results"] + assert filtered["agents"] == traces["agents"] + assert filtered["environment"] == traces["environment"] class TestDetermineCompletionEdgeCases: @@ -811,7 +809,7 @@ def test_completion_empty_eval_data(self): assert evaluator._determine_completion(metrics) is False def test_completion_bargaining_partial_buyer(self): - """_determine_completion should return False if buyer has negative scores.""" + """_determine_completion should return False if buyer has None scores.""" adapter = DummyModelAdapter(model_id="test", responses=[]) evaluator = MultiAgentBenchEvaluator( domain="bargaining", @@ -820,7 +818,7 @@ def test_completion_bargaining_partial_buyer(self): metrics = MultiAgentBenchMetrics( task_evaluation={ "buyer": { - "effectiveness_of_strategies": -1, + "effectiveness_of_strategies": None, "progress_and_outcome": 4, "interaction_dynamics": 4, }, @@ -834,7 +832,7 @@ def test_completion_bargaining_partial_buyer(self): assert evaluator._determine_completion(metrics) is False def test_completion_bargaining_partial_seller(self): - """_determine_completion should return False if seller has negative scores.""" + """_determine_completion should return False if seller has None scores.""" adapter = DummyModelAdapter(model_id="test", responses=[]) evaluator = MultiAgentBenchEvaluator( domain="bargaining", @@ -848,7 +846,7 @@ def test_completion_bargaining_partial_seller(self): "interaction_dynamics": 4, }, "seller": { - "effectiveness_of_strategies": -1, + "effectiveness_of_strategies": None, "progress_and_outcome": 4, "interaction_dynamics": 4, }, From 0fbc9ad7a56529b5490c20d58eb5a28bce144378 Mon Sep 17 00:00:00 2001 From: cemde Date: Wed, 4 Feb 2026 13:26:53 +0100 Subject: [PATCH 11/17] documentation fix --- docs/benchmark/multiagentbench.md | 2 +- maseval/benchmark/multiagentbench/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/benchmark/multiagentbench.md b/docs/benchmark/multiagentbench.md index 55911cc2..793323aa 100644 --- a/docs/benchmark/multiagentbench.md +++ b/docs/benchmark/multiagentbench.md @@ -36,7 +36,7 @@ configure_model_ids(tasks, agent_model_id="gpt-4o") # Create your framework-specific benchmark subclass class MyMultiAgentBenchmark(MultiAgentBenchBenchmark): - def setup_agents(self, agent_data, environment, task, user): + def setup_agents(self, agent_data, environment, task, user, seed_generator=None): # Your framework-specific agent creation agent_configs = task.environment_data.get("agents", []) # Create agents based on configs... diff --git a/maseval/benchmark/multiagentbench/README.md b/maseval/benchmark/multiagentbench/README.md index ccceda3e..3b6581e2 100644 --- a/maseval/benchmark/multiagentbench/README.md +++ b/maseval/benchmark/multiagentbench/README.md @@ -77,7 +77,7 @@ from maseval.benchmark.multiagentbench import ( ) class MyMultiAgentBenchmark(MultiAgentBenchBenchmark): - def setup_agents(self, agent_data, environment, task, user): + def setup_agents(self, agent_data, environment, task, user, seed_generator=None): # Your framework-specific agent creation ... From d1f8d230f75ee404c64570124e274e741ff0589e Mon Sep 17 00:00:00 2001 From: cemde Date: Wed, 4 Feb 2026 13:38:22 +0100 Subject: [PATCH 12/17] [skip ci] fixed changlog --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f1be553c..7bbc9167 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,7 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `MultiAgentBenchBenchmark` abstract base class for framework-agnostic multi-agent evaluation with seeding support for evaluators and agents (PR: #25) - `MarbleMultiAgentBenchBenchmark` for exact MARBLE reproduction mode using native MARBLE agents (note: MARBLE's internal LLM calls bypass MASEval seeding) (PR: #25) - `MultiAgentBenchEnvironment` and `MultiAgentBenchEvaluator` components (PR: #25) -- Data loading utilities: `load_tasks()`, `configure_model_ids()`, `list_scenarios()` (PR: #25) +- Data loading utilities: `load_tasks()`, `configure_model_ids()`, `get_domain_info()`, `ensure_marble_exists()` (PR: #25) - MARBLE adapter: `MarbleAgentAdapter` for wrapping MARBLE agents with MASEval tracing (PR: #25) **Interface** From 5c39c903f859567b41865bdcdf9466c4c8ebaff7 Mon Sep 17 00:00:00 2001 From: cemde Date: Wed, 4 Feb 2026 16:45:25 +0100 Subject: [PATCH 13/17] updated benchmarks list --- BENCHMARKS.md | 2 +- .../benchmark/multiagentbench/PROVENANCE.md | 7 +------ maseval/benchmark/multiagentbench/README.md | 18 +++++++++--------- 3 files changed, 11 insertions(+), 16 deletions(-) diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 4df50d51..bcb15d06 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -40,7 +40,7 @@ MultiAgentBench is a comprehensive benchmark suite for evaluating multi-agent co --- -## 3. GAIA2 +## 4. GAIA2 Gaia2 is a benchmark for evaluating LLM-based agents on dynamic, multi-step scenarios using Meta's ARE (Agent Research Environments) platform. It tests agents across 7 capability dimensions: execution, search, adaptability, time, ambiguity, agent2agent, and noise. diff --git a/maseval/benchmark/multiagentbench/PROVENANCE.md b/maseval/benchmark/multiagentbench/PROVENANCE.md index 50fe413a..9fd516bf 100644 --- a/maseval/benchmark/multiagentbench/PROVENANCE.md +++ b/maseval/benchmark/multiagentbench/PROVENANCE.md @@ -10,6 +10,7 @@ ## Reference **Paper**: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents" + - arXiv: https://arxiv.org/abs/2503.01935 - Authors: Haofei Yu et al. - Publication Date: 2025 @@ -66,9 +67,3 @@ To update MARBLE to a newer version: 4. `git checkout ` 5. Run integration tests 6. Update this file with new version info - -## Last Updated - -- **Date**: 2026-01-19 -- **Updated By**: Claude Code -- **Version Tested**: Initial integration (not yet pinned) diff --git a/maseval/benchmark/multiagentbench/README.md b/maseval/benchmark/multiagentbench/README.md index 3b6581e2..c5b3cdab 100644 --- a/maseval/benchmark/multiagentbench/README.md +++ b/maseval/benchmark/multiagentbench/README.md @@ -135,15 +135,15 @@ for result in results: MultiAgentBench includes 7 domains with different requirements: -| Domain | External Dependencies | Initial Support | -|--------|----------------------|-----------------| -| Research | None | Yes | -| Bargaining | None | Yes | -| Coding | Filesystem access | Yes | -| Web | Network access | Yes | -| WorldSimulation | None | Yes | -| Database | Docker + PostgreSQL | Optional | -| Minecraft | External game server | Deferred | +| Domain | External Dependencies | Initial Support | +| --------------- | --------------------- | --------------- | +| Research | None | Yes | +| Bargaining | None | Yes | +| Coding | Filesystem access | Yes | +| Web | Network access | Yes | +| WorldSimulation | None | Yes | +| Database | Docker + PostgreSQL | Optional | +| Minecraft | External game server | Deferred | ### Domain-Specific Notes From dc53b0e38178b72420388687b5845ed19c7e073d Mon Sep 17 00:00:00 2001 From: cemde Date: Wed, 4 Feb 2026 16:46:26 +0100 Subject: [PATCH 14/17] removed overview files --- INSTRUCTIONS.md | 114 ----- STRATEGY.md | 1032 -------------------------------------- STRATEGY_BACKUP.md | 1120 ------------------------------------------ STRATEGY_CRITIQUE.md | 517 ------------------- 4 files changed, 2783 deletions(-) delete mode 100644 INSTRUCTIONS.md delete mode 100644 STRATEGY.md delete mode 100644 STRATEGY_BACKUP.md delete mode 100644 STRATEGY_CRITIQUE.md diff --git a/INSTRUCTIONS.md b/INSTRUCTIONS.md deleted file mode 100644 index 27d98079..00000000 --- a/INSTRUCTIONS.md +++ /dev/null @@ -1,114 +0,0 @@ -# MultiAgentBench Integration Instructions - -## Phase 0: Before You Start - -### 0.1 Repeat Back Requirements - -List exactly: - -- The classes you will design -- The integration approaches you will evaluate -- The design principles you must follow - -### 0.2 Create a Scoped ToDo List - -List every file/directory you need to examine. Scope constraint: explore MARBLE only as deep as needed for MASEval integration—no deeper. Go into depth on integration-relevant code only. - -### 0.3 Periodically Reread - -These instructions are saved in `INSTRUCTIONS.md`. Reread and re-summarize requirements: - -- Before starting analysis -- After completing each major section -- Before writing final deliverable - ---- - -## Resources - -| Resource | Location | -| ------------------------- | --------------------------------------------- | -| MARBLE repo | https://github.com/ulab-uiuc/MARBLE | -| Local copy | `/Users/cornelius/Repositories/MARBLE` | -| Paper | `/Users/cornelius/Repositories/MARBLE/paper/` | -| These instructions | `INSTRUCTIONS.md` | -| Reference implementations | `macs` and `tau2` benchmarks in MASEval | -| MASEval guidelines | `AGENTS.md` | - -**License**: MIT (verify subdirectories) - ---- - -## Deliverable - -Write `STRATEGY.md` proposing 2–3 integration approaches with tradeoffs and a recommendation. - -Requirements: - -- [ ] Standalone document (readable without this conversation) -- [ ] **Do not implement**—strategy document only - ---- - -## Required Analysis: Hybrid Vendoring Approach - -Deeply evaluate this specific approach: - -**Vendoring**: Clone MARBLE to `maseval/benchmark/multiagentbench/marble/` (gitignored) - -**MASEval wrappers** (in `maseval/benchmark/multiagentbench/`): - -- [ ] `MultiAgentBenchEnvironment(Environment)`: wraps the environment of MultiAgentBench -- [ ] `MultiAgentBenchEvaluator(Evaluator)`: wraps the evaluation of MultiAgentBnech -- [ ] `MultiAgentBenchBenchmark(Benchmark)` Harness to coordinate environment, eval etc. Does NOT implement specific agents. -- [ ] `MarbleAgentAdapter(AgentAdapter)` Implements the multi-agent engine of MARBLE -- [ ] `MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark)` adapts the benchmark to use the MARBLE agent and runs them with the same parameters as the original to reproduce. -- [ ] Utilities: `load_tasks()`, `load_agent_data()`, `configure_model_ids()`, `ensure_data()` - -**Key question**: Do patterns from `tau2` and `macs` transfer smoothly to MultiAgentBench? - ---- - -## Design Principles (Must Address Each) - -- [ ] **R1: Reuse MASEval** — Don't reimplement existing features -- [ ] **R2: Scientific fidelity** — Reproduce original results exactly; preserve semantically relevant details; allow functionally equivalent implementation changes -- [ ] **R3: Fail loudly** — No defensive defaults that silently corrupt results; this targets a fixed dataset -- [ ] **R4: Maintainability** — Plan for long-term upkeep - ---- - -## Phase 1: Study - -- [ ] Read and internalize `AGENTS.md` -- [ ] Study `macs` benchmark implementation -- [ ] Study `tau2` benchmark implementation -- [ ] Understand MASEval core library (especially `Benchmark` class) -- [ ] Read MARBLE paper (sufficient for integration understanding) -- [ ] Examine MARBLE code (scoped to integration needs) - ---- - -## Phase 2: Analyze - -- [ ] Identify 2–3 integration approaches -- [ ] For each approach, document tradeoffs -- [ ] Assess hybrid vendoring approach in depth -- [ ] Determine if `tau2`/`macs` patterns transfer to MultiAgentBench -- [ ] Verify each design principle (R1–R4) is addressed - ---- - -## Phase 3: Write - -- [ ] Draft `STRATEGY.md` -- [ ] **Stop and verify**: Reread `INSTRUCTIONS.md`, confirm all checkboxes above are addressed -- [ ] Finalize document - ---- - -## Process Notes - -- Question everything -- Do not make assumptions—verify in the codebase -- Take time to consider all consequences diff --git a/STRATEGY.md b/STRATEGY.md deleted file mode 100644 index d4da4928..00000000 --- a/STRATEGY.md +++ /dev/null @@ -1,1032 +0,0 @@ -# MARBLE Integration Strategy for MASEval - -## Executive Summary - -This document proposes the integration architecture for bringing MARBLE (Multi-Agent Coordination Backbone with LLM Engine) and its MultiAgentBench benchmark suite into MASEval. - -**Key Architecture**: A dual-purpose design that enables: -1. **Exact MARBLE reproduction** via `MarbleMultiAgentBenchBenchmark` (for scientific validation) -2. **Fair framework comparison** via abstract `MultiAgentBenchBenchmark` base (users implement with their own frameworks) - -This enables critical research questions: **"Which multi-agent framework performs best on the same tasks?"** - -**Key Finding**: Individual MARBLE agents CAN be extracted and wrapped in separate AgentAdapters, providing per-agent trace visibility while preserving MARBLE's coordination logic. - -**License**: ✅ MARBLE's MIT license explicitly permits vendoring/usage with attribution. - ---- - -## Background - -### What is MARBLE? - -MARBLE is a multi-agent coordination framework that: -- Orchestrates multiple LLM-based agents using an **Engine** + **AgentGraph** architecture -- Supports various coordination modes (star, chain, tree, graph) -- Provides **inter-agent communication** via direct message passing (stored in `agent.msg_box`) -- Includes domain-specific environments (Coding, Database, Minecraft, Research, Bargaining, Web, WorldSimulation) -- Uses configuration for agent relationships and task specifications - -### What is MultiAgentBench? - -MultiAgentBench is MARBLE's benchmark suite featuring: -- **7 domains**: Coding, Database, Minecraft, Research, Bargaining, Web, WorldSimulation -- **JSONL task format** with rich metadata (agent relationships, coordination modes, evaluation metrics) -- **500+ tasks** testing collaboration and competition scenarios -- Original paper: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents" (arXiv:2503.01935) - -### MASEval Integration Goals - -1. **Reproduce MARBLE's published results** - Exact reproduction via MarbleMultiAgentBenchBenchmark -2. **Enable multi-agent framework comparison** - Abstract base allows users to implement with any framework -3. **Maintain MASEval's framework-agnostic design** - No hard dependencies on agent frameworks -4. **Reuse MASEval infrastructure** - Callbacks, tracing, parallelization, error handling -5. **Per-agent trace visibility** - Wrap individual agents, not just the engine - ---- - -## Integration Architecture - -### Component Structure - -``` -maseval/benchmark/multiagentbench/ -├── __init__.py -├── README.md # Setup instructions -├── PROVENANCE.md # Track upstream version, license -├── .gitignore # Ignore marble/ directory -│ -├── multiagentbench.py # Core classes: -│ ├── MultiAgentBenchBenchmark # - Abstract base (framework-agnostic) -│ ├── MarbleMultiAgentBenchBenchmark # - MARBLE reproduction -│ └── MultiAgentBenchEvaluator # - Wraps MARBLE metrics -│ -├── environment.py # MultiAgentBenchEnvironment -├── data_loader.py # load_tasks(), configure_model_ids() -├── adapters/ -│ └── marble_adapter.py # MarbleAgentAdapter (per-agent) -│ -└── marble/ # ← Vendored MARBLE (gitignored) - ├── LICENSE # MIT license preserved - ├── agent/ - ├── engine/ - ├── environments/ - ├── evaluator/ - └── ... # Full MARBLE source -``` - -### Class Hierarchy - -```python -# Abstract base - provides task/eval infrastructure for ANY framework -class MultiAgentBenchBenchmark(Benchmark): - """Framework-agnostic base for MultiAgentBench tasks. - - Users implement setup_agents() with their chosen framework. - """ - def setup_environment(...) → MultiAgentBenchEnvironment # Shared - def setup_evaluators(...) → MultiAgentBenchEvaluator # Shared - def setup_agents(...) → ABSTRACT # User implements - -# MARBLE reproduction - wraps individual agents for trace visibility -class MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark): - """Exact MARBLE reproduction for scientific validation.""" - def setup_agents(...): - # Create MARBLE agents from task config - # Wrap EACH agent in MarbleAgentAdapter - # Return list of adapters to MASEval -``` - ---- - -## Agent Wrapping Strategy - -### Per-Agent Adapters (Preferred Pattern) - -MARBLE agents CAN be extracted and wrapped individually. This provides: -- Per-agent trace visibility (matches tau2/macs pattern) -- Better error attribution -- Callback hooks fire per-agent -- Cleaner debugging - -**Requirements for independent agent execution:** -1. Each agent needs an `AgentGraph` reference (even if minimal) -2. Communicating agents must share the same `AgentGraph` -3. Each agent needs an environment reference -4. Call `agent.act(task)` directly - -### How It Works - -```python -class MarbleAgentAdapter(AgentAdapter): - """Wraps a single MARBLE BaseAgent.""" - - def __init__( - self, - agent: "BaseAgent", - name: str, - callbacks: Optional[List[Any]] = None, - ): - super().__init__(agent, name, callbacks) - self._agent = agent - - def _run_agent(self, query: str) -> str: - """Execute MARBLE agent's act() method.""" - result, communication = self._agent.act(query) - return result - - def get_messages(self) -> MessageHistory: - """Extract messages from agent's msg_box.""" - return self._extract_message_history() - - def _extract_message_history(self) -> MessageHistory: - """Extract messages from MARBLE's msg_box structure. - - MARBLE stores messages in agent.msg_box: - msg_box[session_id][other_agent_id] = List[(direction, message)] - direction: 0 = FORWARD_TO (sent), 1 = RECV_FROM (received) - """ - messages = [] - for session_id, conversations in self._agent.msg_box.items(): - for other_agent_id, msg_list in conversations.items(): - for direction, content in msg_list: - messages.append({ - "role": "agent" if direction == 0 else "other", - "content": str(content), - "agent_id": self._agent.agent_id, - "other_agent_id": other_agent_id, - "direction": "sent" if direction == 0 else "received", - "session_id": session_id, - }) - return MessageHistory(messages) - - def gather_traces(self) -> Dict[str, Any]: - """Gather MARBLE-specific execution data.""" - return { - **super().gather_traces(), - "agent_id": self._agent.agent_id, - "agent_type": getattr(self._agent, "agent_type", "BaseAgent"), - "profile": getattr(self._agent, "profile", ""), - "task_history": getattr(self._agent, "task_history", []), - "relationships": self._extract_relationships(), - } - - def _extract_relationships(self) -> List[Dict[str, str]]: - """Extract this agent's relationships from the graph.""" - relationships = [] - if self._agent.agent_graph: - for src, dst, rel_type in self._agent.agent_graph.relationships: - if src == self._agent.agent_id or dst == self._agent.agent_id: - relationships.append({ - "source": src, - "target": dst, - "type": rel_type, - }) - return relationships -``` - -### Coordination Without Engine.start() - -Instead of calling `engine.start()`, we implement coordination in MASEval's `run_agents()`: - -```python -def run_agents( - self, - agents: Sequence[AgentAdapter], - task: Task, - environment: MultiAgentBenchEnvironment, - query: str = "", -) -> Any: - """Execute agents according to coordination mode.""" - coordinate_mode = task.environment_data.get("coordinate_mode", "star") - - if coordinate_mode == "star": - return self._star_coordinate(agents, task, query) - elif coordinate_mode == "chain": - return self._chain_coordinate(agents, task, query) - elif coordinate_mode == "tree": - return self._tree_coordinate(agents, task, query) - elif coordinate_mode == "graph": - return self._graph_coordinate(agents, task, query) - else: - raise ValueError(f"Unknown coordination mode: {coordinate_mode}") - -def _star_coordinate( - self, - agents: Sequence[AgentAdapter], - task: Task, - query: str, -) -> Any: - """Star coordination: central planner assigns tasks to agents.""" - max_iterations = task.environment_data.get("max_iterations", 10) - results = {} - - for iteration in range(max_iterations): - # Assign tasks (simplified - full impl uses EnginePlanner) - task_assignments = self._assign_tasks(agents, task, results) - - if not task_assignments: - break - - # Execute each agent with its assigned task - for agent, agent_task in task_assignments.items(): - result = agent.run(agent_task) - results[agent.name] = result - - # Check termination - if self._should_terminate(results, task): - break - - return self._aggregate_results(results) -``` - ---- - -## Critical MARBLE API Details - -### Message History Storage - -**MARBLE stores messages in `agent.msg_box`, NOT SharedMemory:** - -```python -# BaseAgent structure (base_agent.py) -self.msg_box: DefaultDict[str, DefaultDict[str, List[Tuple[int, str]]]] -# Structure: msg_box[session_id][other_agent_id] = [(direction, message), ...] -# Direction: 0 = FORWARD_TO (sent), 1 = RECV_FROM (received) - -# Serialize via (note: typo in MARBLE) -serialized = agent.seralize_message(session_id="") -``` - -### SharedMemory is NOT Shared - -Despite the name, each MARBLE agent creates its own `SharedMemory` instance: - -```python -# In BaseAgent.__init__() (base_agent.py:72-73) -self.memory = BaseMemory() # Per-agent memory -self.shared_memory = SharedMemory() # Also per-agent, NOT shared! -``` - -**Do NOT rely on SharedMemory for inter-agent state.** Use `msg_box` or environment state instead. - -### AgentGraph is Required - -`agent.act()` requires an AgentGraph reference: - -```python -# In BaseAgent.act() (base_agent.py:145-147) -assert self.agent_graph is not None, \ - "Agent graph is not set. Please set the agent graph using set_agent_graph method first." -``` - -**Solution:** Create AgentGraph during setup and set it on all agents: - -```python -def setup_agents(self, agent_data, environment, task, user): - # Create agents - agents = self._create_agents(task) - - # Create and configure AgentGraph - from .marble.graph.agent_graph import AgentGraph - graph = AgentGraph(agents, self._build_graph_config(task)) - - # Set graph reference on each agent (REQUIRED) - for agent in agents: - agent.set_agent_graph(graph) - - # Wrap in adapters - adapters = [MarbleAgentAdapter(agent, agent.agent_id) for agent in agents] - agents_dict = {a.name: a for a in adapters} - - return adapters, agents_dict -``` - -### Known MARBLE Bug: chain_coordinate() - -**Bug:** `engine.py:702` calls `self.graph.get_agent_profiles_linked()` which does not exist in AgentGraph. - -**Impact:** Chain coordination mode will crash. - -**Workaround:** Either: -1. Avoid chain mode tasks initially -2. Patch MARBLE locally (add the missing method) -3. Implement chain coordination in MASEval's `run_agents()` - ---- - -## Environment Integration - -### Domain-Specific Requirements - -| Domain | External Dependencies | Initial Support | -|--------|----------------------|-----------------| -| Research | None | ✅ Yes | -| Bargaining | None | ✅ Yes | -| Coding | Filesystem access | ✅ Yes | -| Web | Network access | ✅ Yes | -| WorldSimulation | None | ✅ Yes | -| Database | Docker + PostgreSQL | ⚠️ Optional | -| Minecraft | External game server | ❌ Deferred | - -**Strategy:** Start with domains that don't require external services. Add infrastructure-heavy domains as optional extras with clear skip logic. - -### MultiAgentBenchEnvironment - -```python -class MultiAgentBenchEnvironment(Environment): - """Wraps MARBLE environment instances.""" - - # Domains that require external infrastructure - INFRASTRUCTURE_DOMAINS = {"database", "minecraft"} - - def __init__( - self, - task_data: Dict[str, Any], - callbacks: Optional[List[Any]] = None, - ): - self.domain = task_data.get("scenario", "") - self._marble_env: Optional["BaseEnvironment"] = None - super().__init__(task_data, callbacks) - - def setup_state(self, task_data: Dict[str, Any]) -> Dict[str, Any]: - """Initialize state and create MARBLE environment.""" - domain = task_data.get("scenario", "") - env_config = task_data.get("environment", {}) - - # Check infrastructure requirements - if domain in self.INFRASTRUCTURE_DOMAINS: - if not self._check_infrastructure(domain): - raise EnvironmentError( - f"Domain '{domain}' requires external infrastructure. " - f"See README.md for setup instructions.", - component="MultiAgentBenchEnvironment", - ) - - # Create MARBLE environment - self._marble_env = self._create_marble_environment(domain, env_config) - - return { - "domain": domain, - "env_config": env_config, - "marble_env_type": type(self._marble_env).__name__, - } - - def _check_infrastructure(self, domain: str) -> bool: - """Check if required infrastructure is available.""" - if domain == "database": - # Check Docker availability - import shutil - return shutil.which("docker") is not None - elif domain == "minecraft": - # Minecraft requires external server - always fail for now - return False - return True - - def _create_marble_environment( - self, - domain: str, - env_config: Dict[str, Any], - ) -> "BaseEnvironment": - """Create the appropriate MARBLE environment.""" - from .marble.environments.base_env import BaseEnvironment - - # Import domain-specific environments - env_classes = { - "coding": "marble.environments.coding_env.CodingEnvironment", - "database": "marble.environments.db_env.DBEnvironment", - "research": "marble.environments.research_env.ResearchEnvironment", - "bargaining": "marble.environments.bargaining_env.BargainingEnvironment", - "web": "marble.environments.web_env.WebEnvironment", - "worldsimulation": "marble.environments.world_env.WorldSimulationEnvironment", - } - - env_class_path = env_classes.get(domain.lower()) - if not env_class_path: - # Fallback to base environment - return BaseEnvironment(env_config) - - # Dynamic import - module_path, class_name = env_class_path.rsplit(".", 1) - module = __import__(module_path, fromlist=[class_name]) - env_class = getattr(module, class_name) - return env_class(env_config) - - def create_tools(self) -> Dict[str, Any]: - """Extract tools from MARBLE environment for tracing. - - MARBLE environments expose tools via action_handler_descriptions. - We wrap these for MASEval tracing. - """ - if not self._marble_env: - return {} - - tools = {} - for action_name in self._marble_env._action_handlers: - handler = self._marble_env._action_handlers[action_name] - # Wrap handler for tracing - tools[action_name] = self._wrap_tool_for_tracing(action_name, handler) - return tools - - def _wrap_tool_for_tracing( - self, - name: str, - handler: Callable, - ) -> Callable: - """Wrap a MARBLE action handler for MASEval tracing.""" - tool_history = ToolInvocationHistory() - - def traced_handler(**kwargs) -> Any: - try: - result = handler(**kwargs) - tool_history.add_invocation( - inputs=kwargs, - outputs=result, - status="success", - ) - return result - except Exception as e: - tool_history.add_invocation( - inputs=kwargs, - outputs=str(e), - status="error", - ) - raise - - # Attach history for trace collection - traced_handler._history = tool_history - traced_handler._original_name = name - return traced_handler - - def get_tool(self, name: str) -> Optional[Any]: - """Get a specific tool by name.""" - return self.tools.get(name) - - def gather_traces(self) -> Dict[str, Any]: - """Gather traces including tool invocations.""" - traces = super().gather_traces() - traces["domain"] = self.domain - - # Collect tool invocation histories - tool_traces = {} - for name, tool in self.tools.items(): - if hasattr(tool, "_history"): - tool_traces[name] = { - "invocations": tool._history.to_list(), - } - traces["tool_invocations"] = tool_traces - - return traces -``` - ---- - -## Evaluator Design - -### Metrics Mapping - -MARBLE Evaluator produces metrics that must be mapped to MASEval format: - -| MARBLE Metric | Type | MASEval Mapping | -|---------------|------|-----------------| -| `task_completion` | List[0/1] | `passed` (last value) | -| `token_consumption` | List[int] | `total_tokens` (sum) | -| `planning_score` | List[1-5] | `planning_score` (mean) | -| `communication_score` | List[1-5] | `communication_score` (mean) | -| `code_quality` | Dict | `code_quality` (pass-through) | -| `agent_kpis` | Dict[agent_id, int] | `agent_kpis` (pass-through) | - -### MultiAgentBenchEvaluator - -```python -class MultiAgentBenchEvaluator(Evaluator): - """Wraps MARBLE's Evaluator for MASEval integration.""" - - def __init__( - self, - task: Task, - environment: MultiAgentBenchEnvironment, - agents: Sequence[AgentAdapter], - ): - self.task = task - self.environment = environment - self.agents = agents - self.metrics_config = task.evaluation_data.get("metrics", {}) - - # Create MARBLE evaluator - from .marble.evaluator.evaluator import Evaluator as MarbleEvaluator - self._marble_evaluator = MarbleEvaluator(metrics_config=self.metrics_config) - - def filter_traces(self, traces: Dict[str, Any]) -> Dict[str, Any]: - """Extract data needed for MARBLE evaluation.""" - # Get agent traces - agent_traces = traces.get("agents", {}) - - # Get environment state - env_traces = traces.get("environment", {}) - - # Get tool invocations - tool_traces = env_traces.get("tool_invocations", {}) - - return { - "agent_traces": agent_traces, - "environment": env_traces, - "tool_invocations": tool_traces, - "task_id": self.task.id, - } - - def __call__( - self, - traces: Dict[str, Any], - final_answer: Optional[str] = None, - ) -> Dict[str, Any]: - """Run MARBLE evaluation and convert to MASEval format.""" - # Extract MARBLE agents from adapters - marble_agents = [ - adapter._agent for adapter in self.agents - if hasattr(adapter, "_agent") - ] - - # Call MARBLE evaluator update (REQUIRED - not automatic) - if self.environment._marble_env: - self._marble_evaluator.update( - environment=self.environment._marble_env, - agents=marble_agents, - ) - - # Finalize metrics - self._marble_evaluator.finalize() - - # Get raw metrics - raw_metrics = self._marble_evaluator.get_metrics() - - # Convert to MASEval format - return self._convert_metrics(raw_metrics, final_answer) - - def _convert_metrics( - self, - raw_metrics: Dict[str, Any], - final_answer: Optional[str], - ) -> Dict[str, Any]: - """Convert MARBLE metrics to MASEval format.""" - task_completion = raw_metrics.get("task_completion", []) - planning_scores = raw_metrics.get("planning_score", []) - comm_scores = raw_metrics.get("communication_score", []) - token_counts = raw_metrics.get("token_consumption", []) - - # Compute aggregates - passed = task_completion[-1] == 1 if task_completion else False - planning_score = sum(planning_scores) / len(planning_scores) if planning_scores else 0.0 - communication_score = sum(comm_scores) / len(comm_scores) if comm_scores else 0.0 - total_tokens = sum(token_counts) - - return { - "passed": passed, - "task_completion": task_completion[-1] if task_completion else 0, - "planning_score": planning_score, - "communication_score": communication_score, - "total_tokens": total_tokens, - "code_quality": raw_metrics.get("code_quality", {}), - "agent_kpis": raw_metrics.get("agent_kpis", {}), - "total_milestones": raw_metrics.get("total_milestones", 0), - "raw_metrics": raw_metrics, # Include raw for debugging - } -``` - ---- - -## Error Classification - -Map MARBLE errors to MASEval's `TaskExecutionStatus`: - -```python -from maseval import TaskExecutionStatus, AgentError, EnvironmentError - -def classify_marble_error(error: Exception) -> TaskExecutionStatus: - """Classify MARBLE errors into MASEval status codes.""" - error_str = str(error).lower() - - # Infrastructure failures (not agent's fault) - if any(x in error_str for x in ["docker", "connection", "timeout", "rate limit"]): - return TaskExecutionStatus.ENVIRONMENT_ERROR - - # MARBLE internal bugs (not agent's fault) - if "marble" in error_str or "agentgraph" in error_str: - return TaskExecutionStatus.ENVIRONMENT_ERROR - - # Missing dependencies - if "import" in error_str or "module" in error_str: - return TaskExecutionStatus.SETUP_FAILED - - # LLM/API failures - if any(x in error_str for x in ["api", "openai", "anthropic", "quota"]): - return TaskExecutionStatus.ENVIRONMENT_ERROR - - # Agent assertion failures, invalid actions - if any(x in error_str for x in ["assert", "invalid", "not found"]): - return TaskExecutionStatus.AGENT_ERROR - - # Default to agent error (conservative - holds agent accountable) - return TaskExecutionStatus.AGENT_ERROR - - -def wrap_marble_error(error: Exception) -> Exception: - """Wrap MARBLE exceptions in MASEval exception types.""" - status = classify_marble_error(error) - - if status == TaskExecutionStatus.ENVIRONMENT_ERROR: - return EnvironmentError( - str(error), - component="MARBLE", - ) - elif status == TaskExecutionStatus.AGENT_ERROR: - return AgentError( - str(error), - component="MarbleAgent", - ) - else: - return error -``` - ---- - -## Data Loading - -### Task Loading with Validation - -```python -from pathlib import Path -from typing import List, Optional, Union -import json - -VALID_DOMAINS = frozenset({ - "coding", "database", "minecraft", "research", - "bargaining", "web", "worldsimulation", -}) - -def load_tasks( - domain: str, - data_dir: Optional[Path] = None, - limit: Optional[int] = None, -) -> List[Task]: - """Load MultiAgentBench tasks from JSONL. - - Args: - domain: One of the valid domains (see VALID_DOMAINS) - data_dir: Base data directory (default: auto-detect) - limit: Maximum number of tasks - - Returns: - List of Task objects - - Raises: - ValueError: If domain is invalid or required fields missing - FileNotFoundError: If data files not found - """ - # Validate domain - if domain.lower() not in VALID_DOMAINS: - raise ValueError( - f"Invalid domain '{domain}'. Must be one of: {sorted(VALID_DOMAINS)}" - ) - - # Find data directory - data_dir = _resolve_data_dir(data_dir) - jsonl_path = data_dir / domain / f"{domain}_main.jsonl" - - if not jsonl_path.exists(): - raise FileNotFoundError( - f"Task data not found: {jsonl_path}\n" - f"Ensure MARBLE is cloned to multiagentbench/marble/\n" - f"See multiagentbench/README.md for setup instructions." - ) - - tasks = [] - with jsonl_path.open() as f: - for idx, line in enumerate(f): - if limit and idx >= limit: - break - - entry = json.loads(line) - task = _parse_task_entry(entry, domain, idx) - tasks.append(task) - - return tasks - - -def _resolve_data_dir(data_dir: Optional[Path]) -> Path: - """Resolve the MARBLE data directory.""" - if data_dir: - return Path(data_dir) - - # Check standard locations - candidates = [ - Path(__file__).parent / "marble" / "multiagentbench", - Path.cwd() / "marble" / "multiagentbench", - ] - - # Check environment variable - import os - env_dir = os.environ.get("MARBLE_DATA_DIR") - if env_dir: - candidates.insert(0, Path(env_dir)) - - for candidate in candidates: - if candidate.exists(): - return candidate - - raise FileNotFoundError( - "MARBLE data directory not found. Either:\n" - "1. Clone MARBLE to maseval/benchmark/multiagentbench/marble/\n" - "2. Set MARBLE_DATA_DIR environment variable\n" - "See README.md for setup instructions." - ) - - -def _parse_task_entry( - entry: Dict[str, Any], - domain: str, - idx: int, -) -> Task: - """Parse a JSONL entry into a MASEval Task. - - Raises: - ValueError: If required fields are missing (fail loudly, no defaults) - """ - # Required fields - fail if missing - REQUIRED_FIELDS = ["scenario", "task_id", "task", "agents", "environment", "relationships"] - missing = [f for f in REQUIRED_FIELDS if f not in entry] - if missing: - raise ValueError( - f"Task entry {idx} missing required fields: {missing}\n" - f"Entry keys: {list(entry.keys())}" - ) - - # Validate agent specifications - for i, agent_spec in enumerate(entry["agents"]): - if "agent_id" not in agent_spec: - raise ValueError( - f"Agent {i} in task {entry['task_id']} missing 'agent_id'\n" - f"Agent spec: {agent_spec}" - ) - - # Extract task content - task_content = entry["task"] - if isinstance(task_content, dict): - query = task_content.get("content", "") - else: - query = str(task_content) - - if not query: - raise ValueError( - f"Task {entry['task_id']} has empty query/content" - ) - - return Task( - id=f"{domain}_{entry['task_id']}", - query=query, - environment_data={ - "scenario": entry["scenario"], - "coordinate_mode": entry.get("coordinate_mode", "star"), - "relationships": entry["relationships"], - "environment": entry["environment"], - "task": entry["task"], - "agents": entry["agents"], - "max_iterations": entry.get("max_iterations", 10), - # Store raw entry for MARBLE compatibility - "raw_marble_config": entry, - }, - evaluation_data={ - "metrics": entry.get("metrics", {}), - }, - metadata={ - "domain": domain, - "task_id": entry["task_id"], - }, - ) - - -def configure_model_ids( - tasks: List[Task], - *, - agent_model_id: str, - evaluator_model_id: Optional[str] = None, -) -> List[Task]: - """Configure model IDs for MARBLE agents and evaluator. - - Args: - tasks: List of Tasks - agent_model_id: Model for all MARBLE agents - evaluator_model_id: Optional model for LLM-based evaluation - - Returns: - Tasks with model IDs configured (mutated in place) - """ - for task in tasks: - # Set agent model - task.environment_data["llm"] = agent_model_id - - # Set evaluator model - if evaluator_model_id: - task.evaluation_data["model_id"] = evaluator_model_id - - return tasks -``` - ---- - -## Design Principle Compliance - -### R1: Reuse MASEval ✅ - -- Uses `Benchmark`, `Environment`, `Evaluator`, `AgentAdapter` base classes -- Leverages callback system for tracing -- Uses `Task` data structures -- Benefits from parallel execution, progress bars, error handling - -### R2: Scientific Fidelity ✅ - -- Version pinning via commit hash -- Task data preserved from original JSONL format -- Same coordination strategies available -- MARBLE's agent logic preserved in adapters - -### R3: Fail Loudly ✅ - -- No silent fallbacks that change benchmark results -- Explicit validation with clear error messages -- Infrastructure requirements checked upfront -- Missing fields cause immediate failure - -### R4: Maintainability ✅ - -- Clear module boundaries -- Per-agent adapters match tau2/macs pattern -- Documentation of MARBLE quirks -- Update process documented - ---- - -## Implementation Checklist - -### Phase 1: Core Infrastructure -- [ ] Create `maseval/benchmark/multiagentbench/` directory structure -- [ ] Add `.gitignore` excluding `marble/` -- [ ] Write `README.md` with MARBLE setup instructions -- [ ] Write `PROVENANCE.md` documenting MARBLE version and license -- [ ] Implement `_resolve_data_dir()` for flexible data location - -### Phase 2: Data Loading -- [ ] Implement `load_tasks()` with strict validation -- [ ] Implement `_parse_task_entry()` with required field checks -- [ ] Implement `configure_model_ids()` -- [ ] Unit tests for data loading (valid/invalid inputs) - -### Phase 3: Environment -- [ ] Implement `MultiAgentBenchEnvironment` -- [ ] Implement `_create_marble_environment()` for each supported domain -- [ ] Implement `_check_infrastructure()` for Docker-dependent domains -- [ ] Implement `create_tools()` with tracing wrappers -- [ ] Implement `gather_traces()` with tool invocation histories -- [ ] Unit tests for environment setup - -### Phase 4: Agent Adapter -- [ ] Implement `MarbleAgentAdapter` -- [ ] Implement `_extract_message_history()` using `msg_box` -- [ ] Implement `gather_traces()` with relationships -- [ ] Implement `_extract_relationships()` from AgentGraph -- [ ] Unit tests for adapter - -### Phase 5: Benchmark Class -- [ ] Implement `MultiAgentBenchBenchmark` (abstract base) - - [ ] `setup_environment()` - create MultiAgentBenchEnvironment - - [ ] `setup_user()` - return None (no user simulation) - - [ ] `setup_agents()` - ABSTRACT - - [ ] `setup_evaluators()` - create MultiAgentBenchEvaluator -- [ ] Implement `MarbleMultiAgentBenchBenchmark` - - [ ] `setup_agents()` - create agents, AgentGraph, wrap in adapters - - [ ] `_create_agents()` - instantiate MARBLE BaseAgent instances - - [ ] `_build_graph_config()` - create AgentGraph config from task - - [ ] `run_agents()` - implement coordination modes -- [ ] Implement coordination methods: - - [ ] `_star_coordinate()` - - [ ] `_graph_coordinate()` - - [ ] `_tree_coordinate()` (optional - defer if complex) - - [ ] `_chain_coordinate()` (document MARBLE bug, implement workaround) - -### Phase 6: Evaluator -- [ ] Implement `MultiAgentBenchEvaluator` -- [ ] Implement `filter_traces()` - extract agent/env/tool data -- [ ] Implement `__call__()` with explicit `evaluator.update()` call -- [ ] Implement `_convert_metrics()` - MARBLE → MASEval format -- [ ] Unit tests for evaluator - -### Phase 7: Error Handling -- [ ] Implement `classify_marble_error()` -- [ ] Implement `wrap_marble_error()` -- [ ] Add try/except wrappers in adapter and benchmark -- [ ] Test error classification - -### Phase 8: Integration Testing -- [ ] Create `tests/test_benchmarks/test_multiagentbench/` -- [ ] Integration test: Run 1 Research task -- [ ] Integration test: Run 1 Bargaining task -- [ ] Integration test: Run 1 Coding task -- [ ] Verify traces collected correctly -- [ ] Verify metrics computed correctly -- [ ] Verify error handling works - -### Phase 9: Example & Documentation -- [ ] Create `examples/multiagentbench_marble.py` -- [ ] Update main MASEval README with MultiAgentBench section -- [ ] Document known limitations (chain mode bug, Minecraft not supported) - ---- - -## Setup Instructions - -### Getting MARBLE Source - -```bash -cd maseval/benchmark/multiagentbench -git clone https://github.com/ulab-uiuc/MARBLE.git marble -cd marble -git checkout # Pin to tested version -``` - -**Recommended:** Pin to a specific commit after testing. - -### PROVENANCE.md Template - -```markdown -# MARBLE Integration Provenance - -- **Source**: https://github.com/ulab-uiuc/MARBLE -- **Version**: Commit `` (YYYY-MM-DD) -- **License**: MIT (Copyright 2024 Haofei Yu) -- **Vendoring**: Permitted by MIT license with attribution -- **Paper**: arXiv:2503.01935 -- **Last Updated**: YYYY-MM-DD - -## Known Issues in MARBLE - -1. `AgentGraph.get_agent_profiles_linked()` does not exist - breaks chain coordination -2. SharedMemory is per-agent, not actually shared - -## Local Patches Applied - -- None (document any patches here) -``` - ---- - -## Example Usage - -```python -from maseval.benchmark.multiagentbench import ( - load_tasks, - configure_model_ids, - MarbleMultiAgentBenchBenchmark, -) - -# Load tasks from a simple domain (no Docker required) -tasks = load_tasks("research", limit=5) -configure_model_ids(tasks, agent_model_id="gpt-4o") - -# Create benchmark -class MyMarbleBenchmark(MarbleMultiAgentBenchBenchmark): - def get_model_adapter(self, model_id, **kwargs): - from maseval.interface.openai import OpenAIModelAdapter - adapter = OpenAIModelAdapter(model_id) - if "register_name" in kwargs: - self.register("models", kwargs["register_name"], adapter) - return adapter - -# Run -benchmark = MyMarbleBenchmark() -results = benchmark.run(tasks) - -# Print results -for result in results: - print(f"Task: {result['task_id']}") - print(f"Status: {result['status']}") - if result['eval']: - print(f"Passed: {result['eval'][0]['passed']}") - print(f"Planning Score: {result['eval'][0]['planning_score']:.2f}") -``` - ---- - -## Risks and Mitigations - -| Risk | Probability | Impact | Mitigation | -|------|-------------|--------|------------| -| MARBLE API changes | High | High | Pin commit hash, vendored source allows local patches | -| Chain coordination bug | Confirmed | Medium | Document, implement MASEval workaround | -| Docker-dependent domains | Medium | Low | Clear error messages, skip logic, optional support | -| Evaluation discrepancies | Medium | High | Validation study comparing standalone vs MASEval | - ---- - -**Document Version**: 3.0 -**Date**: 2026-01-19 -**Status**: Ready for Implementation diff --git a/STRATEGY_BACKUP.md b/STRATEGY_BACKUP.md deleted file mode 100644 index 32979671..00000000 --- a/STRATEGY_BACKUP.md +++ /dev/null @@ -1,1120 +0,0 @@ -# MARBLE Integration Strategy for MASEval - -## Executive Summary - -This document proposes integration strategies for bringing MARBLE (Multi-Agent Coordination Backbone with LLM Engine) and its MultiAgentBench benchmark suite into MASEval. After analyzing both codebases, I recommend **Approach 2: Hybrid Vendoring with Adapter Layer** as it best balances scientific fidelity, maintainability, and reusability of MASEval infrastructure. - -**Key Findings**: -1. The tau2/macs patterns transfer smoothly to MultiAgentBench with one critical architectural difference: MARBLE's multi-agent engine requires wrapping as an AgentAdapter rather than implementing individual agent setup. -2. The architecture must support **two complementary purposes**: - - **Exact MARBLE reproduction** via `MarbleMultiAgentBenchBenchmark` (for scientific validation) - - **Fair framework comparison** via abstract `MultiAgentBenchBenchmark` base (for LangGraph, smolagents, CrewAI implementations) -3. This dual-purpose design enables critical research questions: "Which multi-agent framework performs best on the same tasks?" - ---- - -## Background - -### What is MARBLE? - -MARBLE is a multi-agent coordination framework that: -- Orchestrates multiple LLM-based agents using an **Engine** + **AgentGraph** architecture -- Supports various coordination modes (chain, hierarchical, graph-based) -- Provides **SharedMemory** for inter-agent communication -- Includes domain-specific environments (Coding, Database, Minecraft, Research) -- Uses YAML-based configuration for agent relationships and task specifications - -### What is MultiAgentBench? - -MultiAgentBench is MARBLE's benchmark suite featuring: -- **5 domains**: Coding, Database, Minecraft, Research, Bargaining -- **JSONL task format** with rich metadata (agent relationships, coordination modes, evaluation metrics) -- **500+ tasks** testing collaboration and competition scenarios -- Original paper: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents" (arXiv:2503.01935) - -### MASEval Integration Goals - -1. Enable reproduction of MARBLE's published results -2. Allow comparative evaluation of different multi-agent architectures -3. Maintain MASEval's framework-agnostic design -4. Reuse existing MASEval infrastructure (callbacks, tracing, parallelization) - ---- - -## Approach 1: Full Dependency Integration - -### Overview -Install MARBLE as a Python package dependency via `pyproject.toml`. - -### Architecture -``` -maseval/ -├── benchmark/ -│ └── multiagentbench/ -│ ├── __init__.py -│ ├── multiagentbench.py # Benchmark classes -│ ├── environment.py # Environment wrapper -│ ├── evaluator.py # Evaluator wrapper -│ ├── data_loader.py # Task loading utilities -│ └── adapters/ -│ └── marble_adapter.py # MarbleAgentAdapter -``` - -**MARBLE installed as dependency:** -```toml -[project.optional-dependencies] -multiagentbench = ["marble-bench>=0.1.0"] -``` - -### Implementation Pattern - -#### 1. Environment Wrapper -```python -class MultiAgentBenchEnvironment(Environment): - """Wraps MARBLE environment instances.""" - - def __init__(self, task_data: Dict[str, Any], callbacks=None): - self.domain = task_data["scenario"] - self.env_config = task_data["environment"] - super().__init__(task_data, callbacks) - - def setup_state(self, task_data: Dict[str, Any]) -> Any: - # Convert MASEval task data to MARBLE environment config - return { - "domain": task_data["scenario"], - "workspace_dir": task_data["environment"].get("workspace_dir", "workspace"), - "max_iterations": task_data["environment"].get("max_iterations", 10), - } - - def create_tools(self) -> Dict[str, Any]: - # MARBLE environments manage tools internally - # Return empty dict as tools are accessed through MARBLE Engine - return {} -``` - -#### 2. Agent Adapter (Critical Component) -```python -class MarbleAgentAdapter(AgentAdapter): - """Wraps MARBLE's multi-agent Engine as a single agent.""" - - def __init__( - self, - engine: "marble.engine.Engine", - name: str = "marble_engine", - callbacks: Optional[List[AgentCallback]] = None - ): - """ - Args: - engine: MARBLE Engine instance (pre-configured with agents, graph, memory) - name: Adapter name for tracing - callbacks: Optional callbacks - """ - self.engine = engine - super().__init__(engine, name, callbacks) - - def _run_agent(self, query: str) -> Any: - """Execute MARBLE's multi-agent coordination.""" - # MARBLE's Engine.start() runs the full multi-agent workflow - self.engine.start() - - # Extract final output from SharedMemory or designated output agent - final_output = self.engine.memory.get("final_answer") - - # Convert MARBLE's execution trace to MessageHistory - messages = self._convert_to_message_history(self.engine) - self.messages = messages - - return final_output - - def _convert_to_message_history(self, engine) -> MessageHistory: - """Convert MARBLE agent traces to MASEval MessageHistory format.""" - messages = [] - - # Extract messages from all agents in the engine - for agent in engine.agents: - agent_messages = agent.get_messages() # Assume MARBLE agents track messages - for msg in agent_messages: - messages.append({ - "role": f"agent_{agent.agent_id}", - "content": msg.content, - "timestamp": msg.timestamp, - "agent_id": agent.agent_id - }) - - return MessageHistory(messages) - - def gather_traces(self) -> Dict[str, Any]: - """Gather MARBLE-specific traces.""" - return { - **super().gather_traces(), - "coordination_mode": self.engine.coordinate_mode, - "agent_graph": self.engine.graph.to_dict(), - "shared_memory": self.engine.memory.to_dict(), - "iterations": self.engine.current_iteration, - "max_iterations": self.engine.max_iterations, - } -``` - -#### 3. Benchmark Harness -```python -class MultiAgentBenchBenchmark(Benchmark): - """Framework-agnostic MultiAgentBench benchmark. - - Users must implement get_model_adapter() for their LLM provider. - For MARBLE reproduction, use MarbleMultiAgentBenchBenchmark. - """ - - def setup_environment( - self, - agent_data: Dict[str, Any], - task: Task - ) -> MultiAgentBenchEnvironment: - return MultiAgentBenchEnvironment( - task_data=task.environment_data, - ) - - def setup_user( - self, - agent_data: Dict[str, Any], - environment: MultiAgentBenchEnvironment, - task: Task - ) -> Optional[User]: - # MultiAgentBench doesn't use external user simulation - return None - - @abstractmethod - def setup_agents( - self, - agent_data: Dict[str, Any], - environment: MultiAgentBenchEnvironment, - task: Task, - user: Optional[User], - ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: - """Must be implemented by subclass for specific multi-agent framework.""" - pass - - def setup_evaluators( - self, - environment: MultiAgentBenchEnvironment, - task: Task, - agents: Sequence[AgentAdapter], - user: Optional[User], - ) -> Sequence[Evaluator]: - return [ - MultiAgentBenchEvaluator( - task=task, - environment=environment, - ) - ] - - def run_agents( - self, - agents: Sequence[AgentAdapter], - task: Task, - environment: MultiAgentBenchEnvironment, - query: str = "", - ) -> Any: - # For MultiAgentBench, run single "agent" (which is the multi-agent engine) - return agents[0].run(query) - - -class MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark): - """MARBLE-specific implementation for reproduction.""" - - def setup_agents( - self, - agent_data: Dict[str, Any], - environment: MultiAgentBenchEnvironment, - task: Task, - user: Optional[User], - ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: - """Create MARBLE Engine and wrap as single agent.""" - from marble.configs.config import Config - from marble.engine.engine import Engine - - # Convert MASEval task to MARBLE config - marble_config = self._build_marble_config(agent_data, task) - - # Create MARBLE Engine (contains multiple agents internally) - engine = Engine(marble_config) - - # Wrap engine as single AgentAdapter - engine_adapter = MarbleAgentAdapter( - engine=engine, - name="marble_engine", - ) - - # Return as single agent (MASEval runs this one adapter) - return [engine_adapter], {"marble_engine": engine_adapter} - - def _build_marble_config( - self, - agent_data: Dict[str, Any], - task: Task - ) -> "Config": - """Convert MASEval Task to MARBLE Config.""" - from marble.configs.config import Config - - config_dict = { - "coordinate_mode": task.environment_data.get("coordinate_mode", "chain"), - "relationships": task.environment_data.get("relationships", []), - "llm": agent_data.get("model_id", "gpt-4"), - "environment": task.environment_data.get("environment", {}), - "task": task.environment_data.get("task", {}), - "agents": task.environment_data.get("agents", []), - "memory": task.environment_data.get("memory", {"type": "SharedMemory"}), - "metrics": task.evaluation_data.get("metrics", {}), - "engine_planner": task.environment_data.get("engine_planner", {}), - } - - return Config.from_dict(config_dict) -``` - -#### 4. Data Loading -```python -def load_tasks( - domain: str, - data_dir: Optional[Path] = None, - limit: Optional[int] = None, -) -> TaskQueue: - """Load MultiAgentBench tasks from JSONL. - - Args: - domain: One of "coding", "database", "minecraft", "research", "bargaining" - data_dir: Base data directory (default: multiagentbench package data) - limit: Maximum number of tasks to load - - Returns: - TaskQueue containing Task objects - """ - data_dir = Path(data_dir) if data_dir else DEFAULT_DATA_DIR - jsonl_path = data_dir / f"{domain}_main.jsonl" - - tasks = [] - with jsonl_path.open() as f: - for idx, line in enumerate(f): - if limit and idx >= limit: - break - - entry = json.loads(line) - - # Convert JSONL entry to MASEval Task - task = Task( - id=f"{domain}_{entry['task_id']}", - query=entry["task"]["content"], - environment_data={ - "scenario": entry["scenario"], - "coordinate_mode": entry["coordinate_mode"], - "relationships": entry["relationships"], - "environment": entry["environment"], - "task": entry["task"], - "agents": entry["agents"], - "memory": entry["memory"], - "engine_planner": entry.get("engine_planner", {}), - }, - evaluation_data={ - "metrics": entry.get("metrics", {}), - }, - metadata={ - "domain": domain, - "task_id": entry["task_id"], - "llm": entry.get("llm", ""), - }, - ) - tasks.append(task) - - return TaskQueue(tasks) - - -def configure_model_ids( - tasks: Union[TaskQueue, List[Task]], - *, - agent_model_id: str, - evaluator_model_id: Optional[str] = None, -) -> Union[TaskQueue, List[Task]]: - """Configure model IDs for MARBLE agents and evaluator.""" - for task in tasks: - # Set model for MARBLE agents - task.environment_data["llm"] = agent_model_id - - # Set evaluator model if LLM-based evaluation needed - if evaluator_model_id: - task.evaluation_data["evaluate_llm"] = evaluator_model_id - - return tasks -``` - -### Pros -✅ **Clean separation**: MARBLE remains an independent package -✅ **Standard Python packaging**: Uses established dependency management -✅ **Easy updates**: Can upgrade MARBLE version via `uv add --optional multiagentbench marble-bench@latest` -✅ **R4 (Maintainability)**: Clear boundary between MASEval and MARBLE code - -### Cons -❌ **MARBLE not on PyPI**: Would need to install from Git or build local wheel -❌ **Version pinning challenges**: MARBLE development may break compatibility -❌ **Dependency bloat**: MARBLE's dependencies become transitive dependencies -❌ **Limited control**: Can't patch MARBLE bugs without forking - -### Design Principle Assessment - -- **R1 (Reuse MASEval)**: ✅ Excellent - uses all MASEval patterns -- **R2 (Scientific fidelity)**: ✅ Good - depends on MARBLE version stability -- **R3 (Fail loudly)**: ✅ Good - validation at config conversion boundary -- **R4 (Maintainability)**: ⚠️ Moderate - depends on MARBLE API stability - ---- - -## Approach 2: Hybrid Vendoring with Adapter Layer (RECOMMENDED) - -### Overview -Clone MARBLE source into `maseval/benchmark/multiagentbench/marble/` (gitignored), create thin adapter layer in MASEval. - -### Architecture -``` -maseval/ -├── benchmark/ -│ └── multiagentbench/ -│ ├── __init__.py -│ ├── multiagentbench.py # Benchmark + Evaluator -│ ├── environment.py # Environment wrapper -│ ├── data_loader.py # load_tasks(), configure_model_ids() -│ ├── .gitignore # Ignore marble/ directory -│ ├── marble/ # ← Vendored MARBLE source (gitignored) -│ │ ├── __init__.py -│ │ ├── agent/ -│ │ ├── engine/ -│ │ ├── environments/ -│ │ ├── evaluator/ -│ │ ├── graph/ -│ │ ├── llms/ -│ │ ├── memory/ -│ │ └── utils/ -│ └── README.md # Setup instructions -``` - -**`.gitignore` in `multiagentbench/`:** -``` -marble/ -``` - -**`README.md` in `multiagentbench/`:** -```markdown -# MultiAgentBench Integration - -## Setup - -1. Clone MARBLE into this directory: - ```bash - cd maseval/benchmark/multiagentbench - git clone https://github.com/ulab-uiuc/MARBLE.git marble - cd marble - git checkout # Pin to tested version - ``` - -2. Install MARBLE dependencies: - ```bash - uv pip install -r marble/pyproject.toml - ``` - -## Data - -MultiAgentBench tasks are located in `marble/multiagentbench/`. -``` - -### Implementation Pattern - -Same as Approach 1, but: -- Import from vendored source: `from .marble.engine.engine import Engine` -- Version control via documented commit hash in README -- Can patch MARBLE locally if needed (document patches in `MARBLE_PATCHES.md`) - -### Pros -✅ **Full control**: Can patch bugs, optimize, or modify MARBLE as needed -✅ **Reproducibility**: Pin exact MARBLE version via commit hash -✅ **No external dependency**: Works offline, no PyPI/Git availability issues -✅ **Flexible testing**: Can test against multiple MARBLE versions easily -✅ **R2 (Scientific fidelity)**: Guaranteed exact MARBLE behavior - -### Cons -❌ **Setup friction**: Users must manually clone MARBLE (documented in README) -❌ **Disk space**: Each MASEval checkout includes full MARBLE source -❌ **Update overhead**: Must manually update vendored copy -❌ **Licensing**: Must verify MARBLE's MIT license allows vendoring - -### Design Principle Assessment - -- **R1 (Reuse MASEval)**: ✅ Excellent - identical adapter code to Approach 1 -- **R2 (Scientific fidelity)**: ✅ Excellent - exact version pinning -- **R3 (Fail loudly)**: ✅ Good - clear setup instructions, missing vendor errors -- **R4 (Maintainability)**: ✅ Good - documented update process, local patches possible - -### Why This is Recommended - -1. **Scientific Reproducibility**: Pinning exact MARBLE commit ensures bit-for-bit reproduction of results -2. **Development Velocity**: Can iterate on integration without waiting for MARBLE releases -3. **Risk Mitigation**: MARBLE is early-stage (v0.1.0) and may have breaking changes -4. **Pragmatic**: MASEval is a research tool, not production software - vendoring is acceptable - ---- - -## Approach 3: Multi-Framework Comparison Architecture - -### Overview -**This is NOT a replacement for MARBLE - it's the comparison mechanism.** The architecture enables: -1. **Exact MARBLE reproduction** via `MarbleMultiAgentBenchBenchmark` -2. **Alternative framework implementations** via custom subclasses (LangGraph, smolagents, CrewAI) -3. **Fair comparison** - all frameworks run on the same tasks, environment, and evaluation - -### Architecture -```python -class MultiAgentBenchBenchmark(Benchmark): - """Abstract base providing task/eval infrastructure for ANY multi-agent framework. - - Subclasses: - - MarbleMultiAgentBenchBenchmark: Exact MARBLE reproduction - - LangGraphMultiAgentBenchBenchmark: LangGraph implementation - - SmolagentsMultiAgentBenchBenchmark: Smolagents implementation - - CrewAIMultiAgentBenchBenchmark: CrewAI implementation - """ - - @abstractmethod - def setup_agents(self, agent_data, environment, task, user): - """Implement multi-agent coordination using your chosen framework.""" - pass -``` - -**Example: LangGraph implementation for comparison** -```python -class LangGraphMultiAgentBench(MultiAgentBenchBenchmark): - """LangGraph implementation of MultiAgentBench tasks. - - Uses same tasks, environment, and evaluation as MARBLE, - but implements multi-agent coordination with LangGraph. - """ - - def setup_agents(self, agent_data, environment, task, user): - from langgraph.graph import StateGraph - - # Create graph-based multi-agent system - graph = StateGraph(...) - - # Add agents from task specification - for agent_spec in task.environment_data["agents"]: - agent = self._create_langgraph_agent(agent_spec) - graph.add_node(agent_spec["agent_id"], agent) - - # Add edges based on relationships - for src, dst, rel_type in task.environment_data["relationships"]: - graph.add_edge(src, dst) - - compiled_graph = graph.compile() - adapter = LangGraphAdapter(compiled_graph, "langgraph_system") - return [adapter], {"langgraph_system": adapter} -``` - -### Purpose: Enable Fair Multi-Agent Framework Comparison - -This architecture enables researchers to ask: -- **"How does MARBLE compare to LangGraph on the same tasks?"** -- **"Which coordination strategy works best for coding tasks?"** -- **"Does hierarchical vs. graph-based coordination matter?"** - -All comparisons use: -- ✅ Same task specifications (from JSONL) -- ✅ Same evaluation metrics (code quality, collaboration) -- ✅ Same environment (Coding, DB, Minecraft, etc.) -- ✅ Only difference: multi-agent coordination implementation - -### Pros -✅ **Enables scientific comparison**: Test multiple multi-agent architectures fairly -✅ **Preserves MARBLE reproduction**: MarbleMultiAgentBenchBenchmark still exists -✅ **Framework flexibility**: Users can implement with any multi-agent library -✅ **R1 (Reuse MASEval)**: Perfect - pure MASEval patterns -✅ **Research value**: Answers "which multi-agent approach is better?" - -### Cons -⚠️ **Implementation effort**: Each framework requires custom adapter -⚠️ **Results divergence**: Different frameworks produce different results (expected) -⚠️ **Coordination semantics**: Not all frameworks can express all coordination patterns - -### Design Principle Assessment - -- **R1 (Reuse MASEval)**: ✅ Perfect - clean abstraction -- **R2 (Scientific fidelity)**: ✅ **FOR MARBLE** (via MarbleMultiAgentBenchBenchmark), ✅ **FOR COMPARISONS** (same tasks/eval) -- **R3 (Fail loudly)**: ✅ Clear separation between reproduction and comparison -- **R4 (Maintainability)**: ✅ Excellent - each framework is independent - -**Verdict**: This is NOT an alternative to MARBLE integration - it's the **comparison layer** that makes MultiAgentBench valuable for multi-agent systems research. The architecture must support BOTH exact MARBLE reproduction AND alternative framework implementations. - ---- - -## Pattern Transfer Analysis: Do tau2/macs Patterns Apply? - -### Summary: YES, with one architectural adaptation - -The tau2/macs integration patterns transfer smoothly to MultiAgentBench **with one critical difference**: the multi-agent engine must be wrapped as a single `AgentAdapter` rather than setting up individual agents. - -### Pattern Comparison - -| Component | MACS/Tau2 Pattern | MultiAgentBench Pattern | Transfer Success | -|-----------|-------------------|-------------------------|------------------| -| **Environment** | Wraps domain-specific environment (tools, database) | Wraps MARBLE domain environment (Coding, DB, Minecraft) | ✅ Direct transfer | -| **Evaluator** | filter_traces() + LLM or deterministic eval | MARBLE evaluator metrics (code quality, collaboration) | ✅ Direct transfer | -| **Data Loading** | load_tasks() from JSON/JSONL | load_tasks() from JSONL | ✅ Direct transfer | -| **Model Config** | configure_model_ids() sets LLM per component | configure_model_ids() sets LLM for all MARBLE agents | ✅ Direct transfer | -| **Benchmark Base** | Abstract base, user implements setup_agents() | Abstract base, user implements setup_agents() | ✅ Direct transfer | -| **Agent Setup** | Create individual AgentAdapters | **DIFFERENT**: Wrap entire MARBLE Engine as single adapter | ⚠️ Requires adaptation | - -### Key Architectural Difference: MarbleAgentAdapter - -**MACS/Tau2 Pattern:** -```python -def setup_agents(self, agent_data, environment, task, user): - # Create individual agent adapters - agent1 = MyAgent("supervisor", tools=environment.tools) - agent2 = MyAgent("worker", tools=environment.tools) - - adapter1 = AgentAdapter(agent1, "supervisor") - adapter2 = AgentAdapter(agent2, "worker") - - return [adapter1], {"supervisor": adapter1, "worker": adapter2} -``` - -**MultiAgentBench Pattern (Approach 1 & 2):** -```python -def setup_agents(self, agent_data, environment, task, user): - # MARBLE Engine contains multiple agents internally - from marble.engine.engine import Engine - from marble.configs.config import Config - - config = self._build_marble_config(task) # Includes agent specs - engine = Engine(config) # Engine manages multiple BaseAgents - - # Wrap entire engine as single adapter - engine_adapter = MarbleAgentAdapter(engine, "marble_engine") - - return [engine_adapter], {"marble_engine": engine_adapter} -``` - -**Why this works:** -- MASEval's `Benchmark.run_agents()` calls `agent.run(query)` on agents in the returned list -- `MarbleAgentAdapter._run_agent()` delegates to `engine.start()`, which coordinates all internal agents -- Individual agent messages are aggregated in `MarbleAgentAdapter.gather_traces()` -- From MASEval's perspective, the multi-agent system appears as a single "black box" agent - -### Benefits of This Pattern - -1. **Preserves MARBLE's coordination logic**: No need to reimplement AgentGraph, SharedMemory, EnginePlanner -2. **Clean abstraction boundary**: MASEval orchestrates benchmark execution, MARBLE handles multi-agent dynamics -3. **Scientific fidelity**: Using MARBLE's Engine exactly as designed ensures reproducible results -4. **Framework comparison**: Easy to swap MARBLE for LangGraph, CrewAI, etc. by implementing different adapters - -### Limitations - -- **Less granular tracing**: Individual agent messages require extracting from Engine internals -- **Single invocation**: MASEval's `max_invocations` doesn't apply to MARBLE's internal iterations - - **Solution**: MARBLE's `max_iterations` in config controls internal agent rounds -- **Callback integration**: MARBLE's internal agent callbacks don't automatically trigger MASEval callbacks - - **Solution**: MarbleAgentAdapter can bridge by polling Engine state in `gather_traces()` - -### Verdict: Strong Pattern Transfer - -✅ **4/5 components** transfer directly -⚠️ **1/5 components** (agent setup) requires architectural adaptation that is **clean and well-motivated** - ---- - -## Critical Implementation Details - -### 1. Message History Conversion - -**Challenge**: MARBLE's agents track messages internally; MASEval expects `MessageHistory` from `AgentAdapter.get_messages()`. - -**Solution**: -```python -class MarbleAgentAdapter(AgentAdapter): - def _convert_to_message_history(self, engine) -> MessageHistory: - """Extract and aggregate messages from all MARBLE agents.""" - messages = [] - - for agent in engine.agents: - # Assume MARBLE agents store messages (may need to add this) - agent_messages = getattr(agent, "messages", []) - - for msg in agent_messages: - messages.append({ - "role": f"agent_{agent.agent_id}", - "content": str(msg), - "agent_id": agent.agent_id, - "timestamp": getattr(msg, "timestamp", None), - }) - - # Include shared memory state - memory_state = engine.memory.to_dict() - messages.append({ - "role": "system", - "content": f"Shared Memory State: {json.dumps(memory_state)}", - }) - - return MessageHistory(messages) -``` - -**Risk**: MARBLE agents may not expose message history. -**Mitigation**: Add message tracking to MARBLE BaseAgent if needed (document patch). - -### 2. Environment Tool Integration - -**Challenge**: MARBLE environments manage tools internally; MASEval expects `Environment.create_tools()`. - -**Solution**: Return empty dict, document that tools are accessed through MARBLE Engine. -```python -class MultiAgentBenchEnvironment(Environment): - def create_tools(self) -> Dict[str, Any]: - # MARBLE environments manage tools internally - # Tools are available to MARBLE agents through Engine - return {} - - def gather_traces(self) -> Dict[str, Any]: - """Override to extract tool traces from MARBLE environment.""" - return { - **super().gather_traces(), - "marble_env_type": self.domain, - "marble_tools": self._extract_marble_tool_traces(), - } -``` - -**R3 (Fail loudly)**: If user calls `environment.get_tool()`, raise: -```python -def get_tool(self, name: str) -> Optional[Any]: - raise NotImplementedError( - "MultiAgentBenchEnvironment tools are managed by MARBLE Engine. " - "Access tools through MARBLE agents, not directly from environment." - ) -``` - -### 3. Evaluator Design - -**Challenge**: MultiAgentBench uses MARBLE's Evaluator with domain-specific metrics (code_quality, test_coverage, collaboration_effectiveness). - -**Solution**: Wrap MARBLE evaluator in MASEval Evaluator pattern. -```python -class MultiAgentBenchEvaluator(Evaluator): - """Wraps MARBLE's Evaluator for MASEval integration.""" - - def __init__(self, task: Task, environment: MultiAgentBenchEnvironment): - self.task = task - self.environment = environment - self.metrics_config = task.evaluation_data.get("metrics", {}) - - # Create MARBLE evaluator - from marble.evaluator.evaluator import Evaluator as MarbleEvaluator - self.marble_evaluator = MarbleEvaluator(metrics_config=self.metrics_config) - - def filter_traces(self, traces: Dict[str, Any]) -> Dict[str, Any]: - """Extract MARBLE-specific execution data.""" - # For coding tasks: extract generated code - # For DB tasks: extract query sequences - # For collaboration tasks: extract agent interaction patterns - - marble_engine = traces["agents"]["marble_engine"] - return { - "engine_traces": marble_engine, - "environment_state": traces.get("environment", {}), - } - - def __call__( - self, - traces: Dict[str, Any], - final_answer: Optional[str] = None - ) -> Dict[str, Any]: - """Run MARBLE evaluation metrics.""" - # Extract relevant data - engine_traces = traces["engine_traces"] - - # Call MARBLE evaluator - results = self.marble_evaluator.evaluate( - output=final_answer, - task_spec=self.task.evaluation_data, - traces=engine_traces, - ) - - return results # Should include code_quality, test_coverage, etc. -``` - -### 4. Data Format Validation - -**R3 (Fail loudly)**: Validate JSONL→Task conversion catches missing fields. - -```python -def load_tasks(domain: str, ...) -> TaskQueue: - REQUIRED_FIELDS = [ - "scenario", "task_id", "task", "agents", - "environment", "relationships" - ] - - for entry in jsonl_entries: - missing = [f for f in REQUIRED_FIELDS if f not in entry] - if missing: - raise ValueError( - f"Task {entry.get('task_id', 'unknown')} missing required fields: {missing}. " - f"Ensure MultiAgentBench JSONL format is correct." - ) - - # Validate agent specifications - for agent_spec in entry["agents"]: - if "agent_id" not in agent_spec: - raise ValueError( - f"Agent in task {entry['task_id']} missing 'agent_id'. " - f"Agent spec: {agent_spec}" - ) -``` - -### 5. Model Configuration - -**Pattern**: Similar to MACS, use `configure_model_ids()` to inject runtime model config. - -```python -def configure_model_ids( - tasks: Union[TaskQueue, List[Task]], - *, - agent_model_id: str, - evaluator_model_id: Optional[str] = None, -) -> Union[TaskQueue, List[Task]]: - """Configure LLM model IDs for MARBLE agents and evaluator. - - Args: - tasks: TaskQueue or list of Tasks - agent_model_id: Model ID for all MARBLE agents (e.g., "gpt-4", "claude-sonnet-4.5") - evaluator_model_id: Optional model ID for LLM-based evaluation metrics - - Returns: - Tasks with model IDs configured in environment_data and evaluation_data - """ - for task in tasks: - # Set global LLM for all MARBLE agents - task.environment_data["llm"] = agent_model_id - - # Individual agents can override via agent_config["llm"] - # This is preserved in task.environment_data["agents"] - - # Set evaluator model if LLM-based metrics enabled - if evaluator_model_id and task.evaluation_data.get("metrics", {}).get("use_llm_eval"): - task.evaluation_data["evaluate_llm"] = evaluator_model_id - - return tasks -``` - ---- - -## Recommendations - -### Primary Recommendation: Approach 2 (Hybrid Vendoring) - -**Adopt Approach 2** for initial integration due to: - -1. **Scientific Fidelity (R2)**: Guarantees exact MARBLE behavior via version pinning -2. **Development Velocity**: Enables rapid iteration without dependency management overhead -3. **Risk Mitigation**: MARBLE is early-stage; vendoring provides stability -4. **Pragmatic**: Research tool priorities favor reproducibility over packaging aesthetics - -### Migration Path: Approach 2 → Approach 1 - -Once MARBLE stabilizes (v1.0+ release, stable API), migrate to Approach 1: - -1. Publish MARBLE to PyPI with semantic versioning -2. Replace vendored source with `[project.optional-dependencies]` entry -3. Keep adapter code identical (no MASEval code changes) -4. Document migration in changelog - -**This is a low-risk transition** because adapter code is identical between approaches. - -### Approach 3 is Essential, Not Optional - -**IMPORTANT**: Approach 3 is NOT a replacement for MARBLE reproduction - it's the **reason MultiAgentBench is valuable for research**. - -The complete integration must support: -1. ✅ **Exact MARBLE reproduction** via `MarbleMultiAgentBenchBenchmark` (Approach 1 or 2) -2. ✅ **Fair framework comparison** via abstract `MultiAgentBenchBenchmark` base (Approach 3) - -**Research value**: Enables questions like: -- "Does MARBLE's coordination outperform LangGraph?" -- "Which multi-agent framework is best for coding tasks?" -- "How important is the coordination strategy vs. the LLM?" - -**Implementation**: Approach 3 is simply an abstract base class - it requires minimal code since the infrastructure (Environment, Evaluator, data loading) is shared with MARBLE reproduction. - ---- - -## Design Principle Compliance - -### R1: Reuse MASEval Infrastructure ✅ - -**Fully compliant.** The integration: -- Uses `Benchmark`, `Environment`, `Evaluator`, `AgentAdapter` base classes -- Leverages callback system for tracing -- Uses `TaskQueue` and `Task` data structures -- Benefits from parallel execution, progress bars, error handling -- No reimplementation of MASEval features - -### R2: Scientific Fidelity ✅ - -**Compliant with caveats.** - -**Preserved**: -- Uses MARBLE's Engine, AgentGraph, SharedMemory exactly as designed -- Version pinning via vendored source (Approach 2) or Git dependency (Approach 1) -- Task data preserved from original JSONL format - -**Trade-offs**: -- **Different execution environment**: MASEval orchestration vs. standalone MARBLE - - *Mitigation*: MarbleAgentAdapter delegates directly to Engine.start() - - *Impact*: Minimal - same agent coordination logic runs -- **Aggregated agent messages**: Individual agent traces must be extracted - - *Mitigation*: Implement comprehensive trace extraction in MarbleAgentAdapter - - *Impact*: Low - agent interactions are preserved, just formatted differently - -**Validation strategy**: -1. Run subset of tasks with both standalone MARBLE and MASEval integration -2. Compare outputs, metrics, and agent behaviors -3. Document any discrepancies -4. If material differences exist, adjust adapter implementation - -### R3: Fail Loudly ✅ - -**Fully compliant.** The integration includes: - -**Validation at boundaries**: -```python -# Data loading -if "agent_id" not in agent_spec: - raise ValueError("Agent missing 'agent_id'") - -# Environment tools -def get_tool(self, name): - raise NotImplementedError("Tools managed by MARBLE Engine") - -# Missing config -if not Path("marble").exists(): - raise FileNotFoundError( - "MARBLE source not found. See multiagentbench/README.md for setup." - ) -``` - -**No defensive defaults**: -- Missing JSONL fields → crash with clear error -- Invalid agent relationships → propagate MARBLE error -- Missing MARBLE dependency → crash at import - -**Clear error messages**: -- Include context (task ID, agent ID, field name) -- Suggest fixes ("See README.md for setup") -- Link to documentation - -### R4: Maintainability ✅ - -**Compliant.** The integration: - -**Clear module boundaries**: -- `data_loader.py`: JSONL → Task conversion (0 dependencies on MARBLE internals) -- `environment.py`: Thin wrapper, delegates to MARBLE -- `multiagentbench.py`: Benchmark + Evaluator, well-documented adapter pattern -- `marble/`: Vendored source, not modified (patches documented separately if needed) - -**Documentation**: -- `README.md`: Setup instructions, MARBLE version pinning -- `PROVENANCE.md`: Track MARBLE commit hash, upstream changes -- Docstrings: Explain adapter rationale, MARBLE delegation - -**Update process**: -```bash -# Update vendored MARBLE -cd maseval/benchmark/multiagentbench/marble -git pull origin main -git checkout - -# Test integration -cd ../../../.. -pytest tests/test_benchmarks/test_multiagentbench/ -v - -# Document update -echo "" > multiagentbench/MARBLE_VERSION.txt -``` - -**Long-term maintenance**: -- **Low coupling**: Adapter layer isolates MASEval from MARBLE internals -- **Testable**: Can mock MARBLE Engine for unit tests -- **Upgradable**: Switching to Approach 1 (dependency) requires zero adapter code changes - ---- - -## Implementation Checklist - -### Phase 1: Core Integration (Approach 2) -- [ ] Create `maseval/benchmark/multiagentbench/` directory -- [ ] Add `.gitignore` to exclude `marble/` -- [ ] Write `README.md` with MARBLE setup instructions -- [ ] Implement `MultiAgentBenchEnvironment(Environment)` -- [ ] Implement `MarbleAgentAdapter(AgentAdapter)` - - [ ] `_run_agent()`: Delegate to Engine.start() - - [ ] `_convert_to_message_history()`: Extract agent messages - - [ ] `gather_traces()`: Include AgentGraph, SharedMemory -- [ ] Implement `MultiAgentBenchBenchmark(Benchmark)` (abstract base) -- [ ] Implement `MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark)` - - [ ] `setup_agents()`: Create MARBLE Engine, wrap in adapter - - [ ] `_build_marble_config()`: Convert Task → MARBLE Config -- [ ] Implement `MultiAgentBenchEvaluator(Evaluator)` - - [ ] Wrap MARBLE's Evaluator - - [ ] Extract metrics (code_quality, test_coverage, collaboration) - -### Phase 2: Data Loading -- [ ] Implement `load_tasks(domain, limit)` in `data_loader.py` - - [ ] Validate JSONL fields (R3: Fail loudly) - - [ ] Convert to Task objects -- [ ] Implement `configure_model_ids(tasks, agent_model_id, evaluator_model_id)` -- [ ] Add tests for data loading with sample JSONL - -### Phase 3: Testing & Validation -- [ ] Create `tests/test_benchmarks/test_multiagentbench/` -- [ ] Unit tests for data loading -- [ ] Unit tests for Config conversion -- [ ] Integration test: Run 1 task from each domain -- [ ] Validation test: Compare results with standalone MARBLE (same task, same model) - -### Phase 4: Documentation & Examples -- [ ] Example script: `examples/multiagentbench_marble.py` -- [ ] Document setup in main MASEval README -- [ ] Add to documentation site (if exists) -- [ ] Write `PROVENANCE.md`: Track MARBLE version, license, upstream - -### Phase 5: Alternative Framework Examples (Essential for Research Value) -- [ ] Implement `LangGraphMultiAgentBench` as example of alternative multi-agent framework -- [ ] Implement `SmolagentsMultiAgentBench` as second comparison point -- [ ] Shows how users can bring their own multi-agent framework -- [ ] Enables comparative research: "MARBLE vs LangGraph vs Smolagents on same tasks" -- [ ] Demonstrates that `MultiAgentBenchBenchmark` abstract base works for ANY framework -- [ ] Document performance comparison methodology - ---- - -## Risks and Mitigations - -### Risk 1: MARBLE API Changes -**Probability**: High (early-stage project) -**Impact**: High (breaks integration) -**Mitigation**: -- Pin exact commit hash in README -- Approach 2 (vendoring) allows local patches -- Automated tests detect breakage -- Document MARBLE version compatibility matrix - -### Risk 2: Message History Extraction -**Probability**: Medium (depends on MARBLE internals) -**Impact**: Medium (affects tracing quality) -**Mitigation**: -- Test message extraction thoroughly -- If MARBLE agents don't expose messages, contribute PR upstream -- Fallback: Extract from SharedMemory state instead - -### Risk 3: License Compatibility -**Probability**: None (VERIFIED ✅) -**Impact**: N/A -**Verification**: -- ✅ MARBLE uses MIT License (Copyright 2024 Haofei Yu) -- ✅ MIT explicitly allows: "use, copy, modify, merge, publish, distribute" without restriction -- ✅ Only requirement: Include copyright notice and license file -**Implementation**: -- Keep `LICENSE` file in vendored `marble/` directory -- Document in `PROVENANCE.md`: "MARBLE is MIT licensed, vendoring permitted with attribution" - -### Risk 4: User Setup Friction -**Probability**: High (manual MARBLE clone) -**Impact**: Low (one-time setup) -**Mitigation**: -- Clear README instructions -- Provide setup script: `bash setup_multiagentbench.sh` -- Fail fast with helpful error if MARBLE missing -- Future: Migrate to Approach 1 when MARBLE stabilizes - -### Risk 5: Evaluation Discrepancies -**Probability**: Medium (different execution context) -**Impact**: High (invalidates scientific fidelity) -**Mitigation**: -- Run validation study: MASEval vs. standalone MARBLE -- Compare metrics, outputs, agent behaviors -- Document any differences -- Adjust adapter if needed -- If irreconcilable, clearly document limitations - ---- - -## Conclusion - -**Adopt Approach 2 (Hybrid Vendoring)** as the recommended integration strategy for MARBLE/MultiAgentBench into MASEval. - -This approach: -- ✅ Satisfies all four design principles (R1-R4) -- ✅ Enables reproduction of MARBLE's published results (via `MarbleMultiAgentBenchBenchmark`) -- ✅ Enables fair multi-agent framework comparison (via abstract `MultiAgentBenchBenchmark` base) -- ✅ Provides version stability during MARBLE's early development -- ✅ Reuses MASEval infrastructure effectively -- ✅ Maintains clear architectural boundaries -- ✅ Allows future migration to dependency-based approach - -**Key architectural insights**: - -1. **MARBLE's multi-agent Engine must be wrapped as a single AgentAdapter**, not decomposed into individual agents. This preserves MARBLE's coordination logic while integrating cleanly with MASEval's orchestration framework. - -2. **The architecture must support dual purposes**: Exact MARBLE reproduction (for scientific validation) AND alternative framework implementations (for comparative research). This is what makes MultiAgentBench valuable - researchers can ask "Which multi-agent approach performs better?" on the same task set. - -3. **License verified**: MARBLE's MIT license explicitly permits vendoring with attribution (copyright notice + LICENSE file in vendored directory). - -The tau2/macs integration patterns transfer successfully with this one architectural adaptation, demonstrating that MASEval's design is flexible enough to accommodate diverse benchmarking paradigms - including multi-agent systems research. - ---- - -## Appendices - -### Appendix A: Complete Code Example - -See implementation checklist for full module structure. Key classes: -- `MarbleAgentAdapter`: Wraps Engine as single agent -- `MarbleMultiAgentBenchBenchmark`: Creates Engine from Task -- `MultiAgentBenchEvaluator`: Wraps MARBLE metrics -- `load_tasks()`: JSONL → Task conversion - -### Appendix B: MARBLE Architecture Summary - -- **Engine**: Main orchestrator -- **BaseAgent**: Individual agent with LLM -- **AgentGraph**: Manages agent relationships and coordination -- **SharedMemory**: Inter-agent communication -- **EnginePlanner**: Plans agent execution order -- **Environments**: Domain-specific (Coding, DB, Minecraft, Research, Web, WorldSimulation) -- **Evaluator**: Computes domain-specific metrics - -### Appendix C: Task Format Mapping - -**JSONL → Task:** -- `task_id` → `Task.id` (prefixed with domain) -- `task.content` → `Task.query` -- `scenario`, `agents`, `environment`, etc. → `Task.environment_data` -- `metrics` → `Task.evaluation_data` -- Remaining fields → `Task.metadata` - -### Appendix D: Alternative Multi-Agent Frameworks - -After MARBLE integration is stable, users can create custom multi-agent benchmarks: -- LangGraph: Graph-based multi-agent workflows -- CrewAI: Role-based agent teams -- AutoGen: Conversational multi-agent systems -- Custom: Domain-specific coordination logic - -Pattern: Subclass `MultiAgentBenchBenchmark`, implement `setup_agents()` with chosen framework. - ---- - -**Document Version**: 1.0 -**Date**: 2026-01-19 -**Author**: Claude (Sonnet 4.5) -**Status**: Final Recommendation diff --git a/STRATEGY_CRITIQUE.md b/STRATEGY_CRITIQUE.md deleted file mode 100644 index 30937736..00000000 --- a/STRATEGY_CRITIQUE.md +++ /dev/null @@ -1,517 +0,0 @@ -# STRATEGY.md Critique: MultiAgentBench Integration - -This document analyzes the proposed STRATEGY.md for integrating MARBLE/MultiAgentBench into MASEval, comparing against established patterns from tau2 and macs benchmarks. - ---- - -## Executive Summary - -The STRATEGY.md proposes a reasonable high-level architecture but contains several issues: - -| Category | Count | Severity | -|----------|-------|----------| -| **Incorrect API Assumptions** | 5 | High | -| **Pattern Violations** | 3 | Medium | -| **Clumsy Implementations** | 4 | Low-Medium | -| **Missing Considerations** | 3 | Medium | - -**Overall Assessment**: The strategy requires significant revision before implementation, particularly around MARBLE API assumptions and the "Engine as single adapter" pattern. - ---- - -## 1. Incorrect MARBLE API Assumptions - -### 1.1 SharedMemory is NOT Actually Shared (HIGH SEVERITY) - -**STRATEGY.md claims:** -```python -# Extract final answer from SharedMemory or designated output -final_answer = self.engine.memory.get("final_answer") -``` - -**Reality**: Each MARBLE agent creates its own `SharedMemory` instance at initialization (`base_agent.py:73`). The memory is NOT shared between agents despite the misleading name. There is no central `engine.memory` that accumulates agent outputs. - -**Evidence from MARBLE source:** -```python -# In BaseAgent.__init__(): -self.memory = BaseMemory() # Per-agent memory -self.shared_memory = SharedMemory() # Also per-agent, NOT shared! -``` - -**Impact**: The proposed `_convert_to_message_history()` and final answer extraction logic will fail silently or return empty data. - -**Recommendation**: Extract final answers from: -- `engine.agents[-1].last_output` (if exists) -- Agent task_history or msg_box -- Environment state after execution - ---- - -### 1.2 Missing `get_agent_profiles_linked()` Method (HIGH SEVERITY) - -**STRATEGY.md implies MARBLE's chain coordination works:** -> "Agents use AgentGraph to check who they can communicate with" - -**Reality**: `engine.py:702` calls `self.graph.get_agent_profiles_linked(current_agent.agent_id)` but this method **does not exist** in `AgentGraph`. This is a bug in MARBLE that will crash chain coordination mode. - -**Impact**: Cannot reliably run tasks with `coordinate_mode: "chain"` using vanilla MARBLE. - -**Recommendation**: -1. Document this limitation -2. Either patch MARBLE locally or avoid chain mode tasks initially -3. File upstream issue - ---- - -### 1.3 Message History Access Pattern is Wrong (HIGH SEVERITY) - -**STRATEGY.md proposes:** -```python -def _convert_to_message_history(self, engine) -> MessageHistory: - for agent in engine.agents: - if hasattr(agent, "messages"): - for msg in agent.messages: - # ... -``` - -**Reality**: MARBLE agents store messages in `msg_box`, not a `messages` attribute: -```python -# BaseAgent structure: -self.msg_box: Dict[session_id, Dict[agent_id, List[Tuple[direction, message]]]] -# Direction: 0 = FORWARD_TO (sent), 1 = RECV_FROM (received) -``` - -Messages are serialized via `agent.seralize_message(session_id)` (note: typo in MARBLE). - -**Recommendation**: Use the actual MARBLE API: -```python -for agent in engine.agents: - serialized = agent.seralize_message("") # Empty session_id for all - # Parse serialized string or access msg_box directly -``` - ---- - -### 1.4 `engine.start()` Return Value Not Specified - -**STRATEGY.md assumes:** -```python -def _run_agent(self, query: str) -> Any: - self.engine.start() - # ... extract results -``` - -**Reality**: `engine.start()` dispatches to coordination methods (`star_coordinate`, `chain_coordinate`, etc.) but the return behavior varies and is not consistently documented. Some modes return results, others mutate state. - -**Recommendation**: Study each coordination mode's return behavior and document clearly. Don't assume uniform behavior. - ---- - -### 1.5 Evaluator.update() Must Be Called Explicitly - -**STRATEGY.md implies:** -> "MARBLE evaluator metrics (code_quality, collaboration)" - -**Reality**: MARBLE's `Evaluator.update(environment, agents)` must be called explicitly after each iteration—the engine does NOT call it automatically. Without explicit calls, metrics remain empty. - -**Impact**: `MultiAgentBenchEvaluator` wrapper may return empty metrics. - -**Recommendation**: Call `marble_evaluator.update()` after engine execution, then `marble_evaluator.finalize()` before reading metrics. - ---- - -## 2. Pattern Violations - -### 2.1 "Engine as Single AgentAdapter" Violates Multi-Agent Visibility (MEDIUM SEVERITY) - -**STRATEGY.md proposes:** -```python -# Wrap entire engine as single adapter -engine_adapter = MarbleAgentAdapter(engine, "marble_engine") -return [engine_adapter], {"marble_engine": engine_adapter} -``` - -**MASEval Pattern (from tau2/macs)**: -- Each agent gets its own `AgentAdapter` -- `agents_dict` maps agent names to individual adapters -- Traces are collected per-agent for analysis - -**Why this matters**: -1. **Trace granularity lost**: Cannot analyze individual agent behavior -2. **Callback hooks incomplete**: `on_agent_step_start/end` fires once for entire system -3. **Error attribution unclear**: Which internal agent failed? -4. **Inconsistent with macs**: MACS explicitly creates adapters per agent in hierarchy - -**Comparison to MACS pattern:** -```python -# MACS approach - each agent is visible -def setup_agents(self, agent_data, environment, task, user): - adapters = [] - agents_dict = {} - for agent_spec in agent_data["agents"]: - agent = create_agent(agent_spec, environment) - adapter = AgentAdapter(agent, agent_spec["id"]) - adapters.append(adapter) - agents_dict[agent_spec["id"]] = adapter - return adapters, agents_dict -``` - -**Justified Reason to Deviate**: MARBLE's coordination logic is tightly coupled in the Engine. Exposing individual agents would require reimplementing coordination, defeating the purpose of scientific fidelity. - -**Recommendation**: Document this as an explicit architectural trade-off. Consider: -1. Keep Engine-as-adapter for `MarbleMultiAgentBenchBenchmark` (reproduction mode) -2. For custom framework implementations, encourage per-agent adapters -3. Enhance `gather_traces()` to expose internal agent details as nested structure - ---- - -### 2.2 Environment.create_tools() Returns Empty Dict (MEDIUM SEVERITY) - -**STRATEGY.md proposes:** -```python -def create_tools(self) -> Dict[str, Any]: - # MARBLE environments manage tools internally - return {} - -def get_tool(self, name: str) -> Optional[Any]: - raise NotImplementedError("Tools managed by MARBLE Engine") -``` - -**MASEval Pattern (from tau2/macs)**: -- `Environment.create_tools()` returns actual tool instances -- Tools are registered and traced via MASEval infrastructure -- Agents receive tools from environment - -**tau2 example:** -```python -def create_tools(self) -> Dict[str, Callable]: - return self.toolkit.get_tools() # Actual callable tools -``` - -**macs example:** -```python -def create_tools(self) -> Dict[str, MACSGenericTool]: - tools = {} - for spec in self.state["tool_specs"]: - tools[spec["name"]] = MACSGenericTool(spec, model) - return tools -``` - -**Why this matters**: -1. **Tool tracing lost**: MASEval won't capture tool invocations -2. **ToolInvocationHistory empty**: No tool usage data in traces -3. **Inconsistent behavior**: Users expect `environment.tools` to work - -**Recommendation**: Either: -1. Extract tool definitions from MARBLE and create MASEval-traceable wrappers -2. Or document clearly that tool tracing requires post-hoc extraction from engine traces - ---- - -### 2.3 User Simulator Returning None (MINOR) - -**STRATEGY.md proposes:** -```python -def setup_user(self, agent_data, environment, task, user): - # → None (MultiAgentBench doesn't use external user simulation) - return None -``` - -**This is acceptable** - tau2 and macs support optional user simulation. However, MARBLE does have some scenarios with user interaction. - -**Recommendation**: Verify which MultiAgentBench domains require user simulation and handle appropriately. - ---- - -## 3. Clumsy Implementations - -### 3.1 Overly Complex Config Conversion - -**STRATEGY.md proposes:** -```python -marble_config = Config.from_dict({ - "coordinate_mode": task.environment_data["coordinate_mode"], - "relationships": task.environment_data["relationships"], - "agents": task.environment_data["agents"], - "environment": task.environment_data["environment"], - "task": {"content": task.query}, - "llm": task.environment_data["llm"], - "memory": {"type": "SharedMemory"}, - "engine_planner": {"initial_progress": "Starting task"}, - "metrics": task.evaluation_data["metrics"], -}) -``` - -**Issue**: This duplicates MARBLE's config loading. MARBLE already has `Config.from_file()` and the JSONL entries ARE MARBLE configs. - -**Cleaner approach:** -```python -# MARBLE configs are stored directly in environment_data -marble_config = Config.from_dict(task.environment_data["raw_marble_config"]) -``` - -Or even simpler - load MARBLE tasks directly and convert to MASEval Task: -```python -def load_tasks(domain): - # Let MARBLE parse its own format - marble_tasks = marble.load_tasks(domain) - return [Task.from_marble(mt) for mt in marble_tasks] -``` - ---- - -### 3.2 Symlink for Data Directory is Fragile - -**STRATEGY.md proposes:** -``` -└── data/ # Symlink to marble/multiagentbench/ -``` - -**Issues**: -1. Symlinks don't work well on Windows -2. Adds complexity to setup -3. Git doesn't track symlinks reliably - -**Cleaner approach (tau2 pattern):** -```python -# In data_loader.py -def _get_data_dir() -> Path: - # Look for MARBLE in known locations - candidates = [ - Path(__file__).parent / "marble" / "multiagentbench", - Path(os.environ.get("MARBLE_DATA_DIR", "")), - ] - for candidate in candidates: - if candidate.exists(): - return candidate - raise FileNotFoundError("MARBLE data not found. See README.md for setup.") -``` - ---- - -### 3.3 Redundant Fallback Strategies for Message History - -**STRATEGY.md proposes three fallback strategies:** -```python -# Strategy 1: Direct message access -# Strategy 2: Fallback to SharedMemory -# Strategy 3: Fallback to agent outputs -``` - -**Issue**: Strategy 2 won't work (SharedMemory isn't shared). This creates dead code and false confidence. - -**Recommendation**: Remove non-functional fallbacks. Be explicit about what actually works: -```python -def _extract_messages(self, engine) -> List[Dict]: - """Extract messages from MARBLE agents. - - Note: MARBLE stores messages in agent.msg_box, not a central location. - """ - messages = [] - for agent in engine.agents: - # Only method that actually works - for session_id, conversations in agent.msg_box.items(): - for other_agent_id, msg_list in conversations.items(): - for direction, content in msg_list: - messages.append({ - "agent_id": agent.agent_id, - "direction": "sent" if direction == 0 else "received", - "to/from": other_agent_id, - "content": content, - }) - return messages -``` - ---- - -### 3.4 Over-Specification in Implementation Checklist - -The checklist has 40+ items spanning 7 phases. Compare to tau2 implementation which is ~500 lines across 4 files. - -**Recommendation**: Simplify to MVP: -1. Phase 1: Data loader + basic environment -2. Phase 2: MarbleAgentAdapter + benchmark class -3. Phase 3: Evaluator integration -4. Phase 4: Tests + example - -Defer user framework examples (LangGraph, smolagents) to later PRs. - ---- - -## 4. Missing Considerations - -### 4.1 No Discussion of Domain-Specific Environments - -**MARBLE has 7 distinct environments:** -- CodingEnvironment (file operations) -- DBEnvironment (Docker + PostgreSQL) -- MinecraftEnvironment (external process) -- ResearchEnvironment -- BargainingEnvironment -- WebEnvironment -- WorldSimulationEnvironment - -**Each has different setup requirements:** -- DBEnvironment requires Docker -- MinecraftEnvironment requires external game server -- CodingEnvironment needs filesystem access - -**STRATEGY.md mentions this but doesn't address:** -- How to handle missing dependencies gracefully -- Which domains to support initially -- How to skip domains without required infrastructure - -**Recommendation**: Start with simpler domains (Research, Bargaining) that don't require external services. Add infrastructure-heavy domains (DB, Minecraft) as optional extras. - ---- - -### 4.2 No Error Classification Strategy - -**MASEval uses structured error attribution:** -```python -class TaskExecutionStatus(Enum): - SUCCESS = "success" - AGENT_ERROR = "agent_error" - ENVIRONMENT_ERROR = "environment_error" - USER_ERROR = "user_error" - TASK_TIMEOUT = "task_timeout" - # ... -``` - -**STRATEGY.md doesn't discuss:** -- How MARBLE errors map to these statuses -- What happens when MARBLE's internal agents fail -- How to distinguish MARBLE bugs from agent failures - -**Recommendation**: Add error classification logic: -```python -def _classify_marble_error(self, error: Exception) -> TaskExecutionStatus: - if "MARBLE" in str(error) or isinstance(error, MarbleInternalError): - return TaskExecutionStatus.ENVIRONMENT_ERROR # MARBLE bug - if "LLM" in str(error) or "rate limit" in str(error).lower(): - return TaskExecutionStatus.ENVIRONMENT_ERROR # External service - return TaskExecutionStatus.AGENT_ERROR # Agent's fault -``` - ---- - -### 4.3 Evaluation Metrics Not Fully Mapped - -**MARBLE Evaluator produces:** -- `task_completion`: List[0/1] -- `token_consumption`: List[int] -- `planning_score`: List[1-5] -- `communication_score`: List[1-5] -- `code_quality`: Dict -- `agent_kpis`: Dict[agent_id, milestone_count] - -**MASEval evaluation format:** -```python -{"metric_name": value, "passed": bool, ...} -``` - -**STRATEGY.md proposes:** -```python -return self.marble_evaluator.evaluate(output, task_spec, traces) -``` - -But doesn't specify how MARBLE's metrics map to MASEval's format. - -**Recommendation**: Define explicit mapping: -```python -def __call__(self, traces, final_answer) -> Dict[str, Any]: - marble_metrics = self.marble_evaluator.get_metrics() - return { - "passed": marble_metrics.get("task_completion", [0])[-1] == 1, - "task_completion": marble_metrics.get("task_completion", [0])[-1], - "planning_score": mean(marble_metrics.get("planning_score", [3])), - "communication_score": mean(marble_metrics.get("communication_score", [3])), - "total_tokens": sum(marble_metrics.get("token_consumption", [0])), - # ... - } -``` - ---- - -## 5. Justified Pattern Deviations - -### 5.1 Vendoring MARBLE Source (JUSTIFIED) - -**Deviation**: Vendoring entire MARBLE repo rather than pip dependency. - -**Justification**: -- MARBLE's pip package has bugs -- Need to pin exact version for reproducibility -- May need local patches -- MIT license explicitly permits this - -**This is the right choice** - tau2 does something similar with its data. - ---- - -### 5.2 Abstract Base + Concrete MARBLE Implementation (JUSTIFIED) - -**Deviation**: Creating `MultiAgentBenchBenchmark` (abstract) and `MarbleMultiAgentBenchBenchmark` (concrete). - -**Justification**: -- Enables framework comparison research -- Matches MASEval's "BYO agent" philosophy -- Same pattern used by tau2 and macs - -**This is well-aligned** with MASEval patterns. - ---- - -### 5.3 Engine-Level Wrapping for Scientific Fidelity (PARTIALLY JUSTIFIED) - -**Deviation**: Wrapping MARBLE Engine as single adapter. - -**Partial Justification**: -- Preserves MARBLE's exact coordination logic -- Required for reproducing published results -- Would require reimplementing MARBLE to do otherwise - -**Concern**: Loses per-agent observability. Should be documented as explicit trade-off with enhanced trace extraction to compensate. - ---- - -## 6. Recommended Changes Before Implementation - -### High Priority (Must Fix) - -1. **Fix SharedMemory assumptions** - It's not shared; update all code that assumes it is -2. **Fix message history extraction** - Use `msg_box` and `seralize_message()` -3. **Document chain mode bug** - `get_agent_profiles_linked()` doesn't exist -4. **Add explicit Evaluator.update() calls** - -### Medium Priority (Should Fix) - -5. **Enhance traces from Engine adapter** - Include internal agent details as nested structure -6. **Simplify config conversion** - Use MARBLE's native loading where possible -7. **Add error classification** - Map MARBLE errors to MASEval statuses -8. **Define metric mapping** - MARBLE metrics → MASEval format - -### Low Priority (Nice to Have) - -9. **Remove symlink approach** - Use path resolution instead -10. **Simplify implementation phases** - Start with MVP -11. **Document domain requirements** - Which need Docker, external services, etc. -12. **Remove non-functional fallbacks** - Dead code creates false confidence - ---- - -## 7. Conclusion - -The STRATEGY.md demonstrates good understanding of MASEval's architecture but makes several incorrect assumptions about MARBLE's API that would cause runtime failures. The "Engine as single adapter" pattern is a reasonable trade-off for scientific fidelity but should be clearly documented with enhanced trace extraction. - -**Recommended next steps:** -1. Revise STRATEGY.md with corrections from this critique -2. Create minimal proof-of-concept with single domain (Research or Bargaining) -3. Validate MARBLE API assumptions with actual execution -4. Expand to other domains incrementally - ---- - -*Document Version: 1.0* -*Date: 2026-01-19* -*Reviewer: Claude (Opus 4.5)* From 02aa0b09aa36c5c196e92d80d9fa09a407dd22c6 Mon Sep 17 00:00:00 2001 From: cemde Date: Wed, 4 Feb 2026 16:50:14 +0100 Subject: [PATCH 15/17] [skip ci] cleaned docs --- docs/benchmark/multiagentbench.md | 86 ------------------------------- 1 file changed, 86 deletions(-) diff --git a/docs/benchmark/multiagentbench.md b/docs/benchmark/multiagentbench.md index 793323aa..1ffabda5 100644 --- a/docs/benchmark/multiagentbench.md +++ b/docs/benchmark/multiagentbench.md @@ -2,8 +2,6 @@ The **MultiAgentBench** benchmark evaluates multi-agent collaboration and competition in LLM-based systems across diverse scenarios including research, negotiation, coding, and more. -## Overview - [MultiAgentBench](https://github.com/ulab-uiuc/MARBLE) (from the MARBLE framework) is designed to evaluate how multiple LLM-based agents collaborate and compete to solve complex tasks. The benchmark features: - **7 diverse domains**: research, bargaining, coding, database, web, worldsimulation, minecraft @@ -15,90 +13,6 @@ Reference Paper: [MultiAgentBench: Evaluating the Collaboration and Competition Check out the [BENCHMARKS.md](https://github.com/parameterlab/MASEval/blob/main/BENCHMARKS.md) file for more information including licenses. -## Quick Start - -```python -from maseval.benchmark.multiagentbench import ( - MultiAgentBenchBenchmark, - MultiAgentBenchEnvironment, - MultiAgentBenchEvaluator, - load_tasks, - configure_model_ids, - ensure_marble_exists, -) - -# Ensure MARBLE is installed (auto-downloads if needed) -ensure_marble_exists() - -# Load and configure tasks -tasks = load_tasks("research", limit=5) -configure_model_ids(tasks, agent_model_id="gpt-4o") - -# Create your framework-specific benchmark subclass -class MyMultiAgentBenchmark(MultiAgentBenchBenchmark): - def setup_agents(self, agent_data, environment, task, user, seed_generator=None): - # Your framework-specific agent creation - agent_configs = task.environment_data.get("agents", []) - # Create agents based on configs... - ... - - def get_model_adapter(self, model_id, **kwargs): - adapter = MyModelAdapter(model_id) - if "register_name" in kwargs: - self.register("models", kwargs["register_name"], adapter) - return adapter - -# Run benchmark -benchmark = MyMultiAgentBenchmark() -results = benchmark.run(tasks, agent_data={}) -``` - -## MARBLE Reproduction Mode - -For exact reproduction of MARBLE's published results, use `MarbleMultiAgentBenchBenchmark` which wraps MARBLE's native agents: - -```python -from maseval.benchmark.multiagentbench import ( - MarbleMultiAgentBenchBenchmark, - load_tasks, - configure_model_ids, - ensure_marble_exists, -) - -# Ensure MARBLE is installed -ensure_marble_exists() - -# Load tasks -tasks = load_tasks("research", limit=5) -configure_model_ids(tasks, agent_model_id="gpt-4o") - -# Create benchmark with model adapter -class MyMarbleBenchmark(MarbleMultiAgentBenchBenchmark): - def get_model_adapter(self, model_id, **kwargs): - from maseval.interface.openai import OpenAIModelAdapter - adapter = OpenAIModelAdapter(model_id) - if "register_name" in kwargs: - self.register("models", kwargs["register_name"], adapter) - return adapter - -benchmark = MyMarbleBenchmark() -results = benchmark.run(tasks, agent_data={}) -``` - -## Available Domains - -| Domain | Description | Infrastructure | -|--------|-------------|----------------| -| `research` | Research idea generation and collaboration | None | -| `bargaining` | Negotiation scenarios (buyer/seller) | None | -| `coding` | Software development collaboration | Filesystem | -| `database` | Database manipulation and querying | Docker + PostgreSQL | -| `web` | Web-based task completion | Network | -| `worldsimulation` | World simulation and interaction | None | -| `minecraft` | Collaborative building | External server | - -## API Reference - ::: maseval.benchmark.multiagentbench.MultiAgentBenchBenchmark ::: maseval.benchmark.multiagentbench.MarbleMultiAgentBenchBenchmark From fd43b893221807ca45cc94a1e74d95947d90d30d Mon Sep 17 00:00:00 2001 From: cemde Date: Wed, 4 Feb 2026 17:01:13 +0100 Subject: [PATCH 16/17] updated multiagentbench code to exclude prompt templates --- .../benchmark/multiagentbench/evaluator.py | 96 ++++--------------- .../prompt_templates/bargaining_buyer.txt | 14 +++ .../prompt_templates/bargaining_seller.txt | 14 +++ .../prompt_templates/coding.txt | 16 ++++ .../prompt_templates/communication.txt | 15 +++ .../prompt_templates/research.txt | 14 +++ .../test_multiagentbench/test_data_loader.py | 8 +- 7 files changed, 97 insertions(+), 80 deletions(-) create mode 100644 maseval/benchmark/multiagentbench/prompt_templates/bargaining_buyer.txt create mode 100644 maseval/benchmark/multiagentbench/prompt_templates/bargaining_seller.txt create mode 100644 maseval/benchmark/multiagentbench/prompt_templates/coding.txt create mode 100644 maseval/benchmark/multiagentbench/prompt_templates/communication.txt create mode 100644 maseval/benchmark/multiagentbench/prompt_templates/research.txt diff --git a/maseval/benchmark/multiagentbench/evaluator.py b/maseval/benchmark/multiagentbench/evaluator.py index 0cba653b..43ed8e30 100644 --- a/maseval/benchmark/multiagentbench/evaluator.py +++ b/maseval/benchmark/multiagentbench/evaluator.py @@ -5,6 +5,7 @@ import json from dataclasses import dataclass, field +from pathlib import Path from typing import Any, Dict, List, Optional from maseval import Evaluator, ModelAdapter @@ -61,6 +62,8 @@ class MultiAgentBenchEvaluator(Evaluator): metrics_config: Configuration for metrics to evaluate """ + DEFAULT_TEMPLATES_DIR = Path(__file__).parent / "prompt_templates" + def __init__( self, domain: str, @@ -82,95 +85,38 @@ def __init__( self.output_format = output_format self._evaluation_prompts = self._load_evaluation_prompts() - def _load_evaluation_prompts(self) -> Dict[str, Any]: - """Load evaluation prompts (matching MARBLE's evaluator_prompts.json).""" - # These prompts mirror MARBLE's evaluation methodology - return { - "communication": { - "prompt": """Evaluate the communication between agents for the following task. + def _load_template(self, filename: str) -> str: + """Load a prompt template from file. -Task: {task} - -Communication Log: -{communications} + Args: + filename: Template filename (without directory) -Rate the communication quality on a scale of 1-5: -1 - Poor: Irrelevant or confusing communication -2 - Below Average: Some relevant communication but lacks clarity -3 - Average: Adequate communication that addresses the task -4 - Good: Clear and relevant communication with good coordination -5 - Excellent: Highly effective communication with perfect coordination + Returns: + Template content as string + """ + template_path = self.DEFAULT_TEMPLATES_DIR / filename + return template_path.read_text() -Respond with a JSON object: {{"rating": }}""" + def _load_evaluation_prompts(self) -> Dict[str, Any]: + """Load evaluation prompts from template files.""" + return { + "communication": { + "prompt": self._load_template("communication.txt"), }, "research": { "task_evaluation": { - "prompt": """Evaluate the following research idea based on innovation, safety, and feasibility. - -Task: {task} - -Research Result: -{result} - -Rate each dimension on a scale of 1-5: -- Innovation: How novel and creative is the research idea? -- Safety: Does the research consider ethical implications and safety? -- Feasibility: How practical and achievable is the proposed research? - -Respond with a JSON object: -{{"innovation": , "safety": , "feasibility": }}""" + "prompt": self._load_template("research.txt"), } }, "bargaining": { "task_evaluation": { - "buyer_prompt": """Evaluate the buyer's negotiation performance. - -Task: {task} - -Negotiation Result: -{result} - -Rate each dimension on a scale of 1-5: -- Effectiveness of Strategies: How well did the buyer negotiate? -- Progress and Outcome: Did the buyer achieve a favorable outcome? -- Interaction Dynamics: How well did the buyer engage with the seller? - -Respond with a JSON object: -{{"effectiveness_of_strategies": , "progress_and_outcome": , "interaction_dynamics": }}""", - "seller_prompt": """Evaluate the seller's negotiation performance. - -Task: {task} - -Negotiation Result: -{result} - -Rate each dimension on a scale of 1-5: -- Effectiveness of Strategies: How well did the seller negotiate? -- Progress and Outcome: Did the seller achieve a favorable outcome? -- Interaction Dynamics: How well did the seller engage with the buyer? - -Respond with a JSON object: -{{"effectiveness_of_strategies": , "progress_and_outcome": , "interaction_dynamics": }}""", + "buyer_prompt": self._load_template("bargaining_buyer.txt"), + "seller_prompt": self._load_template("bargaining_seller.txt"), } }, "coding": { "task_evaluation": { - "prompt": """Evaluate the code quality based on the following criteria. - -Task Description: {task_description} -Implementation Requirements: {requirements} - -Solution: -{solution} - -Rate each dimension on a scale of 1-5: -- Instruction Following: Does the code fulfill all requirements? -- Executability: Is the code syntactically correct and executable? -- Consistency: Is the code consistent in naming, formatting, and logic? -- Quality: Is the code well-documented, clear, and modular? - -Respond with a JSON object: -{{"instruction_following": , "executability": , "consistency": , "quality": }}""" + "prompt": self._load_template("coding.txt"), } }, } diff --git a/maseval/benchmark/multiagentbench/prompt_templates/bargaining_buyer.txt b/maseval/benchmark/multiagentbench/prompt_templates/bargaining_buyer.txt new file mode 100644 index 00000000..fe236bc7 --- /dev/null +++ b/maseval/benchmark/multiagentbench/prompt_templates/bargaining_buyer.txt @@ -0,0 +1,14 @@ +Evaluate the buyer's negotiation performance. + +Task: {task} + +Negotiation Result: +{result} + +Rate each dimension on a scale of 1-5: +- Effectiveness of Strategies: How well did the buyer negotiate? +- Progress and Outcome: Did the buyer achieve a favorable outcome? +- Interaction Dynamics: How well did the buyer engage with the seller? + +Respond with a JSON object: +{{"effectiveness_of_strategies": , "progress_and_outcome": , "interaction_dynamics": }} diff --git a/maseval/benchmark/multiagentbench/prompt_templates/bargaining_seller.txt b/maseval/benchmark/multiagentbench/prompt_templates/bargaining_seller.txt new file mode 100644 index 00000000..68878e71 --- /dev/null +++ b/maseval/benchmark/multiagentbench/prompt_templates/bargaining_seller.txt @@ -0,0 +1,14 @@ +Evaluate the seller's negotiation performance. + +Task: {task} + +Negotiation Result: +{result} + +Rate each dimension on a scale of 1-5: +- Effectiveness of Strategies: How well did the seller negotiate? +- Progress and Outcome: Did the seller achieve a favorable outcome? +- Interaction Dynamics: How well did the seller engage with the buyer? + +Respond with a JSON object: +{{"effectiveness_of_strategies": , "progress_and_outcome": , "interaction_dynamics": }} diff --git a/maseval/benchmark/multiagentbench/prompt_templates/coding.txt b/maseval/benchmark/multiagentbench/prompt_templates/coding.txt new file mode 100644 index 00000000..1fa98713 --- /dev/null +++ b/maseval/benchmark/multiagentbench/prompt_templates/coding.txt @@ -0,0 +1,16 @@ +Evaluate the code quality based on the following criteria. + +Task Description: {task_description} +Implementation Requirements: {requirements} + +Solution: +{solution} + +Rate each dimension on a scale of 1-5: +- Instruction Following: Does the code fulfill all requirements? +- Executability: Is the code syntactically correct and executable? +- Consistency: Is the code consistent in naming, formatting, and logic? +- Quality: Is the code well-documented, clear, and modular? + +Respond with a JSON object: +{{"instruction_following": , "executability": , "consistency": , "quality": }} diff --git a/maseval/benchmark/multiagentbench/prompt_templates/communication.txt b/maseval/benchmark/multiagentbench/prompt_templates/communication.txt new file mode 100644 index 00000000..aca7ccbf --- /dev/null +++ b/maseval/benchmark/multiagentbench/prompt_templates/communication.txt @@ -0,0 +1,15 @@ +Evaluate the communication between agents for the following task. + +Task: {task} + +Communication Log: +{communications} + +Rate the communication quality on a scale of 1-5: +1 - Poor: Irrelevant or confusing communication +2 - Below Average: Some relevant communication but lacks clarity +3 - Average: Adequate communication that addresses the task +4 - Good: Clear and relevant communication with good coordination +5 - Excellent: Highly effective communication with perfect coordination + +Respond with a JSON object: {{"rating": }} diff --git a/maseval/benchmark/multiagentbench/prompt_templates/research.txt b/maseval/benchmark/multiagentbench/prompt_templates/research.txt new file mode 100644 index 00000000..355afa6b --- /dev/null +++ b/maseval/benchmark/multiagentbench/prompt_templates/research.txt @@ -0,0 +1,14 @@ +Evaluate the following research idea based on innovation, safety, and feasibility. + +Task: {task} + +Research Result: +{result} + +Rate each dimension on a scale of 1-5: +- Innovation: How novel and creative is the research idea? +- Safety: Does the research consider ethical implications and safety? +- Feasibility: How practical and achievable is the proposed research? + +Respond with a JSON object: +{{"innovation": , "safety": , "feasibility": }} diff --git a/tests/test_benchmarks/test_multiagentbench/test_data_loader.py b/tests/test_benchmarks/test_multiagentbench/test_data_loader.py index 7c8ecae7..555e9f04 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_data_loader.py +++ b/tests/test_benchmarks/test_multiagentbench/test_data_loader.py @@ -329,11 +329,9 @@ def test_resolve_from_env_var(self): def test_resolve_not_found(self): """_resolve_data_dir should raise when no directory found.""" with patch.dict("os.environ", {}, clear=True): - with patch( - "maseval.benchmark.multiagentbench.data_loader._get_marble_dir", - return_value=Path("/nonexistent/marble"), - ): - with patch("pathlib.Path.cwd", return_value=Path("/nonexistent/cwd")): + with patch("pathlib.Path.cwd", return_value=Path("/nonexistent/cwd")): + # Mock Path.exists to return False for all candidate paths + with patch.object(Path, "exists", return_value=False): with pytest.raises(FileNotFoundError, match="MARBLE data directory not found"): _resolve_data_dir() From c54eda7631d2d03bd26bfdcb23188f1f25df6815 Mon Sep 17 00:00:00 2001 From: cemde Date: Wed, 4 Feb 2026 17:05:19 +0100 Subject: [PATCH 17/17] [skip ci] updated changelog --- CHANGELOG.md | 38 ++++++++++++++++++-------------------- 1 file changed, 18 insertions(+), 20 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ead53224..02fa7acb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,19 +12,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 **Benchmarks** - GAIA2 Benchmark: Integration with Meta's ARE (Agent Research Environments) platform for evaluating LLM-based agents on dynamic, multi-step scenarios (PR: #26) -- `Gaia2Benchmark`, `Gaia2Environment`, `Gaia2Evaluator` components for framework-agnostic evaluation with ARE simulation (PR: #26) -- `DefaultAgentGaia2Benchmark` with ReAct-style agent for direct comparison with ARE reference implementation (PR: #26) -- Tool wrapper (`AREToolWrapper`) for MASEval tracing of ARE tools with simulation time tracking (PR: #26) -- Data loading utilities: `load_tasks()`, `configure_model_ids()` for loading scenarios from HuggingFace (PR: #26) -- Metrics: `compute_gaia2_metrics()` for GSR (Goal Success Rate) computation by capability type (PR: #26) -- Support for 7 capability dimensions: execution, search, adaptability, time, ambiguity, agent2agent, noise (PR: #26) -- Added `gaia2` optional dependency: `pip install maseval[gaia2]` (PR: #26) + - `Gaia2Benchmark`, `Gaia2Environment`, `Gaia2Evaluator` components for framework-agnostic evaluation with ARE simulation (PR: #26) + - `DefaultAgentGaia2Benchmark` with ReAct-style agent for direct comparison with ARE reference implementation (PR: #26) + - Tool wrapper (`AREToolWrapper`) for MASEval tracing of ARE tools with simulation time tracking (PR: #26) + - Data loading utilities: `load_tasks()`, `configure_model_ids()` for loading scenarios from HuggingFace (PR: #26) + - Metrics: `compute_gaia2_metrics()` for GSR (Goal Success Rate) computation by capability type (PR: #26) + - Support for 7 capability dimensions: execution, search, adaptability, time, ambiguity, agent2agent, noise (PR: #26) + - Added `gaia2` optional dependency: `pip install maseval[gaia2]` (PR: #26) + +- MultiAgentBench Benchmark: Integration with MARBLE MultiAgentBench for evaluating multi-agent collaboration across research, bargaining, coding, and database domains (PR: #25) + - `MultiAgentBenchBenchmark` abstract base class for framework-agnostic multi-agent evaluation with seeding support for evaluators and agents (PR: #25) + - `MarbleMultiAgentBenchBenchmark` for exact MARBLE reproduction mode using native MARBLE agents (note: MARBLE's internal LLM calls bypass MASEval seeding) (PR: #25) + - `MultiAgentBenchEnvironment` and `MultiAgentBenchEvaluator` components (PR: #25) + - Data loading utilities: `load_tasks()`, `configure_model_ids()`, `get_domain_info()`, `ensure_marble_exists()` (PR: #25) + - MARBLE adapter: `MarbleAgentAdapter` for wrapping MARBLE agents with MASEval tracing (PR: #25) **Examples** - Gaia2 benchmark example with Google GenAI and OpenAI model support (PR: #26) -**Seeding System** +**Core** - Added `SeedGenerator` abstract base class and `DefaultSeedGenerator` implementation for reproducible benchmark runs via SHA-256-based seed derivation (PR: #24) - Added `seed` and `seed_generator` parameters to `Benchmark.__init__` for enabling reproducibility (PR: #24) @@ -33,21 +40,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Added `SeedingError` exception for providers that don't support seeding (Anthropic models raise this if seed is provided) (PR: #24) - Added seed support to interface adapters: `OpenAIModelAdapter`, `GoogleGenAIModelAdapter`, `LiteLLMModelAdapter`, `HuggingFaceModelAdapter` pass seeds to underlying APIs (PR: #24) -**Benchmarks** - -- MultiAgentBench Benchmark: Integration with MARBLE MultiAgentBench for evaluating multi-agent collaboration across research, bargaining, coding, and database domains (PR: #25) -- `MultiAgentBenchBenchmark` abstract base class for framework-agnostic multi-agent evaluation with seeding support for evaluators and agents (PR: #25) -- `MarbleMultiAgentBenchBenchmark` for exact MARBLE reproduction mode using native MARBLE agents (note: MARBLE's internal LLM calls bypass MASEval seeding) (PR: #25) -- `MultiAgentBenchEnvironment` and `MultiAgentBenchEvaluator` components (PR: #25) -- Data loading utilities: `load_tasks()`, `configure_model_ids()`, `get_domain_info()`, `ensure_marble_exists()` (PR: #25) -- MARBLE adapter: `MarbleAgentAdapter` for wrapping MARBLE agents with MASEval tracing (PR: #25) - **Interface** - CAMEL-AI integration: `CamelAgentAdapter` and `CamelLLMUser` for evaluating CAMEL-AI ChatAgent-based systems (PR: #22) -- Added `CamelAgentUser` for using a CAMEL ChatAgent as the user in agent-to-agent evaluation (PR: #22) -- Added `camel_role_playing_execution_loop()` for benchmarks using CAMEL's RolePlaying semantics (PR: #22) -- Added `CamelRolePlayingTracer` and `CamelWorkforceTracer` for capturing orchestration-level traces from CAMEL's multi-agent systems (PR: #22) + - Added `CamelAgentUser` for using a CAMEL ChatAgent as the user in agent-to-agent evaluation (PR: #22) + - Added `camel_role_playing_execution_loop()` for benchmarks using CAMEL's RolePlaying semantics (PR: #22) + - Added `CamelRolePlayingTracer` and `CamelWorkforceTracer` for capturing orchestration-level traces from CAMEL's multi-agent systems (PR: #22) ### Changed