diff --git a/BENCHMARKS.md b/BENCHMARKS.md index 83c81737..bcb15d06 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -27,7 +27,20 @@ $\tau^2$-bench is a benchmark for evaluating agentic systems in realistic, multi --- -## 3. Gaia2 (ARE) +## 3. MultiAgentBench (MARBLE) + +MultiAgentBench is a comprehensive benchmark suite for evaluating multi-agent collaboration and competition in LLM-based systems. It includes diverse scenarios across multiple domains including research collaboration, negotiation, coding tasks, and more. + +### Source and License + +- **Original Repository:** [https://github.com/ulab-uiuc/MARBLE](https://github.com/ulab-uiuc/MARBLE) +- **Paper:** [MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents](https://arxiv.org/abs/2503.01935) +- **Code License:** MIT +- **Data License:** MIT + +--- + +## 4. GAIA2 Gaia2 is a benchmark for evaluating LLM-based agents on dynamic, multi-step scenarios using Meta's ARE (Agent Research Environments) platform. It tests agents across 7 capability dimensions: execution, search, adaptability, time, ambiguity, agent2agent, and noise. diff --git a/CHANGELOG.md b/CHANGELOG.md index ead3a377..02fa7acb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,19 +12,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 **Benchmarks** - GAIA2 Benchmark: Integration with Meta's ARE (Agent Research Environments) platform for evaluating LLM-based agents on dynamic, multi-step scenarios (PR: #26) -- `Gaia2Benchmark`, `Gaia2Environment`, `Gaia2Evaluator` components for framework-agnostic evaluation with ARE simulation (PR: #26) -- `DefaultAgentGaia2Benchmark` with ReAct-style agent for direct comparison with ARE reference implementation (PR: #26) -- Tool wrapper (`AREToolWrapper`) for MASEval tracing of ARE tools with simulation time tracking (PR: #26) -- Data loading utilities: `load_tasks()`, `configure_model_ids()` for loading scenarios from HuggingFace (PR: #26) -- Metrics: `compute_gaia2_metrics()` for GSR (Goal Success Rate) computation by capability type (PR: #26) -- Support for 7 capability dimensions: execution, search, adaptability, time, ambiguity, agent2agent, noise (PR: #26) -- Added `gaia2` optional dependency: `pip install maseval[gaia2]` (PR: #26) + - `Gaia2Benchmark`, `Gaia2Environment`, `Gaia2Evaluator` components for framework-agnostic evaluation with ARE simulation (PR: #26) + - `DefaultAgentGaia2Benchmark` with ReAct-style agent for direct comparison with ARE reference implementation (PR: #26) + - Tool wrapper (`AREToolWrapper`) for MASEval tracing of ARE tools with simulation time tracking (PR: #26) + - Data loading utilities: `load_tasks()`, `configure_model_ids()` for loading scenarios from HuggingFace (PR: #26) + - Metrics: `compute_gaia2_metrics()` for GSR (Goal Success Rate) computation by capability type (PR: #26) + - Support for 7 capability dimensions: execution, search, adaptability, time, ambiguity, agent2agent, noise (PR: #26) + - Added `gaia2` optional dependency: `pip install maseval[gaia2]` (PR: #26) + +- MultiAgentBench Benchmark: Integration with MARBLE MultiAgentBench for evaluating multi-agent collaboration across research, bargaining, coding, and database domains (PR: #25) + - `MultiAgentBenchBenchmark` abstract base class for framework-agnostic multi-agent evaluation with seeding support for evaluators and agents (PR: #25) + - `MarbleMultiAgentBenchBenchmark` for exact MARBLE reproduction mode using native MARBLE agents (note: MARBLE's internal LLM calls bypass MASEval seeding) (PR: #25) + - `MultiAgentBenchEnvironment` and `MultiAgentBenchEvaluator` components (PR: #25) + - Data loading utilities: `load_tasks()`, `configure_model_ids()`, `get_domain_info()`, `ensure_marble_exists()` (PR: #25) + - MARBLE adapter: `MarbleAgentAdapter` for wrapping MARBLE agents with MASEval tracing (PR: #25) **Examples** - Gaia2 benchmark example with Google GenAI and OpenAI model support (PR: #26) -**Seeding System** +**Core** - Added `SeedGenerator` abstract base class and `DefaultSeedGenerator` implementation for reproducible benchmark runs via SHA-256-based seed derivation (PR: #24) - Added `seed` and `seed_generator` parameters to `Benchmark.__init__` for enabling reproducibility (PR: #24) @@ -36,9 +43,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 **Interface** - CAMEL-AI integration: `CamelAgentAdapter` and `CamelLLMUser` for evaluating CAMEL-AI ChatAgent-based systems (PR: #22) -- Added `CamelAgentUser` for using a CAMEL ChatAgent as the user in agent-to-agent evaluation (PR: #22) -- Added `camel_role_playing_execution_loop()` for benchmarks using CAMEL's RolePlaying semantics (PR: #22) -- Added `CamelRolePlayingTracer` and `CamelWorkforceTracer` for capturing orchestration-level traces from CAMEL's multi-agent systems (PR: #22) + - Added `CamelAgentUser` for using a CAMEL ChatAgent as the user in agent-to-agent evaluation (PR: #22) + - Added `camel_role_playing_execution_loop()` for benchmarks using CAMEL's RolePlaying semantics (PR: #22) + - Added `CamelRolePlayingTracer` and `CamelWorkforceTracer` for capturing orchestration-level traces from CAMEL's multi-agent systems (PR: #22) ### Changed diff --git a/docs/benchmark/multiagentbench.md b/docs/benchmark/multiagentbench.md new file mode 100644 index 00000000..1ffabda5 --- /dev/null +++ b/docs/benchmark/multiagentbench.md @@ -0,0 +1,34 @@ +# MultiAgentBench: Multi-Agent Collaboration Benchmark + +The **MultiAgentBench** benchmark evaluates multi-agent collaboration and competition in LLM-based systems across diverse scenarios including research, negotiation, coding, and more. + +[MultiAgentBench](https://github.com/ulab-uiuc/MARBLE) (from the MARBLE framework) is designed to evaluate how multiple LLM-based agents collaborate and compete to solve complex tasks. The benchmark features: + +- **7 diverse domains**: research, bargaining, coding, database, web, worldsimulation, minecraft +- **Multiple coordination modes**: cooperative, star, tree, hierarchical +- **LLM-based evaluation**: Matches MARBLE's evaluation methodology +- **Framework-agnostic**: Use with any agent framework or MARBLE's native agents + +Reference Paper: [MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents](https://arxiv.org/abs/2503.01935) + +Check out the [BENCHMARKS.md](https://github.com/parameterlab/MASEval/blob/main/BENCHMARKS.md) file for more information including licenses. + +::: maseval.benchmark.multiagentbench.MultiAgentBenchBenchmark + +::: maseval.benchmark.multiagentbench.MarbleMultiAgentBenchBenchmark + +::: maseval.benchmark.multiagentbench.MultiAgentBenchEnvironment + +::: maseval.benchmark.multiagentbench.MultiAgentBenchEvaluator + +::: maseval.benchmark.multiagentbench.MarbleAgentAdapter + +::: maseval.benchmark.multiagentbench.load_tasks + +::: maseval.benchmark.multiagentbench.configure_model_ids + +::: maseval.benchmark.multiagentbench.ensure_marble_exists + +::: maseval.benchmark.multiagentbench.download_marble + +::: maseval.benchmark.multiagentbench.get_domain_info diff --git a/maseval/benchmark/multiagentbench/.gitignore b/maseval/benchmark/multiagentbench/.gitignore new file mode 100644 index 00000000..abb9714c --- /dev/null +++ b/maseval/benchmark/multiagentbench/.gitignore @@ -0,0 +1,10 @@ +# Vendored MARBLE source (clone manually) +marble/ + +# Python cache +__pycache__/ +*.pyc +*.pyo + +# Test artifacts +.pytest_cache/ diff --git a/maseval/benchmark/multiagentbench/PROVENANCE.md b/maseval/benchmark/multiagentbench/PROVENANCE.md new file mode 100644 index 00000000..9fd516bf --- /dev/null +++ b/maseval/benchmark/multiagentbench/PROVENANCE.md @@ -0,0 +1,69 @@ +# MARBLE Integration Provenance + +## Source Information + +- **Source Repository**: https://github.com/ulab-uiuc/MARBLE +- **Version**: Not yet pinned (clone latest and test) +- **License**: MIT (Copyright 2024 Haofei Yu) +- **Vendoring**: Permitted by MIT license with attribution + +## Reference + +**Paper**: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents" + +- arXiv: https://arxiv.org/abs/2503.01935 +- Authors: Haofei Yu et al. +- Publication Date: 2025 + +## License Text (MIT) + +``` +MIT License + +Copyright (c) 2024 Haofei Yu + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. +``` + +## Known Issues in MARBLE + +1. **Missing method**: `AgentGraph.get_agent_profiles_linked()` does not exist but is + called in `engine.py:702`. This breaks chain coordination mode. + +2. **SharedMemory naming**: Despite the name, `SharedMemory` is instantiated per-agent + in `BaseAgent.__init__()` and is NOT shared between agents. Use `msg_box` for + inter-agent communication. + +3. **Environment constructor signature**: Some environments expect different constructor + arguments. Check each environment's `__init__` signature before use. + +## Local Patches Applied + +None currently. Document any patches here if applied. + +## Update Process + +To update MARBLE to a newer version: + +1. `cd maseval/benchmark/multiagentbench/marble` +2. `git fetch origin` +3. `git log --oneline origin/main` (review changes) +4. `git checkout ` +5. Run integration tests +6. Update this file with new version info diff --git a/maseval/benchmark/multiagentbench/README.md b/maseval/benchmark/multiagentbench/README.md new file mode 100644 index 00000000..c5b3cdab --- /dev/null +++ b/maseval/benchmark/multiagentbench/README.md @@ -0,0 +1,182 @@ +# MultiAgentBench Integration + +Framework-agnostic implementation of the MultiAgentBench benchmark suite from MARBLE +(Multi-Agent Coordination Backbone with LLM Engine) for evaluating multi-agent collaboration. + +**Original Paper**: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents" +(arXiv:2503.01935) + +**Original Repository**: https://github.com/ulab-uiuc/MARBLE + +## Setup + +This benchmark requires the MARBLE source code. You can set it up automatically or manually. + +### Option 1: Automatic Setup (Recommended) + +MARBLE will be automatically downloaded when you first use it: + +```python +from maseval.benchmark.multiagentbench import ensure_marble_exists, load_tasks + +# This downloads MARBLE if not present (about 50MB) +ensure_marble_exists() + +# Now load tasks +tasks = load_tasks("research", limit=1) +print(f"Loaded {len(tasks)} task(s)") +``` + +### Option 2: Manual Clone + +If you prefer to clone manually: + +```bash +cd maseval/benchmark/multiagentbench +git clone https://github.com/ulab-uiuc/MARBLE.git marble +cd marble +# Pin to tested version (recommended) +git checkout +``` + +### Install MARBLE Dependencies + +MARBLE requires additional dependencies. Add them to your environment: + +```bash +# If using uv (recommended) +uv add litellm ruamel.yaml + +# Or with pip +pip install litellm ruamel.yaml +``` + +### Verify Setup + +```python +from maseval.benchmark.multiagentbench import load_tasks + +# Should load without error +tasks = load_tasks("research", limit=1) +print(f"Loaded {len(tasks)} task(s)") +``` + +## Usage + +### Basic Usage (Abstract Base) + +The abstract `MultiAgentBenchBenchmark` provides task loading, environment setup, +and evaluation infrastructure. You implement `setup_agents()` with your framework: + +```python +from maseval.benchmark.multiagentbench import ( + MultiAgentBenchBenchmark, + MultiAgentBenchEnvironment, + load_tasks, + configure_model_ids, +) + +class MyMultiAgentBenchmark(MultiAgentBenchBenchmark): + def setup_agents(self, agent_data, environment, task, user, seed_generator=None): + # Your framework-specific agent creation + ... + + def get_model_adapter(self, model_id, **kwargs): + adapter = MyModelAdapter(model_id) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter + +# Load and configure tasks +tasks = load_tasks("research", limit=5) +configure_model_ids(tasks, agent_model_id="gpt-4o") + +# Run +benchmark = MyMultiAgentBenchmark() +results = benchmark.run(tasks) +``` + +### MARBLE Reproduction + +Use `MarbleMultiAgentBenchBenchmark` for exact reproduction of MARBLE's published results: + +```python +from maseval.benchmark.multiagentbench import ( + MarbleMultiAgentBenchBenchmark, + load_tasks, + configure_model_ids, +) + +# Load tasks from a simple domain (no Docker required) +tasks = load_tasks("research", limit=5) +configure_model_ids(tasks, agent_model_id="gpt-4o") + +# Create benchmark with model adapter implementation +class MyMarbleBenchmark(MarbleMultiAgentBenchBenchmark): + def get_model_adapter(self, model_id, **kwargs): + from maseval.interface.openai import OpenAIModelAdapter + adapter = OpenAIModelAdapter(model_id) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter + +benchmark = MyMarbleBenchmark() +results = benchmark.run(tasks) + +# Print results +for result in results: + print(f"Task: {result['task_id']}") + print(f"Status: {result['status']}") + if result['eval']: + print(f"Passed: {result['eval'][0]['passed']}") +``` + +## Domains + +MultiAgentBench includes 7 domains with different requirements: + +| Domain | External Dependencies | Initial Support | +| --------------- | --------------------- | --------------- | +| Research | None | Yes | +| Bargaining | None | Yes | +| Coding | Filesystem access | Yes | +| Web | Network access | Yes | +| WorldSimulation | None | Yes | +| Database | Docker + PostgreSQL | Optional | +| Minecraft | External game server | Deferred | + +### Domain-Specific Notes + +- **Research/Bargaining**: Recommended for initial testing - no infrastructure required +- **Coding**: Creates files in a workspace directory +- **Database**: Requires Docker with PostgreSQL image +- **Minecraft**: Not currently supported (requires external game server) + +## Known Limitations + +1. **Chain coordination mode bug**: MARBLE's `engine.py` references `get_agent_profiles_linked()` + which doesn't exist in `AgentGraph`. Tasks using chain coordination may fail. + +2. **SharedMemory is per-agent**: Despite the name, each MARBLE agent creates its own + `SharedMemory` instance. Use `msg_box` for inter-agent communication. + +3. **Requires manual MARBLE clone**: MARBLE must be cloned manually into the `marble/` + subdirectory (gitignored by default). + +## File Structure + +``` +multiagentbench/ +├── __init__.py # Public API exports +├── README.md # This file +├── PROVENANCE.md # MARBLE version and license info +├── .gitignore # Ignores marble/ directory +├── multiagentbench.py # Benchmark classes +├── environment.py # MultiAgentBenchEnvironment +├── data_loader.py # Task loading utilities +├── adapters/ +│ ├── __init__.py +│ └── marble_adapter.py # MarbleAgentAdapter +└── marble/ # ← Vendored MARBLE (gitignored) + └── ... +``` diff --git a/maseval/benchmark/multiagentbench/__init__.py b/maseval/benchmark/multiagentbench/__init__.py new file mode 100644 index 00000000..cc7eb688 --- /dev/null +++ b/maseval/benchmark/multiagentbench/__init__.py @@ -0,0 +1,149 @@ +"""MultiAgentBench - Multi-Agent Coordination Benchmark from MARBLE. + +Framework-agnostic implementation of the MultiAgentBench benchmark suite for +evaluating multi-agent collaboration and competition in LLM-based systems. + +Original Repository: https://github.com/ulab-uiuc/MARBLE +Paper: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents" + (arXiv:2503.01935) + +Domains: + - research: Research idea generation and collaboration + - bargaining: Negotiation and bargaining scenarios + - coding: Software development collaboration + - database: Database manipulation and querying (requires Docker) + - minecraft: Collaborative building (requires external server) + - web: Web-based task completion + - worldsimulation: World simulation and interaction + +Setup: + This benchmark requires MARBLE source code. It will be automatically + downloaded when you first use `load_tasks()` or you can set it up manually: + + ```python + # Option 1: Automatic download (recommended) + from maseval.benchmark.multiagentbench import ensure_marble_exists + ensure_marble_exists() # Downloads MARBLE if not present + + # Option 2: Manual clone + # cd maseval/benchmark/multiagentbench + # git clone https://github.com/ulab-uiuc/MARBLE.git marble + ``` + + See README.md in this directory for detailed setup instructions. + +Usage: + from maseval.benchmark.multiagentbench import ( + MultiAgentBenchBenchmark, + MarbleMultiAgentBenchBenchmark, + MultiAgentBenchEnvironment, + MultiAgentBenchEvaluator, + MarbleAgentAdapter, + load_tasks, + configure_model_ids, + ensure_marble_exists, + get_domain_info, + VALID_DOMAINS, + ) + + # Ensure MARBLE is installed (auto-downloads if needed) + ensure_marble_exists() + + # Load and configure tasks + tasks = load_tasks("research", limit=5) + configure_model_ids(tasks, agent_model_id="gpt-4o") + + # Create your framework-specific benchmark subclass + class MyMultiAgentBenchmark(MultiAgentBenchBenchmark): + def setup_agents(self, agent_data, environment, task, user): + # Create your agents + ... + + def get_model_adapter(self, model_id, **kwargs): + adapter = MyModelAdapter(model_id) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter + + # Run benchmark + benchmark = MyMultiAgentBenchmark() + results = benchmark.run(tasks, agent_data={}) + +MARBLE Reproduction Mode: + For exact reproduction of MARBLE's published results, use + MarbleMultiAgentBenchBenchmark which wraps MARBLE's native agents: + + ```python + class MyMarbleBenchmark(MarbleMultiAgentBenchBenchmark): + def get_model_adapter(self, model_id, **kwargs): + from maseval.interface.openai import OpenAIModelAdapter + adapter = OpenAIModelAdapter(model_id) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter + + benchmark = MyMarbleBenchmark() + results = benchmark.run(tasks, agent_data={}) + ``` +""" + +# Core benchmark classes +from maseval.benchmark.multiagentbench.multiagentbench import ( + MultiAgentBenchBenchmark, + MarbleMultiAgentBenchBenchmark, +) + +# Environment +from maseval.benchmark.multiagentbench.environment import ( + MultiAgentBenchEnvironment, + INFRASTRUCTURE_DOMAINS, +) + +# Evaluator +from maseval.benchmark.multiagentbench.evaluator import ( + MultiAgentBenchEvaluator, + MultiAgentBenchMetrics, +) + +# Agent adapters +from maseval.benchmark.multiagentbench.adapters import ( + MarbleAgentAdapter, +) +from maseval.benchmark.multiagentbench.adapters.marble_adapter import ( + create_marble_agents, +) + +# Data loading and setup +from maseval.benchmark.multiagentbench.data_loader import ( + load_tasks, + configure_model_ids, + get_domain_info, + ensure_marble_exists, + download_marble, + VALID_DOMAINS, + INFRASTRUCTURE_DOMAINS as INFRASTRUCTURE_REQUIRED_DOMAINS, +) + + +__all__ = [ + # Core benchmark classes + "MultiAgentBenchBenchmark", + "MarbleMultiAgentBenchBenchmark", + # Environment + "MultiAgentBenchEnvironment", + "INFRASTRUCTURE_DOMAINS", + # Evaluator + "MultiAgentBenchEvaluator", + "MultiAgentBenchMetrics", + # Agent adapters + "MarbleAgentAdapter", + "create_marble_agents", + # Data loading and setup + "load_tasks", + "configure_model_ids", + "get_domain_info", + "ensure_marble_exists", + "download_marble", + "VALID_DOMAINS", + "INFRASTRUCTURE_REQUIRED_DOMAINS", +] diff --git a/maseval/benchmark/multiagentbench/_constants.py b/maseval/benchmark/multiagentbench/_constants.py new file mode 100644 index 00000000..f852e5a6 --- /dev/null +++ b/maseval/benchmark/multiagentbench/_constants.py @@ -0,0 +1,8 @@ +"""Shared constants for MultiAgentBench module. + +This module exists to avoid circular imports between multiagentbench.py, +environment.py, and adapters/marble_adapter.py. +""" + +# Shared error message for MARBLE import failures +MARBLE_IMPORT_ERROR = "MARBLE is not available. Clone MARBLE to maseval/benchmark/multiagentbench/marble/\nOriginal error: {error}" diff --git a/maseval/benchmark/multiagentbench/adapters/__init__.py b/maseval/benchmark/multiagentbench/adapters/__init__.py new file mode 100644 index 00000000..a020fed0 --- /dev/null +++ b/maseval/benchmark/multiagentbench/adapters/__init__.py @@ -0,0 +1,7 @@ +"""Agent adapters for MultiAgentBench.""" + +from maseval.benchmark.multiagentbench.adapters.marble_adapter import ( + MarbleAgentAdapter, +) + +__all__ = ["MarbleAgentAdapter"] diff --git a/maseval/benchmark/multiagentbench/adapters/marble_adapter.py b/maseval/benchmark/multiagentbench/adapters/marble_adapter.py new file mode 100644 index 00000000..5d18ea02 --- /dev/null +++ b/maseval/benchmark/multiagentbench/adapters/marble_adapter.py @@ -0,0 +1,220 @@ +"""MARBLE agent adapter for MASEval. + +This module provides an adapter that wraps MARBLE's BaseAgent for use with +MASEval's tracing and evaluation infrastructure. +""" + +from typing import Any, Dict, List, Sequence, Tuple + +from maseval import AgentAdapter, AgentError, MessageHistory +from maseval.benchmark.multiagentbench._constants import MARBLE_IMPORT_ERROR + + +class MarbleAgentAdapter(AgentAdapter): + """Adapter wrapping a MARBLE BaseAgent for MASEval tracing. + + This adapter provides a unified interface to MARBLE agents while + capturing all relevant traces for evaluation. + + Attributes: + agent_id: Unique identifier for the agent + marble_agent: The underlying MARBLE BaseAgent instance + profile: Agent's role profile from MARBLE config + """ + + def __init__( + self, + marble_agent: Any, + agent_id: str, + ): + """Initialize the adapter. + + Args: + marble_agent: MARBLE BaseAgent instance + agent_id: Unique identifier for this agent + """ + self._marble_agent = marble_agent + self._agent_id = agent_id + self._profile = getattr(marble_agent, "profile", "") + self._communication_log: List[Dict[str, Any]] = [] + self._action_log: List[Dict[str, Any]] = [] + super().__init__(agent_instance=marble_agent, name=agent_id) + # Initialize message history + self.messages: MessageHistory = MessageHistory() + + @property + def agent_id(self) -> str: + """Return the agent's unique identifier.""" + return self._agent_id + + @property + def profile(self) -> str: + """Return the agent's profile.""" + return self._profile + + @property + def marble_agent(self) -> Any: + """Return the underlying MARBLE agent.""" + return self._marble_agent + + def _run_agent(self, query: str) -> str: + """Execute the MARBLE agent's act() method. + + Args: + query: The task/query to pass to the agent + + Returns: + String result from the agent + + Raises: + AgentError: If agent execution fails + """ + try: + # Call MARBLE agent's act method + result, communication = self._marble_agent.act(query) + + # Log action + self._action_log.append( + { + "task": query, + "result": result, + "has_communication": communication is not None, + } + ) + + # Log communication if present + if communication is not None: + self._communication_log.append( + { + "session_id": str(getattr(self._marble_agent, "session_id", "")), + "communication": communication, + } + ) + + # Update message history + self.messages.add_message(role="user", content=query) + self.messages.add_message(role="assistant", content=result) + + return result + + except Exception as e: + raise AgentError( + f"MARBLE agent '{self._agent_id}' failed: {e}", + component=f"MarbleAgentAdapter:{self._agent_id}", + ) from e + + def get_token_usage(self) -> int: + """Get the total token usage from the MARBLE agent. + + Returns: + Total tokens used by the agent + """ + if hasattr(self._marble_agent, "get_token_usage"): + return self._marble_agent.get_token_usage() + return 0 + + def get_memory_str(self) -> str: + """Get the agent's memory as a string. + + Returns: + Serialized memory string + """ + memory = getattr(self._marble_agent, "memory", None) + if memory is not None and hasattr(memory, "get_memory_str"): + return memory.get_memory_str() + return "" + + def get_serialized_messages(self, session_id: str = "") -> str: + """Get serialized inter-agent messages. + + Args: + session_id: Optional session ID filter + + Returns: + Serialized message string + """ + # Note: "seralize_message" is misspelled in MARBLE's API (should be "serialize") + if hasattr(self._marble_agent, "seralize_message"): + return self._marble_agent.seralize_message(session_id) + return "" + + def gather_traces(self) -> Dict[str, Any]: + """Gather traces including agent-specific data. + + Returns: + Dict with all agent traces + """ + traces = super().gather_traces() + + # Add MARBLE-specific traces + traces["agent_id"] = self._agent_id + traces["profile"] = self._profile + traces["token_usage"] = self.get_token_usage() + traces["action_log"] = self._action_log + traces["communication_log"] = self._communication_log + traces["memory"] = self.get_memory_str() + traces["relationships"] = getattr(self._marble_agent, "relationships", {}) + + # Add task history if available + traces["task_history"] = getattr(self._marble_agent, "task_history", []) + + return traces + + def gather_config(self) -> Dict[str, Any]: + """Gather agent configuration. + + Returns: + Dict with agent configuration + """ + config = super().gather_config() + + config["agent_id"] = self._agent_id + config["profile"] = self._profile + config["strategy"] = getattr(self._marble_agent, "strategy", "default") + config["llm"] = getattr(self._marble_agent, "llm", "unknown") + + return config + + +def create_marble_agents( + agent_configs: Sequence[Dict[str, Any]], + marble_env: Any, + model: str, +) -> Tuple[List[MarbleAgentAdapter], Dict[str, MarbleAgentAdapter]]: + """Create MarbleAgentAdapters from agent configurations. + + This is a factory function that creates MARBLE BaseAgent instances + and wraps them in MarbleAgentAdapters. + + Args: + agent_configs: List of agent configuration dicts from task data + marble_env: MARBLE environment instance + model: Model ID to use for agents + + Returns: + Tuple of (agents_list, agents_dict) + + Raises: + ImportError: If MARBLE is not available + """ + try: + from ..marble.agent.base_agent import BaseAgent # type: ignore[unresolved-import] + except ImportError as e: + raise ImportError(MARBLE_IMPORT_ERROR.format(error=e)) from e + + agents_list: List[MarbleAgentAdapter] = [] + agents_dict: Dict[str, MarbleAgentAdapter] = {} + + for config in agent_configs: + agent_id = config.get("agent_id", f"agent_{len(agents_list)}") + + # Create MARBLE agent + marble_agent = BaseAgent(config=config, env=marble_env, model=model) + + # Wrap in adapter + adapter = MarbleAgentAdapter(marble_agent=marble_agent, agent_id=agent_id) + + agents_list.append(adapter) + agents_dict[agent_id] = adapter + + return agents_list, agents_dict diff --git a/maseval/benchmark/multiagentbench/data_loader.py b/maseval/benchmark/multiagentbench/data_loader.py new file mode 100644 index 00000000..6c4410f7 --- /dev/null +++ b/maseval/benchmark/multiagentbench/data_loader.py @@ -0,0 +1,435 @@ +"""Data loading utilities for MultiAgentBench tasks. + +This module provides functions for loading and configuring tasks from +MARBLE's MultiAgentBench JSONL files. +""" + +import json +import logging +import os +import subprocess +from pathlib import Path +from typing import Any, Dict, FrozenSet, List, Optional + +from maseval import Task + +logger = logging.getLogger(__name__) + +# MARBLE repository configuration +MARBLE_REPO_URL = "https://github.com/ulab-uiuc/MARBLE.git" +MARBLE_DEFAULT_COMMIT = None # Set to a specific commit hash for reproducibility, or None for latest + +# Valid domain names +VALID_DOMAINS: FrozenSet[str] = frozenset( + { + "coding", + "database", + "minecraft", + "research", + "bargaining", + "web", + "worldsimulation", + } +) + +# Domains requiring external infrastructure +INFRASTRUCTURE_DOMAINS: FrozenSet[str] = frozenset({"database", "minecraft"}) + + +def _get_marble_dir() -> Path: + """Get the default MARBLE installation directory. + + Returns: + Path to marble/ directory relative to this module + """ + return Path(__file__).parent / "marble" + + +def download_marble( + target_dir: Optional[Path] = None, + commit: Optional[str] = None, + force: bool = False, +) -> Path: + """Clone MARBLE repository to the specified directory. + + Args: + target_dir: Directory to clone into. Defaults to marble/ relative to this module. + commit: Specific commit hash to checkout. Defaults to MARBLE_DEFAULT_COMMIT or latest. + force: If True, remove existing directory and re-clone. + + Returns: + Path to the cloned MARBLE directory + + Raises: + RuntimeError: If git clone fails + FileExistsError: If directory exists and force=False + """ + if target_dir is None: + target_dir = _get_marble_dir() + + target_dir = Path(target_dir) + + # Check if already exists + if target_dir.exists(): + if not force: + logger.info(f"MARBLE already exists at {target_dir}") + return target_dir + + # Remove existing directory + import shutil + + logger.info(f"Removing existing MARBLE directory: {target_dir}") + shutil.rmtree(target_dir) + + # Clone repository + logger.info(f"Cloning MARBLE from {MARBLE_REPO_URL} to {target_dir}") + + try: + subprocess.run( + ["git", "clone", MARBLE_REPO_URL, str(target_dir)], + check=True, + capture_output=True, + text=True, + ) + except subprocess.CalledProcessError as e: + raise RuntimeError(f"Failed to clone MARBLE: {e.stderr}") from e + except FileNotFoundError: + raise RuntimeError("git is not installed or not in PATH. Please install git and try again.") + + # Checkout specific commit if requested + checkout_commit = commit or MARBLE_DEFAULT_COMMIT + if checkout_commit: + logger.info(f"Checking out commit: {checkout_commit}") + try: + subprocess.run( + ["git", "checkout", checkout_commit], + cwd=target_dir, + check=True, + capture_output=True, + text=True, + ) + except subprocess.CalledProcessError as e: + raise RuntimeError(f"Failed to checkout commit {checkout_commit}: {e.stderr}") from e + + logger.info(f"MARBLE successfully installed at {target_dir}") + return target_dir + + +def ensure_marble_exists(auto_download: bool = True) -> Path: + """Ensure MARBLE is available, optionally downloading it. + + This function checks if MARBLE is installed and optionally downloads it + if not present. + + Args: + auto_download: If True, automatically download MARBLE if not found. + If False, raise an error if MARBLE is not found. + + Returns: + Path to the MARBLE directory + + Raises: + FileNotFoundError: If MARBLE is not found and auto_download=False + + Example: + >>> marble_dir = ensure_marble_exists() + >>> # MARBLE is now available at marble_dir + """ + marble_dir = _get_marble_dir() + + # Check if MARBLE exists and has the expected structure + if marble_dir.exists() and (marble_dir / "multiagentbench").exists(): + return marble_dir + + if not auto_download: + raise FileNotFoundError( + f"MARBLE not found at {marble_dir}.\n" + "Run `ensure_marble_exists(auto_download=True)` to download automatically,\n" + "or manually clone: git clone https://github.com/ulab-uiuc/MARBLE.git marble" + ) + + return download_marble(marble_dir) + + +def _resolve_data_dir(data_dir: Optional[Path] = None) -> Path: + """Resolve the MARBLE data directory. + + Searches for MARBLE data in the following order: + 1. Provided data_dir argument + 2. MARBLE_DATA_DIR environment variable + 3. marble/multiagentbench/ relative to this module + 4. Current working directory / marble/multiagentbench/ + + Args: + data_dir: Optional explicit data directory + + Returns: + Path to the MARBLE multiagentbench data directory + + Raises: + FileNotFoundError: If no valid data directory is found + """ + if data_dir is not None: + path = Path(data_dir) + if path.exists(): + return path + raise FileNotFoundError(f"Specified data_dir does not exist: {data_dir}") + + # Check environment variable + env_dir = os.environ.get("MARBLE_DATA_DIR") + + candidates = [] + if env_dir: + candidates.append(Path(env_dir)) + + # Relative to this module (vendored MARBLE) + candidates.append(Path(__file__).parent / "marble" / "multiagentbench") + + # Current working directory + candidates.append(Path.cwd() / "marble" / "multiagentbench") + + for candidate in candidates: + if candidate.exists(): + return candidate + + raise FileNotFoundError( + "MARBLE data directory not found. Either:\n" + "1. Clone MARBLE to maseval/benchmark/multiagentbench/marble/\n" + "2. Set MARBLE_DATA_DIR environment variable\n" + "See multiagentbench/README.md for setup instructions." + ) + + +def _parse_task_entry(entry: Dict[str, Any], domain: str, idx: int) -> Task: + """Parse a JSONL entry into a MASEval Task. + + Args: + entry: Raw JSONL entry dict + domain: Domain name + idx: Entry index (for error messages) + + Returns: + MASEval Task object + + Raises: + ValueError: If required fields are missing + """ + # Required fields - fail if missing + REQUIRED_FIELDS = ["scenario", "task_id", "task", "agents", "relationships"] + missing = [f for f in REQUIRED_FIELDS if f not in entry] + if missing: + raise ValueError(f"Task entry {idx} missing required fields: {missing}\nEntry keys: {list(entry.keys())}") + + # Validate agent specifications + for i, agent_spec in enumerate(entry["agents"]): + if "agent_id" not in agent_spec: + raise ValueError(f"Agent {i} in task {entry['task_id']} missing 'agent_id'\nAgent spec: {agent_spec}") + + # Extract task content + task_content = entry["task"] + if isinstance(task_content, dict): + query = task_content.get("content", "") + output_format = task_content.get("output_format", "") + else: + query = str(task_content) + output_format = "" + + if not query: + raise ValueError(f"Task {entry['task_id']} has empty query/content") + + # Extract environment config + env_config = entry.get("environment", {}) + + # Extract coordination mode + coordinate_mode = entry.get("coordinate_mode", "") + if not coordinate_mode: + # Default based on domain if not specified + coordinate_mode = "star" + + # Build Task object + return Task( + id=f"{domain}_{entry['task_id']}", + query=query, + environment_data={ + "scenario": entry["scenario"], + "coordinate_mode": coordinate_mode, + "relationships": entry["relationships"], + "environment": env_config, + "task": entry["task"], + "agents": entry["agents"], + "max_iterations": env_config.get("max_iterations") or 10, + "engine_planner": entry.get("engine_planner", {}), + "memory": entry.get("memory", {}), + "output": entry.get("output", {}), + # Store raw entry for MARBLE compatibility + "raw_marble_config": entry, + }, + evaluation_data={ + "metrics": entry.get("metrics", {}), + "output_format": output_format, + }, + metadata={ + "domain": domain, + "task_id": entry["task_id"], + "scenario": entry["scenario"], + }, + ) + + +def load_tasks( + domain: str, + data_dir: Optional[Path] = None, + limit: Optional[int] = None, +) -> List[Task]: + """Load MultiAgentBench tasks from JSONL files. + + Args: + domain: Domain name (one of: coding, database, minecraft, research, + bargaining, web, worldsimulation) + data_dir: Optional path to MARBLE data directory + limit: Maximum number of tasks to load (None for all) + + Returns: + List of Task objects + + Raises: + ValueError: If domain is invalid + FileNotFoundError: If data files not found + + Example: + >>> tasks = load_tasks("research", limit=5) + >>> len(tasks) + 5 + >>> tasks[0].metadata["domain"] + 'research' + """ + # Normalize and validate domain + domain_lower = domain.lower() + if domain_lower not in VALID_DOMAINS: + raise ValueError(f"Invalid domain '{domain}'. Must be one of: {sorted(VALID_DOMAINS)}") + + # Find data directory + resolved_data_dir = _resolve_data_dir(data_dir) + jsonl_path = resolved_data_dir / domain_lower / f"{domain_lower}_main.jsonl" + + if not jsonl_path.exists(): + raise FileNotFoundError( + f"Task data not found: {jsonl_path}\n" + f"Ensure MARBLE is cloned to multiagentbench/marble/\n" + f"See multiagentbench/README.md for setup instructions." + ) + + tasks = [] + with jsonl_path.open(encoding="utf-8") as f: + for idx, line in enumerate(f): + if limit is not None and idx >= limit: + break + + line = line.strip() + if not line: + continue + + entry = json.loads(line) + task = _parse_task_entry(entry, domain_lower, idx) + tasks.append(task) + + return tasks + + +def configure_model_ids( + tasks: List[Task], + *, + agent_model_id: str, + evaluator_model_id: Optional[str] = None, +) -> List[Task]: + """Configure model IDs for MARBLE agents and evaluator. + + Modifies tasks in-place to set the LLM model IDs used by agents + and optionally the evaluator. + + Args: + tasks: List of Tasks to configure + agent_model_id: Model ID for all MARBLE agents (e.g., "gpt-4o") + evaluator_model_id: Optional model ID for LLM-based evaluation + + Returns: + The input tasks (modified in-place) + + Example: + >>> tasks = load_tasks("research", limit=5) + >>> configure_model_ids(tasks, agent_model_id="gpt-4o") + >>> tasks[0].environment_data["llm"] + 'gpt-4o' + """ + for task in tasks: + # Set agent model + task.environment_data["llm"] = agent_model_id + + # Set evaluator model if provided + if evaluator_model_id: + task.evaluation_data["model_id"] = evaluator_model_id + else: + # Default to agent model for evaluation + task.evaluation_data["model_id"] = agent_model_id + + return tasks + + +def get_domain_info(domain: str) -> Dict[str, Any]: + """Get information about a domain. + + Args: + domain: Domain name + + Returns: + Dict with domain information including: + - requires_infrastructure: Whether external services needed + - description: Brief domain description + - coordination_mode: Default coordination mode + + Raises: + ValueError: If domain is invalid + """ + domain_lower = domain.lower() + if domain_lower not in VALID_DOMAINS: + raise ValueError(f"Invalid domain '{domain}'. Must be one of: {sorted(VALID_DOMAINS)}") + + domain_info = { + "coding": { + "requires_infrastructure": False, + "description": "Software development collaboration tasks", + "coordination_mode": "tree", + }, + "database": { + "requires_infrastructure": True, + "description": "Database manipulation and querying tasks (requires Docker)", + "coordination_mode": "star", + }, + "minecraft": { + "requires_infrastructure": True, + "description": "Collaborative building in Minecraft (requires game server)", + "coordination_mode": "cooperative", + }, + "research": { + "requires_infrastructure": False, + "description": "Research idea generation and collaboration", + "coordination_mode": "cooperative", + }, + "bargaining": { + "requires_infrastructure": False, + "description": "Negotiation and bargaining scenarios", + "coordination_mode": "cooperative", + }, + "web": { + "requires_infrastructure": False, + "description": "Web-based task completion", + "coordination_mode": "star", + }, + "worldsimulation": { + "requires_infrastructure": False, + "description": "World simulation and interaction tasks", + "coordination_mode": "cooperative", + }, + } + + return domain_info[domain_lower] diff --git a/maseval/benchmark/multiagentbench/environment.py b/maseval/benchmark/multiagentbench/environment.py new file mode 100644 index 00000000..5be86308 --- /dev/null +++ b/maseval/benchmark/multiagentbench/environment.py @@ -0,0 +1,360 @@ +"""MultiAgentBench Environment implementation. + +This module provides the MASEval Environment wrapper for MARBLE environments. +""" + +import shutil +from typing import Any, Callable, Dict, Optional + +from maseval import Environment, EnvironmentError, ToolInvocationHistory +from maseval.benchmark.multiagentbench._constants import MARBLE_IMPORT_ERROR + + +# Domains requiring external infrastructure +INFRASTRUCTURE_DOMAINS = frozenset({"database", "minecraft"}) + + +class MultiAgentBenchEnvironment(Environment): + """MASEval Environment wrapper for MARBLE environments. + + This environment wraps MARBLE's domain-specific environments (Research, + Bargaining, Coding, etc.) and exposes their tools through MASEval's + tracing infrastructure. + + Attributes: + domain: The domain name (e.g., "research", "bargaining") + marble_env: The underlying MARBLE environment instance + """ + + def __init__( + self, + task_data: Dict[str, Any], + ): + """Initialize the environment. + + Args: + task_data: Task data containing environment configuration + + Raises: + EnvironmentError: If required infrastructure is unavailable + """ + self.domain = task_data.get("scenario", "") + self._marble_env: Optional[Any] = None + self._tool_histories: Dict[str, ToolInvocationHistory] = {} + super().__init__(task_data) + + def setup_state(self, task_data: Dict[str, Any]) -> Dict[str, Any]: + """Initialize state and optionally create MARBLE environment. + + Args: + task_data: Task data containing environment configuration + + Returns: + Initial state dictionary + + Raises: + EnvironmentError: If required infrastructure is unavailable + """ + domain = task_data.get("scenario", "") + env_config = task_data.get("environment", {}) + task_config = task_data.get("task", {}) + + # Check infrastructure requirements + if domain.lower() in INFRASTRUCTURE_DOMAINS: + if not self._check_infrastructure(domain): + raise EnvironmentError( + f"Domain '{domain}' requires external infrastructure. See README.md for setup instructions.", + component="MultiAgentBenchEnvironment", + ) + + # Build config for MARBLE environment + marble_config = { + "description": env_config.get("description", f"{domain} environment"), + "task_description": task_config.get("content", "") if isinstance(task_config, dict) else str(task_config), + "ground_truth": env_config.get("ground_truth", ""), + "max_iterations": env_config.get("max_iterations") or task_data.get("max_iterations", 10), + } + + # Try to create MARBLE environment (may fail if MARBLE not available) + try: + self._marble_env = self._create_marble_environment(domain, marble_config) + except ImportError: + # MARBLE not available - store config for later use + self._marble_env = None + + return { + "domain": domain, + "env_config": env_config, + "task_config": task_config, + "marble_env_type": type(self._marble_env).__name__ if self._marble_env else "None", + "max_iterations": marble_config["max_iterations"], + } + + def _check_infrastructure(self, domain: str) -> bool: + """Check if required infrastructure is available. + + Args: + domain: Domain name + + Returns: + True if infrastructure is available, False otherwise + """ + domain_lower = domain.lower() + + if domain_lower == "database": + # Check Docker availability + return shutil.which("docker") is not None + + if domain_lower == "minecraft": + # Minecraft requires external server - always fail for now + return False + + return True + + def _create_marble_environment( + self, + domain: str, + config: Dict[str, Any], + ) -> Any: + """Create the appropriate MARBLE environment. + + Args: + domain: Domain name + config: Environment configuration + + Returns: + MARBLE environment instance + + Raises: + ImportError: If MARBLE is not available + """ + domain_lower = domain.lower() + env_name = config.get("name", domain_lower) + + # Import MARBLE environments + try: + from .marble.environments.base_env import BaseEnvironment # type: ignore[unresolved-import] + except ImportError as e: + raise ImportError(MARBLE_IMPORT_ERROR.format(error=e)) from e + + # Map domains to environment classes + env_mapping: Dict[str, str] = { + "coding": "marble.environments.coding_env.CodingEnvironment", + "database": "marble.environments.db_env.DBEnvironment", + "research": "marble.environments.research_env.ResearchEnvironment", + "bargaining": "marble.environments.bargaining_env.BargainingEnvironment", + "web": "marble.environments.web_env.WebEnvironment", + "worldsimulation": "marble.environments.world_env.WorldSimulationEnvironment", + "minecraft": "marble.environments.minecraft_env.MinecraftEnvironment", + } + + env_class_path = env_mapping.get(domain_lower) + + if env_class_path is None: + # Use base environment for unknown domains + return BaseEnvironment(env_name, config) + + try: + # Dynamic import of domain-specific environment + module_path, class_name = env_class_path.rsplit(".", 1) + # Adjust import path for vendored MARBLE + module_path = module_path.replace("marble.", ".marble.", 1) + module = __import__(module_path, globals(), locals(), [class_name], 1) + env_class = getattr(module, class_name) + return env_class(env_name, config) + except (ImportError, AttributeError): + # Fall back to base environment + return BaseEnvironment(env_name, config) + + def create_tools(self) -> Dict[str, Callable]: + """Create tools from MARBLE environment for MASEval tracing. + + MARBLE environments expose tools via action_handler_descriptions. + This method wraps them for MASEval's tracing infrastructure. + + Returns: + Dict mapping tool names to wrapped callables + """ + if self._marble_env is None: + return {} + + tools: Dict[str, Callable] = {} + + # Get action handlers from MARBLE environment + action_handlers = getattr(self._marble_env, "_action_handlers", {}) + + for action_name, handler in action_handlers.items(): + # Create tool history for this action + history = ToolInvocationHistory() + self._tool_histories[action_name] = history + + # Wrap handler for tracing + wrapped = self._wrap_tool_for_tracing(action_name, handler, history) + tools[action_name] = wrapped + + return tools + + def _wrap_tool_for_tracing( + self, + name: str, + handler: Callable, + history: ToolInvocationHistory, + ) -> Callable: + """Wrap a MARBLE action handler for MASEval tracing. + + Args: + name: Tool name + handler: Original handler callable + history: ToolInvocationHistory to record invocations + + Returns: + Wrapped callable that records invocations + """ + + def traced_handler(**kwargs: Any) -> Any: + try: + result = handler(**kwargs) + history.add_invocation( + inputs=kwargs, + outputs=result, + status="success", + ) + return result + except Exception as e: + history.add_invocation( + inputs=kwargs, + outputs=str(e), + status="error", + ) + raise + + # Attach metadata for inspection + traced_handler._original_name = name # type: ignore[attr-defined] + traced_handler._history = history # type: ignore[attr-defined] + + return traced_handler + + def get_tool(self, name: str) -> Optional[Callable]: + """Get a specific tool by name. + + Args: + name: Tool name + + Returns: + Tool callable if found, None otherwise + """ + return self.tools.get(name) + + def get_tool_descriptions(self) -> Dict[str, Any]: + """Get tool descriptions in OpenAI function format. + + Returns: + Dict mapping tool names to their OpenAI-format descriptions + """ + if self._marble_env is None: + return {} + + return getattr(self._marble_env, "action_handler_descriptions", {}) + + def apply_action( + self, + agent_id: Optional[str], + action_name: str, + arguments: Dict[str, Any], + ) -> Dict[str, Any]: + """Execute an action in the MARBLE environment. + + Args: + agent_id: ID of the agent performing the action + action_name: Name of the action to execute + arguments: Arguments for the action + + Returns: + Action result dictionary + + Raises: + EnvironmentError: If MARBLE environment is not available + """ + if self._marble_env is None: + raise EnvironmentError( + "MARBLE environment not available. Cannot execute actions.", + component="MultiAgentBenchEnvironment", + ) + + return self._marble_env.apply_action(agent_id, action_name, arguments) + + def is_done(self) -> bool: + """Check if the environment has reached a terminal state. + + Returns: + True if done, False otherwise + """ + if self._marble_env is None: + return False + return self._marble_env.is_done() + + def is_task_completed(self) -> bool: + """Check if the task has been completed successfully. + + Returns: + True if task completed, False otherwise + """ + if self._marble_env is None: + return False + return self._marble_env.is_task_completed() + + def get_marble_state(self) -> Dict[str, Any]: + """Get the current MARBLE environment state. + + Returns: + State dictionary from MARBLE environment + """ + if self._marble_env is None: + return {} + return self._marble_env.get_state() + + def gather_traces(self) -> Dict[str, Any]: + """Gather traces including tool invocations. + + Returns: + Dict with environment traces + """ + traces = super().gather_traces() + + # Add domain-specific info + traces["domain"] = self.domain + traces["marble_env_type"] = type(self._marble_env).__name__ if self._marble_env else "None" + + # Add MARBLE state if available + if self._marble_env is not None: + traces["marble_state"] = self.get_marble_state() + traces["is_done"] = self.is_done() + traces["is_task_completed"] = self.is_task_completed() + + # Collect tool invocation histories + tool_traces = {} + for name, history in self._tool_histories.items(): + invocations = history.to_list() + tool_traces[name] = { + "invocations": invocations, + "invocation_count": len(invocations), + } + traces["tool_invocations"] = tool_traces + + return traces + + def gather_config(self) -> Dict[str, Any]: + """Gather environment configuration. + + Returns: + Dict with environment configuration + """ + config = super().gather_config() + + config["domain"] = self.domain + config["marble_env_type"] = type(self._marble_env).__name__ if self._marble_env else "None" + + # Add tool descriptions + config["tool_descriptions"] = self.get_tool_descriptions() + + return config diff --git a/maseval/benchmark/multiagentbench/evaluator.py b/maseval/benchmark/multiagentbench/evaluator.py new file mode 100644 index 00000000..43ed8e30 --- /dev/null +++ b/maseval/benchmark/multiagentbench/evaluator.py @@ -0,0 +1,485 @@ +"""MultiAgentBench evaluator implementation. + +This module provides evaluation metrics matching MARBLE's evaluation methodology. +""" + +import json +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Dict, List, Optional + +from maseval import Evaluator, ModelAdapter + +# Sentinel value for "evaluation not performed" or "evaluation failed" +# Using None instead of -1 to avoid confusion with valid scores +SCORE_NOT_EVALUATED: None = None + + +@dataclass +class MultiAgentBenchMetrics: + """Metrics collected during MultiAgentBench evaluation. + + Attributes: + task_completion: Whether the task was completed + token_consumption: Total tokens used + communication_score: Score for inter-agent communication (1-5), None if not evaluated + task_evaluation: Domain-specific evaluation results + agent_kpis: Per-agent key performance indicators + total_milestones: Number of milestones achieved + """ + + task_completion: bool = False + token_consumption: int = 0 + communication_score: Optional[float] = None + task_evaluation: Dict[str, Any] = field(default_factory=dict) + agent_kpis: Dict[str, int] = field(default_factory=dict) + total_milestones: int = 0 + + def to_dict(self) -> Dict[str, Any]: + """Convert metrics to dictionary.""" + return { + "task_completion": self.task_completion, + "token_consumption": self.token_consumption, + "communication_score": self.communication_score, + "task_evaluation": self.task_evaluation, + "agent_kpis": self.agent_kpis, + "total_milestones": self.total_milestones, + } + + +class MultiAgentBenchEvaluator(Evaluator): + """Evaluator for MultiAgentBench tasks matching MARBLE's methodology. + + This evaluator implements MARBLE's LLM-based evaluation metrics: + - Task completion assessment + - Communication quality scoring + - Planning/coordination scoring + - Domain-specific task evaluation (research, bargaining, etc.) + + Attributes: + domain: The benchmark domain (research, bargaining, etc.) + model_adapter: Model adapter for LLM-based evaluation + metrics_config: Configuration for metrics to evaluate + """ + + DEFAULT_TEMPLATES_DIR = Path(__file__).parent / "prompt_templates" + + def __init__( + self, + domain: str, + model_adapter: ModelAdapter, + metrics_config: Optional[Dict[str, Any]] = None, + output_format: str = "", + ): + """Initialize the evaluator. + + Args: + domain: Benchmark domain (research, bargaining, etc.) + model_adapter: Model adapter for LLM evaluation + metrics_config: Configuration for evaluation metrics + output_format: Expected output format for task evaluation + """ + self.domain = domain.lower() + self.model_adapter = model_adapter + self.metrics_config = metrics_config or {} + self.output_format = output_format + self._evaluation_prompts = self._load_evaluation_prompts() + + def _load_template(self, filename: str) -> str: + """Load a prompt template from file. + + Args: + filename: Template filename (without directory) + + Returns: + Template content as string + """ + template_path = self.DEFAULT_TEMPLATES_DIR / filename + return template_path.read_text() + + def _load_evaluation_prompts(self) -> Dict[str, Any]: + """Load evaluation prompts from template files.""" + return { + "communication": { + "prompt": self._load_template("communication.txt"), + }, + "research": { + "task_evaluation": { + "prompt": self._load_template("research.txt"), + } + }, + "bargaining": { + "task_evaluation": { + "buyer_prompt": self._load_template("bargaining_buyer.txt"), + "seller_prompt": self._load_template("bargaining_seller.txt"), + } + }, + "coding": { + "task_evaluation": { + "prompt": self._load_template("coding.txt"), + } + }, + } + + def filter_traces(self, traces: Dict[str, Any]) -> Dict[str, Any]: + """Filter traces for evaluation. + + Args: + traces: All collected traces + + Returns: + Filtered traces relevant for evaluation + """ + return { + "agents": traces.get("agents", {}), + "environment": traces.get("environment", {}), + "communications": self._extract_communications(traces), + } + + def _extract_communications(self, traces: Dict[str, Any]) -> str: + """Extract communication logs from traces. + + Args: + traces: Execution traces + + Returns: + Formatted communication string + """ + communications: List[str] = [] + + # Extract from agent traces + agent_traces = traces.get("agents", {}) + for agent_id, agent_trace in agent_traces.items(): + comm_log = agent_trace.get("communication_log", []) + for entry in comm_log: + comm = entry.get("communication", "") + if comm: + communications.append(f"[{agent_id}]: {comm}") + + return "\n".join(communications) if communications else "No communications recorded." + + def _extract_results(self, traces: Dict[str, Any]) -> str: + """Extract agent results from traces. + + Args: + traces: Execution traces + + Returns: + Formatted results string + """ + results: List[str] = [] + + agent_traces = traces.get("agents", {}) + for agent_id, agent_trace in agent_traces.items(): + action_log = agent_trace.get("action_log", []) + for entry in action_log: + result = entry.get("result", "") + if result: + results.append(f"[{agent_id}]: {result}") + + return "\n".join(results) if results else "No results recorded." + + def __call__( # type: ignore[override] + self, traces: Dict[str, Any], final_answer: Any + ) -> Dict[str, Any]: + """Evaluate the task execution. + + Args: + traces: Filtered execution traces + final_answer: Final output from agents (dict, list, str, or None) + + Returns: + Evaluation results dictionary + """ + metrics = MultiAgentBenchMetrics() + + # Extract filtered data + filtered = self.filter_traces(traces) + communications = filtered["communications"] + + # Calculate token consumption + metrics.token_consumption = self._calculate_token_consumption(traces) + + # Evaluate communication if present + if communications != "No communications recorded.": + metrics.communication_score = self._evaluate_communication(self._get_task_description(traces), communications) + + # Domain-specific evaluation + task_desc = self._get_task_description(traces) + final_result = self._format_final_answer(final_answer) + + if self.domain == "research": + metrics.task_evaluation = self._evaluate_research(task_desc, final_result) + elif self.domain in ("bargaining", "worldsimulation"): + metrics.task_evaluation = self._evaluate_bargaining(task_desc, final_result) + elif self.domain == "coding": + metrics.task_evaluation = self._evaluate_coding(task_desc, final_result) + elif self.domain == "database": + metrics.task_evaluation = self._evaluate_database(task_desc, final_result) + else: + # Default: check if task has a completion marker + metrics.task_completion = bool(final_result) + + # Set task completion based on evaluation + metrics.task_completion = self._determine_completion(metrics) + + return { + "passed": metrics.task_completion, + "metrics": metrics.to_dict(), + "domain": self.domain, + } + + def _get_task_description(self, traces: Dict[str, Any]) -> str: + """Get task description from traces.""" + env_traces = traces.get("environment", {}) + state = env_traces.get("marble_state", {}) + return state.get("task_description", "") + + def _format_final_answer(self, final_answer: Any) -> str: + """Format final answer for evaluation.""" + if isinstance(final_answer, dict): + # Handle structured output from run_agents + results = final_answer.get("agent_results", []) + if results: + return "\n".join(f"[{r.get('agent_id', 'unknown')}]: {r.get('result', '')}" for r in results) + return json.dumps(final_answer) + elif isinstance(final_answer, list): + return "\n".join(f"[{r.get('agent_id', 'unknown')}]: {r.get('result', '')}" for r in final_answer if isinstance(r, dict)) + return str(final_answer) if final_answer else "" + + def _calculate_token_consumption(self, traces: Dict[str, Any]) -> int: + """Calculate total token consumption.""" + total = 0 + + agent_traces = traces.get("agents", {}) + for agent_trace in agent_traces.values(): + token_usage = agent_trace.get("token_usage", 0) + if isinstance(token_usage, int): + total += token_usage + + return total + + def _evaluate_communication(self, task: str, communications: str) -> Optional[float]: + """Evaluate communication quality using LLM.""" + prompt_template = self._evaluation_prompts["communication"]["prompt"] + prompt = prompt_template.format(task=task, communications=communications) + + try: + response = self.model_adapter.generate(prompt) + return self._parse_score(response) + except Exception: + return None + + def _evaluate_research(self, task: str, result: str) -> Dict[str, Any]: + """Evaluate research task output.""" + prompt_template = self._evaluation_prompts["research"]["task_evaluation"]["prompt"] + prompt = prompt_template.format(task=task, result=result) + + try: + response = self.model_adapter.generate(prompt) + return self._parse_research_ratings(response) + except Exception: + return {"innovation": None, "safety": None, "feasibility": None} + + def _evaluate_bargaining(self, task: str, result: str) -> Dict[str, Any]: + """Evaluate bargaining/world simulation task output.""" + # Evaluate both buyer and seller perspectives + buyer_prompt = self._evaluation_prompts["bargaining"]["task_evaluation"]["buyer_prompt"] + seller_prompt = self._evaluation_prompts["bargaining"]["task_evaluation"]["seller_prompt"] + + ratings = {"buyer": {}, "seller": {}} + + try: + buyer_response = self.model_adapter.generate(buyer_prompt.format(task=task, result=result)) + ratings["buyer"] = self._parse_bargaining_ratings(buyer_response) + except Exception: + ratings["buyer"] = { + "effectiveness_of_strategies": None, + "progress_and_outcome": None, + "interaction_dynamics": None, + } + + try: + seller_response = self.model_adapter.generate(seller_prompt.format(task=task, result=result)) + ratings["seller"] = self._parse_bargaining_ratings(seller_response) + except Exception: + ratings["seller"] = { + "effectiveness_of_strategies": None, + "progress_and_outcome": None, + "interaction_dynamics": None, + } + + return ratings + + def _evaluate_coding(self, task: str, result: str) -> Dict[str, Any]: + """Evaluate coding task output.""" + prompt_template = self._evaluation_prompts["coding"]["task_evaluation"]["prompt"] + + # For coding, we need requirements and solution separately + # If not available, use task as description and result as solution + prompt = prompt_template.format( + task_description=task, + requirements="See task description", + solution=result, + ) + + try: + response = self.model_adapter.generate(prompt) + return self._parse_coding_ratings(response) + except Exception: + return { + "instruction_following": None, + "executability": None, + "consistency": None, + "quality": None, + } + + def _evaluate_database(self, task: str, result: str) -> Dict[str, Any]: + """Evaluate database task output. + + Database tasks have ground truth labels that would be compared + separately. Here we just store the prediction. + """ + return { + "predicted": result, + "root_cause": [], # Would be filled from task data + } + + def _parse_score(self, response: str) -> Optional[float]: + """Parse a single score from LLM response. + + Returns: + Score as float (1-5), or None if parsing fails + """ + try: + content = response.strip() + + # Remove markdown code block markers + if content.startswith("```json"): + content = content[7:] + if content.startswith("```"): + content = content[3:] + if content.endswith("```"): + content = content[:-3] + content = content.strip() + + # Find JSON object + json_start = content.find("{") + json_end = content.rfind("}") + 1 + + if json_start >= 0 and json_end > json_start: + json_str = content[json_start:json_end] + rating_data = json.loads(json_str) + if isinstance(rating_data, dict) and "rating" in rating_data: + score = int(rating_data["rating"]) + if 1 <= score <= 5: + return float(score) + + return None + + except Exception: + return None + + def _parse_research_ratings(self, response: str) -> Dict[str, Optional[int]]: + """Parse research evaluation ratings.""" + try: + content = response.strip() + json_start = content.find("{") + json_end = content.rfind("}") + 1 + + if json_start >= 0 and json_end > json_start: + json_str = content[json_start:json_end] + ratings = json.loads(json_str) + return {k: int(v) for k, v in ratings.items()} + except Exception: + pass + + return {"innovation": None, "safety": None, "feasibility": None} + + def _parse_bargaining_ratings(self, response: str) -> Dict[str, Optional[int]]: + """Parse bargaining evaluation ratings.""" + try: + content = response.strip() + json_start = content.find("{") + json_end = content.rfind("}") + 1 + + if json_start >= 0 and json_end > json_start: + json_str = content[json_start:json_end] + ratings = json.loads(json_str) + return { + "effectiveness_of_strategies": int(ratings["effectiveness_of_strategies"]) + if "effectiveness_of_strategies" in ratings + else None, + "progress_and_outcome": int(ratings["progress_and_outcome"]) if "progress_and_outcome" in ratings else None, + "interaction_dynamics": int(ratings["interaction_dynamics"]) if "interaction_dynamics" in ratings else None, + } + except Exception: + pass + + return { + "effectiveness_of_strategies": None, + "progress_and_outcome": None, + "interaction_dynamics": None, + } + + def _parse_coding_ratings(self, response: str) -> Dict[str, Optional[int]]: + """Parse coding evaluation ratings.""" + try: + content = response.strip() + json_start = content.find("{") + json_end = content.rfind("}") + 1 + + if json_start >= 0 and json_end > json_start: + json_str = content[json_start:json_end] + ratings = json.loads(json_str) + return { + "instruction_following": int(ratings["instruction_following"]) if "instruction_following" in ratings else None, + "executability": int(ratings["executability"]) if "executability" in ratings else None, + "consistency": int(ratings["consistency"]) if "consistency" in ratings else None, + "quality": int(ratings["quality"]) if "quality" in ratings else None, + } + except Exception: + pass + + return { + "instruction_following": None, + "executability": None, + "consistency": None, + "quality": None, + } + + def _determine_completion(self, metrics: MultiAgentBenchMetrics) -> bool: + """Determine if task was completed based on metrics. + + A task is considered completed if all required scores are present (not None) + and positive (> 0). + """ + eval_data = metrics.task_evaluation + + if not eval_data: + return False + + def _all_scores_valid(scores: List[Any]) -> bool: + """Check all scores are present and positive.""" + return all(s is not None and s > 0 for s in scores) + + if self.domain == "research": + scores = [eval_data.get(k) for k in ["innovation", "safety", "feasibility"]] + return _all_scores_valid(scores) + + elif self.domain in ("bargaining", "worldsimulation"): + buyer = eval_data.get("buyer", {}) + seller = eval_data.get("seller", {}) + buyer_scores = [buyer.get(k) for k in ["effectiveness_of_strategies", "progress_and_outcome", "interaction_dynamics"]] + seller_scores = [seller.get(k) for k in ["effectiveness_of_strategies", "progress_and_outcome", "interaction_dynamics"]] + return _all_scores_valid(buyer_scores) and _all_scores_valid(seller_scores) + + elif self.domain == "coding": + scores = [eval_data.get(k) for k in ["instruction_following", "executability", "consistency", "quality"]] + return _all_scores_valid(scores) + + elif self.domain == "database": + # Database completion is determined by comparing prediction to labels + return bool(eval_data.get("predicted")) + + return False diff --git a/maseval/benchmark/multiagentbench/multiagentbench.py b/maseval/benchmark/multiagentbench/multiagentbench.py new file mode 100644 index 00000000..e20a0a3d --- /dev/null +++ b/maseval/benchmark/multiagentbench/multiagentbench.py @@ -0,0 +1,520 @@ +"""MultiAgentBench benchmark implementations. + +This module provides benchmark classes for the MARBLE MultiAgentBench suite: +- MultiAgentBenchBenchmark: Abstract base for framework-agnostic evaluation +- MarbleMultiAgentBenchBenchmark: Exact MARBLE reproduction mode +""" + +import logging +from abc import abstractmethod +from types import SimpleNamespace +from typing import TYPE_CHECKING, Any, Dict, List, Optional, Sequence, Tuple + +from maseval import ( + AgentAdapter, + Benchmark, + Environment, + Evaluator, + ModelAdapter, + Task, + User, +) +from maseval.core.callback import BenchmarkCallback + +from maseval.benchmark.multiagentbench._constants import MARBLE_IMPORT_ERROR +from maseval.benchmark.multiagentbench.environment import MultiAgentBenchEnvironment +from maseval.benchmark.multiagentbench.evaluator import ( + MultiAgentBenchEvaluator, +) + +if TYPE_CHECKING: + from maseval.benchmark.multiagentbench.adapters.marble_adapter import MarbleAgentAdapter + +logger = logging.getLogger(__name__) + + +class MultiAgentBenchBenchmark(Benchmark): + """Abstract base class for framework-agnostic MultiAgentBench evaluation. + + This benchmark provides the infrastructure for evaluating multi-agent systems + on MARBLE's MultiAgentBench tasks. Subclasses implement `setup_agents()` with + their specific agent framework. + + The benchmark supports: + - Multiple coordination modes (star, cooperative, tree, hierarchical) + - Multiple domains (research, bargaining, coding, database, etc.) + - LLM-based evaluation matching MARBLE's metrics + - Comprehensive tracing of agent interactions + + Example: + ```python + class MyMultiAgentBenchmark(MultiAgentBenchBenchmark): + def setup_agents(self, agent_data, environment, task, user, seed_generator=None): + # Derive seeds for agents if seeding is enabled + agent_seeds = {} + if seed_generator is not None: + agents_gen = seed_generator.child("agents") + for config in task.environment_data.get("agents", []): + agent_id = config.get("agent_id") + agent_seeds[agent_id] = agents_gen.derive_seed(agent_id) + + # Create agents using your framework with seeds + agents_list = [] + agents_dict = {} + for config in task.environment_data.get("agents", []): + agent_id = config.get("agent_id") + model = self.get_model_adapter( + agent_data.get("model_id", "gpt-4o"), + register_name=f"agent_{agent_id}", + seed=agent_seeds.get(agent_id), + ) + # Create your agent with the seeded model... + ... + return agents_list, agents_dict + + def get_model_adapter(self, model_id, **kwargs): + seed = kwargs.pop("seed", None) + adapter = MyModelAdapter(model_id, seed=seed) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter + + benchmark = MyMultiAgentBenchmark(seed=42) # Enable seeding + results = benchmark.run(tasks, agent_data={"model_id": "gpt-4o"}) + ``` + """ + + def __init__( + self, + callbacks: Optional[List[BenchmarkCallback]] = None, + n_task_repeats: int = 1, + max_invocations: int = 10, + num_workers: int = 1, + fail_on_setup_error: bool = False, + fail_on_task_error: bool = False, + fail_on_evaluation_error: bool = False, + progress_bar: bool | str = True, + seed: Optional[int] = None, + seed_generator=None, + ): + """Initialize the benchmark. + + Args: + callbacks: Optional list of callbacks + n_task_repeats: Number of times to repeat each task + max_invocations: Maximum agent invocations per task + num_workers: Number of parallel workers + fail_on_setup_error: Raise on setup errors + fail_on_task_error: Raise on task errors + fail_on_evaluation_error: Raise on evaluation errors + progress_bar: Progress bar configuration + seed: Global seed for reproducible benchmark runs + seed_generator: Custom seed generator (takes precedence over seed) + """ + super().__init__( + callbacks=callbacks, + n_task_repeats=n_task_repeats, + max_invocations=max_invocations, + num_workers=num_workers, + fail_on_setup_error=fail_on_setup_error, + fail_on_task_error=fail_on_task_error, + fail_on_evaluation_error=fail_on_evaluation_error, + progress_bar=progress_bar, + seed=seed, + seed_generator=seed_generator, + ) + + def setup_environment( + self, + agent_data: Dict[str, Any], + task: Task, + seed_generator=None, + ) -> Environment: + """Create the MultiAgentBench environment. + + Args: + agent_data: Agent configuration + task: The task to set up + seed_generator: Optional seed generator for reproducibility + + Returns: + MultiAgentBenchEnvironment instance + """ + return MultiAgentBenchEnvironment( + task_data=task.environment_data, + ) + + def setup_user( + self, + agent_data: Dict[str, Any], + environment: Environment, + task: Task, + seed_generator=None, + ) -> Optional[User]: + """MultiAgentBench tasks don't use user simulators. + + The multi-agent coordination replaces user interaction. + + Args: + agent_data: Agent configuration + environment: The environment instance + task: The task + seed_generator: Optional seed generator (unused) + + Returns: + None + """ + return None + + @abstractmethod + def setup_agents( + self, + agent_data: Dict[str, Any], + environment: Environment, + task: Task, + user: Optional[User], + seed_generator=None, + ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: + """Create agents for the task (implement in subclass). + + Subclasses should: + 1. Read agent specifications from task.environment_data["agents"] + 2. Derive seeds from seed_generator for each agent's model + 3. Create agents using their framework with seeded models + 4. Wrap them in AgentAdapter + 5. Set up relationships from task.environment_data["relationships"] + + Args: + agent_data: Agent configuration (model IDs, etc.) + environment: The environment instance + task: The task containing agent specs + user: User simulator (None for MultiAgentBench) + seed_generator: Optional seed generator for deriving deterministic seeds. + Use `seed_generator.child("agents")` to create a namespace, then + `derive_seed(agent_id)` for each agent's model. + + Returns: + Tuple of (agents_to_run, agents_dict) + + Example: + ```python + def setup_agents(self, agent_data, environment, task, user, seed_generator=None): + agents_gen = seed_generator.child("agents") if seed_generator else None + + for config in task.environment_data.get("agents", []): + agent_id = config.get("agent_id") + seed = agents_gen.derive_seed(agent_id) if agents_gen else None + model = self.get_model_adapter(model_id, seed=seed) + # Create agent with seeded model... + ``` + """ + pass + + def setup_evaluators( + self, + environment: Environment, + task: Task, + agents: Sequence[AgentAdapter], + user: Optional[User], + seed_generator=None, + ) -> Sequence[Evaluator]: + """Create evaluators for the task. + + Args: + environment: The environment + task: The task with evaluation data + agents: The agents + user: User simulator (None for MultiAgentBench) + seed_generator: Optional seed generator for reproducibility + + Returns: + List of evaluators + """ + # Get evaluation model ID from task or default + eval_model_id = task.evaluation_data.get("model_id", "gpt-4o-mini") + + # Derive seed for evaluator model if seeding is enabled + evaluator_seed = None + if seed_generator is not None: + eval_gen = seed_generator.child("evaluators") + evaluator_seed = eval_gen.derive_seed("multiagentbench_evaluator") + + # Create model adapter for evaluation + model_adapter = self.get_model_adapter( + eval_model_id, + register_name="evaluator_model", + seed=evaluator_seed, + ) + + # Get domain-specific evaluation configuration + domain = task.environment_data.get("scenario", "") + metrics_config = task.evaluation_data.get("metrics", {}) + output_format = task.evaluation_data.get("output_format", "") + + return [ + MultiAgentBenchEvaluator( + domain=domain, + model_adapter=model_adapter, + metrics_config=metrics_config, + output_format=output_format, + ) + ] + + @abstractmethod + def get_model_adapter(self, model_id: str, **kwargs: Any) -> ModelAdapter: + """Provide a model adapter (implement in subclass). + + Args: + model_id: Model identifier + **kwargs: Additional arguments including register_name + + Returns: + ModelAdapter instance + """ + pass + + def run_agents( + self, + agents: Sequence[AgentAdapter], + task: Task, + environment: Environment, + query: str, + ) -> Dict[str, Any]: + """Execute the multi-agent system. + + For MultiAgentBench, this runs all agents on the task and + collects their outputs. + + Args: + agents: Agents to run + task: The task + environment: The environment + query: The query/task content + + Returns: + Dict with agent_results, communications, and coordination_mode + """ + results: List[Dict[str, Any]] = [] + communications: List[str] = [] + + coordination_mode = task.environment_data.get("coordinate_mode", "cooperative") + + for agent in agents: + result = agent.run(query) + agent_id = getattr(agent, "agent_id", str(agent)) + + results.append( + { + "agent_id": agent_id, + "result": result, + } + ) + + # Collect communication logs if available + if hasattr(agent, "get_serialized_messages"): + comm = agent.get_serialized_messages() # type: ignore[operator] + if comm: + communications.append(comm) + + return { + "agent_results": results, + "communications": communications, + "coordination_mode": coordination_mode, + } + + def evaluate( + self, + evaluators: Sequence[Evaluator], + agents: Dict[str, AgentAdapter], + final_answer: Any, + traces: Dict[str, Any], + ) -> List[Dict[str, Any]]: + """Execute evaluators on the results. + + Args: + evaluators: The evaluators + agents: Dict of all agents + final_answer: The combined agent outputs + traces: Execution traces + + Returns: + List of evaluation results + """ + results = [] + + for evaluator in evaluators: + # MultiAgentBenchEvaluator expects traces in a specific format + result = evaluator(traces, final_answer) + results.append(result) + + return results + + +class MarbleMultiAgentBenchBenchmark(MultiAgentBenchBenchmark): + """MARBLE reproduction mode for MultiAgentBench. + + This benchmark uses MARBLE's native agents and engine for exact + reproduction of published results. It wraps MARBLE components + in MASEval adapters for unified tracing. + + Example: + ```python + from maseval.benchmark.multiagentbench import ( + MarbleMultiAgentBenchBenchmark, + load_tasks, + configure_model_ids, + ) + + class MyMarbleBenchmark(MarbleMultiAgentBenchBenchmark): + def get_model_adapter(self, model_id, **kwargs): + from maseval.interface.openai import OpenAIModelAdapter + adapter = OpenAIModelAdapter(model_id) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter + + tasks = load_tasks("research", limit=5) + configure_model_ids(tasks, agent_model_id="gpt-4o") + + benchmark = MyMarbleBenchmark() + results = benchmark.run(tasks, agent_data={}) + ``` + """ + + def setup_agents( + self, + agent_data: Dict[str, Any], + environment: Environment, + task: Task, + user: Optional[User], + seed_generator=None, + ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: + """Create MARBLE agents wrapped in MASEval adapters. + + Note: + MARBLE agents use their own internal LLM handling with a model ID string, + not MASEval's ModelAdapter. This means seed_generator cannot be applied + to agent LLM calls in this implementation. For reproducible agent behavior, + use `MultiAgentBenchBenchmark` with a custom `setup_agents` that creates + agents using seeded MASEval ModelAdapters. + + Args: + agent_data: Agent configuration + environment: The environment + task: The task with agent specifications + user: User simulator (None) + seed_generator: Optional seed generator (not used for MARBLE agents, + but seeding is applied to evaluators) + + Returns: + Tuple of (agents_to_run, agents_dict) + """ + from maseval.benchmark.multiagentbench.adapters.marble_adapter import ( + create_marble_agents, + ) + + # Get agent configurations from task + agent_configs = task.environment_data.get("agents", []) + model_id = task.environment_data.get("llm", "gpt-4o-mini") + + # Get MARBLE environment from our wrapper + marble_env = None + if isinstance(environment, MultiAgentBenchEnvironment): + marble_env = environment._marble_env + + # Create MARBLE environment if not available + if marble_env is None: + marble_env = self._create_marble_env(task) + + # Create agents using factory function + agents_list, agents_dict = create_marble_agents( + agent_configs=agent_configs, + marble_env=marble_env, + model=model_id, + ) + + # Set up agent graph for inter-agent communication + self._setup_agent_graph(agents_dict, task, marble_env) + + # Register agents for tracing + for agent_id, adapter in agents_dict.items(): + self.register("agents", agent_id, adapter) + + return agents_list, agents_dict # type: ignore[return-value] + + def _create_marble_env(self, task: Task) -> Any: + """Create a MARBLE environment for the task. + + Args: + task: The task with environment configuration + + Returns: + MARBLE environment instance + """ + try: + from .marble.environments.base_env import BaseEnvironment # type: ignore[unresolved-import] + except ImportError as e: + raise ImportError(MARBLE_IMPORT_ERROR.format(error=e)) from e + + env_config = task.environment_data.get("environment", {}) + task_config = task.environment_data.get("task", {}) + + config = { + "description": f"{task.environment_data.get('scenario', '')} environment", + "task_description": task_config.get("content", "") if isinstance(task_config, dict) else str(task_config), + "max_iterations": env_config.get("max_iterations") or task.environment_data.get("max_iterations", 10), + } + + return BaseEnvironment(name=config["description"], config=config) + + def _setup_agent_graph( + self, + agents_dict: Dict[str, "MarbleAgentAdapter"], + task: Task, + marble_env: Any, + ) -> None: + """Set up MARBLE's AgentGraph for inter-agent communication. + + Args: + agents_dict: Dict of MARBLE agent adapters + task: Task with relationship data + marble_env: MARBLE environment + """ + try: + from .marble.graph.agent_graph import AgentGraph # type: ignore[unresolved-import] + except ImportError: + # MARBLE not available, skip graph setup + return + + # Extract MARBLE agents from adapters + marble_agents = [adapter.marble_agent for adapter in agents_dict.values()] + + # Build config for AgentGraph (MARBLE expects an object with these attributes) + relationships = task.environment_data.get("relationships", []) + coordination_mode = task.environment_data.get("coordinate_mode", "cooperative") + config = SimpleNamespace(coordination_mode=coordination_mode, relationships=relationships) + + try: + # Create agent graph + graph = AgentGraph(marble_agents, config) # type: ignore + + # Set graph on all agents + for agent in marble_agents: + agent.set_agent_graph(graph) + + except (ValueError, KeyError, AttributeError) as e: + # Graph creation failed, agents will work without inter-agent communication + logger.warning("AgentGraph setup failed, agents will run without inter-agent communication: %s", e) + + @abstractmethod + def get_model_adapter(self, model_id: str, **kwargs: Any) -> ModelAdapter: + """Provide a model adapter (implement in subclass). + + Args: + model_id: Model identifier + **kwargs: Additional arguments + + Returns: + ModelAdapter instance + """ + pass diff --git a/maseval/benchmark/multiagentbench/prompt_templates/bargaining_buyer.txt b/maseval/benchmark/multiagentbench/prompt_templates/bargaining_buyer.txt new file mode 100644 index 00000000..fe236bc7 --- /dev/null +++ b/maseval/benchmark/multiagentbench/prompt_templates/bargaining_buyer.txt @@ -0,0 +1,14 @@ +Evaluate the buyer's negotiation performance. + +Task: {task} + +Negotiation Result: +{result} + +Rate each dimension on a scale of 1-5: +- Effectiveness of Strategies: How well did the buyer negotiate? +- Progress and Outcome: Did the buyer achieve a favorable outcome? +- Interaction Dynamics: How well did the buyer engage with the seller? + +Respond with a JSON object: +{{"effectiveness_of_strategies": , "progress_and_outcome": , "interaction_dynamics": }} diff --git a/maseval/benchmark/multiagentbench/prompt_templates/bargaining_seller.txt b/maseval/benchmark/multiagentbench/prompt_templates/bargaining_seller.txt new file mode 100644 index 00000000..68878e71 --- /dev/null +++ b/maseval/benchmark/multiagentbench/prompt_templates/bargaining_seller.txt @@ -0,0 +1,14 @@ +Evaluate the seller's negotiation performance. + +Task: {task} + +Negotiation Result: +{result} + +Rate each dimension on a scale of 1-5: +- Effectiveness of Strategies: How well did the seller negotiate? +- Progress and Outcome: Did the seller achieve a favorable outcome? +- Interaction Dynamics: How well did the seller engage with the buyer? + +Respond with a JSON object: +{{"effectiveness_of_strategies": , "progress_and_outcome": , "interaction_dynamics": }} diff --git a/maseval/benchmark/multiagentbench/prompt_templates/coding.txt b/maseval/benchmark/multiagentbench/prompt_templates/coding.txt new file mode 100644 index 00000000..1fa98713 --- /dev/null +++ b/maseval/benchmark/multiagentbench/prompt_templates/coding.txt @@ -0,0 +1,16 @@ +Evaluate the code quality based on the following criteria. + +Task Description: {task_description} +Implementation Requirements: {requirements} + +Solution: +{solution} + +Rate each dimension on a scale of 1-5: +- Instruction Following: Does the code fulfill all requirements? +- Executability: Is the code syntactically correct and executable? +- Consistency: Is the code consistent in naming, formatting, and logic? +- Quality: Is the code well-documented, clear, and modular? + +Respond with a JSON object: +{{"instruction_following": , "executability": , "consistency": , "quality": }} diff --git a/maseval/benchmark/multiagentbench/prompt_templates/communication.txt b/maseval/benchmark/multiagentbench/prompt_templates/communication.txt new file mode 100644 index 00000000..aca7ccbf --- /dev/null +++ b/maseval/benchmark/multiagentbench/prompt_templates/communication.txt @@ -0,0 +1,15 @@ +Evaluate the communication between agents for the following task. + +Task: {task} + +Communication Log: +{communications} + +Rate the communication quality on a scale of 1-5: +1 - Poor: Irrelevant or confusing communication +2 - Below Average: Some relevant communication but lacks clarity +3 - Average: Adequate communication that addresses the task +4 - Good: Clear and relevant communication with good coordination +5 - Excellent: Highly effective communication with perfect coordination + +Respond with a JSON object: {{"rating": }} diff --git a/maseval/benchmark/multiagentbench/prompt_templates/research.txt b/maseval/benchmark/multiagentbench/prompt_templates/research.txt new file mode 100644 index 00000000..355afa6b --- /dev/null +++ b/maseval/benchmark/multiagentbench/prompt_templates/research.txt @@ -0,0 +1,14 @@ +Evaluate the following research idea based on innovation, safety, and feasibility. + +Task: {task} + +Research Result: +{result} + +Rate each dimension on a scale of 1-5: +- Innovation: How novel and creative is the research idea? +- Safety: Does the research consider ethical implications and safety? +- Feasibility: How practical and achievable is the proposed research? + +Respond with a JSON object: +{{"innovation": , "safety": , "feasibility": }} diff --git a/mkdocs.yml b/mkdocs.yml index 68ffac2b..b0bc6653 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -129,5 +129,6 @@ nav: - OpenAI: interface/inference/openai.md - Benchmarks: - MACS: benchmark/macs.md + - MultiAgentBench: benchmark/multiagentbench.md - Tau2: benchmark/tau2.md - Gaia2: benchmark/gaia2.md diff --git a/tests/test_benchmarks/test_multiagentbench/__init__.py b/tests/test_benchmarks/test_multiagentbench/__init__.py new file mode 100644 index 00000000..eaf3f093 --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/__init__.py @@ -0,0 +1 @@ +"""Tests for MultiAgentBench benchmark.""" diff --git a/tests/test_benchmarks/test_multiagentbench/conftest.py b/tests/test_benchmarks/test_multiagentbench/conftest.py new file mode 100644 index 00000000..5d875ef6 --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/conftest.py @@ -0,0 +1,281 @@ +"""Shared fixtures for MultiAgentBench tests. + +Fixture Hierarchy +----------------- +- tests/conftest.py: Generic fixtures (DummyModelAdapter, dummy_model, etc.) + These are automatically available via pytest's conftest inheritance. +- tests/test_benchmarks/test_multiagentbench/conftest.py: MultiAgentBench-specific fixtures + +MultiAgentBench-Specific Components +----------------------------------- +- MultiAgentBenchAgentAdapter: Test adapter for multi-agent scenarios +- ConcreteMultiAgentBenchBenchmark: Concrete implementation for testing +- Sample task fixtures for different domains +""" + +import pytest +from typing import Any, Dict, List, Optional, Sequence, Tuple +from unittest.mock import MagicMock + +from conftest import DummyModelAdapter +from maseval import AgentAdapter, Task, MessageHistory + + +# ============================================================================= +# Sample Task Data +# ============================================================================= + + +@pytest.fixture +def sample_research_task_data() -> Dict[str, Any]: + """Sample task data for research domain.""" + return { + "scenario": "research", + "task_id": 1, + "agents": [ + { + "agent_id": "agent1", + "profile": "I am a researcher focused on machine learning.", + "type": "BaseAgent", + }, + { + "agent_id": "agent2", + "profile": "I am a researcher focused on NLP.", + "type": "BaseAgent", + }, + ], + "coordinate_mode": "cooperative", + "relationships": [ + ["agent1", "agent2", "collaborate with"], + ], + "environment": { + "type": "Research", + "max_iterations": 10, + }, + "task": { + "content": "Collaborate to generate a research idea about federated learning.", + "output_format": "Present the idea in 5Q format.", + }, + "llm": "gpt-4o-mini", + "memory": {"type": "SharedMemory"}, + "metrics": { + "diversity_of_perspectives": True, + "engagement_level": True, + "relevance": True, + }, + "max_iterations": 10, + "engine_planner": {"initial_progress": "Starting research collaboration."}, + "output": {"format": "jsonl"}, + "raw_marble_config": {}, + } + + +@pytest.fixture +def sample_bargaining_task_data() -> Dict[str, Any]: + """Sample task data for bargaining domain.""" + return { + "scenario": "bargaining", + "task_id": 1, + "agents": [ + { + "agent_id": "buyer", + "profile": "I am a buyer looking for the best price.", + "type": "BaseAgent", + }, + { + "agent_id": "seller", + "profile": "I am a seller trying to maximize profit.", + "type": "BaseAgent", + }, + ], + "coordinate_mode": "cooperative", + "relationships": [ + ["buyer", "seller", "negotiate with"], + ], + "environment": { + "type": "WorldSimulation", + "max_iterations": 10, + }, + "task": { + "content": "Negotiate the price of a used laptop.", + "output_format": "Final agreed price or disagreement.", + }, + "llm": "gpt-4o-mini", + "memory": {"type": "SharedMemory"}, + "metrics": {}, + "max_iterations": 10, + "engine_planner": {}, + "output": {}, + "raw_marble_config": {}, + } + + +@pytest.fixture +def sample_research_task(sample_research_task_data: Dict[str, Any]) -> Task: + """Create a sample research Task.""" + return Task( + id="research_1", + query=sample_research_task_data["task"]["content"], + environment_data=sample_research_task_data, + evaluation_data={ + "metrics": sample_research_task_data.get("metrics", {}), + "output_format": sample_research_task_data["task"].get("output_format", ""), + "model_id": "gpt-4o-mini", + }, + metadata={ + "domain": "research", + "task_id": 1, + }, + ) + + +@pytest.fixture +def sample_bargaining_task(sample_bargaining_task_data: Dict[str, Any]) -> Task: + """Create a sample bargaining Task.""" + return Task( + id="bargaining_1", + query=sample_bargaining_task_data["task"]["content"], + environment_data=sample_bargaining_task_data, + evaluation_data={ + "metrics": sample_bargaining_task_data.get("metrics", {}), + "output_format": sample_bargaining_task_data["task"].get("output_format", ""), + "model_id": "gpt-4o-mini", + }, + metadata={ + "domain": "bargaining", + "task_id": 1, + }, + ) + + +# ============================================================================= +# Mock Components +# ============================================================================= + + +class MultiAgentBenchAgentAdapter(AgentAdapter): + """Test agent adapter for MultiAgentBench tests. + + Provides controllable responses without needing a real agent implementation. + """ + + def __init__( + self, + agent_id: str = "test_agent", + profile: str = "Test agent profile", + ): + super().__init__(agent_instance=MagicMock(), name=agent_id) + self._agent_id = agent_id + self._profile = profile + self._responses: List[str] = [] + self._call_count = 0 + self.run_calls: List[str] = [] + + @property + def agent_id(self) -> str: + return self._agent_id + + @property + def profile(self) -> str: + return self._profile + + def set_responses(self, responses: List[str]) -> None: + """Set canned responses for the agent.""" + self._responses = responses + + def _run_agent(self, query: str) -> MessageHistory: + self.run_calls.append(query) + if self._responses: + response = self._responses[self._call_count % len(self._responses)] + self._call_count += 1 + else: + response = f"Agent {self._agent_id} response to: {query[:50]}..." + return MessageHistory([{"role": "assistant", "content": response}]) + + def get_token_usage(self) -> int: + return self._call_count * 100 + + def gather_traces(self) -> Dict[str, Any]: + traces = super().gather_traces() + traces["agent_id"] = self._agent_id + traces["profile"] = self._profile + traces["token_usage"] = self.get_token_usage() + traces["action_log"] = [{"task": q, "result": f"Response {i}", "has_communication": False} for i, q in enumerate(self.run_calls)] + traces["communication_log"] = [] + return traces + + +# ============================================================================= +# Concrete Benchmark Implementation +# ============================================================================= + + +@pytest.fixture +def concrete_multiagentbench_benchmark(): + """Create a concrete MultiAgentBenchBenchmark for testing.""" + from maseval.benchmark.multiagentbench import MultiAgentBenchBenchmark + + class ConcreteMultiAgentBenchBenchmark(MultiAgentBenchBenchmark): + """Concrete implementation for testing.""" + + def __init__( + self, + model_factory: Optional[Any] = None, + **kwargs: Any, + ): + if model_factory is None: + self._model_factory = lambda model_name: DummyModelAdapter( + model_id=f"test-model-{model_name}", + responses=['{"rating": 4}'], + ) + elif callable(model_factory): + self._model_factory = model_factory + else: + self._model_factory = lambda model_name: model_factory + super().__init__(**kwargs) + + def get_model_adapter(self, model_id: str, **kwargs): + factory_key = kwargs.get("register_name", model_id) + adapter = self._model_factory(factory_key) + register_name = kwargs.get("register_name") + if register_name: + try: + self.register("models", register_name, adapter) + except ValueError: + pass + return adapter + + def setup_agents( + self, + agent_data: Dict[str, Any], + environment: Any, + task: Task, + user: Optional[Any], + seed_generator=None, + ) -> Tuple[Sequence[AgentAdapter], Dict[str, AgentAdapter]]: + agent_configs = task.environment_data.get("agents", []) + agents_list: List[AgentAdapter] = [] + agents_dict: Dict[str, AgentAdapter] = {} + + for config in agent_configs: + agent_id = config.get("agent_id", f"agent_{len(agents_list)}") + profile = config.get("profile", "") + adapter = MultiAgentBenchAgentAdapter( + agent_id=agent_id, + profile=profile, + ) + agents_list.append(adapter) + agents_dict[agent_id] = adapter + + return agents_list, agents_dict + + return ConcreteMultiAgentBenchBenchmark + + +@pytest.fixture +def benchmark_instance(concrete_multiagentbench_benchmark): + """Create a benchmark instance with default settings.""" + return concrete_multiagentbench_benchmark( + progress_bar=False, + max_invocations=1, + ) diff --git a/tests/test_benchmarks/test_multiagentbench/test_benchmark.py b/tests/test_benchmarks/test_multiagentbench/test_benchmark.py new file mode 100644 index 00000000..6acf67f1 --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/test_benchmark.py @@ -0,0 +1,625 @@ +"""Tests for MultiAgentBench benchmark classes.""" + +import pytest +from unittest.mock import MagicMock + +from maseval import Task +from maseval.benchmark.multiagentbench import ( + MarbleMultiAgentBenchBenchmark, + MultiAgentBenchEnvironment, + MultiAgentBenchEvaluator, +) + + +class TestMultiAgentBenchBenchmark: + """Tests for MultiAgentBenchBenchmark abstract class.""" + + def test_setup_environment_returns_correct_type( + self, + benchmark_instance, + sample_research_task: Task, + ): + """setup_environment should return MultiAgentBenchEnvironment.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + + assert isinstance(env, MultiAgentBenchEnvironment) + + def test_setup_user_returns_none( + self, + benchmark_instance, + sample_research_task: Task, + ): + """setup_user should return None (no user simulator for multi-agent).""" + env = benchmark_instance.setup_environment({}, sample_research_task) + user = benchmark_instance.setup_user({}, env, sample_research_task) + + assert user is None + + def test_setup_evaluators_returns_evaluator( + self, + benchmark_instance, + sample_research_task: Task, + ): + """setup_evaluators should return MultiAgentBenchEvaluator.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + agents, _ = benchmark_instance.setup_agents({}, env, sample_research_task, None) + evaluators = benchmark_instance.setup_evaluators(env, sample_research_task, agents, None) + + assert len(evaluators) == 1 + assert isinstance(evaluators[0], MultiAgentBenchEvaluator) + + def test_setup_evaluators_uses_seed_generator( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """setup_evaluators should pass seed to model adapter when seed_generator provided.""" + from unittest.mock import patch + from maseval.core.seeding import DefaultSeedGenerator + + # Create a benchmark and mock get_model_adapter + benchmark = concrete_multiagentbench_benchmark(progress_bar=False) + env = benchmark.setup_environment({}, sample_research_task) + agents, _ = benchmark.setup_agents({}, env, sample_research_task, None) + + # Create a seed generator + seed_gen = DefaultSeedGenerator(global_seed=42) + task_seed_gen = seed_gen.for_task("test_task").for_repetition(0) + + # Patch get_model_adapter to capture the seed argument + with patch.object(benchmark, "get_model_adapter", wraps=benchmark.get_model_adapter) as mock_get_model: + benchmark.setup_evaluators(env, sample_research_task, agents, None, seed_generator=task_seed_gen) + + # Verify get_model_adapter was called with a seed + mock_get_model.assert_called_once() + call_kwargs = mock_get_model.call_args.kwargs + assert "seed" in call_kwargs + assert call_kwargs["seed"] is not None + assert isinstance(call_kwargs["seed"], int) + + def test_setup_evaluators_no_seed_without_generator( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """setup_evaluators should pass None seed when no seed_generator provided.""" + from unittest.mock import patch + + benchmark = concrete_multiagentbench_benchmark(progress_bar=False) + env = benchmark.setup_environment({}, sample_research_task) + agents, _ = benchmark.setup_agents({}, env, sample_research_task, None) + + with patch.object(benchmark, "get_model_adapter", wraps=benchmark.get_model_adapter) as mock_get_model: + benchmark.setup_evaluators(env, sample_research_task, agents, None, seed_generator=None) + + mock_get_model.assert_called_once() + call_kwargs = mock_get_model.call_args.kwargs + assert call_kwargs.get("seed") is None + + def test_setup_evaluators_seed_is_deterministic( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """Same global seed should produce same derived seed.""" + from unittest.mock import patch + from maseval.core.seeding import DefaultSeedGenerator + + seeds_collected = [] + + for _ in range(2): + benchmark = concrete_multiagentbench_benchmark(progress_bar=False) + env = benchmark.setup_environment({}, sample_research_task) + agents, _ = benchmark.setup_agents({}, env, sample_research_task, None) + + # Same global seed each time + seed_gen = DefaultSeedGenerator(global_seed=42) + task_seed_gen = seed_gen.for_task("test_task").for_repetition(0) + + with patch.object(benchmark, "get_model_adapter", wraps=benchmark.get_model_adapter) as mock_get_model: + benchmark.setup_evaluators(env, sample_research_task, agents, None, seed_generator=task_seed_gen) + seeds_collected.append(mock_get_model.call_args.kwargs["seed"]) + + # Same seed both times + assert seeds_collected[0] == seeds_collected[1] + + def test_setup_evaluators_different_global_seeds_produce_different_seeds( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """Different global seeds should produce different derived seeds.""" + from unittest.mock import patch + from maseval.core.seeding import DefaultSeedGenerator + + seeds_collected = [] + + for global_seed in [42, 123]: + benchmark = concrete_multiagentbench_benchmark(progress_bar=False) + env = benchmark.setup_environment({}, sample_research_task) + agents, _ = benchmark.setup_agents({}, env, sample_research_task, None) + + seed_gen = DefaultSeedGenerator(global_seed=global_seed) + task_seed_gen = seed_gen.for_task("test_task").for_repetition(0) + + with patch.object(benchmark, "get_model_adapter", wraps=benchmark.get_model_adapter) as mock_get_model: + benchmark.setup_evaluators(env, sample_research_task, agents, None, seed_generator=task_seed_gen) + seeds_collected.append(mock_get_model.call_args.kwargs["seed"]) + + # Different seeds + assert seeds_collected[0] != seeds_collected[1] + + def test_benchmark_run_with_seed_passes_seed_to_evaluators( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """Full benchmark.run() with seed should pass seed through to evaluators.""" + from unittest.mock import patch + + benchmark = concrete_multiagentbench_benchmark(progress_bar=False, seed=42) + + seeds_passed = [] + original_get_model_adapter = benchmark.get_model_adapter + + def capturing_get_model_adapter(model_id, **kwargs): + seeds_passed.append(kwargs.get("seed")) + return original_get_model_adapter(model_id, **kwargs) + + with patch.object(benchmark, "get_model_adapter", side_effect=capturing_get_model_adapter): + benchmark.run(tasks=[sample_research_task], agent_data={}) + + # At least one call should have a seed (the evaluator) + assert any(s is not None for s in seeds_passed), "Expected at least one seeded model adapter call" + + def test_benchmark_run_without_seed_passes_none( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """Full benchmark.run() without seed should pass None to evaluators.""" + from unittest.mock import patch + + benchmark = concrete_multiagentbench_benchmark(progress_bar=False) # No seed + + seeds_passed = [] + original_get_model_adapter = benchmark.get_model_adapter + + def capturing_get_model_adapter(model_id, **kwargs): + seeds_passed.append(kwargs.get("seed")) + return original_get_model_adapter(model_id, **kwargs) + + with patch.object(benchmark, "get_model_adapter", side_effect=capturing_get_model_adapter): + benchmark.run(tasks=[sample_research_task], agent_data={}) + + # All calls should have None seed + assert all(s is None for s in seeds_passed), "Expected all model adapter calls to have None seed" + + def test_setup_agents_creates_correct_count( + self, + benchmark_instance, + sample_research_task: Task, + ): + """setup_agents should create correct number of agents.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + agents_list, agents_dict = benchmark_instance.setup_agents({}, env, sample_research_task, None) + + # sample_research_task has 2 agents + assert len(agents_list) == 2 + assert len(agents_dict) == 2 + assert "agent1" in agents_dict + assert "agent2" in agents_dict + + def test_run_agents_executes_all_agents( + self, + benchmark_instance, + sample_research_task: Task, + ): + """run_agents should execute all agents and return results.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + agents_list, _ = benchmark_instance.setup_agents({}, env, sample_research_task, None) + + results = benchmark_instance.run_agents( + agents_list, + sample_research_task, + env, + sample_research_task.query, + ) + + assert isinstance(results, dict) + assert "agent_results" in results + assert "communications" in results + assert "coordination_mode" in results + assert len(results["agent_results"]) == 2 + assert all("agent_id" in r for r in results["agent_results"]) + assert all("result" in r for r in results["agent_results"]) + + def test_evaluate_calls_evaluators( + self, + benchmark_instance, + sample_research_task: Task, + ): + """evaluate should call all evaluators.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + agents_list, agents_dict = benchmark_instance.setup_agents({}, env, sample_research_task, None) + evaluators = benchmark_instance.setup_evaluators(env, sample_research_task, agents_list, None) + + final_answer = [{"agent_id": "agent1", "result": "Done"}] + traces = { + "agents": {"agent1": {"token_usage": 100, "action_log": [], "communication_log": []}}, + "environment": {}, + } + + results = benchmark_instance.evaluate(evaluators, agents_dict, final_answer, traces) + + assert len(results) == 1 + assert "passed" in results[0] + + +class TestBenchmarkIntegration: + """Integration tests for benchmark execution.""" + + def test_run_single_task( + self, + benchmark_instance, + sample_research_task: Task, + ): + """Benchmark should run a single task end-to-end.""" + results = benchmark_instance.run( + tasks=[sample_research_task], + agent_data={}, + ) + + assert len(results) == 1 + assert results[0]["task_id"] == "research_1" + assert "status" in results[0] + assert "traces" in results[0] + assert "eval" in results[0] + + def test_run_multiple_tasks( + self, + benchmark_instance, + sample_research_task: Task, + sample_bargaining_task: Task, + ): + """Benchmark should run multiple tasks.""" + results = benchmark_instance.run( + tasks=[sample_research_task, sample_bargaining_task], + agent_data={}, + ) + + assert len(results) == 2 + + def test_run_with_task_repeats( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """Benchmark should repeat tasks when n_task_repeats > 1.""" + benchmark = concrete_multiagentbench_benchmark( + n_task_repeats=3, + progress_bar=False, + ) + results = benchmark.run( + tasks=[sample_research_task], + agent_data={}, + ) + + assert len(results) == 3 + assert all(r["task_id"] == "research_1" for r in results) + assert [r["repeat_idx"] for r in results] == [0, 1, 2] + + def test_run_collects_traces( + self, + benchmark_instance, + sample_research_task: Task, + ): + """Benchmark should collect traces from all components.""" + results = benchmark_instance.run( + tasks=[sample_research_task], + agent_data={}, + ) + + traces = results[0]["traces"] + assert "agents" in traces + assert "environment" in traces or traces.get("environment") is None + + def test_run_handles_setup_error( + self, + concrete_multiagentbench_benchmark, + sample_research_task: Task, + ): + """Benchmark should handle setup errors gracefully.""" + + class FailingBenchmark(concrete_multiagentbench_benchmark): + def setup_environment(self, agent_data, task, seed_generator=None): + raise RuntimeError("Setup failed") + + benchmark = FailingBenchmark(progress_bar=False) + results = benchmark.run( + tasks=[sample_research_task], + agent_data={}, + ) + + assert len(results) == 1 + assert results[0]["status"] == "setup_failed" + assert "error" in results[0] + + +class TestAgentCreation: + """Tests for agent creation and configuration.""" + + def test_agents_have_correct_ids( + self, + benchmark_instance, + sample_research_task: Task, + ): + """Created agents should have IDs from task config.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + _, agents_dict = benchmark_instance.setup_agents({}, env, sample_research_task, None) + + assert "agent1" in agents_dict + assert "agent2" in agents_dict + + def test_agents_have_profiles( + self, + benchmark_instance, + sample_research_task: Task, + ): + """Created agents should have profiles from task config.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + _, agents_dict = benchmark_instance.setup_agents({}, env, sample_research_task, None) + + agent1 = agents_dict["agent1"] + assert hasattr(agent1, "profile") + assert "machine learning" in agent1.profile.lower() + + +class TestEvaluatorConfiguration: + """Tests for evaluator configuration.""" + + def test_evaluator_uses_task_model_id( + self, + benchmark_instance, + sample_research_task: Task, + ): + """Evaluator should use model_id from task evaluation_data.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + agents, _ = benchmark_instance.setup_agents({}, env, sample_research_task, None) + evaluators = benchmark_instance.setup_evaluators(env, sample_research_task, agents, None) + + evaluator = evaluators[0] + assert evaluator.domain == "research" + + def test_evaluator_domain_from_task( + self, + benchmark_instance, + sample_bargaining_task: Task, + ): + """Evaluator should get domain from task environment_data.""" + env = benchmark_instance.setup_environment({}, sample_bargaining_task) + agents, _ = benchmark_instance.setup_agents({}, env, sample_bargaining_task, None) + evaluators = benchmark_instance.setup_evaluators(env, sample_bargaining_task, agents, None) + + evaluator = evaluators[0] + assert evaluator.domain == "bargaining" + + +class TestMarbleMultiAgentBenchBenchmark: + """Tests for MarbleMultiAgentBenchBenchmark class.""" + + @pytest.fixture + def marble_benchmark_class(self): + """Create a concrete MarbleMultiAgentBenchBenchmark class.""" + from conftest import DummyModelAdapter + + class ConcreteMarbleBenchmark(MarbleMultiAgentBenchBenchmark): + def get_model_adapter(self, model_id, **kwargs): + adapter = DummyModelAdapter( + model_id=model_id, + responses=['{"rating": 4}'], + ) + register_name = kwargs.get("register_name") + if register_name: + try: + self.register("models", register_name, adapter) + except ValueError: + pass + return adapter + + return ConcreteMarbleBenchmark + + def test_setup_agents_raises_import_error( + self, + marble_benchmark_class, + sample_research_task: Task, + ): + """setup_agents should raise ImportError when MARBLE not available.""" + benchmark = marble_benchmark_class(progress_bar=False) + env = benchmark.setup_environment({}, sample_research_task) + + with pytest.raises(ImportError, match="MARBLE is not available"): + benchmark.setup_agents({}, env, sample_research_task, None) + + def test_create_marble_env_raises_import_error( + self, + marble_benchmark_class, + sample_research_task: Task, + ): + """_create_marble_env should raise ImportError when MARBLE not available.""" + benchmark = marble_benchmark_class(progress_bar=False) + + with pytest.raises(ImportError, match="MARBLE is not available"): + benchmark._create_marble_env(sample_research_task) + + def test_setup_agent_graph_silently_fails( + self, + marble_benchmark_class, + sample_research_task: Task, + ): + """_setup_agent_graph should not raise when MARBLE not available.""" + benchmark = marble_benchmark_class(progress_bar=False) + + # Should not raise, just return silently + benchmark._setup_agent_graph({}, sample_research_task, None) + + def test_run_agents_returns_structured_output( + self, + marble_benchmark_class, + sample_research_task: Task, + ): + """run_agents should return structured output with agent_results.""" + + benchmark = marble_benchmark_class(progress_bar=False) + env = benchmark.setup_environment({}, sample_research_task) + + # Create mock agents + mock_agent1 = MagicMock() + mock_agent1.run.return_value = "Result from agent1" + mock_agent1.agent_id = "agent1" + + mock_agent2 = MagicMock() + mock_agent2.run.return_value = "Result from agent2" + mock_agent2.agent_id = "agent2" + mock_agent2.get_serialized_messages.return_value = "Communication log" + + result = benchmark.run_agents( + [mock_agent1, mock_agent2], + sample_research_task, + env, + sample_research_task.query, + ) + + assert "agent_results" in result + assert "communications" in result + assert "coordination_mode" in result + assert len(result["agent_results"]) == 2 + assert result["agent_results"][0]["agent_id"] == "agent1" + assert result["agent_results"][1]["agent_id"] == "agent2" + + def test_run_agents_collects_communications( + self, + marble_benchmark_class, + sample_research_task: Task, + ): + """run_agents should collect communications from agents.""" + benchmark = marble_benchmark_class(progress_bar=False) + env = benchmark.setup_environment({}, sample_research_task) + + # Create mock agent with get_serialized_messages + mock_agent = MagicMock() + mock_agent.run.return_value = "Result" + mock_agent.agent_id = "agent1" + mock_agent.get_serialized_messages.return_value = "Hello from agent1" + + result = benchmark.run_agents( + [mock_agent], + sample_research_task, + env, + sample_research_task.query, + ) + + assert "Hello from agent1" in result["communications"] + + +class TestBenchmarkWithDifferentCoordinationModes: + """Tests for different coordination modes.""" + + def test_run_agents_with_cooperative_mode( + self, + benchmark_instance, + sample_research_task: Task, + ): + """run_agents should work with cooperative coordination.""" + # sample_research_task uses cooperative mode by default + env = benchmark_instance.setup_environment({}, sample_research_task) + agents_list, _ = benchmark_instance.setup_agents({}, env, sample_research_task, None) + + results = benchmark_instance.run_agents( + agents_list, + sample_research_task, + env, + sample_research_task.query, + ) + + assert len(results["agent_results"]) == 2 + assert results["coordination_mode"] == "cooperative" + + def test_run_agents_with_star_mode(self, benchmark_instance): + """run_agents should work with star coordination.""" + task_data = { + "scenario": "research", + "task_id": 1, + "agents": [ + {"agent_id": "central", "profile": "Central coordinator"}, + {"agent_id": "worker1", "profile": "Worker 1"}, + ], + "coordinate_mode": "star", + "relationships": [["central", "worker1", "coordinates"]], + "environment": {"max_iterations": 10}, + "task": {"content": "Research task", "output_format": "5Q"}, + "max_iterations": 10, + } + task = Task( + id="test_star", + query="Research task", + environment_data=task_data, + evaluation_data={"model_id": "gpt-4o-mini"}, + metadata={"domain": "research"}, + ) + + env = benchmark_instance.setup_environment({}, task) + agents_list, _ = benchmark_instance.setup_agents({}, env, task, None) + + results = benchmark_instance.run_agents(agents_list, task, env, task.query) + + assert len(results["agent_results"]) == 2 + assert results["coordination_mode"] == "star" + + +class TestBenchmarkWithEmptyAgents: + """Tests for edge cases with agents.""" + + def test_run_agents_with_empty_list( + self, + benchmark_instance, + sample_research_task: Task, + ): + """run_agents should handle empty agent list.""" + env = benchmark_instance.setup_environment({}, sample_research_task) + + results = benchmark_instance.run_agents( + [], + sample_research_task, + env, + sample_research_task.query, + ) + + assert results["agent_results"] == [] + assert results["communications"] == [] + + def test_setup_agents_with_no_agents_in_task(self, benchmark_instance): + """setup_agents should handle task with no agents.""" + task_data = { + "scenario": "research", + "task_id": 1, + "agents": [], # No agents + "coordinate_mode": "cooperative", + "relationships": [], + "environment": {"max_iterations": 10}, + "task": {"content": "Research task"}, + "max_iterations": 10, + } + task = Task( + id="test_no_agents", + query="Research task", + environment_data=task_data, + evaluation_data={"model_id": "gpt-4o-mini"}, + metadata={"domain": "research"}, + ) + + env = benchmark_instance.setup_environment({}, task) + agents_list, agents_dict = benchmark_instance.setup_agents({}, env, task, None) + + assert len(agents_list) == 0 + assert len(agents_dict) == 0 diff --git a/tests/test_benchmarks/test_multiagentbench/test_data_loader.py b/tests/test_benchmarks/test_multiagentbench/test_data_loader.py new file mode 100644 index 00000000..555e9f04 --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/test_data_loader.py @@ -0,0 +1,563 @@ +"""Tests for MultiAgentBench data loading functionality.""" + +import json +import subprocess +import tempfile +from pathlib import Path +from typing import Any, Dict +from unittest.mock import MagicMock, patch + +import pytest + +from maseval import Task +from maseval.benchmark.multiagentbench.data_loader import ( + load_tasks, + configure_model_ids, + get_domain_info, + download_marble, + ensure_marble_exists, + _get_marble_dir, + VALID_DOMAINS, + _parse_task_entry, + _resolve_data_dir, +) + + +class TestValidDomains: + """Tests for domain validation.""" + + def test_valid_domains_contains_expected(self): + """VALID_DOMAINS should contain all expected domains.""" + expected = {"coding", "database", "minecraft", "research", "bargaining", "web", "worldsimulation"} + assert expected == VALID_DOMAINS + + def test_valid_domains_is_frozen(self): + """VALID_DOMAINS should be immutable.""" + assert isinstance(VALID_DOMAINS, frozenset) + + +class TestGetDomainInfo: + """Tests for get_domain_info function.""" + + def test_research_domain_info(self): + """get_domain_info should return correct info for research.""" + info = get_domain_info("research") + assert info["requires_infrastructure"] is False + assert info["coordination_mode"] == "cooperative" + assert "description" in info + + def test_database_requires_infrastructure(self): + """Database domain should require infrastructure.""" + info = get_domain_info("database") + assert info["requires_infrastructure"] is True + + def test_minecraft_requires_infrastructure(self): + """Minecraft domain should require infrastructure.""" + info = get_domain_info("minecraft") + assert info["requires_infrastructure"] is True + + def test_invalid_domain_raises(self): + """get_domain_info should raise for invalid domain.""" + with pytest.raises(ValueError, match="Invalid domain"): + get_domain_info("invalid_domain") + + def test_case_insensitive(self): + """get_domain_info should be case-insensitive.""" + info_lower = get_domain_info("research") + info_upper = get_domain_info("RESEARCH") + assert info_lower == info_upper + + +class TestParseTaskEntry: + """Tests for _parse_task_entry function.""" + + def test_parse_minimal_entry(self): + """_parse_task_entry should parse a minimal valid entry.""" + entry = { + "scenario": "research", + "task_id": 1, + "task": {"content": "Do research", "output_format": "5Q format"}, + "agents": [{"agent_id": "agent1", "profile": "Researcher"}], + "relationships": [["agent1", "agent1", "self"]], + } + task = _parse_task_entry(entry, "research", 0) + + assert isinstance(task, Task) + assert task.id == "research_1" + assert task.query == "Do research" + assert task.environment_data["scenario"] == "research" + assert len(task.environment_data["agents"]) == 1 + + def test_parse_entry_missing_required_field(self): + """_parse_task_entry should raise for missing required fields.""" + entry = { + "scenario": "research", + "task_id": 1, + # Missing "task", "agents", "relationships" + } + with pytest.raises(ValueError, match="missing required fields"): + _parse_task_entry(entry, "research", 0) + + def test_parse_entry_missing_agent_id(self): + """_parse_task_entry should raise if agent missing agent_id.""" + entry = { + "scenario": "research", + "task_id": 1, + "task": {"content": "Do research"}, + "agents": [{"profile": "Researcher"}], # Missing agent_id + "relationships": [], + } + with pytest.raises(ValueError, match="missing 'agent_id'"): + _parse_task_entry(entry, "research", 0) + + def test_parse_entry_empty_query(self): + """_parse_task_entry should raise for empty query.""" + entry = { + "scenario": "research", + "task_id": 1, + "task": {"content": "", "output_format": "5Q format"}, + "agents": [{"agent_id": "agent1"}], + "relationships": [], + } + with pytest.raises(ValueError, match="empty query"): + _parse_task_entry(entry, "research", 0) + + def test_parse_entry_with_string_task(self): + """_parse_task_entry should handle task as string.""" + entry = { + "scenario": "research", + "task_id": 1, + "task": "Do research task", + "agents": [{"agent_id": "agent1"}], + "relationships": [], + } + task = _parse_task_entry(entry, "research", 0) + assert task.query == "Do research task" + + def test_parse_entry_preserves_metadata(self): + """_parse_task_entry should preserve metadata correctly.""" + entry = { + "scenario": "bargaining", + "task_id": 42, + "task": {"content": "Negotiate"}, + "agents": [{"agent_id": "buyer"}], + "relationships": [], + "coordinate_mode": "star", + "environment": {"max_iterations": 20}, + } + task = _parse_task_entry(entry, "bargaining", 0) + + assert task.metadata["domain"] == "bargaining" + assert task.metadata["task_id"] == 42 + assert task.environment_data["coordinate_mode"] == "star" + assert task.environment_data["max_iterations"] == 20 + + +class TestLoadTasks: + """Tests for load_tasks function.""" + + def test_load_tasks_invalid_domain(self): + """load_tasks should raise for invalid domain.""" + with pytest.raises(ValueError, match="Invalid domain"): + load_tasks("invalid_domain") + + def test_load_tasks_missing_data_dir(self): + """load_tasks should raise if data directory not found.""" + with patch.dict("os.environ", {}, clear=True): + with pytest.raises(FileNotFoundError, match="does not exist"): + load_tasks("research", data_dir=Path("/nonexistent/path")) + + def test_load_tasks_with_mock_data(self): + """load_tasks should load tasks from JSONL file.""" + # Create temporary JSONL file + with tempfile.TemporaryDirectory() as tmpdir: + research_dir = Path(tmpdir) / "research" + research_dir.mkdir() + jsonl_path = research_dir / "research_main.jsonl" + + # Write sample task + task_data = { + "scenario": "research", + "task_id": 1, + "task": {"content": "Research task", "output_format": "5Q"}, + "agents": [{"agent_id": "agent1", "profile": "Researcher"}], + "relationships": [], + } + with jsonl_path.open("w") as f: + f.write(json.dumps(task_data) + "\n") + + tasks = load_tasks("research", data_dir=Path(tmpdir)) + + assert len(tasks) == 1 + assert tasks[0].query == "Research task" + + def test_load_tasks_with_limit(self): + """load_tasks should respect limit parameter.""" + with tempfile.TemporaryDirectory() as tmpdir: + research_dir = Path(tmpdir) / "research" + research_dir.mkdir() + jsonl_path = research_dir / "research_main.jsonl" + + # Write multiple tasks + with jsonl_path.open("w") as f: + for i in range(5): + task_data = { + "scenario": "research", + "task_id": i + 1, + "task": {"content": f"Research task {i + 1}"}, + "agents": [{"agent_id": f"agent{i + 1}"}], + "relationships": [], + } + f.write(json.dumps(task_data) + "\n") + + tasks = load_tasks("research", data_dir=Path(tmpdir), limit=2) + + assert len(tasks) == 2 + + def test_load_tasks_case_insensitive_domain(self): + """load_tasks should handle domain case-insensitively.""" + with tempfile.TemporaryDirectory() as tmpdir: + research_dir = Path(tmpdir) / "research" + research_dir.mkdir() + jsonl_path = research_dir / "research_main.jsonl" + + task_data = { + "scenario": "research", + "task_id": 1, + "task": {"content": "Test"}, + "agents": [{"agent_id": "agent1"}], + "relationships": [], + } + with jsonl_path.open("w") as f: + f.write(json.dumps(task_data) + "\n") + + tasks_lower = load_tasks("research", data_dir=Path(tmpdir)) + tasks_upper = load_tasks("RESEARCH", data_dir=Path(tmpdir)) + + assert len(tasks_lower) == len(tasks_upper) == 1 + + +class TestConfigureModelIds: + """Tests for configure_model_ids function.""" + + def test_configure_model_ids_sets_llm(self): + """configure_model_ids should set llm in environment_data.""" + task = Task( + id="test_1", + query="Test query", + environment_data={"scenario": "research"}, + evaluation_data={}, + metadata={}, + ) + tasks = [task] + + configure_model_ids(tasks, agent_model_id="gpt-4o") + + assert tasks[0].environment_data["llm"] == "gpt-4o" + + def test_configure_model_ids_sets_evaluator_model(self): + """configure_model_ids should set evaluator model_id.""" + task = Task( + id="test_1", + query="Test query", + environment_data={}, + evaluation_data={}, + metadata={}, + ) + tasks = [task] + + configure_model_ids( + tasks, + agent_model_id="gpt-4o", + evaluator_model_id="gpt-4o-mini", + ) + + assert tasks[0].evaluation_data["model_id"] == "gpt-4o-mini" + + def test_configure_model_ids_defaults_evaluator_to_agent(self): + """configure_model_ids should default evaluator model to agent model.""" + task = Task( + id="test_1", + query="Test query", + environment_data={}, + evaluation_data={}, + metadata={}, + ) + tasks = [task] + + configure_model_ids(tasks, agent_model_id="gpt-4o") + + assert tasks[0].evaluation_data["model_id"] == "gpt-4o" + + def test_configure_model_ids_returns_tasks(self): + """configure_model_ids should return the input tasks.""" + task = Task( + id="test_1", + query="Test query", + environment_data={}, + evaluation_data={}, + metadata={}, + ) + tasks = [task] + + result = configure_model_ids(tasks, agent_model_id="gpt-4o") + + assert result is tasks + + +class TestResolveDataDir: + """Tests for _resolve_data_dir function.""" + + def test_resolve_explicit_path(self): + """_resolve_data_dir should use explicit path if provided.""" + with tempfile.TemporaryDirectory() as tmpdir: + result = _resolve_data_dir(Path(tmpdir)) + assert result == Path(tmpdir) + + def test_resolve_nonexistent_explicit_path(self): + """_resolve_data_dir should raise for nonexistent explicit path.""" + with pytest.raises(FileNotFoundError): + _resolve_data_dir(Path("/nonexistent/path")) + + def test_resolve_from_env_var(self): + """_resolve_data_dir should use MARBLE_DATA_DIR env var.""" + with tempfile.TemporaryDirectory() as tmpdir: + with patch.dict("os.environ", {"MARBLE_DATA_DIR": tmpdir}): + result = _resolve_data_dir() + assert result == Path(tmpdir) + + def test_resolve_not_found(self): + """_resolve_data_dir should raise when no directory found.""" + with patch.dict("os.environ", {}, clear=True): + with patch("pathlib.Path.cwd", return_value=Path("/nonexistent/cwd")): + # Mock Path.exists to return False for all candidate paths + with patch.object(Path, "exists", return_value=False): + with pytest.raises(FileNotFoundError, match="MARBLE data directory not found"): + _resolve_data_dir() + + +class TestGetMarbleDir: + """Tests for _get_marble_dir function.""" + + def test_returns_path_relative_to_module(self): + """_get_marble_dir should return path relative to module.""" + result = _get_marble_dir() + assert result.name == "marble" + assert "multiagentbench" in str(result.parent) + + +class TestDownloadMarble: + """Tests for download_marble function.""" + + def test_download_marble_already_exists(self): + """download_marble should return existing path if not force.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + marble_dir.mkdir() + + result = download_marble(target_dir=marble_dir, force=False) + + assert result == marble_dir + + def test_download_marble_force_removes_existing(self): + """download_marble should remove existing dir when force=True.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + marble_dir.mkdir() + (marble_dir / "test_file.txt").write_text("test") + + with patch("subprocess.run") as mock_run: + mock_run.return_value = MagicMock(returncode=0) + download_marble(target_dir=marble_dir, force=True) + + # Directory should have been removed and git clone called + mock_run.assert_called() + + def test_download_marble_git_clone_called(self): + """download_marble should call git clone.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + + with patch("subprocess.run") as mock_run: + mock_run.return_value = MagicMock(returncode=0) + download_marble(target_dir=marble_dir) + + # Verify git clone was called + calls = mock_run.call_args_list + assert len(calls) >= 1 + clone_call = calls[0] + assert "git" in clone_call[0][0] + assert "clone" in clone_call[0][0] + + def test_download_marble_with_commit(self): + """download_marble should checkout specific commit if provided.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + + with patch("subprocess.run") as mock_run: + mock_run.return_value = MagicMock(returncode=0) + download_marble(target_dir=marble_dir, commit="abc123") + + # Verify git checkout was called + calls = mock_run.call_args_list + assert len(calls) >= 2 + checkout_call = calls[1] + assert "checkout" in checkout_call[0][0] + assert "abc123" in checkout_call[0][0] + + def test_download_marble_clone_fails(self): + """download_marble should raise RuntimeError on clone failure.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + + with patch("subprocess.run") as mock_run: + mock_run.side_effect = subprocess.CalledProcessError(1, "git clone", stderr="Clone failed") + + with pytest.raises(RuntimeError, match="Failed to clone MARBLE"): + download_marble(target_dir=marble_dir) + + def test_download_marble_git_not_found(self): + """download_marble should raise RuntimeError if git not installed.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + + with patch("subprocess.run") as mock_run: + mock_run.side_effect = FileNotFoundError() + + with pytest.raises(RuntimeError, match="git is not installed"): + download_marble(target_dir=marble_dir) + + def test_download_marble_checkout_fails(self): + """download_marble should raise RuntimeError on checkout failure.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + + def mock_run_side_effect(*args, **kwargs): + cmd = args[0] + if "checkout" in cmd: + raise subprocess.CalledProcessError(1, "git checkout", stderr="Checkout failed") + return MagicMock(returncode=0) + + with patch("subprocess.run", side_effect=mock_run_side_effect): + with pytest.raises(RuntimeError, match="Failed to checkout commit"): + download_marble(target_dir=marble_dir, commit="invalid") + + +class TestEnsureMarbleExists: + """Tests for ensure_marble_exists function.""" + + def test_ensure_marble_exists_already_present(self): + """ensure_marble_exists should return path if MARBLE exists.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + marble_dir.mkdir() + (marble_dir / "multiagentbench").mkdir() + + with patch( + "maseval.benchmark.multiagentbench.data_loader._get_marble_dir", + return_value=marble_dir, + ): + result = ensure_marble_exists(auto_download=False) + + assert result == marble_dir + + def test_ensure_marble_exists_not_present_no_download(self): + """ensure_marble_exists should raise if not present and auto_download=False.""" + with patch( + "maseval.benchmark.multiagentbench.data_loader._get_marble_dir", + return_value=Path("/nonexistent/marble"), + ): + with pytest.raises(FileNotFoundError, match="MARBLE not found"): + ensure_marble_exists(auto_download=False) + + def test_ensure_marble_exists_auto_download(self): + """ensure_marble_exists should download if not present and auto_download=True.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_dir = Path(tmpdir) / "marble" + + with patch( + "maseval.benchmark.multiagentbench.data_loader._get_marble_dir", + return_value=marble_dir, + ): + with patch( + "maseval.benchmark.multiagentbench.data_loader.download_marble", + return_value=marble_dir, + ) as mock_download: + result = ensure_marble_exists(auto_download=True) + + mock_download.assert_called_once_with(marble_dir) + assert result == marble_dir + + +class TestLoadTasksEdgeCases: + """Edge case tests for load_tasks.""" + + def test_load_tasks_empty_lines(self): + """load_tasks should skip empty lines in JSONL.""" + with tempfile.TemporaryDirectory() as tmpdir: + research_dir = Path(tmpdir) / "research" + research_dir.mkdir() + jsonl_path = research_dir / "research_main.jsonl" + + # Write tasks with empty lines + with jsonl_path.open("w") as f: + task_data: Dict[str, Any] = { + "scenario": "research", + "task_id": 1, + "task": {"content": "Task 1"}, + "agents": [{"agent_id": "agent1"}], + "relationships": [], + } + f.write(json.dumps(task_data) + "\n") + f.write("\n") # Empty line + f.write(" \n") # Whitespace-only line + task_data["task_id"] = 2 + task_data["task"]["content"] = "Task 2" # type: ignore[index] + f.write(json.dumps(task_data) + "\n") + + tasks = load_tasks("research", data_dir=Path(tmpdir)) + + assert len(tasks) == 2 + + def test_load_tasks_file_not_found(self): + """load_tasks should raise FileNotFoundError if JSONL file missing.""" + with tempfile.TemporaryDirectory() as tmpdir: + # Create directory but not the JSONL file + research_dir = Path(tmpdir) / "research" + research_dir.mkdir() + + with pytest.raises(FileNotFoundError, match="Task data not found"): + load_tasks("research", data_dir=Path(tmpdir)) + + +class TestGetDomainInfoAllDomains: + """Test get_domain_info for all domains.""" + + @pytest.mark.parametrize( + "domain", + ["coding", "database", "minecraft", "research", "bargaining", "web", "worldsimulation"], + ) + def test_all_domains_have_info(self, domain): + """All valid domains should return info.""" + info = get_domain_info(domain) + assert "requires_infrastructure" in info + assert "description" in info + assert "coordination_mode" in info + + def test_coding_domain_info(self): + """Coding domain should have tree coordination.""" + info = get_domain_info("coding") + assert info["coordination_mode"] == "tree" + assert info["requires_infrastructure"] is False + + def test_web_domain_info(self): + """Web domain should have star coordination.""" + info = get_domain_info("web") + assert info["coordination_mode"] == "star" + assert info["requires_infrastructure"] is False + + def test_worldsimulation_domain_info(self): + """WorldSimulation domain should have cooperative coordination.""" + info = get_domain_info("worldsimulation") + assert info["coordination_mode"] == "cooperative" + assert info["requires_infrastructure"] is False diff --git a/tests/test_benchmarks/test_multiagentbench/test_environment.py b/tests/test_benchmarks/test_multiagentbench/test_environment.py new file mode 100644 index 00000000..8cddce60 --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/test_environment.py @@ -0,0 +1,378 @@ +"""Tests for MultiAgentBench environment.""" + +import pytest +from typing import Any, Dict +from unittest.mock import patch, MagicMock + +from maseval.benchmark.multiagentbench.environment import ( + MultiAgentBenchEnvironment, + INFRASTRUCTURE_DOMAINS, +) +from maseval import EnvironmentError + + +class TestInfrastructureDomains: + """Tests for infrastructure domain constants.""" + + def test_infrastructure_domains_contains_expected(self): + """INFRASTRUCTURE_DOMAINS should contain expected domains.""" + assert "database" in INFRASTRUCTURE_DOMAINS + assert "minecraft" in INFRASTRUCTURE_DOMAINS + + def test_infrastructure_domains_excludes_simple(self): + """INFRASTRUCTURE_DOMAINS should not include simple domains.""" + assert "research" not in INFRASTRUCTURE_DOMAINS + assert "bargaining" not in INFRASTRUCTURE_DOMAINS + + +class TestMultiAgentBenchEnvironment: + """Tests for MultiAgentBenchEnvironment class.""" + + def test_init_with_research_task(self, sample_research_task_data: Dict[str, Any]): + """Environment should initialize for research domain.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + assert env.domain == "research" + assert env.state is not None + + def test_init_with_bargaining_task(self, sample_bargaining_task_data: Dict[str, Any]): + """Environment should initialize for bargaining domain.""" + env = MultiAgentBenchEnvironment(task_data=sample_bargaining_task_data) + + assert env.domain == "bargaining" + + def test_setup_state_extracts_domain(self, sample_research_task_data: Dict[str, Any]): + """setup_state should extract domain from task data.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + state = env.state + + assert state["domain"] == "research" + + def test_setup_state_extracts_max_iterations(self, sample_research_task_data: Dict[str, Any]): + """setup_state should extract max_iterations.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + state = env.state + + assert state["max_iterations"] == 10 + + def test_is_done_initially_false(self, sample_research_task_data: Dict[str, Any]): + """is_done should return False initially.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + # Without MARBLE env, always returns False + assert env.is_done() is False + + def test_is_task_completed_initially_false(self, sample_research_task_data: Dict[str, Any]): + """is_task_completed should return False initially.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + # Without MARBLE env, always returns False + assert env.is_task_completed() is False + + def test_get_marble_state_empty_without_marble(self, sample_research_task_data: Dict[str, Any]): + """get_marble_state should return empty dict without MARBLE.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + assert env.get_marble_state() == {} + + def test_get_tool_descriptions_empty_without_marble(self, sample_research_task_data: Dict[str, Any]): + """get_tool_descriptions should return empty dict without MARBLE.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + assert env.get_tool_descriptions() == {} + + def test_create_tools_empty_without_marble(self, sample_research_task_data: Dict[str, Any]): + """create_tools should return empty dict without MARBLE.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + tools = env.create_tools() + + assert tools == {} + + def test_gather_traces_includes_domain(self, sample_research_task_data: Dict[str, Any]): + """gather_traces should include domain information.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + traces = env.gather_traces() + + assert traces["domain"] == "research" + assert "tool_invocations" in traces + + def test_gather_config_includes_domain(self, sample_research_task_data: Dict[str, Any]): + """gather_config should include domain information.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + config = env.gather_config() + + assert config["domain"] == "research" + assert "tool_descriptions" in config + + +class TestInfrastructureCheck: + """Tests for infrastructure checking.""" + + def test_database_without_docker_raises(self): + """Environment should raise for database without Docker.""" + task_data = { + "scenario": "database", + "environment": {"type": "DB"}, + "task": {"content": "Query database"}, + "agents": [{"agent_id": "agent1"}], + } + + with patch("shutil.which", return_value=None): + with pytest.raises(EnvironmentError, match="requires external infrastructure"): + MultiAgentBenchEnvironment(task_data=task_data) + + def test_database_with_docker_succeeds(self): + """Environment should succeed for database with Docker.""" + task_data = { + "scenario": "database", + "environment": {"type": "DB"}, + "task": {"content": "Query database"}, + "agents": [{"agent_id": "agent1"}], + } + + with patch("shutil.which", return_value="/usr/bin/docker"): + # Should not raise, but MARBLE env creation may still fail + try: + env = MultiAgentBenchEnvironment(task_data=task_data) + assert env.domain == "database" + except ImportError: + # Expected if MARBLE not available + pass + + def test_minecraft_always_raises(self): + """Environment should raise for minecraft (not supported).""" + task_data = { + "scenario": "minecraft", + "environment": {"type": "Minecraft"}, + "task": {"content": "Build something"}, + "agents": [{"agent_id": "agent1"}], + } + + with pytest.raises(EnvironmentError, match="requires external infrastructure"): + MultiAgentBenchEnvironment(task_data=task_data) + + +class TestApplyAction: + """Tests for apply_action method.""" + + def test_apply_action_without_marble_raises(self, sample_research_task_data: Dict[str, Any]): + """apply_action should raise without MARBLE environment.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + with pytest.raises(EnvironmentError, match="not available"): + env.apply_action("agent1", "some_action", {"arg": "value"}) + + +class TestWithMockedMarbleEnv: + """Tests with mocked MARBLE environment.""" + + def test_is_done_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """is_done should delegate to MARBLE env when available.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + # Mock MARBLE env + mock_marble_env = MagicMock() + mock_marble_env.is_done.return_value = True + env._marble_env = mock_marble_env + + assert env.is_done() is True + mock_marble_env.is_done.assert_called_once() + + def test_is_task_completed_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """is_task_completed should delegate to MARBLE env when available.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_marble_env = MagicMock() + mock_marble_env.is_task_completed.return_value = True + env._marble_env = mock_marble_env + + assert env.is_task_completed() is True + mock_marble_env.is_task_completed.assert_called_once() + + def test_get_marble_state_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """get_marble_state should return state from MARBLE env.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_marble_env = MagicMock() + mock_marble_env.get_state.return_value = {"key": "value"} + env._marble_env = mock_marble_env + + assert env.get_marble_state() == {"key": "value"} + mock_marble_env.get_state.assert_called_once() + + def test_get_tool_descriptions_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """get_tool_descriptions should return descriptions from MARBLE env.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_marble_env = MagicMock() + mock_marble_env.action_handler_descriptions = {"tool1": {"desc": "Tool 1"}} + env._marble_env = mock_marble_env + + assert env.get_tool_descriptions() == {"tool1": {"desc": "Tool 1"}} + + def test_apply_action_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """apply_action should delegate to MARBLE env when available.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_marble_env = MagicMock() + mock_marble_env.apply_action.return_value = {"result": "success"} + env._marble_env = mock_marble_env + + result = env.apply_action("agent1", "action1", {"arg": "value"}) + + assert result == {"result": "success"} + mock_marble_env.apply_action.assert_called_once_with("agent1", "action1", {"arg": "value"}) + + def test_gather_traces_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """gather_traces should include MARBLE state when available.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_marble_env = MagicMock() + mock_marble_env.get_state.return_value = {"state_key": "state_value"} + mock_marble_env.is_done.return_value = False + mock_marble_env.is_task_completed.return_value = False + env._marble_env = mock_marble_env + + traces = env.gather_traces() + + assert traces["marble_state"] == {"state_key": "state_value"} + assert traces["is_done"] is False + assert traces["is_task_completed"] is False + assert traces["marble_env_type"] == "MagicMock" + + def test_create_tools_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """create_tools should wrap MARBLE action handlers.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + # Create mock MARBLE env with action handlers + mock_handler = MagicMock(return_value="handler_result") + mock_marble_env = MagicMock() + mock_marble_env._action_handlers = {"test_action": mock_handler} + env._marble_env = mock_marble_env + + tools = env.create_tools() + + assert "test_action" in tools + # Test the wrapped handler + result = tools["test_action"](arg1="value1") + assert result == "handler_result" + mock_handler.assert_called_once_with(arg1="value1") + + def test_create_tools_traces_invocations(self, sample_research_task_data: Dict[str, Any]): + """create_tools should trace tool invocations.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_handler = MagicMock(return_value="result") + mock_marble_env = MagicMock() + mock_marble_env._action_handlers = {"test_action": mock_handler} + env._marble_env = mock_marble_env + + tools = env.create_tools() + tools["test_action"](input="test") + + # Check that invocation was traced + assert "test_action" in env._tool_histories + history = env._tool_histories["test_action"] + invocations = history.to_list() + assert len(invocations) == 1 + + def test_create_tools_traces_errors(self, sample_research_task_data: Dict[str, Any]): + """create_tools should trace tool errors.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_handler = MagicMock(side_effect=RuntimeError("Handler error")) + mock_marble_env = MagicMock() + mock_marble_env._action_handlers = {"test_action": mock_handler} + env._marble_env = mock_marble_env + + tools = env.create_tools() + + with pytest.raises(RuntimeError): + tools["test_action"](input="test") + + # Check that error was traced + history = env._tool_histories["test_action"] + invocations = history.to_list() + assert len(invocations) == 1 + assert invocations[0]["status"] == "error" + + +class TestGetTool: + """Tests for get_tool method.""" + + def test_get_tool_returns_none_if_not_found(self, sample_research_task_data: Dict[str, Any]): + """get_tool should return None for unknown tool.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + assert env.get_tool("unknown_tool") is None + + def test_get_tool_returns_tool_if_found(self, sample_research_task_data: Dict[str, Any]): + """get_tool should return tool if it exists.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_handler = MagicMock() + mock_marble_env = MagicMock() + mock_marble_env._action_handlers = {"test_tool": mock_handler} + env._marble_env = mock_marble_env + + # Create tools and update env.tools (which is normally set during __init__) + env.tools = env.create_tools() + + tool = env.get_tool("test_tool") + assert tool is not None + + +class TestGatherConfig: + """Tests for gather_config method.""" + + def test_gather_config_with_marble_env(self, sample_research_task_data: Dict[str, Any]): + """gather_config should include tool descriptions.""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + mock_marble_env = MagicMock() + mock_marble_env.action_handler_descriptions = {"tool1": {"name": "Tool 1"}} + env._marble_env = mock_marble_env + + config = env.gather_config() + + assert config["tool_descriptions"] == {"tool1": {"name": "Tool 1"}} + assert config["marble_env_type"] == "MagicMock" + + +class TestSetupStateEdgeCases: + """Tests for setup_state edge cases.""" + + def test_setup_state_with_string_task(self): + """setup_state should handle task as string.""" + task_data = { + "scenario": "research", + "environment": {}, + "task": "String task content", + "agents": [{"agent_id": "agent1"}], + } + + env = MultiAgentBenchEnvironment(task_data=task_data) + assert env.state is not None + + def test_setup_state_default_max_iterations(self): + """setup_state should default max_iterations to 10.""" + task_data = { + "scenario": "research", + "environment": {}, + "task": {"content": "Task"}, + "agents": [{"agent_id": "agent1"}], + } + + env = MultiAgentBenchEnvironment(task_data=task_data) + assert env.state["max_iterations"] == 10 + + def test_setup_state_uses_env_max_iterations(self): + """setup_state should use environment max_iterations if provided.""" + task_data = { + "scenario": "research", + "environment": {"max_iterations": 25}, + "task": {"content": "Task"}, + "agents": [{"agent_id": "agent1"}], + } + + env = MultiAgentBenchEnvironment(task_data=task_data) + assert env.state["max_iterations"] == 25 diff --git a/tests/test_benchmarks/test_multiagentbench/test_evaluator.py b/tests/test_benchmarks/test_multiagentbench/test_evaluator.py new file mode 100644 index 00000000..a6327713 --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/test_evaluator.py @@ -0,0 +1,855 @@ +"""Tests for MultiAgentBench evaluator.""" + +import pytest + +from conftest import DummyModelAdapter +from maseval.benchmark.multiagentbench.evaluator import ( + MultiAgentBenchEvaluator, + MultiAgentBenchMetrics, +) + + +class TestMultiAgentBenchMetrics: + """Tests for MultiAgentBenchMetrics dataclass.""" + + def test_default_values(self): + """Metrics should have sensible defaults.""" + metrics = MultiAgentBenchMetrics() + + assert metrics.task_completion is False + assert metrics.token_consumption == 0 + assert metrics.communication_score is None + assert metrics.task_evaluation == {} + assert metrics.agent_kpis == {} + assert metrics.total_milestones == 0 + + def test_to_dict(self): + """to_dict should serialize all fields.""" + metrics = MultiAgentBenchMetrics( + task_completion=True, + token_consumption=1000, + communication_score=5.0, + ) + d = metrics.to_dict() + + assert d["task_completion"] is True + assert d["token_consumption"] == 1000 + assert d["communication_score"] == 5.0 + + +class TestMultiAgentBenchEvaluator: + """Tests for MultiAgentBenchEvaluator class.""" + + @pytest.fixture + def mock_model_adapter(self): + """Create a mock model adapter.""" + return DummyModelAdapter( + model_id="test-model", + responses=['{"rating": 4}'], + ) + + @pytest.fixture + def research_evaluator(self, mock_model_adapter): + """Create a research domain evaluator.""" + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=mock_model_adapter, + ) + + @pytest.fixture + def bargaining_evaluator(self, mock_model_adapter): + """Create a bargaining domain evaluator.""" + return MultiAgentBenchEvaluator( + domain="bargaining", + model_adapter=mock_model_adapter, + ) + + def test_init_normalizes_domain(self, mock_model_adapter): + """Evaluator should normalize domain to lowercase.""" + evaluator = MultiAgentBenchEvaluator( + domain="RESEARCH", + model_adapter=mock_model_adapter, + ) + assert evaluator.domain == "research" + + def test_filter_traces_extracts_agents(self, research_evaluator): + """filter_traces should extract agent traces.""" + traces = { + "agents": {"agent1": {"action_log": [{"task": "test", "result": "done"}]}}, + "environment": {}, + } + filtered = research_evaluator.filter_traces(traces) + + assert "agents" in filtered + assert "agent1" in filtered["agents"] + + def test_extract_communications_formats_correctly(self, research_evaluator): + """_extract_communications should format communication logs.""" + traces = { + "agents": { + "agent1": {"communication_log": [{"communication": "Hello from agent1"}]}, + "agent2": {"communication_log": [{"communication": "Hello from agent2"}]}, + } + } + comms = research_evaluator._extract_communications(traces) + + assert "[agent1]: Hello from agent1" in comms + assert "[agent2]: Hello from agent2" in comms + + def test_extract_communications_empty(self, research_evaluator): + """_extract_communications should handle empty logs.""" + traces = {"agents": {}} + comms = research_evaluator._extract_communications(traces) + + assert comms == "No communications recorded." + + def test_extract_results_formats_correctly(self, research_evaluator): + """_extract_results should format action logs.""" + traces = {"agents": {"agent1": {"action_log": [{"result": "Completed task A"}]}}} + results = research_evaluator._extract_results(traces) + + assert "[agent1]: Completed task A" in results + + def test_calculate_token_consumption(self, research_evaluator): + """_calculate_token_consumption should sum agent token usage.""" + traces = { + "agents": { + "agent1": {"token_usage": 500}, + "agent2": {"token_usage": 300}, + } + } + total = research_evaluator._calculate_token_consumption(traces) + + assert total == 800 + + def test_parse_score_valid_json(self, research_evaluator): + """_parse_score should parse valid JSON response.""" + response = '{"rating": 4}' + score = research_evaluator._parse_score(response) + + assert score == 4.0 + + def test_parse_score_with_markdown(self, research_evaluator): + """_parse_score should handle markdown code blocks.""" + response = '```json\n{"rating": 5}\n```' + score = research_evaluator._parse_score(response) + + assert score == 5.0 + + def test_parse_score_no_json_returns_none(self, research_evaluator): + """_parse_score should return None when no valid JSON found.""" + response = "The rating is 4 out of 5" + score = research_evaluator._parse_score(response) + + # No regex fallback - returns None for non-JSON responses + assert score is None + + def test_parse_score_default(self, research_evaluator): + """_parse_score should return None when parsing fails.""" + response = "No score here" + score = research_evaluator._parse_score(response) + + assert score is None + + def test_parse_research_ratings_valid(self, research_evaluator): + """_parse_research_ratings should parse valid ratings.""" + response = '{"innovation": 4, "safety": 3, "feasibility": 5}' + ratings = research_evaluator._parse_research_ratings(response) + + assert ratings["innovation"] == 4 + assert ratings["safety"] == 3 + assert ratings["feasibility"] == 5 + + def test_parse_research_ratings_invalid(self, research_evaluator): + """_parse_research_ratings should return None for invalid.""" + response = "Invalid response" + ratings = research_evaluator._parse_research_ratings(response) + + assert ratings["innovation"] is None + assert ratings["safety"] is None + assert ratings["feasibility"] is None + + def test_determine_completion_research_positive(self, research_evaluator): + """_determine_completion should return True for positive research scores.""" + metrics = MultiAgentBenchMetrics(task_evaluation={"innovation": 4, "safety": 3, "feasibility": 5}) + assert research_evaluator._determine_completion(metrics) is True + + def test_determine_completion_research_negative(self, research_evaluator): + """_determine_completion should return False for None scores.""" + metrics = MultiAgentBenchMetrics(task_evaluation={"innovation": None, "safety": 3, "feasibility": 5}) + assert research_evaluator._determine_completion(metrics) is False + + def test_call_returns_expected_structure(self, research_evaluator): + """__call__ should return expected result structure.""" + traces = { + "agents": {"agent1": {"token_usage": 100, "action_log": [], "communication_log": []}}, + "environment": {}, + } + final_answer = [{"agent_id": "agent1", "result": "Done"}] + + # Mock model adapter to return research ratings + research_evaluator.model_adapter = DummyModelAdapter( + model_id="test", + responses=['{"innovation": 4, "safety": 4, "feasibility": 4}'], + ) + + result = research_evaluator(traces, final_answer) + + assert "passed" in result + assert "metrics" in result + assert "domain" in result + assert result["domain"] == "research" + + def test_format_final_answer_dict(self, research_evaluator): + """_format_final_answer should format dict input.""" + final_answer = { + "agent_results": [ + {"agent_id": "agent1", "result": "Result 1"}, + {"agent_id": "agent2", "result": "Result 2"}, + ] + } + formatted = research_evaluator._format_final_answer(final_answer) + + assert "[agent1]: Result 1" in formatted + assert "[agent2]: Result 2" in formatted + + def test_format_final_answer_list(self, research_evaluator): + """_format_final_answer should format list input.""" + final_answer = [ + {"agent_id": "agent1", "result": "Result 1"}, + ] + formatted = research_evaluator._format_final_answer(final_answer) + + assert "[agent1]: Result 1" in formatted + + def test_format_final_answer_string(self, research_evaluator): + """_format_final_answer should pass through string.""" + formatted = research_evaluator._format_final_answer("Simple result") + assert formatted == "Simple result" + + def test_format_final_answer_none(self, research_evaluator): + """_format_final_answer should handle None.""" + formatted = research_evaluator._format_final_answer(None) + assert formatted == "" + + +class TestBargainingEvaluation: + """Tests for bargaining-specific evaluation.""" + + @pytest.fixture + def bargaining_evaluator(self): + """Create evaluator with bargaining-specific responses.""" + adapter = DummyModelAdapter( + model_id="test", + responses=[ + '{"effectiveness_of_strategies": 4, "progress_and_outcome": 3, "interaction_dynamics": 5}', + '{"effectiveness_of_strategies": 3, "progress_and_outcome": 4, "interaction_dynamics": 4}', + ], + ) + return MultiAgentBenchEvaluator( + domain="bargaining", + model_adapter=adapter, + ) + + def test_evaluate_bargaining_returns_buyer_seller(self, bargaining_evaluator): + """_evaluate_bargaining should return buyer and seller ratings.""" + ratings = bargaining_evaluator._evaluate_bargaining("Task", "Result") + + assert "buyer" in ratings + assert "seller" in ratings + assert "effectiveness_of_strategies" in ratings["buyer"] + + def test_determine_completion_bargaining_positive(self, bargaining_evaluator): + """_determine_completion should work for bargaining domain.""" + metrics = MultiAgentBenchMetrics( + task_evaluation={ + "buyer": { + "effectiveness_of_strategies": 4, + "progress_and_outcome": 3, + "interaction_dynamics": 5, + }, + "seller": { + "effectiveness_of_strategies": 3, + "progress_and_outcome": 4, + "interaction_dynamics": 4, + }, + } + ) + assert bargaining_evaluator._determine_completion(metrics) is True + + +class TestCodingEvaluation: + """Tests for coding-specific evaluation.""" + + @pytest.fixture + def coding_evaluator(self): + """Create evaluator with coding-specific responses.""" + adapter = DummyModelAdapter( + model_id="test", + responses=[ + '{"instruction_following": 5, "executability": 4, "consistency": 4, "quality": 3}', + ], + ) + return MultiAgentBenchEvaluator( + domain="coding", + model_adapter=adapter, + ) + + def test_evaluate_coding_returns_all_metrics(self, coding_evaluator): + """_evaluate_coding should return all coding metrics.""" + ratings = coding_evaluator._evaluate_coding("Task", "Solution") + + assert "instruction_following" in ratings + assert "executability" in ratings + assert "consistency" in ratings + assert "quality" in ratings + + def test_determine_completion_coding_positive(self, coding_evaluator): + """_determine_completion should work for coding domain.""" + metrics = MultiAgentBenchMetrics( + task_evaluation={ + "instruction_following": 5, + "executability": 4, + "consistency": 4, + "quality": 3, + } + ) + assert coding_evaluator._determine_completion(metrics) is True + + def test_determine_completion_coding_negative(self, coding_evaluator): + """_determine_completion should return False for None coding scores.""" + metrics = MultiAgentBenchMetrics( + task_evaluation={ + "instruction_following": 5, + "executability": None, + "consistency": 4, + "quality": 3, + } + ) + assert coding_evaluator._determine_completion(metrics) is False + + def test_parse_coding_ratings_invalid(self, coding_evaluator): + """_parse_coding_ratings should return None for invalid response.""" + ratings = coding_evaluator._parse_coding_ratings("Invalid JSON") + assert ratings["instruction_following"] is None + assert ratings["executability"] is None + assert ratings["consistency"] is None + assert ratings["quality"] is None + + +class TestDatabaseEvaluation: + """Tests for database-specific evaluation.""" + + @pytest.fixture + def database_evaluator(self): + """Create evaluator for database domain.""" + adapter = DummyModelAdapter( + model_id="test", + responses=['{"rating": 4}'], + ) + return MultiAgentBenchEvaluator( + domain="database", + model_adapter=adapter, + ) + + def test_evaluate_database_returns_structure(self, database_evaluator): + """_evaluate_database should return predicted result.""" + result = database_evaluator._evaluate_database("Query task", "SELECT * FROM users") + + assert "predicted" in result + assert result["predicted"] == "SELECT * FROM users" + assert "root_cause" in result + assert result["root_cause"] == [] + + def test_determine_completion_database_with_prediction(self, database_evaluator): + """_determine_completion should return True for database with prediction.""" + metrics = MultiAgentBenchMetrics(task_evaluation={"predicted": "SELECT * FROM users", "root_cause": []}) + assert database_evaluator._determine_completion(metrics) is True + + def test_determine_completion_database_empty_prediction(self, database_evaluator): + """_determine_completion should return False for empty prediction.""" + metrics = MultiAgentBenchMetrics(task_evaluation={"predicted": "", "root_cause": []}) + assert database_evaluator._determine_completion(metrics) is False + + def test_call_database_domain(self, database_evaluator): + """__call__ should work for database domain.""" + traces = { + "agents": {"agent1": {"token_usage": 100, "action_log": [], "communication_log": []}}, + "environment": {"marble_state": {"task_description": "Find query issue"}}, + } + final_answer = [{"agent_id": "agent1", "result": "SELECT * FROM orders"}] + + result = database_evaluator(traces, final_answer) + + assert result["domain"] == "database" + assert "passed" in result + + +class TestWorldSimulationEvaluation: + """Tests for worldsimulation domain (alias for bargaining).""" + + @pytest.fixture + def worldsim_evaluator(self): + """Create evaluator for worldsimulation domain.""" + adapter = DummyModelAdapter( + model_id="test", + responses=[ + '{"effectiveness_of_strategies": 4, "progress_and_outcome": 4, "interaction_dynamics": 4}', + '{"effectiveness_of_strategies": 4, "progress_and_outcome": 4, "interaction_dynamics": 4}', + ], + ) + return MultiAgentBenchEvaluator( + domain="worldsimulation", + model_adapter=adapter, + ) + + def test_worldsimulation_uses_bargaining_eval(self, worldsim_evaluator): + """worldsimulation domain should use bargaining evaluation.""" + traces = { + "agents": {"agent1": {"token_usage": 100, "action_log": [], "communication_log": []}}, + "environment": {}, + } + final_answer = "Simulation result" + + result = worldsim_evaluator(traces, final_answer) + + assert result["domain"] == "worldsimulation" + assert "buyer" in result["metrics"]["task_evaluation"] + assert "seller" in result["metrics"]["task_evaluation"] + + def test_determine_completion_worldsimulation(self, worldsim_evaluator): + """_determine_completion should work for worldsimulation domain.""" + metrics = MultiAgentBenchMetrics( + task_evaluation={ + "buyer": { + "effectiveness_of_strategies": 4, + "progress_and_outcome": 4, + "interaction_dynamics": 4, + }, + "seller": { + "effectiveness_of_strategies": 4, + "progress_and_outcome": 4, + "interaction_dynamics": 4, + }, + } + ) + assert worldsim_evaluator._determine_completion(metrics) is True + + +class TestUnknownDomainEvaluation: + """Tests for unknown domain handling.""" + + @pytest.fixture + def unknown_evaluator(self): + """Create evaluator for unknown domain.""" + adapter = DummyModelAdapter( + model_id="test", + responses=['{"rating": 4}'], + ) + return MultiAgentBenchEvaluator( + domain="unknown_domain", + model_adapter=adapter, + ) + + def test_unknown_domain_sets_completion_from_result(self, unknown_evaluator): + """Unknown domain should set completion based on final result presence.""" + traces = { + "agents": {}, + "environment": {}, + } + final_answer = "Some result" + + result = unknown_evaluator(traces, final_answer) + + assert result["domain"] == "unknown_domain" + # Should still have metrics + assert "metrics" in result + + def test_determine_completion_unknown_domain(self, unknown_evaluator): + """_determine_completion should return False for unknown domain.""" + metrics = MultiAgentBenchMetrics(task_evaluation={}) + assert unknown_evaluator._determine_completion(metrics) is False + + +class TestCommunicationEvaluation: + """Tests for communication evaluation.""" + + @pytest.fixture + def evaluator_with_comm_response(self): + """Create evaluator with communication rating response.""" + adapter = DummyModelAdapter( + model_id="test", + responses=['{"rating": 4}', '{"innovation": 4, "safety": 4, "feasibility": 4}'], + ) + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + + def test_evaluate_communication_returns_score(self, evaluator_with_comm_response): + """_evaluate_communication should return score from LLM.""" + score = evaluator_with_comm_response._evaluate_communication("Research task", "[agent1]: Hello\n[agent2]: Hi") + assert score == 4.0 + + def test_evaluate_communication_on_error(self): + """_evaluate_communication should return None on error.""" + from unittest.mock import MagicMock + + # Create mock that raises exception on generate + mock_adapter = MagicMock() + mock_adapter.generate.side_effect = RuntimeError("Model failed") + + evaluator = MultiAgentBenchEvaluator( + domain="research", + model_adapter=mock_adapter, + ) + + score = evaluator._evaluate_communication("Task", "Comms") + assert score is None + + def test_call_with_communications(self, evaluator_with_comm_response): + """__call__ should evaluate communications when present.""" + traces = { + "agents": { + "agent1": { + "token_usage": 100, + "action_log": [], + "communication_log": [{"communication": "Hello from agent1"}], + } + }, + "environment": {}, + } + final_answer = "Result" + + result = evaluator_with_comm_response(traces, final_answer) + + # Communication score should be evaluated (not -1) + assert result["metrics"]["communication_score"] == 4.0 + + +class TestExceptionHandling: + """Tests for exception handling in evaluation methods.""" + + @pytest.fixture + def failing_evaluator(self): + """Create evaluator with adapter that fails.""" + from unittest.mock import MagicMock + + # Create mock that raises exception on generate + mock_adapter = MagicMock() + mock_adapter.generate.side_effect = RuntimeError("Model failed") + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=mock_adapter, + ) + + def test_evaluate_research_on_error(self, failing_evaluator): + """_evaluate_research should return default values on error.""" + result = failing_evaluator._evaluate_research("Task", "Result") + assert result["innovation"] is None + assert result["safety"] is None + assert result["feasibility"] is None + + def test_evaluate_bargaining_on_error(self, failing_evaluator): + """_evaluate_bargaining should return default values on error.""" + failing_evaluator.domain = "bargaining" + result = failing_evaluator._evaluate_bargaining("Task", "Result") + assert result["buyer"]["effectiveness_of_strategies"] is None + assert result["seller"]["effectiveness_of_strategies"] is None + + def test_evaluate_coding_on_error(self, failing_evaluator): + """_evaluate_coding should return default values on error.""" + failing_evaluator.domain = "coding" + result = failing_evaluator._evaluate_coding("Task", "Result") + assert result["instruction_following"] is None + assert result["executability"] is None + + +class TestParsingEdgeCases: + """Tests for parsing edge cases.""" + + @pytest.fixture + def evaluator(self): + """Create evaluator for parsing tests.""" + adapter = DummyModelAdapter(model_id="test", responses=['{"rating": 4}']) + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + + def test_parse_score_out_of_range(self, evaluator): + """_parse_score should reject scores outside 1-5 range.""" + response = '{"rating": 10}' + score = evaluator._parse_score(response) + # Should return None since score is out of range + assert score is None + + def test_parse_score_with_just_code_block(self, evaluator): + """_parse_score should handle code block without json marker.""" + response = '```\n{"rating": 3}\n```' + score = evaluator._parse_score(response) + assert score == 3.0 # Valid score parsed from JSON + + def test_parse_score_with_text_before_json(self, evaluator): + """_parse_score should find JSON in text.""" + response = 'Here is my rating: {"rating": 5}' + score = evaluator._parse_score(response) + assert score == 5.0 + + def test_parse_bargaining_ratings_partial(self, evaluator): + """_parse_bargaining_ratings should handle partial ratings.""" + response = '{"effectiveness_of_strategies": 4}' # Missing other fields + ratings = evaluator._parse_bargaining_ratings(response) + assert ratings["effectiveness_of_strategies"] == 4 + assert ratings["progress_and_outcome"] is None + + def test_parse_bargaining_ratings_invalid(self, evaluator): + """_parse_bargaining_ratings should return None for invalid.""" + ratings = evaluator._parse_bargaining_ratings("Invalid") + assert ratings["effectiveness_of_strategies"] is None + + +class TestFormatFinalAnswerEdgeCases: + """Tests for _format_final_answer edge cases.""" + + @pytest.fixture + def evaluator(self): + """Create evaluator for format tests.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + + def test_format_dict_without_agent_results(self, evaluator): + """_format_final_answer should JSON dump dict without agent_results.""" + final_answer = {"status": "completed", "data": "some data"} + formatted = evaluator._format_final_answer(final_answer) + assert "status" in formatted + assert "completed" in formatted + + def test_format_dict_with_empty_agent_results(self, evaluator): + """_format_final_answer should handle empty agent_results.""" + final_answer = {"agent_results": []} + formatted = evaluator._format_final_answer(final_answer) + # Empty results should JSON dump the dict + assert formatted == '{"agent_results": []}' + + def test_format_list_with_non_dict_items(self, evaluator): + """_format_final_answer should skip non-dict items in list.""" + final_answer = [ + {"agent_id": "agent1", "result": "Result 1"}, + "not a dict", + {"agent_id": "agent2", "result": "Result 2"}, + ] + formatted = evaluator._format_final_answer(final_answer) + assert "[agent1]: Result 1" in formatted + assert "[agent2]: Result 2" in formatted + assert "not a dict" not in formatted + + +class TestTokenConsumptionEdgeCases: + """Tests for token consumption calculation edge cases.""" + + @pytest.fixture + def evaluator(self): + """Create evaluator for token tests.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + + def test_token_consumption_with_non_int(self, evaluator): + """_calculate_token_consumption should skip non-int values.""" + traces = { + "agents": { + "agent1": {"token_usage": 500}, + "agent2": {"token_usage": "invalid"}, # Non-int + "agent3": {"token_usage": 300}, + } + } + total = evaluator._calculate_token_consumption(traces) + assert total == 800 # Only int values counted + + def test_token_consumption_empty_agents(self, evaluator): + """_calculate_token_consumption should return 0 for no agents.""" + traces = {"agents": {}} + total = evaluator._calculate_token_consumption(traces) + assert total == 0 + + def test_token_consumption_missing_key(self, evaluator): + """_calculate_token_consumption should handle missing token_usage.""" + traces = { + "agents": { + "agent1": {"action_log": []}, # No token_usage + } + } + total = evaluator._calculate_token_consumption(traces) + assert total == 0 + + +class TestGetTaskDescription: + """Tests for _get_task_description method.""" + + @pytest.fixture + def evaluator(self): + """Create evaluator for task description tests.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + + def test_get_task_description_from_traces(self, evaluator): + """_get_task_description should extract from traces.""" + traces = {"environment": {"marble_state": {"task_description": "Research AI safety"}}} + desc = evaluator._get_task_description(traces) + assert desc == "Research AI safety" + + def test_get_task_description_missing_env(self, evaluator): + """_get_task_description should return empty for missing env.""" + traces = {} + desc = evaluator._get_task_description(traces) + assert desc == "" + + def test_get_task_description_missing_state(self, evaluator): + """_get_task_description should return empty for missing marble_state.""" + traces = {"environment": {}} + desc = evaluator._get_task_description(traces) + assert desc == "" + + +class TestExtractResultsEdgeCases: + """Tests for _extract_results edge cases.""" + + @pytest.fixture + def evaluator(self): + """Create evaluator for extract results tests.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + + def test_extract_results_empty_result(self, evaluator): + """_extract_results should skip empty results.""" + traces = { + "agents": { + "agent1": {"action_log": [{"result": ""}]}, + } + } + results = evaluator._extract_results(traces) + assert results == "No results recorded." + + def test_extract_results_multiple_actions(self, evaluator): + """_extract_results should include all actions.""" + traces = { + "agents": { + "agent1": { + "action_log": [ + {"result": "Step 1 done"}, + {"result": "Step 2 done"}, + ] + } + } + } + results = evaluator._extract_results(traces) + assert "Step 1 done" in results + assert "Step 2 done" in results + + +class TestFilterTracesEdgeCases: + """Tests for filter_traces edge cases.""" + + @pytest.fixture + def evaluator(self): + """Create evaluator for filter traces tests.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + + def test_filter_traces_empty(self, evaluator): + """filter_traces should handle empty traces.""" + filtered = evaluator.filter_traces({}) + assert filtered["agents"] == {} + assert filtered["environment"] == {} + assert "communications" in filtered + + def test_filter_traces_extracts_all(self, evaluator): + """filter_traces should extract all relevant data.""" + traces = { + "agents": { + "agent1": { + "action_log": [{"result": "Done"}], + "communication_log": [{"communication": "Hello"}], + } + }, + "environment": {"domain": "research"}, + } + filtered = evaluator.filter_traces(traces) + assert "[agent1]: Hello" in filtered["communications"] + assert filtered["agents"] == traces["agents"] + assert filtered["environment"] == traces["environment"] + + +class TestDetermineCompletionEdgeCases: + """Tests for _determine_completion edge cases.""" + + def test_completion_empty_eval_data(self): + """_determine_completion should return False for empty eval data.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + evaluator = MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + metrics = MultiAgentBenchMetrics(task_evaluation={}) + assert evaluator._determine_completion(metrics) is False + + def test_completion_bargaining_partial_buyer(self): + """_determine_completion should return False if buyer has None scores.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + evaluator = MultiAgentBenchEvaluator( + domain="bargaining", + model_adapter=adapter, + ) + metrics = MultiAgentBenchMetrics( + task_evaluation={ + "buyer": { + "effectiveness_of_strategies": None, + "progress_and_outcome": 4, + "interaction_dynamics": 4, + }, + "seller": { + "effectiveness_of_strategies": 4, + "progress_and_outcome": 4, + "interaction_dynamics": 4, + }, + } + ) + assert evaluator._determine_completion(metrics) is False + + def test_completion_bargaining_partial_seller(self): + """_determine_completion should return False if seller has None scores.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + evaluator = MultiAgentBenchEvaluator( + domain="bargaining", + model_adapter=adapter, + ) + metrics = MultiAgentBenchMetrics( + task_evaluation={ + "buyer": { + "effectiveness_of_strategies": 4, + "progress_and_outcome": 4, + "interaction_dynamics": 4, + }, + "seller": { + "effectiveness_of_strategies": None, + "progress_and_outcome": 4, + "interaction_dynamics": 4, + }, + } + ) + assert evaluator._determine_completion(metrics) is False diff --git a/tests/test_benchmarks/test_multiagentbench/test_marble_adapter.py b/tests/test_benchmarks/test_multiagentbench/test_marble_adapter.py new file mode 100644 index 00000000..93ef2535 --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/test_marble_adapter.py @@ -0,0 +1,207 @@ +"""Tests for MarbleAgentAdapter.""" + +import pytest +from unittest.mock import MagicMock + +from maseval.benchmark.multiagentbench.adapters.marble_adapter import ( + MarbleAgentAdapter, +) +from maseval import AgentError + + +class TestMarbleAgentAdapter: + """Tests for MarbleAgentAdapter class.""" + + @pytest.fixture + def mock_marble_agent(self): + """Create a mock MARBLE agent.""" + agent = MagicMock() + agent.profile = "Test researcher profile" + agent.strategy = "cot" + agent.llm = "gpt-4o" + agent.relationships = {"agent2": "collaborator"} + agent.task_history = ["task1", "task2"] + agent.session_id = "session_123" + agent.act.return_value = ("Test result", "Communication message") + agent.get_token_usage.return_value = 500 + agent.seralize_message.return_value = "Serialized messages" + return agent + + @pytest.fixture + def adapter(self, mock_marble_agent): + """Create a MarbleAgentAdapter.""" + return MarbleAgentAdapter( + marble_agent=mock_marble_agent, + agent_id="test_agent", + ) + + def test_init(self, adapter, mock_marble_agent): + """Adapter should initialize correctly.""" + assert adapter.agent_id == "test_agent" + assert adapter.profile == "Test researcher profile" + assert adapter.marble_agent is mock_marble_agent + + def test_agent_id_property(self, adapter): + """agent_id property should return correct value.""" + assert adapter.agent_id == "test_agent" + + def test_profile_property(self, adapter): + """profile property should return correct value.""" + assert adapter.profile == "Test researcher profile" + + def test_marble_agent_property(self, adapter, mock_marble_agent): + """marble_agent property should return the underlying agent.""" + assert adapter.marble_agent is mock_marble_agent + + def test_run_agent_success(self, adapter, mock_marble_agent): + """_run_agent should execute MARBLE agent's act method.""" + result = adapter._run_agent("Test query") + + mock_marble_agent.act.assert_called_once_with("Test query") + assert result == "Test result" + + def test_run_agent_logs_action(self, adapter, mock_marble_agent): + """_run_agent should log actions.""" + adapter._run_agent("Test query") + + assert len(adapter._action_log) == 1 + assert adapter._action_log[0]["task"] == "Test query" + assert adapter._action_log[0]["result"] == "Test result" + assert adapter._action_log[0]["has_communication"] is True + + def test_run_agent_logs_communication(self, adapter, mock_marble_agent): + """_run_agent should log communications when present.""" + adapter._run_agent("Test query") + + assert len(adapter._communication_log) == 1 + assert adapter._communication_log[0]["communication"] == "Communication message" + + def test_run_agent_no_communication(self, adapter, mock_marble_agent): + """_run_agent should handle None communication.""" + mock_marble_agent.act.return_value = ("Result", None) + adapter._run_agent("Test query") + + assert len(adapter._communication_log) == 0 + assert adapter._action_log[0]["has_communication"] is False + + def test_run_agent_updates_messages(self, adapter, mock_marble_agent): + """_run_agent should update message history.""" + adapter._run_agent("Test query") + + messages = adapter.get_messages() + assert len(messages) == 2 + assert messages[0]["role"] == "user" + assert messages[0]["content"] == "Test query" + assert messages[1]["role"] == "assistant" + assert messages[1]["content"] == "Test result" + + def test_run_agent_error(self, adapter, mock_marble_agent): + """_run_agent should wrap exceptions in AgentError.""" + mock_marble_agent.act.side_effect = RuntimeError("Agent failed") + + with pytest.raises(AgentError, match="MARBLE agent 'test_agent' failed"): + adapter._run_agent("Test query") + + def test_get_token_usage(self, adapter, mock_marble_agent): + """get_token_usage should return token count from MARBLE agent.""" + assert adapter.get_token_usage() == 500 + mock_marble_agent.get_token_usage.assert_called_once() + + def test_get_token_usage_no_method(self, adapter, mock_marble_agent): + """get_token_usage should return 0 if method not available.""" + del mock_marble_agent.get_token_usage + assert adapter.get_token_usage() == 0 + + def test_get_memory_str(self, adapter, mock_marble_agent): + """get_memory_str should return memory string.""" + mock_memory = MagicMock() + mock_memory.get_memory_str.return_value = "Memory contents" + mock_marble_agent.memory = mock_memory + + assert adapter.get_memory_str() == "Memory contents" + + def test_get_memory_str_no_memory(self, adapter, mock_marble_agent): + """get_memory_str should return empty string if no memory.""" + mock_marble_agent.memory = None + assert adapter.get_memory_str() == "" + + def test_get_memory_str_no_method(self, adapter, mock_marble_agent): + """get_memory_str should return empty string if no get_memory_str method.""" + mock_marble_agent.memory = MagicMock(spec=[]) # No methods + assert adapter.get_memory_str() == "" + + def test_get_serialized_messages(self, adapter, mock_marble_agent): + """get_serialized_messages should return serialized messages.""" + result = adapter.get_serialized_messages("session_1") + + mock_marble_agent.seralize_message.assert_called_once_with("session_1") + assert result == "Serialized messages" + + def test_get_serialized_messages_no_method(self, adapter, mock_marble_agent): + """get_serialized_messages should return empty string if no method.""" + del mock_marble_agent.seralize_message + assert adapter.get_serialized_messages() == "" + + def test_gather_traces(self, adapter, mock_marble_agent): + """gather_traces should include MARBLE-specific data.""" + # Run agent to generate some data + adapter._run_agent("Test query") + + traces = adapter.gather_traces() + + assert traces["agent_id"] == "test_agent" + assert traces["profile"] == "Test researcher profile" + assert traces["token_usage"] == 500 + assert len(traces["action_log"]) == 1 + assert len(traces["communication_log"]) == 1 + assert traces["relationships"] == {"agent2": "collaborator"} + assert traces["task_history"] == ["task1", "task2"] + + def test_gather_traces_no_relationships(self, adapter, mock_marble_agent): + """gather_traces should handle missing relationships.""" + del mock_marble_agent.relationships + traces = adapter.gather_traces() + assert traces["relationships"] == {} + + def test_gather_traces_no_task_history(self, adapter, mock_marble_agent): + """gather_traces should handle missing task_history.""" + del mock_marble_agent.task_history + traces = adapter.gather_traces() + assert traces["task_history"] == [] + + def test_gather_config(self, adapter, mock_marble_agent): + """gather_config should include MARBLE configuration.""" + config = adapter.gather_config() + + assert config["agent_id"] == "test_agent" + assert config["profile"] == "Test researcher profile" + assert config["strategy"] == "cot" + assert config["llm"] == "gpt-4o" + + def test_gather_config_missing_attrs(self, adapter, mock_marble_agent): + """gather_config should handle missing attributes.""" + del mock_marble_agent.strategy + del mock_marble_agent.llm + + config = adapter.gather_config() + + assert config["strategy"] == "default" + assert config["llm"] == "unknown" + + +class TestCreateMarbleAgentsImportError: + """Tests for create_marble_agents when MARBLE is not available.""" + + def test_create_marble_agents_import_error(self): + """create_marble_agents should raise ImportError when MARBLE not available.""" + from maseval.benchmark.multiagentbench.adapters.marble_adapter import ( + create_marble_agents, + ) + + # This will fail because MARBLE is not installed + with pytest.raises(ImportError, match="MARBLE is not available"): + create_marble_agents( + agent_configs=[{"agent_id": "test"}], + marble_env=MagicMock(), + model="gpt-4o", + )