Added integration: LlamaIndex (#7)

* added llama index integration

* updated docs of agent interfaces to be more consistent across frameworks.

Author: cemde

CHANGELOG.md

@@ -9,6 +9,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- [LlamaIndex](https://github.com/run-llama/llama_index) integration: `LlamaIndexAgentAdapter` and `LlamaIndexUser` for evaluating LlamaIndex workflow-based agents (PR: #7)
+
+  - Supports async workflow execution with proper event loop handling
+
 - The `logs` property inside `SmolAgentAdapter` and `LanggraphAgentAdapter` are now properly filled. (PR: #3)
 
 ### Changed

README.md

@@ -45,16 +45,23 @@ The package is published on PyPI as `maseval`. To install the stable release for
 pip install maseval
 ```
 
-If you want the optional integrations used by the examples (smolagents, langchain, etc.), install the examples extras:
+If you want the optional integrations used by the examples (smolagents, langgraph, llamaindex, etc.), install the examples extras:
 
 ```bash
 pip install "maseval[examples]"
 ```
 
-Or install only the smolagents integration:
+Or install specific framework integrations:
 
 ```bash
+# Smolagents
 pip install "maseval[smolagents]"
+
+# LangGraph
+pip install "maseval[langgraph]"
+
+# LlamaIndex
+pip install "maseval[llamaindex]"
 ```
 
 ## Example

docs/interface/agents/langgraph.md

@@ -1,6 +1,22 @@
 # LangGraph
 
-Adapter for the LangGraph agent framework. The page renders the adapter API when
-the optional dependency is available in the build environment.
+Adapter for the `LangGraph` agent framework.
+
+- [Documentation](https://langchain-ai.github.io/langgraph/)
+- [Code Repository](https://github.com/langchain-ai/langgraph)
+
+## Installation
+
+```bash
+pip install maseval[langgraph]
+```
+
+Alternatively, install langgraph directly:
+
+```bash
+pip install langgraph
+```
+
+## API Reference
 
 ::: maseval.interface.agents.langgraph

docs/interface/agents/llamaindex.md

@@ -0,0 +1,22 @@
+# LlamaIndex
+
+Adapter implementing commonly used functions for `LlamaIndex`'s workflow-based agent system.
+
+- [Documentation](https://docs.llamaindex.ai/)
+- [Code Repository](https://developers.llamaindex.ai/python/framework/)
+
+## Installation
+
+```bash
+pip install maseval[llamaindex]
+```
+
+Alternatively, install llama-index-core directly:
+
+```bash
+pip install llama-index-core
+```
+
+## API Reference
+
+::: maseval.interface.agents.llamaindex

docs/interface/agents/smolagents.md

@@ -1,8 +1,22 @@
-# Huggingface `Smolagents`
+# SmolAgents
 
-Adapter implementing commonly used functions for the Huggingface
-[smolagents](https://huggingface.co/docs/smolagents/en/index) package. The
-API will be rendered when the optional dependency is installed in the build
-environment.
+Adapter implementing commonly used functions for the HuggingFace `smolagents` package.
+
+- [Documentation](https://huggingface.co/docs/smolagents/en/index)
+- [Code Repository](https://github.com/huggingface/smolagents)
+
+## Installation
+
+```bash
+pip install maseval[smolagents]
+```
+
+Alternatively, install smolagents directly:
+
+```bash
+pip install smolagents
+```
+
+## API Reference
 
 ::: maseval.interface.agents.smolagents

docs/reference/agent.md

@@ -8,4 +8,6 @@ Adapters that integrate MASEval with agent frameworks:
 
 ::: maseval.interface.agents.smolagents.SmolAgentAdapter
 
-<!-- ::: maseval.interface.agents.langgraph -->
+::: maseval.interface.agents.langgraph.LangGraphAgentAdapter
+
+::: maseval.interface.agents.llamaindex.LlamaIndexAgentAdapter

docs/reference/user.md

@@ -9,3 +9,7 @@ The User is initialized with a persona and a scenario, both of which are typical
 Some integrations provide convenience user/tool implementations for specific agent frameworks. For example:
 
 ::: maseval.interface.agents.smolagents.SmolAgentUser
+
+::: maseval.interface.agents.langgraph.LangGraphUser
+
+::: maseval.interface.agents.llamaindex.LlamaIndexUser

maseval/interface/agents/__init__.py

@@ -20,3 +20,11 @@
     __all__.extend(["LangGraphAgentAdapter", "LangGraphUser"])
 except ImportError:
     pass
+
+# Conditionally import llamaindex
+try:
+    from .llamaindex import LlamaIndexAgentAdapter, LlamaIndexUser  # noqa: F401
+
+    __all__.extend(["LlamaIndexAgentAdapter", "LlamaIndexUser"])
+except ImportError:
+    pass

maseval/interface/agents/langgraph.py

@@ -32,33 +32,88 @@ def _check_langgraph_installed():
 class LangGraphAgentAdapter(AgentAdapter):
     """An AgentAdapter for LangGraph CompiledGraph agents.
 
-    Requires langgraph to be installed.
+    This adapter integrates LangGraph's compiled graphs with MASEval's benchmarking framework,
+    converting LangChain/LangGraph message types to OpenAI-compatible MessageHistory format.
+    It preserves tool calls, tool responses, multi-modal content, and supports both stateless
+    and stateful (checkpointed) graph execution.
+
+    LangGraph graphs can operate in two modes:
+
+    - **Stateless**: Messages from invoke() result are cached in the adapter for access
+    - **Stateful**: With checkpointer and thread_id, messages are fetched from persistent state
+
+    The adapter automatically handles both modes, preferring persistent state when available
+    and falling back to cached results for stateless graphs.
+
+    How to use:
+        1. **Create a LangGraph graph** with state and nodes
+        2. **Compile the graph** (optionally with checkpointer for state persistence)
+        3. **Wrap with LangGraphAgentAdapter** to enable MASEval integration
+        4. **Use in benchmarks** or call directly for testing
+        5. **Access traces and config** for analysis and debugging
+
+        Example workflow:
+            ```python
+            from maseval.interface.agents.langgraph import LangGraphAgentAdapter
+            from langgraph.graph import StateGraph, MessagesState
+            from langgraph.checkpoint.memory import MemorySaver
+
+            # Define your graph
+            def chatbot(state: MessagesState):
+                # Your agent logic
+                return {"messages": [response]}
+
+            # Build graph
+            graph = StateGraph(MessagesState)
+            graph.add_node("chatbot", chatbot)
+            graph.set_entry_point("chatbot")
+            graph.set_finish_point("chatbot")
+
+            # Compile (stateless)
+            compiled_graph = graph.compile()
+            agent_adapter = LangGraphAgentAdapter(compiled_graph, "agent_name")
+
+            # Or compile with checkpointer (stateful)
+            memory = MemorySaver()
+            compiled_graph = graph.compile(checkpointer=memory)
+            config = {"configurable": {"thread_id": "session_1"}}
+            agent_adapter = LangGraphAgentAdapter(
+                compiled_graph,
+                "agent_name",
+                config=config
+            )
 
-    This adapter converts LangChain/LangGraph message types to MASEval's
-    OpenAI-compatible MessageHistory format. It preserves tool calls, tool
-    responses, and multi-modal content.
+            # Run agent
+            result = agent_adapter.run("What's the weather?")
 
-    LangGraph graphs can be stateless or stateful (with checkpointer). This
-    adapter supports both modes:
-    - Stateless: Messages from invoke() result are cached in adapter
-    - Stateful: Messages fetched from graph state if config/thread_id provided
+            # Access message history in OpenAI format
+            for msg in agent_adapter.get_messages():
+                print(f"{msg['role']}: {msg['content']}")
 
-    Example:
-        ```python
-        from maseval.interface.agents.langgraph import LangGraphAgentAdapter
-        from langgraph.graph import StateGraph
+            # Gather execution traces
+            traces = agent_adapter.gather_traces()
+            if 'total_tokens' in traces:
+                print(f"Total tokens: {traces['total_tokens']}")
 
-        # Create a LangGraph graph
-        graph = StateGraph(...)
-        compiled_graph = graph.compile()
+            # Use in benchmark
+            benchmark = MyBenchmark(agent_data={"agent": agent_adapter})
+            results = benchmark.run(tasks)
+            ```
 
-        agent_adapter = LangGraphAgentAdapter(compiled_graph, "agent_name")
-        result = agent_adapter.run("What's the weather?")
+        For stateful graphs, the adapter preserves conversation context across multiple
+        calls using the same thread_id, enabling multi-turn interactions.
 
-        # Access message history
-        for msg in agent_adapter.get_messages():
-            print(msg['role'], msg['content'])
-        ```
+    Message Format:
+        LangGraph uses LangChain message types. The adapter converts to `maseval` / OpenAI format.
+
+        Tool calls are preserved with metadata and converted to OpenAI's tool call format.
+
+    Token Usage:
+        If LangChain messages include `usage_metadata`, the adapter automatically extracts
+        and aggregates token counts. This is available for models that provide usage information.
+
+    Requires:
+        langgraph to be installed: `pip install maseval[langgraph]`
     """
 
     def __init__(self, agent_instance, name: str, callbacks=None, config=None):