Skip to content

In Memory Memory Service missing from Evaluation Generator #2084

@sjhatfield

Description

@sjhatfield

** Please make sure you read the contribution guide and file the issues in the right place. **
Contribution guide.

Describe the bug
When the root agent uses a save to memory callback (code below) and the PreloadMemoryTool from tools/preload_memory_tool.py, then evaluation in the command line throws a ValueError 'Memory service is not available.' (full traceback below).

To Reproduce

Create a basic root agent like:

from google.adk.agents import Agent

async def save_to_memory_callback(callback_context: CallbackContext) -> None:
    if hasattr(callback_context._invocation_context, "session"):
        session = callback_context._invocation_context.session
    else:
        return

    if hasattr(callback_context._invocation_context, "memory_service"):
        memory_service = callback_context._invocation_context.memory_service
    else:
        return

    if hasattr(session, "events") and len(session.events) >= 2:
        # At least 2 events: user query + agent response
        await memory_service.add_session_to_memory(session)

root_agent = Agent(
    model="gemini-2.5-flash",
    name="root_agent",
    description="A coordinator agent that manages health coach tools/agents.",
    instruction="You coordinate a number of health tools/agents",
    after_agent_callback=[save_to_memory_callback],
    tools=[PreloadMemoryTool()],
)

and then create any evaluation test case and run

adk eval app path/to/test/case.json

and you will get an error similar to the following full traceback:

Traceback (most recent call last):
File "/home/runner/.cache/pypoetry/virtualenvs/health-coach-GZ0pTjfG-py3.11/lib/python3.11/site-packages/google/adk/cli/cli_eval.py", line 166, in run_evals
await EvaluationGenerator._generate_inferences_from_root_agent(
File "/home/runner/.cache/pypoetry/virtualenvs/health-coach-GZ0pTjfG-py3.11/lib/python3.11/site-packages/google/adk/evaluation/evaluation_generator.py", line 185, in _generate_inferences_from_root_agent
async for event in runner.run_async(
File "/home/runner/.cache/pypoetry/virtualenvs/health-coach-GZ0pTjfG-py3.11/lib/python3.11/site-packages/google/adk/runners.py", line 233, in run_async
async for event in self._exec_with_plugin(
File "/home/runner/.cache/pypoetry/virtualenvs/health-coach-GZ0pTjfG-py3.11/lib/python3.11/site-packages/google/adk/runners.py", line 273, in _exec_with_plugin
async for event in execute_fn(invocation_context):
File "/home/runner/.cache/pypoetry/virtualenvs/health-coach-GZ0pTjfG-py3.11/lib/python3.11/site-packages/google/adk/runners.py", line 230, in execute
async for event in ctx.agent.run_async(ctx):
File "/home/runner/.cache/pypoetry/virtualenvs/health-coach-GZ0pTjfG-py3.11/lib/python3.11/site-packages/google/adk/agents/base_agent.py", line 208, in run_async
async for event in self._run_async_impl(ctx):
File "/home/runner/.cache/pypoetry/virtualenvs/health-coach-GZ0pTjfG-py3.11/lib/python3.11/site-packages/google/adk/agents/llm_agent.py", line 283, in _run_async_impl
async for event in self._llm_flow.run_async(ctx):
File "/home/runner/.cache/pypoetry/virtualenvs/health-coach-GZ0pTjfG-py3.11/lib/python3.11/site-packages/google/adk/flows/llm_flows/base_llm_flow.py", line 283, in run_async
async for event in self._run_one_step_async(invocation_context):
File "/home/runner/.cache/pypoetry/virtualenvs/health-coach-GZ0pTjfG-py3.11/lib/python3.11/site-packages/google/adk/flows/llm_flows/base_llm_flow.py", line 299, in _run_one_step_async
async for event in self._preprocess_async(invocation_context, llm_request):
File "/home/runner/.cache/pypoetry/virtualenvs/health-coach-GZ0pTjfG-py3.11/lib/python3.11/site-packages/google/adk/flows/llm_flows/base_llm_flow.py", line 342, in _preprocess_async
await tool.process_llm_request(
File "/home/runner/.cache/pypoetry/virtualenvs/health-coach-GZ0pTjfG-py3.11/lib/python3.11/site-packages/google/adk/tools/preload_memory_tool.py", line 56, in process_llm_request
response = await tool_context.search_memory(user_query)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/runner/.cache/pypoetry/virtualenvs/health-coach-GZ0pTjfG-py3.11/lib/python3.11/site-packages/google/adk/tools/tool_context.py", line 75, in search_memory
Error: raise ValueError('Memory service is not available.')
ValueError: Memory service is not available.

Expected behavior
The evaluation generator should create an InMemoryMemoryService to be used during evaluation.

Desktop (please complete the following information):

  • OS: MacOS Sonoma 14.7.5
  • Python version(python -V): 3.11.7
  • ADK version(pip show google-adk): 1.7.0

Additional context
I have a fix and can open a PR.

Metadata

Metadata

Assignees

Labels

eval[Component] This issue is related to evaluation

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions