Question: GPU/Docker Execution and Tool/MCP Integration Support

[passing2961]

Hi,

I’m wondering whether OpenEvolve supports LLM interaction with an external execution environment such as a GPU cluster or a Docker-based server. For example, in a task like “discovering an optimized mixture-of-experts architecture for mathematical reasoning,” the system would need to fine-tune or evaluate candidate model architectures on a GPU environment. Does OpenEvolve support such GPU-backed execution, and is it also possible to call external tools or MCP servers during the evolutionary process?

Thanks!

[codelion]

GPU/Docker Execution

LLM Operations

OpenEvolve supports any OpenAI-compatible API endpoint via the api_base configuration. You can point to:

• Self-hosted GPU servers running vLLM, TGI, Ollama, or similar
• Docker containers with OpenAI-compatible APIs
• Cloud GPU endpoints (Together AI, Fireworks, etc.)

Example config:

llm:
  api_base: "http://your-gpu-server:8000/v1"
  api_key: "your-key"
  models:
    - name: "your-model"
      weight: 1.0

Evaluation on GPU/Docker

The evaluator is fully customizable. Your evaluator.py can call remote GPU resources. For your MoE architecture discovery use case:

def evaluate(program_path):
    # Read the candidate architecture
    with open(program_path) as f:
        architecture_code = f.read()

    # Submit to your GPU cluster/Docker for training
    job_id = submit_training_job(architecture_code, gpu_cluster_url)
    results = wait_for_job(job_id)

    return {
        "accuracy": results["accuracy"],
        "efficiency": results["params"] / results["flops"],
        "combined_score": compute_final_score(results)
    }

Configure longer timeouts for training jobs:

evaluator:
  timeout: 3600  # 1 hour for training jobs

Tool/MCP Integration

You can use https://github.com/algorithmicsuperintelligence/optillm as a proxy to enable tool calling and MCP server integration with any model. OptiLLM sits between OpenEvolve and your LLM, adding capabilities like:

• Tool/function calling support
• MCP server connections
• Various inference optimization techniques

Setup:

1. Configure MCP servers in ~/.optillm/mcp_config.json:

{
  "servers": {
    "my-gpu-tools": {
      "transport": "sse",
      "url": "http://your-mcp-server:8080/sse",
      "headers": {
        "Authorization": "Bearer ${API_TOKEN}"
      }
    }
  }
}

2. Point OpenEvolve to OptiLLM and use the mcp- prefix:

llm:
  api_base: "http://localhost:8000/v1"  # OptiLLM proxy
  models:
    - name: "mcp-gpt-4o"  # mcp prefix enables MCP plugin
      weight: 1.0

This allows the LLM to call external tools and MCP servers during code generation.

[passing2961]

@codelion Thank you very much for your kind and detailed response—it was extremely helpful.

I have one further question. What if the task or question involves training GPT-2? In that case, we would need to prepare a Docker container with GPU support. While it is possible to rely on cloud GPU services for fine-tuning—since many providers now offer managed fine-tuning (e.g., Together AI)—what if we instead want to train the model on my local server, which has two A100 GPUs (40GB each)?

Is this setup supported? If so, could you explain how to set it up once again?

[codelion]

I think there are couple of options:

1. We can run OpenEvolve directly on your GPU server

The simplest approach - install OpenEvolve on the machine with the A100s. Your evaluator can directly use the local GPUs:

def evaluate(program_path):
    import torch
    # Your training code runs directly on local GPUs
    device = "cuda" if torch.cuda.is_available() else "cpu"
    # ... train GPT-2, return metrics

2. Remote submission from a different machine

If OpenEvolve runs elsewhere, set up a simple job API on your GPU server:

# evaluator.py (runs on OpenEvolve machine)
def evaluate(program_path):
    import requests
    with open(program_path) as f:
        code = f.read()

    # Submit to your GPU server's API
    response = requests.post(
        "http://your-a100-server:5000/train",
        json={"code": code}
    )
    return response.json()  # {"accuracy": ..., "loss": ...}

On your A100 server, run a Flask/FastAPI endpoint that receives code, trains, and returns results.

[passing2961]

Both solutions are really helpful to my current ongoing research, where I try to expand new OpenEvolve tasks that require to use GPU servers. Really appreciate it. Especially, I love the second way.