Running instruction?

[jacklxc]

Hi, is there any instruction to run this refactor benchmark?

[paul-gauthier]

Thanks for your interest in aider and the refactor benchmark.

You can run it like the main aider benchmark:

https://github.com/Aider-AI/aider/tree/main/benchmark

You just need to use --exercises-dir to point it at the refactor exercises.

[jacklxc]

Thanks for your quick reply. I was diving into the source code but still confused. Can you explain what is the outputs of the LLM and how is the output applied to the original file? Is the LLM output the full updated file content or only the difference between the original file content and the updated file content?

I am trying to use my own LLM for this benchmark, thank you.

[yiling-chen]

Hi @paul-gauthier , simply chaning --exercises-dir does not work. The data loading code (i.e. get_exercise_dirs is specifically written for polyglot benchmark. I am wondering if you have an official implementation of how to evaluate refactor benchmark? Thanks.

[doodlebro]

Figured out how to get this running with the main Aider benchmark. You have to create the .meta folder and config files for each task. You also have to fix imports from benchmark.refactor_tools since each test uses this utility. My kludgey workaround is to copy benchmark/refactor_tools.py into each task folder.

Here is a short script I used to help, assumes main aider git has been cloned to ~/aider, refactor-benchmark has been cloned to tmp.benchmarks:

import os
import shutil
import json

def process_directory(directory):
    # Create .meta directory
    meta_dir = os.path.join(directory, ".meta")
    os.makedirs(meta_dir, exist_ok=True)

    # Identify the {{PROBLEM}} value
    problem = None
    for filename in os.listdir(directory):
        if filename.endswith(".py") and not filename.endswith("_test.py"):
            problem = filename.split(".")[0]
            break

    if problem is None:
        print(f"No valid problem file found in directory: {directory}")
        return

    # Create config.json file
    config = {
        "authors": [],
        "contributors": [],
        "files": {
            "solution": [f"{problem}.py"],
            "test": [f"{problem}_test.py"],
            "example": [".meta/example.py"]
        },
        "blurb": "",
        "source": "",
        "source_url": ""
    }

    config_file_path = os.path.join(meta_dir, "config.json")
    with open(config_file_path, "w") as config_file:
        json.dump(config, config_file, indent=4)

    # Create blank 'example.py' file in .meta directory
    example_file_path = os.path.join(meta_dir, "example.py")
    with open(example_file_path, "w") as example_file:
        pass  # This creates an empty file

    # Create benchmark directory and copy ~/aider/benchmark/refactor_tools.py to it
    benchmark_tools_dir_path = os.path.join(directory, "benchmark")
    os.makedirs(benchmark_tools_dir_path, exist_ok=True)
    shutil.copy("~/aider/benchmark/refactor_tools.py", benchmark_tools_dir_path)

def main():
    # Replace with the path to the parent directory
    parent_directory = "refactor-benchmark"

    for child_dir in os.listdir(parent_directory):
        child_path = os.path.join(parent_directory, child_dir)
        if os.path.isdir(child_path):
            process_directory(child_path)

if __name__ == "__main__":
    main()