IMPLEMENTATION_PLAN.md AutoRAG – AI Scientist Implementation Plan Objective
Build an autonomous experimentation system that behaves like an AI researcher optimizing RAG systems.
The system must:
Build modular RAG pipelines.
Experiment with pipeline structures and parameters.
Evaluate pipeline performance.
Learn from experiment history.
Adapt experimentation strategy over time.
Discover the best-performing RAG configuration.
The system should simulate an AI scientist running experiments overnight.
System Capabilities
The system must support:
• Pipeline architecture changes • Parameter optimization • Experiment reasoning • Experiment memory • Strategy adaptation • Automatic reporting
High-Level System Flow baseline pipeline ↓ experiment proposed by AI ↓ pipeline built ↓ evaluation run ↓ results logged ↓ AI analyzes results ↓ strategy updated ↓ next experiment proposed Phase 1 – Project Structure
Create the following directory structure.
autorag/
rag/ chunking.py embedding.py retrieval.py reranking.py query_expansion.py generation.py
pipeline/ pipeline_builder.py
evaluation/ semantic_score.py llm_judge.py evaluator.py
agent/ planner.py strategist.py memory.py
experiments/ logger.py logs.json best_pipeline.json
analysis/ parameter_importance.py experiment_analysis.py
data/ docs/ eval/
run_autorag.py export_results.py Phase 2 – Implement RAG Components
All pipeline components must support configurable parameters.
Chunking
File:
rag/chunking.py
Implement:
• fixed chunking • sentence chunking • semantic chunking
Parameters:
chunk_method chunk_size chunk_overlap Embeddings
File:
rag/embedding.py
Supported models:
nomic-embed-text bge-small-en-v1.5 all-MiniLM-L6-v2
Functions:
embed_documents() embed_query() Retrieval
File:
rag/retrieval.py
Implement:
Vector retrieval
BM25 retrieval
Hybrid retrieval
Parameters:
retrieval_method top_k Reranking
File:
rag/reranking.py
Supported models:
bge-reranker-large bce-reranker-base
Parameters:
reranker_model rerank_top_k Query Expansion
File:
rag/query_expansion.py
Implement:
none rewrite_query multi_query
Parameters:
query_expansion_method multi_query_count Generation
File:
rag/generation.py
Use generation model:
qwen2.5:7b
Function:
generate_answer(query, context) Phase 3 – Evaluation System
Evaluation produces a single experiment score.
Semantic Similarity
File:
evaluation/semantic_score.py
Compute embedding similarity between generated answer and ground truth.
LLM Judge
File:
evaluation/llm_judge.py
Model used:
dolphin3:latest
The LLM must score answer correctness from 0–10 and also provide concise reasoning for pipeline mutations.
Final Evaluator
File:
evaluation/evaluator.py
Score formula:
score = 0.6 * semantic_similarity + 0.4 * llm_judge_score
Return average score across evaluation dataset.
Phase 4 – Pipeline Builder
File:
pipeline/pipeline_builder.py
Build pipelines from configuration objects.
Example configuration:
{ embedding_model chunk_method chunk_size chunk_overlap retrieval_method top_k reranker_model rerank_top_k query_expansion_method multi_query_count }
Pipeline builder must dynamically assemble the RAG pipeline from prebuilt code blocks rather than generating fresh source code per experiment.
Phase 5 – Baseline Pipeline
Before experimentation begins, run a baseline pipeline.
Example baseline:
embedding_model = nomic chunk_method = fixed chunk_size = 400 chunk_overlap = 50 retrieval_method = vector top_k = 5 reranker_model = none query_expansion_method = none
Run evaluation and store baseline score.
Phase 6 – Experiment Memory
File:
agent/memory.py
The system must store:
all previous experiments scores pipeline configurations parameter values reasoning
Memory enables the AI to learn from past experiments.
Phase 7 – Parameter Importance Analysis
File:
analysis/parameter_importance.py
Analyze which parameters influence performance.
Example output:
chunk_size strongly affects accuracy rerank_top_k moderately affects accuracy query expansion weak effect
This helps the AI focus future experiments.
Phase 8 – Experiment Strategist
File:
agent/strategist.py
The strategist determines how experiments should be explored.
Strategies include:
• exploration • exploitation • parameter tuning • pipeline mutation
The system must sometimes:
try new architectures
and sometimes:
refine best pipelines Phase 9 – AI Experiment Planner
File:
agent/planner.py
The planner proposes experiments using:
• experiment history • parameter importance • best pipeline so far
Example output:
{ config: {...}, reasoning: "...", hypothesis: "..." } Phase 10 – Experiment Logger
File:
experiments/logger.py
Each experiment must log:
experiment_id pipeline_config reasoning_before evaluation_score analysis_after timestamp
Logs stored in:
experiments/logs.json Phase 11 – Autonomous Experiment Loop
File:
run_autorag.py
Loop:
load experiment memory
while running:
planner proposes experiment
log reasoning_before
build pipeline
run evaluation
compute score
log experiment
analyze results
update memory
update strategy
Phase 12 – Adaptive Experiment Strategy
The system must adapt experimentation behavior.
Early phase:
explore many configurations
Later phase:
refine best pipelines
This mimics scientific experimentation.
Phase 13 – Stop Behavior
When the user stops execution:
terminate experiment loop save experiment logs identify best pipeline
Save best pipeline to:
experiments/best_pipeline.json Phase 14 – Export Full Experiment Report
File:
export_results.py
Generate:
experiment_report.txt
Report must include:
• baseline pipeline • best discovered pipeline • experiment history • reasoning before experiments • reasoning after experiments • evaluation scores • parameter importance analysis • final conclusions
Final System Behavior
The completed system will:
• build modular RAG pipelines • run automated experiments • analyze experiment outcomes • adapt experimentation strategies • discover optimal RAG architectures
The system effectively behaves like an AI researcher improving retrieval pipelines through experimentation.