update RL code

Author: Luciennnnnnn

README.md

@@ -28,9 +28,17 @@
 - **Versatile Applications**: Ready to use as a best-in-class reranker to improve editing outputs, or as a high-fidelity reward signal for **stable and effective Reinforcement Learning (RL) fine-tuning**.
 
 ## 🔥 News
-- **2025-10-16**: Training datasets [EditScore-Reward-Data](https://huggingface.co/datasets/EditScore/EditScore-Reward-Data) and [EditScore-RL-Data](https://huggingface.co/datasets/EditScore/EditScore-RL-Data) are available.
-- **2025-10-15**: **EditScore** is now available on PyPI — install it easily with `pip install editscore`.
-- **2025-10-15**: Best-of-N inference scripts for OmniGen2, Flux-dev-Kontext, and Qwen-Image-Edit are now available! See [this](#apply-editscore-to-image-editing) for details.
+- **2025-10-22**: **Introducing Our Reinforcement Learning Training Framework!**
+  We're excited to release our complete RL pipeline, the result of a massive effort to simplify fine-tuning for image editing models. Key features include:
+    - **Ready-to-Use RL Dataset**: Includes the complete dataset used in the EditScore project, along with clear usage guidelines and preparation scripts.
+    - **An Easy-to-Use Reward Model**: Seamlessly integrate **EditScore** as a reward signal.
+    - **A Scalable Reward Server**: Built with native multi-node support for high-throughput training.
+    - **Flexible Training Code**: Supports distributed training, variable image resolutions and mixed tasks (t2i, edit, in-context generation) out-of-the-box.
+  Dive into our comprehensive guide on [RL Fine-Tuning](examples/OmniGen2-RL#application-2-reinforcement-fine-tuning) to get started.
+
+- 2025-10-16: Training datasets [EditScore-Reward-Data](https://huggingface.co/datasets/EditScore/EditScore-Reward-Data) and [EditScore-RL-Data](https://huggingface.co/datasets/EditScore/EditScore-RL-Data) are available.
+- 2025-10-15: **EditScore** is now available on PyPI — install it easily with `pip install editscore`.
+- 2025-10-15: Best-of-N inference scripts for OmniGen2, Flux-dev-Kontext, and Qwen-Image-Edit are now available! See [this](#apply-editscore-to-image-editing) for details.
 - 2025-09-30: We release **OmniGen2-EditScore7B**, unlocking online RL For Image Editing via high-fidelity EditScore. LoRA weights are available at [Hugging Face](https://huggingface.co/OmniGen2/OmniGen2-EditScore7B) and [ModelScope](https://www.modelscope.cn/models/OmniGen2/OmniGen2-EditScore7B).
 - 2025-09-30: We are excited to release **EditScore** and **EditReward-Bench**! Model weights and the benchmark dataset are now publicly available. You can access them on Hugging Face: [Models Collection](https://huggingface.co/collections/EditScore/editscore-68d8e27ee676981221db3cfe) and [Benchmark Dataset](https://huggingface.co/datasets/EditScore/EditReward-Bench), and on ModelScope: [Models Collection](https://www.modelscope.cn/collections/EditScore-8b0d53aa945d4e) and [Benchmark Dataset](https://www.modelscope.cn/datasets/EditScore/EditReward-Bench).
 

evaluate.sh

@@ -2,9 +2,6 @@
 SHELL_FOLDER=$(cd "$(dirname "$0")";pwd)
 cd $SHELL_FOLDER
 
-source "$(dirname $(which conda))/../etc/profile.d/conda.sh"
-conda activate py3.12+pytorch2.7.1+cu126
-
 python evaluation.py \
 --benchmark_dir EditScore/EditReward-Bench \
 --result_dir results/EditScore-7B \

evaluate_32B_vllm.sh

@@ -2,23 +2,20 @@
 SHELL_FOLDER=$(cd "$(dirname "$0")";pwd)
 cd $SHELL_FOLDER
 
-source "$(dirname $(which conda))/../etc/profile.d/conda.sh"
-conda activate py3.12+pytorch2.7.1+cu126
-
-# python evaluation.py \
-# --benchmark_dir EditScore/EditReward-Bench \
-# --result_dir results/EditScore-72B \
-# --backbone qwen25vl_vllm \
-# --model_name_or_path Qwen/Qwen2.5-VL-72B-Instruct \
-# --enable_lora \
-# --lora_path /share/project/jiahao/LLaMA-Factory2/output/lora_72B_extract \
-# --score_range 25 \
-# --max_workers 1 \
-# --max_model_len 4096 \
-# --max_num_seqs 1 \
-# --max_num_batched_tokens 4096 \
-# --tensor_parallel_size 4 \
-# --num_pass 1
+python evaluation.py \
+--benchmark_dir EditScore/EditReward-Bench \
+--result_dir results/EditScore-72B \
+--backbone qwen25vl_vllm \
+--model_name_or_path Qwen/Qwen2.5-VL-72B-Instruct \
+--enable_lora \
+--lora_path EditScore/EditScore-72B \
+--score_range 25 \
+--max_workers 1 \
+--max_model_len 4096 \
+--max_num_seqs 1 \
+--max_num_batched_tokens 4096 \
+--tensor_parallel_size 4 \
+--num_pass 1
 
 python calculate_statistics.py \
 --result_dir results/EditScore-72B/qwen25vl_vllm

evaluate_72B_vllm.sh

@@ -2,23 +2,20 @@
 SHELL_FOLDER=$(cd "$(dirname "$0")";pwd)
 cd $SHELL_FOLDER
 
-source "$(dirname $(which conda))/../etc/profile.d/conda.sh"
-conda activate py3.12+pytorch2.7.1+cu126
-
-# python evaluation.py \
-# --benchmark_dir EditScore/EditReward-Bench \
-# --result_dir results/EditScore-7B \
-# --backbone qwen25vl_vllm \
-# --model_name_or_path Qwen/Qwen2.5-VL-7B-Instruct \
-# --enable_lora \
-# --lora_path EditScore/EditScore-7B \
-# --score_range 25 \
-# --max_workers 1 \
-# --max_model_len 4096 \
-# --max_num_seqs 1 \
-# --max_num_batched_tokens 4096 \
-# --tensor_parallel_size 1 \
-# --num_pass 1
+python evaluation.py \
+--benchmark_dir EditScore/EditReward-Bench \
+--result_dir results/EditScore-7B \
+--backbone qwen25vl_vllm \
+--model_name_or_path Qwen/Qwen2.5-VL-7B-Instruct \
+--enable_lora \
+--lora_path EditScore/EditScore-7B \
+--score_range 25 \
+--max_workers 1 \
+--max_model_len 4096 \
+--max_num_seqs 1 \
+--max_num_batched_tokens 4096 \
+--tensor_parallel_size 1 \
+--num_pass 1
 
 python calculate_statistics.py \
 --result_dir results/EditScore-7B/qwen25vl_vllm

evaluate_72B_vllm_2.sh

@@ -2,7 +2,6 @@ ratio_type: inside_ratio
 
 data:
   - 
-    # path: '/path/to/EditScore-RL-Data/rl_abs_9tasks.jsonl'
-    path: '/share/project/chenyuan/data2/EditScore-RL-Data-v4/rl_abs_9tasks.jsonl'
+    path: '/path/to/EditScore-RL-Data/rl_abs_9tasks.jsonl'
     type: 'edit'
     ratio: !!float 1

evaluate_vllm.sh

@@ -73,12 +73,16 @@ for ((i=0; i<num_gpus_per_machine; i++)); do
     --result_dir evaluation/GEdit-Bench/results/FLUX-Kontext-dev/results_gs${guidance_scale}_16samples \
     --save_dir evaluation/GEdit-Bench/results/FLUX-Kontext-dev/results_gs${guidance_scale}_16samples_pass1 \
     --num_samples 16 \
-    --backbone qwen25vl \
-    --model_variant GRM-v4 \
-    --model_path /share/project/jiahao/LLaMA-Factory2/output/merge_v7-2_8models_omnigen2-4samples_gpt4-1_range_0to25 \
+    --backbone qwen25vl_vllm \
+    --model_name_or_path Qwen/Qwen2.5-VL-7B-Instruct \
+    --enable_lora \
+    --lora_path EditScore/EditScore-7B \
+    --score_range 25 \
     --max_workers 1 \
     --max_model_len 4096 \
-    --context_version v2 \
+    --max_num_seqs 1 \
+    --max_num_batched_tokens 4096 \
+    --tensor_parallel_size 1 \
     --num_pass 1 \
     --start_index ${start_idx} --end_index ${end_idx} \
     > logs/gedit_FLUX-Kontext-dev_gs${guidance_scale}_16samples_select_best_pass1_${start_idx}_${end_idx}.log 2>&1 &

examples/OmniGen2-RL/data_configs/train/example/edit/all.yml

@@ -73,12 +73,16 @@ for ((i=0; i<num_gpus_per_machine; i++)); do
     --result_dir evaluation/GEdit-Bench/results/FLUX-Kontext-dev/results_gs${guidance_scale}_16samples \
     --save_dir evaluation/GEdit-Bench/results/FLUX-Kontext-dev/results_gs${guidance_scale}_16samples_pass4 \
     --num_samples 16 \
-    --backbone qwen25vl \
-    --model_variant GRM-v4 \
-    --model_path /share/project/jiahao/LLaMA-Factory2/output/merge_v7-2_8models_omnigen2-4samples_gpt4-1_range_0to25 \
+    --backbone qwen25vl_vllm \
+    --model_name_or_path Qwen/Qwen2.5-VL-7B-Instruct \
+    --enable_lora \
+    --lora_path EditScore/EditScore-7B \
+    --score_range 25 \
     --max_workers 1 \
     --max_model_len 4096 \
-    --context_version v2 \
+    --max_num_seqs 1 \
+    --max_num_batched_tokens 4096 \
+    --tensor_parallel_size 1 \
     --num_pass 4 \
     --start_index ${start_idx} --end_index ${end_idx} \
     > logs/gedit_FLUX-Kontext-dev_gs${guidance_scale}_16samples_select_best_pass4_${start_idx}_${end_idx}.log 2>&1 &