Merge T4 GRPO experiment into L20-CodeForge

Author: Kevin-Li-2025

README.md

@@ -85,6 +85,9 @@ The repository contains:
 - Candidate health audits for syntax, entrypoint, and execution failure modes.
 - A trajectory schema for repo-repair agents and model training data.
 - SFT, DPO, reward-function, and GRPO/RLVR scaffolding.
+- A migrated T4 GRPO reasoning experiment under
+  [`experiments/grpo_t4/`](experiments/grpo_t4/), retained as an experiment
+  artifact rather than a separate public repo.
 - Single-L20 setup scripts and GPU sanity checks.
 
 ## Quickstart

experiments/grpo_t4/.gitignore

@@ -0,0 +1,27 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.ipynb_checkpoints
+.venv/
+venv/
+ENV/
+env/
+
+# OS-specific
+.DS_Store
+.DS_Store?
+._*
+.Spotlight-V100
+.Trashes
+ehthumbs.db
+Thumbs.db
+
+# Kaggle metadata and generated outputs
+kernel-metadata.json
+grpo_notebook.ipynb
+inference_notebook.ipynb
+grpo_runs/
+grpo_r1_adapter/
+my_model/
+*.log

experiments/grpo_t4/README.md

@@ -0,0 +1,102 @@
+# DeepSeek-R1-Mini-T4: GRPO Reinforcement Learning Alignment Pipeline
+
+> Migrated from
+> [`Kevin-Li-2025/grpo-deepseek-r1-t4`](https://github.com/Kevin-Li-2025/grpo-deepseek-r1-t4)
+> into L20-CodeForge as `experiments/grpo_t4/`. Generated Unsloth compiled-cache
+> files were intentionally not migrated; this directory keeps only the scripts,
+> metrics, plots, and written results needed to preserve the experiment.
+
+This repository contains a complete pipeline to train and evaluate a reasoning LLM using **GRPO (Group Relative Policy Optimization)** on resource-constrained hardware (e.g., Kaggle's free Tesla T4 GPUs).
+
+It implements a DeepSeek-R1 style reinforcement learning loop that teaches a 1.5B parameters model to reason step-by-step using `<reasoning>` and `<answer>` tags, optimized for training and inference stability.
+
+---
+
+## 🚀 Key Features
+
+*   **GRPO Training Implementation**: Direct policy reinforcement optimization using custom rule-based rewards without requiring a Critic model (saving up to 50% VRAM).
+*   **Multi-Aspect Reward Design**:
+    1.  `format_reward`: Enforces thinking layout containing strict `<reasoning>` and `<answer>` blocks.
+    2.  `correctness_reward`: Parses mathematical responses and validates calculations against ground truth (GSM8K).
+    3.  `length_penalty`: Mitigates "thought-length hacking" (penalizes redundant loops).
+*   **VRAM-Optimized Environment Setup**: Configured to run on standard T4 GPUs by disabling memory-heavy vLLM dependencies, enabling gradient checkpointing, and using `Unsloth` 4-bit quantization.
+*   **Automated Cloud Execution**: Includes wrapper scripts to convert Python code into Jupyter Notebooks and programmatically deploy them to Kaggle Kernels.
+
+---
+
+## 📐 GRPO Architecture
+
+```mermaid
+graph TD
+    Prompt[Input Math Problem] --> Model[Qwen2.5-Math-1.5B Policy]
+    Model -->|Generate G=4 completions| Outputs[Completions 1, 2, 3, 4]
+    Outputs --> R1[Format Reward: <reasoning> and <answer> validation]
+    Outputs --> R2[Correctness Reward: Match ground truth output]
+    Outputs --> R3[Length Penalty: Exclude redundant outputs]
+    R1 & R2 & R3 --> GRPO[Compute Relative Advantages & Update Weights]
+    GRPO --> Model
+```
+
+---
+
+## 📁 Repository Structure
+
+```
+├── grpo_trainer.py       # Main GRPO training script using TRL and Unsloth
+├── push_kernel.py        # Converts training script to Notebook and pushes to Kaggle
+├── inference_test.py     # Inference evaluation script loading the merged model
+├── push_inference.py     # Deploys inference test to Kaggle referencing the training outputs
+└── .gitignore            # Git exclusion rules
+```
+
+---
+
+## 🛠️ Usage Instructions
+
+### 1. Prerequisite: Kaggle CLI
+Install the Kaggle CLI and set up your authentication key:
+```bash
+pip install kaggle
+mkdir -p ~/.kaggle
+echo '{"username":"YOUR_USERNAME","key":"YOUR_KEY"}' > ~/.kaggle/kaggle.json
+chmod 600 ~/.kaggle/kaggle.json
+```
+
+### 2. Triggering the Training Job
+Run the push script to convert `grpo_trainer.py` to a Kaggle Notebook format and deploy it to a remote GPU container:
+```bash
+python3 push_kernel.py
+```
+Monitor the run using:
+```bash
+kaggle kernels status YOUR_USERNAME/grpo-deepseek-r1-t4
+```
+
+### 3. Running Inference Verification
+Once the training kernel completes and saves the merged weights, deploy the inference evaluation:
+```bash
+python3 push_inference.py
+```
+
+---
+
+## 🎯 Verification Results
+
+During testing, the aligned `Qwen2.5-Math-1.5B` model generated perfect reasoning trajectories. Here is a sample evaluation output:
+
+### Problem: Algebra Equation
+*   **Question**: `Solve for x: 3x + 7 = 22. Provide the numeric value.`
+*   **Generated Response**:
+    ```html
+    To solve the equation 3x + 7 = 22 for x, we will follow these steps:
+
+    1. Isolate the term with the variable x by subtracting 7 from both sides:
+       3x + 7 - 7 = 22 - 7
+       3x = 15
+
+    2. Solve for x by dividing both sides of the equation by 3:
+       3x/3 = 15/3
+       x = 5
+
+    So, the solution to the equation is \boxed{5}.
+    ```

experiments/grpo_t4/benchmark.py

@@ -0,0 +1,122 @@
+import os
+import re
+import argparse
+import torch
+from unsloth import FastLanguageModel
+from datasets import load_dataset
+from tqdm import tqdm
+
+def extract_gt(text):
+    if "####" in text:
+        return text.split("####")[-1].strip()
+    return None
+
+def extract_answer(content):
+    # Extract prediction inside <answer>...</answer>
+    pred_match = re.search(r"<answer>(.*?)</answer>", content, re.DOTALL)
+    if pred_match:
+        return pred_match.group(1).strip()
+    return None
+
+def verify_format(content):
+    # Verify presence of <reasoning> and <answer> blocks
+    pattern = r"<reasoning>.*?</reasoning>\s*<answer>.*?</answer>"
+    return 1.0 if re.search(pattern, content, re.DOTALL) else 0.0
+
+def run_benchmark(model_name, num_samples=30):
+    print(f"Loading model: {model_name}...")
+    max_seq_length = 1024
+
+    model, tokenizer = FastLanguageModel.from_pretrained(
+        model_name = model_name,
+        max_seq_length = max_seq_length,
+        load_in_4bit = True,
+        fast_inference = False
+    )
+    model.eval()
+
+    print("Loading GSM8K test dataset...")
+    dataset = load_dataset("openai/gsm8k", "main", split=f"test[:{num_samples}]")
+
+    SYSTEM_PROMPT = """
+Respond in the following format:
+<reasoning>
+...
+</reasoning>
+<answer>
+...
+</answer>
+"""
+
+    correct = 0
+    format_compliant = 0
+    total_thought_length = 0
+
+    print(f"\nEvaluating on {num_samples} samples...")
+    for item in tqdm(dataset):
+        question = item["question"]
+        raw_ans = item["answer"]
+        gt = extract_gt(raw_ans)
+
+        messages = [
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user", "content": question}
+        ]
+
+        inputs = tokenizer.apply_chat_template(
+            messages,
+            tokenize=True,
+            add_generation_prompt=True,
+            return_tensors="pt"
+        ).to("cuda")
+
+        with torch.no_grad():
+            outputs = model.generate(
+                input_ids=inputs,
+                max_new_tokens=512,
+                temperature=0.6,
+                top_p=0.95,
+                use_cache=True
+            )
+
+        generated_tokens = outputs[0][len(inputs[0]):]
+        response = tokenizer.decode(generated_tokens, skip_special_tokens=True)
+
+        # Evaluate metrics
+        pred = extract_answer(response)
+        is_compliant = verify_format(response)
+
+        # Calculate thought length (characters inside <reasoning> tags)
+        reasoning_match = re.search(r"<reasoning>(.*?)</reasoning>", response, re.DOTALL)
+        thought_len = len(reasoning_match.group(1)) if reasoning_match else 0
+        total_thought_length += thought_len
+
+        if is_compliant:
+            format_compliant += 1
+        if gt and pred and gt == pred:
+            correct += 1
+
+    accuracy = (correct / num_samples) * 100.0
+    compliance = (format_compliant / num_samples) * 100.0
+    avg_thought_len = total_thought_length / num_samples
+
+    print("\n--- Benchmark Results ---")
+    print(f"Model: {model_name}")
+    print(f"Accuracy: {accuracy:.2f}% ({correct}/{num_samples})")
+    print(f"Format Compliance: {compliance:.2f}% ({format_compliant}/{num_samples})")
+    print(f"Average Thought Length: {avg_thought_len:.2f} chars")
+
+    # Return metrics for report generation
+    return {
+        "accuracy": accuracy,
+        "compliance": compliance,
+        "thought_len": avg_thought_len
+    }
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model", type=str, required=True, help="Path or repo name of the model to evaluate")
+    parser.add_argument("--samples", type=int, default=30, help="Number of test samples")
+    args = parser.parse_args()
+
+    run_benchmark(args.model, args.samples)