🎮 AMD Minesweeper RL: GRPO & Gymnasium Framework

🏆 AMD-RL Hackathon IIT Delhi 2026

An advanced Reinforcement Learning framework designed to train Large Language Models (LLMs) to play Minesweeper using Group Relative Policy Optimization (GRPO) and Gymnasium environments. The system leverages Monte Carlo Tree Search (MCTS) to generate expert gameplay trajectories paired with detailed Chain-of-Thought (CoT) reasoning, training the agent through a multi-axis reward function optimized for logical deduction, safety, and efficiency.

---

🗺️ Repository Architecture

This repository is structured into two main components: the Gymnasium environment wrapper and the LLM training/evaluation harness.

.
├── README.md                           # Main project documentation (this file)
└── minesweeper-gym/                    # Primary workspace directory
    ├── gym-minesweeper/                # Core Gymnasium package
    │   ├── minesweeper/
    │   │   └── minesweeper.py          # Standardized Minesweeper Env (Discreet/MultiDiscreet)
    │   ├── rewards.py                  # GRPO environment-based reward logic (4-axis + 12-criteria)
    │   ├── train_grpo.py               # Basic GRPO pipeline with Qwen3-4B
    │   ├── setup.py                    # Gymnasium installation script
    │   └── requirements_train.txt      # Python dependencies for training
    │
    ├── minesweeper/                    # Live Agent and Execution harness
    │   ├── agent_server.py             # File-watcher based persistent agent server
    │   ├── minesweeper_agent.py        # Prompter & response parser for LLM
    │   ├── minesweeper_model.py        # llama.cpp server OpenAI API client
    │   └── play_game.py                # Console game player and interactive evaluator
    │
    ├── Plan_1.md                       # Design outline for the multi-axis reward system
    ├── minesweeper_game.py             # Ground-truth game rules, MCTS solver & CoT generators
    ├── generate_synthetic_dataset.py   # Dataset generator generating expert MCTS+CoT trajectories
    ├── verify_dataset.py               # Validation script verifying game seeds & CoT correctness
    ├── train_minesweeper_grpo.py       # Fine-tuning script (Unsloth + GRPO on BF16 models)
    └── minesweeper_config.yaml         # Inference configuration parameters

---

🚀 Key Innovations & Features

🧠 1. MCTS Expert Trajectory Generator

Pure logical solvers can get stuck or perform poor guesswork when determinism is unavailable. We implement a hybrid Monte Carlo Tree Search (MCTS) engine within minesweeper_game.py.

• Deterministic Deductions: The solver automatically unmasks guaranteed safe cells and flags guaranteed mines by examining immediate neighborhood constraints.
• Probabilistic Playouts: When no certain moves exist, the MCTS engine conducts simulated rollout playouts to compute survival win rates and select the statistically optimal coordinate to reveal.

📝 2. Scaled Chain-of-Thought (CoT) Engine

To align LLMs towards thinking step-by-step (matching models like DeepSeek R1), our ReasoningEngine outputs structured reasoning traces reflecting game complexities:

• Low-complexity (small boards): Immediate, focused neighborhood explanations.
• High-complexity (large boards): Detailed multi-step cross-constraint propagation, risk assessments, and win-rate statistics from the MCTS simulations.

📊 3. Multi-Axis Reward Architecture (PDF §8.2)

The reward computed at each step balances long-term strategic safety against play efficiency: $$R_{total} = w_1 \cdot R_{safety} + w_2 \cdot R_{info} + w_3 \cdot R_{efficiency} + w_4 \cdot R_{flag_quality}$$

• Weights: $w_1 = 5.0$, $w_2 = 2.0$, $w_3 = 1.0$, $w_4 = 1.0$

Note

CoT Reasoning verification is kept decoupled from the environment rules—it is evaluated during training in the GRPO reward function where the LLM's text output is fully visible, while the game engine evaluates pure physical actions.

1. Safety ($R_{safety}$): High reward for revealing provably safe cells ($+1.0$), moderate for survival guesses ($+0.2$), large penalty for mine hits ($-100$, terminal), and game win bonus ($+50$).
2. Information Gain ($R_{info}$): Sigmoid-capped reward encouraging cascades that reveal large areas of the board: $$R_{info} = \text{sigmoid}\left(\frac{\text{revealed}}{\text{total_closed}}\right) \times 2.0$$ Redundant reveals yield a penalty of $-0.5$.
3. Efficiency ($R_{efficiency}$): Normalizes step cost across varying board layouts by caching Bechtel's Board Value (3BV) (minimum clicks to solve without flags): $$\text{Penalty per step} = -\frac{1.0}{3BV}$$
4. Flag Quality ($R_{flag}$): Encourages correct mine flags ($+1.0$) and penalizes safe cell flagging ($-2.0$).

---

🎯 12-Criteria Scoring System & Training Rewards

We combine the 4-axis environmental rewards with the core 12 Scoring Criteria designed specifically for LLM game play:

#	Criterion	Score	Rationale
1	Flag cell that IS a mine	+15.0	Encourages active identification of targets
2	Flag cell that is NOT a mine	-10.0	Penalizes incorrect flagging
3	Reveal cell that IS a mine	-25.0	Heavily penalizes terminal failures (Game Over)
4a	Reveal safe (random guess)	+10.0	Soft reward for surviving uncertainty
4b	Reveal safe (logical deduction)	+15.0	Promotes analytical safety play
5	Flag already-flagged cell	-12.0	Prevents infinite loops/redundant state mutations
6	Reveal already-revealed cell	-12.0	Discourages wasting moves
7	Out of bounds move	-15.0	Enforces structural parsing and boundary awareness
8	Total flags > total mines	-10.0	Enforces resource management constraints
9	Invalid JSON response	-50.0 / -5.0	Format gate: filters out malformed textual completions
10	Win the game	+100.0	Large sparse reward for successful environment completion
11	Reveal a flagged cell	-8.0	Discourages safety-violating clicks
12	Flag a revealed cell	-8.0	Prevents redundant cell markings

---

🛠️ Installation & Environment Setup

Ensure you have a Python 3.10+ environment installed.

# Clone the repository and navigate to the directory
cd AMD-RL_HACKATHON_IITD_2026/minesweeper-gym

# Setup virtual environment
python3 -m venv venv
source venv/bin/activate

# Install Gymnasium Package in editable mode
cd gym-minesweeper
pip install -e .

# Install Training and Inference dependencies
pip install -r requirements_train.txt

---

🏃 Workflow: How to Generate, Train, and Run

Step 1: Generate the Expert CoT Dataset

Generate a high-quality expert demonstration dataset of $10,000$ game configurations. This script uses the MCTS engine to play and logs the game history along with scaled reasoning.

python generate_synthetic_dataset.py

This produces structured chunks in minesweeper_dataset_400/ to avoid massive file reads and OOM issues during DataLoader loads.

Step 2: Validate Dataset Consistency

Check if the generated dataset seeds match game-replays and if the CoT reasoning logic aligns with the mathematical constraints on the board:

python verify_dataset.py

Step 3: Run GRPO Fine-Tuning

Using Unsloth for extremely fast PEFT/LoRA loading in BF16 precision, train the LLM using Group Relative Policy Optimization (GRPO). GRPOTrainer computes rewards on batches of completions using the 12 gameplay scoring criteria and details:

python train_minesweeper_grpo.py

Evaluations run every 50 steps using MinesweeperEvalCallback which plays live validation games on random seeds to plot genuine win-rate developments.

---

🎮 Evaluating & Playing Interactively

You can evaluate trained checkpoints (or GGUF/llama.cpp models) in real-time.

1. Persistent Agent Server (File-Watcher Mode)

The interactive system is split to support fast model inference. Start the server which keeps the LLM loaded in VRAM, watching the directory for input files:

python -m minesweeper.agent_server --config minesweeper_config.yaml

It reads states written to inputs/game_state.json and outputs predictions atomically to outputs/action.json.

2. Launch the Game Runner

Run a series of games and watch the agent play in the terminal. The runner automatically formats gymnasium observations into JSON and pipes them to the server:

# Run 5 games on a 6x6 board with 5 mines, displaying agent play steps
python minesweeper/play_game.py --board-size 6 --mines 5 --games 5 --show

---

📈 Verification Outputs & Stats

The training evaluation loop logs clear metrics showing:

1. Format Success Rate: Verification of the structured JSON syntax matching the schema constraints.
2. Move Safety Index: Ratio of safe cells selected vs. mines hit.
3. Active Win Rate: Recorded percentage of boards solved successfully by the model under 50-move constraints.

Important

To adjust LLM inference speed, maximum output tokens, and stats output, modify values in minesweeper_config.yaml.