Reward Hacking of SWE-Bench

[prnake]

Thanks for uploading trajectory, and starting from the finding on https://x.com/xeophon/status/2006969664346501589 I was wondering how many traj was afftected by future commits. I did a quick check with gemini-3-flash and found that about 122/500(24.40%) were affected, which isn't very accurate and may need further manual checking.

Future trajectories: future_trajectories.jsonl.zip

From https://github.com/xiaomimimo/mimo-v2-flash/blob/main/paper.pdf:

[csfjing]

Clarification: Regarding the Performance of IQuest-Coder-V1

We would like to express our sincere gratitude to @xeophon,@zephyr_z9, @jyangballin, and @paradite_ from the community for their rigorous feedback and for bringing the concerns of 1) "future commit" on SWEBench Verified, and 2) the slow inference speed to our attention. Such engagement is vital for ensuring the integrity and transparency of AI evaluation in the open-source ecosystem.

Following recent discussions concerning potential data leakage via "future commits" we conducted a thorough investigation. Here is our analysis, the steps taken to address the issue, and the updated evaluation results.

1. Investigation: Shortcut Learning

Upon reviewing the execution trajectories from our initial submission, the model exhibited patterns of shortcut learning, which was unexpected.

Specifically, the model autonomously utilized git commands (such as git log or git show) to access commit histories that post-dated the issue creation. While our training data was strictly cleaned to exclude these fixes, the inference-time environment provided a loophole—potentially due to outdated Docker images—that the model exploited to "look ahead" (as noted by @jyangballin; see the leakage issue discussions at SWE-bench/SWE-bench#465, and official environment fix PR at SWE-bench/SWE-bench#471).

2. Mitigation and Re-evaluation

Following the official guideline, we use the latest evaluation repository to prevent any unintentional information retrieval during inference (@jyangballin suggests that the latest version of the official repository has fixed the issue).

3. Updated Results

The table below compares our initial reported scores with the results from our re-evaluation:

Metric	Initial Score （old version）	Updated Score (new version)
SWE-bench Verified	81.4	76.2

Note: While the scores reflect an adjustment, they represent a more accurate and robust measure of the model's genuine problem-solving capabilities without shortcuts and environment-side assistance.

4. The Lesson of Robust Evaluation

The "future commit" issue is not only a subtle point that is overlooked by us in evaluation, but an experience of a profound lesson for the team. We believe such an issue of negligence can serve as a valuable case study for the community, and we welcome collaboration to further refine relevant evaluation methodologies together.

5. Positioning: A Coding Specialist, Not a Generalist

IQuest-Coder is designed as a highly focused, specialized model for code—it is an expert, not a generalist. We have aggressively optimized its capabilities for programming competitions and software engineering.

This trade-off is intentional: compared to general-purpose LLMs, IQuest-Coder may lack broader world knowledge and general conversational fluency.

6. Size and "Vibe": Our Path Forward

With 40B parameters, IQuest-Coder is relatively "compact" compared to massive models. We recognize that a gap still exists in "Vibe Coding" and multi-turn dialogue compared to top-tier commercial models.

Regarding community feedback on inference speed: because IQuest-Coder utilizes a dual-inference loop to ensure high-quality outputs, the latency is naturally higher than vanilla single-pass inference. To accelerate the model, we has developed a specialized vLLM PR (vllm-project/vllm#31575). We highly recommend using this version, as it includes specific kernel optimizations designed by our team to significantly boost efficiency.

Furthermore, since the loop architecture exhibits certain incompatibilities with widely used quantization methods (e.g., GPTQ, AWQ) and is still being refined, we currently recommend against applying quantization to the loop model at this stage.

7. Thoroughly Open Source: No Secrets, Just Support

To give back to the community, we are embracing total transparency. We are not holding anything back—we have released not only the 40B Instruct model but also the 40B Base and even the Stage 1 Base versions!

Our goal is to provide a powerful tool for individual developers, researchers, and those who prioritize privacy and local deployment on consumer-grade GPUs. We hope these foundational weights empower the community to build, innovate, and customize IQuest-Coder for their own needs.

Finally, we sincerely apologize for the missing of the fix brought by the update of the SWE-bench repository, which led to the use of deprecated versions of images.

[embedding-shapes]

To give back to the community, we are embracing total transparency. We are not holding anything back—we have released not only the 40B Instruct model but also the 40B Base and even the Stage 1 Base versions!

This is amazing, thank you for doing so! Considering the "We are not holding anything back", I cannot find references to what exact datasets you've used for the various training stages. There are some references like "we curated a massive dataset primarily sourced from Common Crawl" and similar, but "primarily" is doing a lot of heavy lifting, so eagerly looking forward for 100% transparent references to what exact data you've used for the training stages here. Again, thank you for what been released so far, and looking forward to diving deeper into what you've built ❤️

[foldl]

@csfjing Happy new year! A busy and wonderful start of 2026. Are you going to release the 7B & 14B models? 40B is too heavy for GPU-poor.

[bryancnkln]

They created a SOTA model specifically designed for coding in 40B. It’s astonishing that people are complaining about a 40B model when you can’t achieve this level of coding proficiency without using GLM 4.7 which is how much larger? Thank you! 🥇

[prnake]

@foldl Are you planning to release the new 76.2 SWE score trajectories?

[embedding-shapes]

They created a SOTA model specifically designed for coding in 40B. It’s astonishing that people are complaining about a 40B model

People complain about all sorts of stuff, all the time, just ignore them and use what you find is useful? 🤷