Fixing fine-tuning script issues

[alanprior]

Fix #1064 — fine-tuning silently worse than the base model

What & why

Users reported that FinetunedTabPFN{Classifier,Regressor} made performance worse, not better. Root cause was two bugs that made fine-tuning look broken regardless of the actual training, plus example hyperparameters that no longer suit the current default model:

1. Wrong model generation. The fine-tuner was hard-coded to TabPFN V2.5, while a default TabPFNClassifier()/TabPFNRegressor() (the natural baseline, and the examples' "Default"
   print) loads V3. So users were comparing a fine-tuned older model against a newer base — worse by construction.
2. Degraded-weights fallback. With early stopping, if no epoch beat the default model on validation, the wrapper returned the last (degraded) epoch's weights instead of the original
   — i.e. it could hand back a model strictly worse than the base it started from.

After the fixes, fine-tuning is never silently worse than base, and the shipped examples now improve over base:

• Higgs: ROC 0.8211 → 0.8279, log-loss 0.5157 → 0.5057
• California: MSE 0.1388 → 0.1363

Issue

#1064

Motivation and Context

---

Public API Changes

• No Public API changes

---

How Has This Been Tested?

The script now produces improved results, apart from some internal evaluation.

Checklist

• The changes have been tested locally.
• Documentation has been updated (if the public API or usage changes).
• A changelog entry has been added (see changelog/README.md), or "no changelog needed" label requested.
• The code follows the project's style guidelines.
• I have considered the impact of these changes on the public API.

---

[gemini-code-assist]

Code Review

This pull request updates the fine-tuning wrappers to dynamically target the package default model version instead of a hardcoded older version, and adds support for an optional model_version override. It also ensures that if fine-tuning with early stopping does not yield any improvement over the default model, the original base weights are restored. The reviewer identified a potential synchronization issue in distributed data parallel (DDP) training where the default model state is only snapshot on the main process, which could lead to weight desynchronization across ranks if no epoch improves; they suggested seeding the snapshot on all ranks instead.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[psinger-prior]

lgtm!