Skip to content

REPOSITORIES.md

Latest commit

 

History

History
1007 lines (841 loc) · 53.2 KB

REPOSITORIES.md

File metadata and controls

1007 lines (841 loc) · 53.2 KB
Raw

Repositories — context corpus

This folder is a read-only reference corpus. Every .txt file here is a flat dump of an upstream repository or webpage, kept locally so the pipeline code can be grepped against canonical implementations without round-tripping to the internet. Never edit these files. If a snapshot is stale, replace the whole file with a fresh dump using the same filename, so existing greps in the codebase keep working.

Listed alphabetically below. Where a repo corresponds to a paper in papers/, the paper is linked. Where the file is a fresh dump of a public GitHub repo, the upstream URL is linked.

Overview table

File Lines GitHub Paper What it gives us
Huggingface TabPFN.txt 494 tabpfn_2_5, tabpfn_2_6 Hollmann 2025, Grinsztajn 2026 Primary citation source for checkpoint provenance (synthetic vs. real-finetuned, layer counts, intended limits, licence).
NanoTabPFN.txt 895 automl/nanoTabPFN Pfefferle 2025 Cleanest end-to-end reference of a PFN training loop. Structural template for src/train/.
On Finetuning Tabular Foundation Models.txt 87,914 yandex-research/tabpfn-finetuning Rubachev 2025 Yandex research repo for TabPFNv2 full/PEFT finetuning: experiment configs, reports, LoRA utilities, vendored TabPFN changes, and practical LR/early-stopping recipes.
PFNS.txt 20,743 SamuelGabriel/PFNs Müller 2021 Implementations of every encoder step that runs inside every TabPFN forward pass (NaN handling, normalisation, …). Tells us what sanitize.py should not duplicate.
PFNs4BO.txt 6,488 automl/PFNs4BO Müller 2023 PFN-as-Bayesian-optimisation surrogate. Tangential to credit-risk; useful only if we wrap a PFN around our own HP search.
TabDPT.txt 2,874 layer6ai-labs/TabDPT-inference Ma 2026 Inference code for the real-data-only competitor to TabPFN. Comparison baseline.
TabPFN .txt 63,974 PriorLabs/tabPFN Hollmann 2023, Hollmann 2025, Grinsztajn 2026 Canonical sklearn-style API, all checkpoint metadata, the multi-table finetuning machinery (get_preprocessed_dataset_chunks, DatasetCollectionWithPreprocessing, FinetunedTabPFN*). Primary code reference for src/train/.
TabPFN Client.txt 8,916 PriorLabs/tabpfn-client — Hosted-API HTTP client. Not used in our self-hosted pretraining; only for benchmarking against the API.
TabPFN Docs.txt 7,797 GitHub source removed — refresh manually from docs.priorlabs.ai/overview — The docs.priorlabs.ai source. Documents intent of every config knob; faster to grep than the implementation in TabPFN .txt.
TabPFN Drift-Resilient.txt 17,844 automl/Drift-Resilient_TabPFN Helli 2024 Drift-aware training augmentation. Highly relevant for credit-risk's macro-cycle drift; consider folding into src/train/.
TabPFN Extensions.txt 17,415 PriorLabs/tabpfn-extensions — AutoTabPFN post-hoc ensembling, RF-PFN, embeddings, HPO. Source of evaluation baselines.
TabPFN V2 Finetuning.txt 3,697 PriorLabs/TabPFN/examples Rubachev 2025 The finetune_classifier.py and finetune_regressor.py reference scripts. Canonical "load checkpoint → backward pass → save checkpoint" sequence.
TabPFN Wide.txt 2,388 not-a-feature/TabPFN-Wide Kolberg 2026 Continued-pretraining for extreme-feature-count regimes via a feature-widening prior (it argues against feature reduction; not the source of our agglomeration).
TabTune.txt 134,411 Lexsi-Labs/TabTune Lexsi-Labs 2025 Unified sklearn-style wrapper around the non-TabPFN tabular foundation models (TabICL, OrionMSP/Bix, Mitra, ContextTab, TabDPT, LimiX) plus TabPFNv2.6 native FT, ensembling, distillation, and a TabularLeaderboard. Useful as a future source of additional eval baselines beyond what src/model/ currently wraps; not adopted now because it doesn't natively support Real-TabPFN-style multi-dataset continued pretraining (which is the core training stage of this project). See "Should we use TabTune?" below for the full call.
TransformersCanDoBayesianInference.txt 6,869 SamuelGabriel/PFNs (early, same upstream as PFNS.txt) Müller 2021 Code for the original PFN paper. Mostly historical; useful for explaining what a PFN is.
VSC Documentation.txt 39,358 hpcleuven/VscDocumentation — Full Sphinx source of the VSC supercomputer documentation. SLURM job scripting, Mindwell B200 / wICE H100+A100 partitions, Lustre/GPFS + project-staging storage tiers, account / credit management. The reference when writing the SLURM scripts under scripts/slurm/.

Layout

repositories/
├── REPOSITORIES.md                          (this file)
├── Huggingface TabPFN.txt                   (    494 lines)
├── NanoTabPFN.txt                           (    895 lines)
├── On Finetuning Tabular Foundation Models.txt ( 87,914 lines)
├── PFNS.txt                                 ( 20,743 lines)
├── PFNs4BO.txt                              (  6,488 lines)
├── TabDPT.txt                               (  2,874 lines)
├── TabPFN .txt                              ( 63,974 lines)
├── TabPFN Client.txt                        (  8,916 lines)
├── TabPFN Docs.txt                          (  7,797 lines)
├── TabPFN Drift-Resilient.txt               ( 17,844 lines)
├── TabPFN Extensions.txt                    ( 17,415 lines)
├── TabPFN V2 Finetuning.txt                 (  3,697 lines)
├── TabPFN Wide.txt                          (  2,388 lines)
├── TabTune.txt                              (134,411 lines)
├── TransformersCanDoBayesianInference.txt   (  6,869 lines)
└── VSC Documentation.txt                    ( 39,358 lines)

Huggingface TabPFN.txt

Upstream: HuggingFace model cards for Prior-Labs/tabpfn_2_5 and Prior-Labs/tabpfn_2_6.

Related papers: 2025 — Hollmann et al. — Accurate predictions on small data with a tabular foundation model, 2026 — Grinsztajn et al. — TabPFN-2.5.

What it is. Concatenation of the public HuggingFace model-card READMEs for Prior-Labs/tabpfn_2_5 and Prior-Labs/tabpfn_2_6 (the source URLs are written out as comments inside the file). Each card is included twice in the dump (with and without # Source: headers); the content is identical and any grep against the file will still hit.

Why it matters here. The model cards are the primary published source for which checkpoint is real-finetuned vs. synthetic-only, the layer counts (v2.5 = 18–24, v2.6 = 24), the licence terms, and the citation. Every fact in docs/CHECKPOINTS.md is cross-checked against this file.

Contents in detail.

  • YAML front matter — licence (tabpfn-2.5-license-v1.1 / tabpfn-2.6-license-v1.0), pipeline tag, gated-access fields, thematic tags including finance.
  • Model overview — TabPFN-2.x = transformer-based foundation model, in-context learning, single forward pass.
  • Architecture — v2.5: "TabPFNv2-like alternating attention with 18-24 layers"; v2.6: "TabPFNv2-like alternating attention with 24 layers".
  • Training data and priors —
    • v2.5: "TabPFN-2.5: trained purely on synthetic tabular tasks / Real-TabPFN-2.5: continued pre-training on real-world datasets (for details please see Appendix C.1 of the model tech report)."
    • v2.6: "TabPFN-2.6 is trained purely on synthetic tabular tasks." (No real-finetuned variant. Decisive evidence that v2.6 default is the synthetic-only base.)
  • The complete v2.5 checkpoint catalogue with one-line descriptions per checkpoint, identical to what you'll find inside TabPFN .txt:736-751. The 🌍 emoji marks the real-finetuned variants. v2.6 has only the _default checkpoints listed.
  • Intended use / limitations — ≤ 50,000 samples and ≤ 2,000 features; not for unstructured data.
  • Licensing — research-only with an enterprise option for commercial use.

When to grep this file: when you need a primary citation for any checkpoint provenance claim (v2.5/v2.6, real vs synthetic, intended sample/feature limits). For pipeline implementation, prefer TabPFN .txt (more detailed) or TabPFN Docs.txt (more recent prose).

NanoTabPFN.txt

Upstream: github.com/automl/nanoTabPFN.

Related paper: 2025 — Pfefferle et al. — nanoTabPFN: A Lightweight and Educational Reimplementation of TabPFN.

What it is. A minimal reference implementation of TabPFN — the "how-it-works-in-under-1000-lines" educational version, trimmed of the production package's ergonomics (sklearn API surface, ensembles, distributed training, autocast handling, multiple checkpoints) so that the core training and inference loop is visible at a glance.

Why it's the highest-signal file in this folder for our work: Real-TabPFN's source is not public, so the closest thing we have to "a continued-pretraining loop in a few hundred lines that we can read end-to-end" lives here.

Contents in detail:

  • Lines ~150–260 — Data loading and preprocessing. Defines get_feature_preprocessor(X) which fits a ColumnTransformer that separates numerical from categorical columns by checking, for each column, whether pd.to_numeric(errors='coerce').notna().sum() equals the non-NaN count. Numerical columns get coerced to numeric arrays; categoricals get an OrdinalEncoder(handle_unknown= 'use_encoded_value', unknown_value=np.nan). Constant columns (≤ 1 unique non-NaN value) are dropped. This is the minimum feature preprocessing TabPFN inputs need, and it matches the rules we are putting into src/data/sanitize.py.
  • get_openml_datasets(...) — illustrates the OpenML download path for evaluation (TabArena task IDs hardcoded), with stratified subsampling via train_test_split(stratify=y).
  • Lines ~520–700 — Model definition. The full TabPFN-style alternating-attention transformer in pure PyTorch: alternating attention "between samples" and "between features" within the same layer, plus a target embedding head. forward(src, train_test_split_index) shows exactly how the context-vs-query split is consumed: the model sees one tensor of shape (B, N, F) and an integer index that says "rows [0:split_idx) are context with labels, rows [split_idx:] are query with masked labels".
  • Lines 758–832 — Training loop. The pretraining loop in 70 lines: schedulefree.AdamWScheduleFree, learning rate 4e-3, cross-entropy loss reshaped to (B*N_query, n_classes), gradient-norm clip at 1.0, periodic eval. This is the structural reference for src/train/train.py once we get there.
  • Lines 835–880 — PriorDumpDataLoader. Loads pre-baked synthetic prior datasets from an HDF5 file. Fields: X (B, N_max, F_max), y (B, N_max), num_features, num_datapoints, single_eval_pos (= train/test split index), max_num_classes. Padding-aware: each batch slices to the per-batch max feature count and max sequence length. Reference format only — since the 2026-05-20 refactor our training pipeline reads sanitized CSVs from data/processed/ directly and assembles the same (X_context, y_context, X_query, y_query) quadruple on the fly inside src/train/dataloader.py::_build_step_batch (with a fresh random subsample every epoch). No .npz cache step.

When to grep this file: any time you need the "what does the training loop actually look like" answer — model forward semantics, loss shape, gradient handling, optimizer setup, batch layout.

On Finetuning Tabular Foundation Models.txt

Upstream: github.com/yandex-research/tabpfn-finetuning.

Related paper: 2025 - Rubachev et al. - On Finetuning Tabular Foundation Models.

What it is. A Yandex Research code dump for the Rubachev et al. TabPFNv2 finetuning study. It is much larger than the Prior Labs example scripts because it includes experiment grids, tuned configs, result reports, RTDL-style data/model utilities, LoRA utilities, and a vendored/modified TabPFN implementation.

Why it matters here. The paper's main practical conclusion is that full finetuning is the strongest and simplest TabPFNv2 adaptation baseline: it converges quickly, tends to beat PEFT variants, and can be run on datasets up to roughly 50K rows on an 80GB GPU. The mechanism result is also directly relevant to CreditPFN: finetuning improves the alignment between test-row query representations and in-context training-row key representations, so attention weights better track target similarity. For our project, this repo is the best public source for the LR grid, patience, prediction length, and A100 memory assumptions behind single-dataset gradient adaptation.

Two facts that frame comparisons against it: (1) the 342 archived report.json grid runs inspected below are full-FT runs, but the paper also reports LoRA and other partial/PEFT ablations. Do not turn the contents of that one result directory into a claim that the paper did not evaluate LoRA. (2) each run adapts to one target table. Multi-dataset corpus continued pretraining is Garg et al.'s Real-TabPFN setting and CreditPFN's setting, not Rubachev's experimental design.

Contents in detail:

  • README.md around line 1862 - project overview, arXiv link (2506.08982), uv setup, checkpoint download instructions, and the headline finding: full finetuning is the practical default for TabPFNv2. The README BibTeX still says arXiv:2024.08982; treat the paper text and arXiv link (2506.08982, 2025) as canonical.
  • pyproject.toml around line 1946 - Python pinned to ==3.11.11, with CUDA 12.4 extras and dependencies including PyTorch, CatBoost, XGBoost, Optuna, RTDL utilities, and loralib.
  • exp/full-finetune/*/*.toml - reproducible experiment grids. The adult config at line 2212 calls bin.tabpfnv2_finetune.main, uses n_trials = 10, brute-force learning rates from about 5e-6 to 5e-4, batch_size = 1, epoch_size = 10, seq_len_pred = 1024, and finetune_mode = "full".
  • report.json / summary.json blocks - practical run reports: A100-SXM4-80GB hardware, roughly 7.2M or 11.1M trainable parameters depending on the config, early-stopped best steps, and zero-shot vs. finetuned metrics.
  • lib/data.py - expects arrays such as X_num.npy, X_bin.npy, X_cat.npy, Y.npy, and split-default/{train,val,test}_idx.npy. Useful when designing future adapters between our .npz cache and RTDL-style tabular loaders.
  • lib/deep.py - RTDL numerical embedding utilities such as piecewise-linear embeddings, one-hot helpers, optimization helpers, and parameter-count utilities.
  • lib/tabpfn/lora_utils.py around line 85367 - LoRA injection helpers for TabPFN internals, including replacements for MHA, linear layers, and embeddings. This is the local reference if we later benchmark LoRA against full continued pretraining.
  • lib/tabpfn/preprocessing.py around line 85798 - vendored TabPFN preprocessing configs (none, numeric, onehot, ordinal, ordinal_shuffled, etc.) and the package-level feature preprocessing choices.

Verified config facts (aggregated across all 342 report.json runs, 2026-06-23) — and how CreditPFN deliberately differs. These are the reference values; our deviations are intentional and noted:

Knob Reference (all 342 runs) CreditPFN Why we differ
optimizer AdamW AdamW match
weight_decay 0.0 0.0 (was 0.01; fixed 2026-06-23) match — decay-to-origin fights the prior
LR tuned 5e-6…5e-4, median ~3.9e-5 grid {3e-7, 1e-6, 1e-5, 3e-5} low rung matches Garg; high rung approaches Rubachev's median
LR schedule none (constant) warmup→cosine deliberate; consider a constant rung
L2-SP / anchor not used λ=0.003 (full-FT only) follows Garg's Real-TabPFN corpus-CPT recipe, not Rubachev
batch_size 1 1 match
epoch policy n_epochs=-1 + early stop (patience 16) fixed 50 epochs + divergence-abort deliberate (val-noise with ~10 datasets)
ensemble/step clf 2 / reg 8 (official wrappers) PD 2 / LGD 8 match
loss CE (clf) / bar-dist NLL (reg) same match

The main optimization lesson from this repository is that Rubachev uses weight_decay=0.0; stacking decay-to-origin on an L2-SP anchor would be a separate, untested regularization choice. This says nothing about Garg's AdamW weight decay, which the Real-TabPFN paper does not report. Garg does explicitly use L2-SP λ=0.003, warmup→cosine, and LR 3e-7. Keep the two papers separate. (The README body is unrecoverable from this dump — a charmap codec error replaced it — so prose claims must be read from the PDF, not this .txt.)

Important caveat. The dump references bin.tabpfnv2_finetune.main, but this .txt snapshot contains no FILE: bin/... sections. Use it as a high-signal reference for configs, results, LoRA/vendored TabPFN changes, and paper-grounded hyperparameters; do not assume the local dump is a complete runnable clone.

When to grep this file: when choosing single-dataset finetuning hyperparameters, comparing full finetuning vs. LoRA/prefix variants, or checking how Rubachev et al. structure their data arrays, tuning reports, and TabPFNv2 internals.

PFNS.txt

Upstream: github.com/SamuelGabriel/PFNs (formerly automl/PFNs; the old slug no longer resolves directly).

Related papers: 2021 — Müller et al. — Transformers Can Do Bayesian Inference (the foundational PFN framework that this code implements).

What it is. The full Prior-Labs PFNs repo — the underlying "Prior-fitted Network" framework that TabPFN is built on top of. Contains the canonical implementations of the internal encoder steps that run inside every TabPFN forward pass.

Why it matters here: It tells us what sanitize.py should not duplicate. The model already handles a lot of preprocessing internally; if our pipeline does it a second time we either waste work or (worse) double-transform features in ways that drift the distribution off the model's training prior.

Contents in detail:

  • Lines ~5700–5800 — Encoder-step composition. Shows how the standard TabPFN-2.x model assembles its input encoder as a sequence:
    • NanHandlingEncoderStep — handles missing values (emits an explicit indicator and replaces NaN with a learned default).
    • LinearInputEncoderStep — linear projection of features into embedding space.
    • VariableNumFeaturesEncoderStep — pads/shuffles the feature dimension so a model trained on F-max features can ingest any F ≤ F-max.
    • ConstantNormalizationInputEncoderStep and InputNormalizationEncoderStep — per-column normalisation fitted on the context rows only and applied to query rows.
  • Lines ~6048–6310 — Each encoder step's full implementation.
    • LinearInputEncoderStep (line 6048).
    • ConstantNormalizationInputEncoderStep (6108).
    • NanHandlingEncoderStep (6143): replaces NaN with nan_indicator, +inf with inf_indicator, -inf with neg_inf_indicator, then concatenates a binary "was-this-NaN" flag along the feature dimension. Confirms that our sanitize step (h) — replace ±inf with NaN — is correct.
    • VariableNumFeaturesEncoderStep (6211).
    • InputNormalizationEncoderStep (6290) — per-column mean/std computed on the context split only.
  • Lines ~16335–16415 — Standalone preprocessing transforms that run before the model, as ensemble members at inference time: PowerTransformer(method='yeo-johnson') / PowerTransformer(method='box-cox') / QuantileTransformer(output_distribution='normal') / RobustScaler(unit_variance=True). These are inference-time ensemble preprocessing, not training-time preprocessing.

When to grep this file: to confirm whether something is the model's internal job or our pipeline's job; to look up what the encoder steps actually do to NaNs / infs / constants; to see the canonical ensemble of preprocessing options at inference.

PFNs4BO.txt

Upstream: Same repo as PFNs (BO branch / github.com/automl/PFNs4BO).

Related paper: 2023 — Müller et al. — PFNs4BO: In-Context Learning for Bayesian Optimization.

What it is. PFNs adapted to Bayesian optimisation. Implements an acquisition function on top of a PFN posterior — useful for hyperparameter search but unrelated to credit-risk pretraining.

When to grep this file: only if you later want to use a PFN as a surrogate model for tuning your own continued-pretraining hyperparameters. No relevant preprocessing or training-loop machinery beyond what's already in PFNS.txt.

TabDPT.txt

Upstream: github.com/layer6ai-labs/TabDPT-inference (inference code; full training code is in the sibling repo layer6ai-labs/TabDPT-training).

Related paper: 2026 — Ma et al. — TabDPT: Scaling Tabular Foundation Models on Real Data.

What it is. TabDPT is the real-data-only counterpart to TabPFN: a transformer trained on real tables sampled from OpenML via retrieval-augmented self-supervision, with no synthetic prior. The dump in this folder is the inference repo — load weights from HuggingFace and predict on a new dataset via a sklearn-style API.

Why it matters. TabDPT and TabPFN bracket the synthetic-vs-real spectrum from opposite ends. Real-TabPFN (and hence CreditPFN) sits in the middle. Having TabDPT locally lets us include it as an inference baseline in the eval harness: compare CreditPFN's scores against TabPFN-2.6, TabDPT, TabICL, and the published Real-TabPFN-2.5 weights, all in one place.

Contents in detail.

  • src/tabdpt/classifier.py, regressor.py, estimator.py, model.py, utils.py — the inference path.
  • tabdpt_datasets/openml.py — OpenML dataset loaders that they used at training time and ship for reproducibility.
  • tabdpt_datasets/data_splits/{cls,reg}_datasets.csv — the exact CSV manifest of which OpenML datasets they trained on. Useful for contamination checking — any dataset on this list must NOT appear in our held-out evaluation set if we want a fair comparison.
  • tests/cls_example.py, reg_example.py — minimum working example.

When to grep this file: when implementing the TabDPT baseline in src/eval/, or when checking that our credit-risk held-out splits don't overlap with TabDPT's training corpus.

TabPFN .txt

Upstream: github.com/PriorLabs/tabPFN (the main tabpfn Python package).

Related papers: 2023 — Hollmann et al. — TabPFN, 2025 — Hollmann et al. — Accurate predictions on small data, 2026 — Grinsztajn et al. — TabPFN-2.5.

What it is. The user-facing sklearn-style API (TabPFNClassifier, TabPFNRegressor), the checkpoint-loading infrastructure, the inference-time ensembling, and the documented list of every released .ckpt and what it's good for. Plus the finetuning wrappers (FinetunedTabPFNClassifier / FinetunedTabPFNRegressor) and the multi-table machinery (get_preprocessed_dataset_chunks, DatasetCollectionWithPreprocessing, fit_from_preprocessed) that our src/train/ will compose on.

Why it matters: this is the single source of truth for checkpoint provenance (which files are synthetic-only, which are real-finetuned), the public configuration knobs (PreprocessorConfig, ModelInterfaceConfig), and the supported input shapes / dtypes / NaN handling at the API boundary.

Contents in detail:

  • Lines 649–650 — README block listing the default v2.5 classifier and regressor checkpoint URLs.
  • Lines 736–751 — The complete TabPFN-2.5 checkpoint catalogue with one-line descriptions: which is real-finetuned (🌍 emoji), which is synthetic, which specialises for "large features" (large-features-L up to 500 features, large-features-XL up to 1000), which for "large samples" (>30K), which for low-skew regression targets, etc. This is what docs/CHECKPOINTS.md is built from — when in doubt, this section of TabPFN .txt is ground truth.
  • Lines 2606–2628 — Programmatic checkpoint name registry used inside the package to validate model_path arguments.
  • Lines 6952–6985 — tabpfn-v2- (v2.0) checkpoint registry for the older v2.0 model family.
  • Lines 17301–17400 — DatasetCollectionWithPreprocessing, the torch.utils.data.Dataset subclass that lazily preprocesses each dataset on __getitem__, returning a ClassifierBatch or RegressorBatch.
  • Lines 17702–17761 — shuffle_and_chunk_data, TabPFN's own per-dataset row sub-sampler. Stratified for multiclass, non-stratified for regression. Reference behaviour: our src/train/dataloader.py::_build_step_batch performs the same shape (subsample → 80/20 ctx/qry split → ordinal-encode) on the fly per epoch, against the sanitized CSV directly.
  • Lines 17764–17881 — get_preprocessed_dataset_chunks, the helper that accepts a list of datasets and produces a DatasetCollectionWithPreprocessing ready for a multi-table training loop.
  • Lines 18062, 18999, 19616 — FinetunedTabPFNBase, FinetunedTabPFNClassifier, FinetunedTabPFNRegressor — the official sklearn-compatible finetuning wrappers.
  • PreprocessorConfig definitions — exposes name='none' | 'safepower' | 'quantile_uni_coarse' | 'quantile_uni' | 'robust_scaler' | …. These are inference-time ensemble names.
  • save_tabpfn_model utility (around line 710) — the reverse direction: how to dump a model object back to a .ckpt. PR #930 (changelog ~line 435) fixed it to set architecture_name="tabpfn_v3" for v3 configs and to persist inference_config_, which is exactly the gap our own save_finetuned had to close (see the checkpoint-format note below).
  • Architecture registry (architectures/tabpfn_v2_6.py, tabpfn_v3.py; inference_config.py) — the architecture_name strings the loader recognises and the InferenceConfig schema (pydantic, extra="forbid").

The 4-key checkpoint contract (critical for src/train/model.py). A TabPFN checkpoint is a single torch.save dict with exactly four keys: state_dict, config (asdict of the pydantic ArchitectureConfig), architecture_name, and inference_config (asdict of InferenceConfig). Valid architecture_name values are tabpfn_v2, tabpfn_v2_5, tabpfn_v2_6, tabpfn_v3 (a missing key defaults to tabpfn_v2; the legacy "base" string remaps to tabpfn_v2_5). Our save_finetuned writes all four keys; the only robustness gap is that the architecture_name fallback leans on the private upstream symbol _resolve_architecture_name (hardened, and src/train/model.py now RAISES rather than mislabelling an unknown architecture as "base" — 2026-06-23 audit fix). The weights_only=False load patch in src/model/tabpfn_models.py is necessary because PyTorch ≥ 2.6 rejects the embedded pydantic config objects by default.

v2.6-vs-v3 criterion handling (regressors). v3's model.forward takes test_targets_MB, so the loader STRIPS the criterion.* keys and rebuilds the bar-distribution from the model's regression_borders buffer; v2.6 has no test_targets_MB, so the loader REQUIRES the criterion.* keys. Our save_finetuned writes criterion.* for regressors — required by v2.6, harmlessly stripped by v3 — so a saved regressor checkpoint round-trips correctly for both architectures.

When to grep this file: checkpoint names, the 4-key checkpoint schema, public API surface, inference-time preprocessing names, validation / error messages the package raises, the multi-table finetuning machinery.

TabPFN Client.txt

Upstream: github.com/PriorLabs/tabpfn-client.

Related paper: none.

What it is. The HTTP client for Prior Labs' hosted inference API.

When to grep this file: only if you're benchmarking against the hosted API. Not relevant for our self-hosted pretraining workflow on VSC.

TabPFN Docs.txt

Upstream: docs.priorlabs.ai/overview. Prior Labs no longer publishes the docs GitHub source, so this file cannot be refreshed by src/utils/refresh_repositories.py. It's listed in SKIP_NON_GIT and must be refreshed by hand (copy the relevant pages from the live docs site) when the on-disk snapshot goes stale.

Related papers: indirectly all of the TabPFN papers — the docs sit on top of the package described above.

What it is. Flat dump of the TabPFN documentation repository — the Markdown sources that the docs.priorlabs.ai site is built from. ~7,800 lines covering every documented capability, hyperparameter, integration, and fine-tuning recipe.

Why it's the highest-signal recent addition: it contains the official documentation of TabPFN's preprocessing knobs, the fine-tuning wrappers shipped inside the package, and the design intent behind every configurable parameter. For Stage 1–4 design, this is the most authoritative non-code reference.

Contents in detail (with line numbers worth bookmarking):

  • overview.mdx (around lines 50–200) — what TabPFN is and what its capabilities are.
  • models.mdx (around lines 1100–1180) — version comparison table.
  • improving-performance/preprocessing.mdx (lines 6331–6404) — most important section for our Stage 3 (sanitize.py) design. Documents:
    • PREPROCESS_TRANSFORMS: ensemble preprocessing names ("quantile_uni", "squashing_scaler_default", "safepower", "quantile_uni_coarse", "kdi", "robust", "none").
    • categorical_name: encoding options.
    • max_features_per_estimator: default 500.
    • REGRESSION_Y_PREPROCESS_TRANSFORMS: target transforms for regression ("none", "safepower", "quantile_norm", "quantile_uni", "1_plus_log").
    • OUTLIER_REMOVAL_STD: default "auto" which resolves to 12.0 for classification and None for regression — see the "Outlier handling" section below.
    • POLYNOMIAL_FEATURES, FINGERPRINT_FEATURE, SUBSAMPLE_SAMPLES. None of these belong in config/data.yaml — they are inference-time levers.
  • capabilities/fine-tuning.mdx (lines 4224–4450) — documentation of the official FinetunedTabPFNClassifier and FinetunedTabPFNRegressor wrappers. Important caveat at line 4246: "The fine-tuning process decouples the preprocessing pipeline to generate transformed tensors that mirror the preprocessing configurations used during inference, ensuring the model optimizes on the exact same data variations it encounters when making predictions."
  • improving-performance/feature-engineering.mdx, feature-selection.mdx, model-parameters.mdx — softmax temperature, balanced-probability handling, imbalance handling.
  • extensions/*.mdx — hpo, many-class, post-hoc-ensembles, rf-pfn. Reference for the evaluation protocol later.
  • api-reference/*.mdx — hosted-API only.
  • integrations/*.mdx — Databricks, Azure Foundry, SageMaker, MLflow, n8n.
  • use-cases/*.mdx — including finance.mdx.

When to grep this file: for the intent and contract of any documented TabPFN configuration option. Faster than grepping TabPFN .txt (which is the implementation) when you just need "what does this parameter do".

TabPFN Drift-Resilient.txt

Upstream: github.com/automl/Drift-Resilient_TabPFN.

Related paper: 2024 — Helli et al. — Drift-Resilient TabPFN.

What it is. A research repo that fine-tunes / specialises TabPFN for distribution-shift robustness — explicitly modelling "training distribution drifts at inference time" rather than assuming i.i.d.

Why it's interesting for credit risk specifically: credit-risk data is defined by macroeconomic regime drift. PD and LGD distributions shift dramatically between expansion and recession. A model that's been continued-pretrained on a credit-risk corpus might benefit from drift-aware training-time augmentations borrowed from this repo. Not a Stage 1–4 concern, but worth grepping during the training-loop design.

Contents in detail: drift-aware loss formulations, distribution-shift simulation as a training-time augmentation, extra evaluation protocols (eval on artificially shifted test sets).

When to grep this file: when designing training-time augmentations or evaluation under regime shift.

TabPFN Extensions.txt

Upstream: github.com/PriorLabs/tabpfn-extensions.

Related paper: none directly; references the AutoTabPFN ensemble idea documented across multiple TabPFN papers.

What it is. tabpfn-extensions — official add-on package (post-hoc ensembling, RF-PFN hybrids, embeddings, hyperparameter search via OpenAutoML, a "many-class classifier" wrapper for >10 target classes, a fingerprint-feature tool, etc.).

Why it's relevant: at evaluation time, the standard reporting package compares plain TabPFN vs. AutoTabPFN (post-hoc ensemble). We will probably compare our CreditPFN against both, so the ensemble mechanics matter for fair comparison.

Contents in detail: post-hoc ensemble definition (uses AutoGluon under the hood), RF-PFN tree-based hybrid (rf_pfn extension — good baseline since classical PD/LGD modelling traditionally uses gradient-boosted trees and random forests).

When to grep this file: when designing the evaluation protocol or building strong baselines.

TabPFN V2 Finetuning.txt

Upstream: github.com/PriorLabs/TabPFN/tree/main/examples (the finetune_classifier.py and finetune_regressor.py example scripts).

Related paper: 2025 — Rubachev et al. — On Finetuning Tabular Foundation Models.

What it is. Sebastian Pineda's TabPFN-V2-Finetuning recipe — the closest public analog of Real-TabPFN's continued-pretraining, but with one key difference: this is dataset-specific finetuning ("finetune TabPFN-v2 to one downstream dataset") rather than continued pretraining on a whole corpus.

Why it's a critical reference: it shows the exact mechanics of loading a v2 checkpoint, making a forward/backward pass through it, and saving the result — which is the unit-of-work for continued pretraining too. Just call the same loop in a sweep over many datasets.

Official wrapper defaults (verified, lines 183–188 / 339–348). These are the upstream FinetunedTabPFN* settings our train.yaml grid is calibrated against:

Wrapper epochs lr n_estimators_finetune ctx+query samples
FinetunedTabPFNClassifier 30 2e-5 2 —
FinetunedTabPFNRegressor 30 1e-5 8 n_finetune_ctx_plus_query_samples = 20_000

CreditPFN uses n_estimators_finetune = 2 on both tracks (the reference regressor uses 8 — a candidate to revisit for LGD) and a fixed-50-epoch schedule with divergence-abort instead of the 30-epoch default.

Contents in detail:

  • Lines ~85, 367, 418, 718 — save_path_to_fine_tuned_model = "./fine_tuned_model.ckpt" and surrounding torch.save({"state_dict": …, "optimizer_state": …, "scheduler_state": …}, path). Note this is a training-state save (weights + optimizer + scheduler, for resuming a run); it is not the 4-key inference checkpoint (state_dict, config, architecture_name, inference_config) that the package loader consumes and that our save_finetuned writes — see the TabPFN .txt section for that contract.
  • Lines ~407–441 — from tabpfn.config import ModelInterfaceConfig, PreprocessorConfig and the no_preprocessing_inference_config pattern.
  • Lines 468, 562 — OrdinalEncoder(handle_unknown= 'use_encoded_value', unknown_value=-1). Confirms the categorical encoding contract.
  • Lines 538–700 — preprocess_dummy_data(...) end-to-end: load OpenML task → split → fit OrdinalEncoder on train → transform query → cast to torch tensors → device placement.
  • Loss / backward / optimizer setup (search for loss.backward).

When to grep this file: for the canonical "load checkpoint → attach optimizer → forward pass → backward pass → save checkpoint" sequence for real TabPFN-2 weights, not the toy model.

TabPFN Wide.txt

Upstream: github.com/not-a-feature/TabPFN-Wide (the canonical repo for the TabPFN-Wide paper, Kolberg et al. 2026).

Related paper: 2026 — Kolberg et al. — TabPFN-Wide.

What it is. TabPFN-Wide — modifications for high-dimensional inputs, e.g. multi-omics or wide credit-bureau datasets with hundreds to thousands of columns.

Why it matters: several credit datasets in our raw corpus already have >100 columns, so we need a story for wide data. TabPFN-Wide's answer is the opposite of ours: instead of reducing features, it widens the synthetic prior so the model natively handles hundreds-to-thousands of columns, and it uses FeatureAgglomeration only as a baseline it compares against, not as its method. Our sanitize.py step is unsupervised feature selection (keep real columns) — an independent, pragmatic cap; useful to read this repo for how the widening alternative works, not as the source of our design.

Contents in detail:

  • Lines ~170–890 — Multiple load_state_dict patterns for loading both the standard TabPFN checkpoint and Wide-modified checkpoints.
  • Lines 1746–1801 — load_checkpoint(self) method showing how the Wide trainer loads a training-state checkpoint (state_dict + optimizer state + scheduler state). Template for resumable training in our SLURM jobs.
  • Lines 2244, 2294, 2312 — FeatureAgglomeration usage. Two variants: FeatureAgglomeration(n_clusters=n_features) (default Euclidean+Ward) at line 2294, and FeatureAgglomeration( n_clusters=n_features, metric='precomputed', linkage='complete') at line 2312. Confirms the idiom we adopt for sanitize step (i).

When to grep this file: dimensionality-reduction strategies, checkpoint resumption, anything wide/high-feature.

TabTune.txt

Upstream: github.com/Lexsi-Labs/TabTune.

Related paper: arXiv:2511.02802 — Lexsi-Labs 2025 — TabTune: A Unified Library for Inference and Fine-Tuning of Tabular Foundation Models.

What it is. A unified, sklearn-style wrapper that exposes ~10 recent tabular foundation models — TabPFN-v2 / -v2.6, TabICL, TabICLv2, OrionMSP v1.0 / v1.5, OrionBix, Mitra, ContextTab, TabDPT, LimiX — under a single TabularPipeline.fit() / .predict() / .evaluate() / .save() / .load() API. Built on three components: DataProcessor (model-aware preprocessing), TuningManager (per-model training strategy), TabularPipeline (end-to-end driver), plus a TabularLeaderboard for head-to-head comparison. Each model gets four tuning strategies: zero-shot inference, episodic meta-learning FT (the standard way to finetune ICL models), supervised FT, and PEFT/LoRA. Recent additions: 6-strategy ensembling (TabularEnsemble), distillation (TabDistiller — TFM → MLP / LightGBM / XGBoost / CatBoost), and TabPFNv2.6 native FT (the official Prior Labs FinetunedTabPFN* API, surfaced through TabTune's unified pipeline).

Should we use TabTune? No, not for the training stage; possibly for the eval stage in the future. Reasoning:

  1. Training stage doesn't fit. Real-TabPFN-style continued pretraining iterates one transformer over a corpus of datasets — an outer loop over many parents, each contributing context/ query batches. TabTune's pipelines are per-dataset: one TabularPipeline.fit() call ↔ one dataset. Adopting TabTune for training would replace our cross-dataset loop with a series of per-dataset finetunes, which is a different research question. Our src/train/loop.py already implements the cross-dataset variant — reading sanitized CSVs directly via ProcessedDatasetLoader and drawing a fresh random subsample per epoch; no change needed.

  2. Eval stage is potentially attractive. The user's plan compares TabPFN variants against XGBoost/CatBoost/LogReg/LinReg. TabTune would give us the option to also compare against TabICL, OrionMSP, Mitra, ContextTab, TabDPT, and LimiX with a single library — much cheaper than wrapping each one ourselves. The TabularLeaderboard API in particular looks like a near-drop-in alternative to our src/eval/benchmark.py.

  3. Cost. TabTune pulls in every model's runtime as a dependency — torch, transformers, several specific repo wheels. That's a non-trivial install footprint on VSC. Worth doing only if the thesis / paper actually compares against ≥ 3 of the non-TabPFN foundation models.

Recommendation: keep TabTune in the repo for reference, do not adopt it now. Revisit when the eval baseline list grows beyond {XGBoost, CatBoost, LogReg/LinReg, TabPFN-untuned, TabPFN-trained}. Specifically: if we add TabICL or any non-TabPFN foundation model as a comparison, swap our hand-written wrapper for TabTune at that point rather than building the 4th wrapper from scratch.

When to grep this file: when designing a new eval baseline that would otherwise need its own model-specific wrapper. The TuningManager and DataProcessor source code in particular shows how each non-TabPFN model expects its inputs preprocessed (integer-encoded categoricals for TabPFN vs. text-tokenised columns for ContextTab vs. column-attention masks for TabICL, …).

TransformersCanDoBayesianInference.txt

Upstream: github.com/SamuelGabriel/PFNs (early snapshot of the same repo PFNS.txt covers; the repo moved from automl/PFNs to its current location).

Related paper: 2021 — Müller et al. — Transformers Can Do Bayesian Inference.

What it is. Code accompanying Müller et al. 2021 — the original PFN paper. Mostly historical context for understanding what a "Prior-fitted Network" is: a transformer trained to perform posterior inference for a particular Bayesian prior, by sampling synthetic datasets from that prior and training the model to map context → query predictions.

When to grep this file: when you need to write a paragraph explaining PFNs in your thesis / a defence / a report. Not relevant for pipeline implementation.

VSC Documentation.txt

Upstream: github.com/hpcleuven/VscDocumentation (the source of the VSC documentation site).

Related paper: none — this is supercomputer infrastructure documentation, not a research artefact.

What it is. Flat dump of the entire Sphinx-generated VSC (Vlaams Supercomputer Centrum / Flemish Supercomputer Centre) user documentation. ~39,000 lines of .rst source covering account management, SSH/MFA setup, Genius / wICE / Mindwell cluster hardware, SLURM job scripting, storage tiers, scientific software modules, and the various ways to acknowledge the VSC.

Why it matters here. Continued pretraining runs on the Mindwell B200 cluster; data prep and eval run on wICE. When writing the SLURM scripts under scripts/slurm/ — partitions, GPU allocation, walltimes, the Lustre/GPFS storage split — this dump is the canonical reference. Verified facts (line numbers as of the 2026-06-23 dump):

  • Mindwell (lines 37570–37642): partition gpu_b200, 3 nodes × 8 = 24 B200 GPUs, 192 GiB VRAM each, AMD EPYC Turin hosts; max 24 cores + ~190 GiB CPU mem per GPU. Account lp_mindwell_pilot (free pilot). --clusters=mindwell. GPFS scratch.
  • wICE (lines 37959–38084): gpu_h100 (5×4 = 20 H100, 80 GiB, 16 cores/GPU) and gpu_a100 (4×4 = 16 A100, 80 GiB, 18 cores/GPU). Lustre scratch. CPU batch partition for data prep.
  • Storage (lines 37357–37873): $VSC_HOME 3 GiB / $VSC_DATA 75 GiB (both NFS, backed up, all-cluster) / $VSC_SCRATCH 500 GiB (Lustre on wICE+Genius, GPFS on Mindwell; purged after 30 days — and mv/rsync -a don't refresh atime, so stage with cp). Project ("staging") storage lives under $VSC_PROJECT_LUSTRE1 (stg_XXXXX, Lustre) and $VSC_PROJECT_GPFS1 (Mindwell, GPFS) — single-copy, not guaranteed backed up. Hard rule: Mindwell jobs must use GPFS, wICE/Genius jobs must use Lustre for sustained I/O.
  • Scheduling (lines 13343–14243, 35463–35563): --clusters= is mandatory; GPU walltime caps at 72 h (no gpu_*_long); shorter walltime ⇒ better backfill priority; sam-balance -A <acct> checks credits (B200 437.5 / H100 569.4 / A100 141.7 per GPU-min).
  • Cross-cluster: --dependency (afterok) across -M clusters is undocumented/unverified — we treat it as unsupported and submit data (wICE) / train (Mindwell) / eval (wICE) independently, sequenced through the shared staging tier.

When to grep this file: writing/updating a SLURM script, debugging a job-submission error, choosing a partition or GPU type, or sizing storage. See docs/VSC_GUIDE.md for the distilled deployment recipe.

TabPFN V2 Finetuning.txt ↔ On Finetuning Tabular Foundation Models.txt ↔ NanoTabPFN.txt ↔ TabPFN Wide.txt — how they relate

NanoTabPFN Wide V2 Finetuning On Finetuning
Model definition toy reimpl full + wide modifications uses real package vendored/modified package
Loads real .ckpt? no (trains from scratch) yes yes yes
Has training loop? yes (synthetic prior) yes (sweep over real datasets) yes (single-dataset finetune) configs/results for single-dataset finetune
Closest to our use case training-loop structure dataset-sweep structure checkpoint mechanics hyperparameters + PEFT/full-finetune evidence

We will end up combining all four: V2-Finetuning's checkpoint loading + On-Finetuning's hyperparameter evidence + Wide's resumable training-state pattern + NanoTabPFN's clear training-loop scaffolding, applied to a multi-dataset corpus in the Real-TabPFN spirit.

Outlier handling: what TabPFN actually does (verified)

This is important enough to factor out into its own subsection, because it directly determines how sanitize.py should treat extreme values. The implementation lives in TabPFN .txt:15795-15828 (function remove_outliers) and TabPFN .txt:6273-6314 (the public knob OUTLIER_REMOVAL_STD).

# TabPFN .txt:6273-6274
_REGRESSION_DEFAULT_OUTLIER_REMOVAL_STD: float | None = None
_CLASSIFICATION_DEFAULT_OUTLIER_REMOVAL_STD: float = 12.0
# TabPFN .txt:15795 (paraphrased)
def remove_outliers(X, n_sigma=4, normalize_positions=-1, ...):
    # 1. Compute per-column mean/std using ONLY the context split.
    # 2. Mark cells outside [mean ± n_sigma·std] as NaN.
    # 3. Re-compute mean/std from the now-cleaned data.
    # 4. Re-derive the [lower, upper] bounds from those robust stats.
    # 5. Apply a SOFT log-squash (not hard clip):
    X = max(-log(1+|X|) + lower, X)
    X = min( log(1+|X|) + upper, X)

Three takeaways for our pipeline:

  1. TabPFN's z-score normalization is not a substitute for outlier handling. Z-scoring is sensitive to the very outliers it's trying to normalise — a single 1e9 value pins the mean and std to ~1e9 and crushes everything else to ~0. That's why the package itself ships an outlier removal step before / alongside the normalization step.
  2. The default threshold is 12σ for classification, and outlier removal is disabled by default for regression. A [0.5%, 99.5%] quantile cut would be ~ ±2.6σ for Gaussians, which is far more aggressive than what TabPFN does at inference and would create a train-vs-inference distribution mismatch.
  3. OUTLIER_REMOVAL_STD is an inference-time parameter — when we pretrain by feeding tensors directly to the underlying torch model, this step is not automatically applied for us. Our sanitize.py therefore only normalises ±inf → NaN; we delegate true outlier handling to the package's own machinery at inference time.

Fine-tuning wrappers: official package machinery

Independent of the data pipeline, TabPFN Docs.txt:4224-4450 and TabPFN .txt:2035-2188 document FinetunedTabPFNClassifier / FinetunedTabPFNRegressor. These are the supported entry points for gradient-based adaptation of TabPFN. Our src/train/loop.py goes one level lower than these wrappers: it calls the underlying PerFeatureTransformer.forward(x, y, ...) directly with batches assembled on the fly by ProcessedDatasetLoader, which lets us iterate over a corpus of datasets (Real-TabPFN-style) rather than fine-tune on one dataset at a time. Our per-epoch subsample size is set in config/data.yaml (in the consuming project) (finetuning.max_rows_per_epoch: 20000 for v3/default, 9000 for v2.6), in the same spirit as the upstream FinetunedTabPFNClassifier per-fit row cap.

Refreshing this folder

  • For the upstream code repos (TabPFN .txt, PFNS.txt, etc.), re-grab a fresh dump from GitHub (e.g. via code2txt or a manual concat of find . -name "*.py" -exec cat {}) and overwrite the existing file with the same filename so existing greps in the codebase keep resolving.
  • For the docs and HuggingFace cards, refresh TabPFN Docs.txt and Huggingface TabPFN.txt manually from their upstream sources.