The classifier assigns each of 17 inherited retinal dystrophy (IRD) phenotypic modules to a gene, trained on 434 labeled genes - a small dataset by deep learning standards, and an imbalanced one: module sizes range from 10 to 63 genes. Macro-F1 (F1 averaged equally across all 17 modules, regardless of size) was chosen as the primary metric specifically because of that imbalance. A model that performs well only on the largest modules while failing on rare ones is not useful here - in a clinical or discovery context, the rare modules matter just as much as the common ones. Evaluation throughout uses 5-fold stratified cross-validation, with a working target of macro-F1 > 0.35 set early as a threshold for "meaningfully better than noise."
A methodological note worth stating plainly: before any of the results below were considered valid, a checkpoint-selection bug was identified and fixed. The training script had been saving each fold's checkpoint at the epoch of minimum validation loss, but under class-weighted cross-entropy loss, the epoch of minimum loss does not reliably coincide with the epoch of maximum validation macro-F1 - the two objectives can diverge. Correcting the checkpoint criterion to track macro-F1 directly raised apparent performance substantially, from an artifact of roughly 0.18 to the ~0.29 baseline reported below. This is flagged explicitly because it is an instructive distinction: that jump reflects a measurement fix, not a model improvement, and conflating the two would have been a quiet error baked into every subsequent comparison.
The first architecture combined two modalities: normalized phylogenetic profiling (NPP, 1905-dimensional cross-species conservation vectors) and ESM2 protein language model embeddings (1280-dimensional). Once the corrected baseline was established, nine controlled experiments followed, each changing exactly one variable at a time: architecture capacity (reduced, then set to a midpoint), regularization strength (higher dropout and weight decay), data augmentation (Gaussian noise injection on both modalities), learning rate (lowered, then scheduled with cosine annealing), and the class-weighting scheme itself.
The result across all nine runs was a strikingly narrow band: validation macro-F1 never left the range of roughly 0.24-0.29, and never exceeded the very first corrected baseline. No single lever - smaller models, stronger regularization, augmented data, slower or scheduled learning rates - produced a reliable improvement beyond that ceiling. Differences between individual runs were generally within one standard deviation of each other (~±0.05), meaning most of the apparent variation was consistent with ordinary fold-to-fold noise rather than genuine signal.
That stability was itself the useful finding. Nine independent, well-motivated attempts to extract more signal through training-level interventions converged on the same ceiling, which is reasonably strong evidence that the bottleneck was not how the model was being trained, but how much information NPP and ESM2 actually contained about module membership. That conclusion is what motivated the next phase, rather than a tenth or eleventh variation on the same two inputs.
A few module-level patterns held consistently across Phase 1 worth noting on their own: the two largest modules (63 and 56 genes) underperformed relative to their size in every run, while several mid-sized modules (notably modules 5, 9, 10, and 11) consistently scored among the highest F1 values regardless of which training configuration was tested - early evidence that signal quality, not training-set size, was driving per-module performance. One module in particular (16 genes) failed outright - F1 = 0.00 in seven of the nine Phase 1 runs - and was flagged for follow-up rather than written off; that follow-up is described below. Separately, an ablation that removed the inverse-frequency class weighting confirmed the weighting scheme was doing real work: without it, the smallest and hardest module's F1 collapsed to zero while larger modules saw only marginal gains, validating that the weighting was correctly calibrated for the project's actual goal of recovering rare modules rather than simply maximizing overall accuracy.
With architecture and training-level variation exhausted as productive levers, the next experiment changed the input itself: a third tower was added, built from STRING protein-protein interaction (PPI) network embeddings (full methodology and biological validation in the companion PPI document). A complete ablation then tested every combination of the three modalities - seven configurations in total: each modality in isolation, every pairwise combination, and the full three-tower fusion.
The effect was immediate and large. The non-PPI baseline (NPP + ESM2, re-run within this same experiment as an internal control) reproduced the Phase 1 ceiling almost exactly - validation macro-F1 of 0.29, matching the original baseline within rounding, which confirmed the ceiling was a real, reproducible property of those two modalities rather than an artifact of one particular run. Every configuration that included the PPI tower, by contrast, broke decisively past that ceiling. The full three-tower fusion reached 0.40 ± 0.02. More strikingly, PPI features used alone - with no NPP or ESM2 input at all - reached 0.44 ± 0.05, the single best result obtained anywhere in the project, using a tower with roughly 30 times fewer parameters than the full fusion model. This headline number deserves an important caveat, developed in full below: it is measured on a gene set that is not representative of the genes this model is ultimately meant to discover.
In absolute terms, a macro-F1 of 0.44 across 17 imbalanced classes, from only 434 labeled training genes, represents roughly a 7.5-fold improvement over a near-uniform random baseline (~0.06). That framing matters: this is a genuinely useful predictive signal for prioritizing candidate genes, not a solved classification problem. Several modules remain weak under every configuration tested so far and should not be treated as reliably classified.
The clearest single illustration is the module that had failed completely throughout Phase 1. With PPI features alone, its F1 jumped to 0.56 - the first strong result for that module across all ten training runs in the project, in either phase. That specific module is discussed further, with a concrete biological hypothesis, in the PPI methodology document; from a purely modeling standpoint, it is the cleanest demonstration that this modality contributes signal the other two structurally could not provide. Elsewhere, one module reached F1 = 0.86 when NPP and PPI were combined - the highest single-module score recorded anywhere in the project - while the modules that were already strong in Phase 1 (5, 9, 10, 11) stayed strong across nearly every modality combination, suggesting their underlying signal is somewhat redundant across feature types rather than tied to any one of them.
The most scientifically interesting result of Phase 2 is also the least settled one, and the explanation that matters most here has less to do with model architecture than with what a curated interaction database actually measures. The PPI-only model - roughly 13,600 parameters - outperformed the full three-tower fusion model - roughly 312,000 parameters - on both cross-validated and held-out evaluation metrics, and did so with a substantially smaller gap between training and validation macro-F1 (0.14, versus 0.42 for the full fusion model) - meaning the advantage isn't simply a better-fit-but-still-overconfident model. STRING, like essentially every curated protein-protein interaction resource, is built substantially from published experimental and literature-derived evidence. A gene's representation in the network is therefore a function not only of its true biological connectivity, but of how much research attention it has received - well-studied genes accumulate more documented interactions simply because more labs have looked for them, independent of whether the underlying biology is genuinely richer than a less-studied gene's. This is a well-known property of interactome databases, generally discussed under the heading of study or ascertainment bias.
This matters directly here because the 434 labeled training genes are, by construction, already-established IRD genes - characterized well enough to have been confidently assigned to a phenotypic module in the first place. They are, almost definitionally, among the more heavily studied genes in the genome relative to an arbitrary draw from the full gene set. If PPI-only's strong cross-validated performance is driven in part by these particular genes having unusually rich, well-resolved network neighborhoods - rather than PPI topology being intrinsically more informative about IRD module membership than evolutionary conservation or protein sequence - then that advantage should not be expected to transfer with the same magnitude to genuinely novel, poorly-characterized candidate genes discovered genome-wide. Those are, by the nature of the discovery task itself, exactly the genes least likely to have been extensively studied, and therefore most likely to have a sparse or thin PPI representation even where formal STRING coverage technically exists - a non-imputed coverage flag confirms a gene cleared a minimum connectivity threshold, not that its represented neighborhood is biologically complete.
This reframes the result rather than undermining it: PPI-only's advantage during training validation is real and reproducible, but it is measured on a gene population that is not representative of the genome-wide population the model is ultimately applied to. The practical implication is that PPI-only's predictions should be trusted somewhat more cautiously for genuinely obscure or understudied candidate genes than the headline 0.44 macro-F1 figure alone would suggest - and a moderate-confidence PPI-only prediction for an understudied gene may, for this exact reason, be just as scientifically interesting as a high-confidence one for a well-studied gene, rather than less so. This is part of the reasoning for carrying the full three-tower fusion model forward into genome-wide inference alongside PPI-only rather than relying on PPI-only exclusively, and it is discussed further, with its implications for candidate interpretation, in the inference methodology document.
Two more conventional explanations remain plausible alongside this one and are not mutually exclusive with it. The fusion model's much larger parameter count, trained on only 434 genes, leaves more room for overfitting even under identical regularization. It is also possible that naively concatenating three modalities introduces interference rather than synergy - a single linear fusion layer may need a more deliberate mechanism, such as attention-based weighting or per-modality gating, to combine three feature types this different in nature. Fold-level variance is a further complicating factor: PPI-only's own standard deviation (±0.05) is larger than the fusion model's (±0.02), so it is not uniformly more stable - it simply lands higher on average. Disentangling how much of the PPI-only advantage is study bias versus these architectural factors would require further controlled experiments, most directly genome-wide validation against truly independent evidence rather than additional cross-validation on the same labeled set.
In practice, both variants were carried forward into genome-wide inference rather than discarding one. The PPI-only model is the stronger predictor where PPI coverage exists, but roughly 7% of genes genome-wide have no usable STRING network signal and rely on imputation; for those genes specifically, the full fusion model's NPP and ESM2 inputs remain informative where PPI alone would be guessing from an imputed placeholder.
A planned extension is to add tissue expression data (gene expression consensus profiles, sourced from the Human Protein Atlas rather than GTEx, since GTEx's retinal tissue coverage is a known gap relevant specifically to this disease area) as a fourth, independent modality. The intent is to use it for post-hoc validation of genome-wide candidates rather than as additional training input - keeping it as an orthogonal signal that wasn't part of what the model learned from, rather than folding it into training and losing that independence.