Describe the bug
When add_inactive_sensors=True is used with the IceCube Detector classes that standardize charge with a bare log10(x) (IceCube86, IceCubeKaggle, IceCubeUpgrade), the resulting node features contain -inf in the charge column.
Inactive sensors are padded from the geometry table in DataRepresentation._attach_inactive_sensors, and the IceCube geometry tables store charge = 0 for every sensor. Those zero charges then go through _charge:
def _charge(self, x: torch.tensor) -> torch.tensor:
return torch.log10(x) # log10(0) = -inf
so every inactive sensor becomes -inf. The forward pass does not raise, but it hands the model non-finite inputs, which NaN-out the loss/gradients as soon as you train on it.
This is independent of #898: it triggers with add_inactive_sensors alone, without any perturbation — but it shares the same root cause (log10 of a non-positive charge).
To Reproduce
Minimal and self-contained — no dataset needed, the single active pulse is built straight from the geometry table:
import torch
from graphnet.models.graphs import GraphDefinition
from graphnet.models.detector import IceCube86
detector = IceCube86()
features = list(detector.feature_map().keys())
charge_idx = features.index("charge")
# A single real pulse: one sensor from the geometry table, given a positive charge.
x = detector.geometry_table[features].to_numpy(dtype=float)[:1]
x[:, charge_idx] = 1.0
graph = GraphDefinition(detector=detector, add_inactive_sensors=True)(x, features)
charge = graph.x[:, charge_idx]
print("total nodes :", graph.x.shape[0])
print("non-finite charges :", int((~torch.isfinite(charge)).sum()))
print("all charges finite :", bool(torch.isfinite(charge).all()))
Output on main (ced21ab):
total nodes : 5407
non-finite charges : 5406
all charges finite : False
Expected behavior
Padded inactive sensors (charge 0) should map to a finite value. Flooring the charge before log10 — as proposed in #899, torch.log10(torch.clamp(x, min=1e-2)) — keeps the transform identical for all valid charges and maps inactive sensors to the bottom of the charge scale (log10(1e-2) = -2) instead of -inf.
Affected detectors
IceCube86, IceCubeKaggle, IceCubeUpgrade — all use bare log10(x) and produce -inf.
IceCubeDeepCore — uses the identity for charge, unaffected.
NuBenchDetector — its geometry tables have no charge column, so add_inactive_sensors raises KeyError: "['charge'] not in index" instead; that is a separate problem.
Related: #898, #899
Describe the bug
When
add_inactive_sensors=Trueis used with the IceCubeDetectorclasses that standardize charge with a barelog10(x)(IceCube86,IceCubeKaggle,IceCubeUpgrade), the resulting node features contain-infin the charge column.Inactive sensors are padded from the geometry table in
DataRepresentation._attach_inactive_sensors, and the IceCube geometry tables storecharge = 0for every sensor. Those zero charges then go through_charge:so every inactive sensor becomes
-inf. The forward pass does not raise, but it hands the model non-finite inputs, which NaN-out the loss/gradients as soon as you train on it.This is independent of #898: it triggers with
add_inactive_sensorsalone, without any perturbation — but it shares the same root cause (log10of a non-positive charge).To Reproduce
Minimal and self-contained — no dataset needed, the single active pulse is built straight from the geometry table:
Output on
main(ced21ab):Expected behavior
Padded inactive sensors (charge
0) should map to a finite value. Flooring the charge beforelog10— as proposed in #899,torch.log10(torch.clamp(x, min=1e-2))— keeps the transform identical for all valid charges and maps inactive sensors to the bottom of the charge scale (log10(1e-2) = -2) instead of-inf.Affected detectors
IceCube86,IceCubeKaggle,IceCubeUpgrade— all use barelog10(x)and produce-inf.IceCubeDeepCore— uses the identity for charge, unaffected.NuBenchDetector— its geometry tables have nochargecolumn, soadd_inactive_sensorsraisesKeyError: "['charge'] not in index"instead; that is a separate problem.Related: #898, #899