Skip to content

add_inactive_sensors=True yields -inf charge for IceCube detectors (log10(0)) #903

Description

@sevmag

Describe the bug

When add_inactive_sensors=True is used with the IceCube Detector classes that standardize charge with a bare log10(x) (IceCube86, IceCubeKaggle, IceCubeUpgrade), the resulting node features contain -inf in the charge column.

Inactive sensors are padded from the geometry table in DataRepresentation._attach_inactive_sensors, and the IceCube geometry tables store charge = 0 for every sensor. Those zero charges then go through _charge:

def _charge(self, x: torch.tensor) -> torch.tensor:
    return torch.log10(x)   # log10(0) = -inf

so every inactive sensor becomes -inf. The forward pass does not raise, but it hands the model non-finite inputs, which NaN-out the loss/gradients as soon as you train on it.

This is independent of #898: it triggers with add_inactive_sensors alone, without any perturbation — but it shares the same root cause (log10 of a non-positive charge).

To Reproduce

Minimal and self-contained — no dataset needed, the single active pulse is built straight from the geometry table:

import torch
from graphnet.models.graphs import GraphDefinition
from graphnet.models.detector import IceCube86

detector = IceCube86()
features = list(detector.feature_map().keys())
charge_idx = features.index("charge")

# A single real pulse: one sensor from the geometry table, given a positive charge.
x = detector.geometry_table[features].to_numpy(dtype=float)[:1]
x[:, charge_idx] = 1.0

graph = GraphDefinition(detector=detector, add_inactive_sensors=True)(x, features)

charge = graph.x[:, charge_idx]
print("total nodes        :", graph.x.shape[0])
print("non-finite charges :", int((~torch.isfinite(charge)).sum()))
print("all charges finite :", bool(torch.isfinite(charge).all()))

Output on main (ced21ab):

total nodes        : 5407
non-finite charges : 5406
all charges finite : False

Expected behavior

Padded inactive sensors (charge 0) should map to a finite value. Flooring the charge before log10 — as proposed in #899, torch.log10(torch.clamp(x, min=1e-2)) — keeps the transform identical for all valid charges and maps inactive sensors to the bottom of the charge scale (log10(1e-2) = -2) instead of -inf.

Affected detectors

  • IceCube86, IceCubeKaggle, IceCubeUpgrade — all use bare log10(x) and produce -inf.
  • IceCubeDeepCore — uses the identity for charge, unaffected.
  • NuBenchDetector — its geometry tables have no charge column, so add_inactive_sensors raises KeyError: "['charge'] not in index" instead; that is a separate problem.

Related: #898, #899

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions