Skip to content

Add check_spike_times_without_nans - #689

Merged
bendichter merged 11 commits into
devfrom
deal_with_nan_in_spike_times
Jun 30, 2026
Merged

Add check_spike_times_without_nans#689
bendichter merged 11 commits into
devfrom
deal_with_nan_in_spike_times

Conversation

@h-mayorquin

Copy link
Copy Markdown
Collaborator

Spike times are physical event timestamps representing when action potentials occurred. A spike either happened at a specific time or it didn't, so NaN is never a meaningful value. NaN in spike times typically indicates a conversion bug or array padding that was not cleaned up before writing to NWB.

Unlike other spike time checks that sample the first N elements, this check scans the full array because NaN values can appear at any position in the data.

@h-mayorquin h-mayorquin self-assigned this Mar 24, 2026
Comment thread docs/checks_by_importance.rst Outdated
@rly

rly commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

This looks good to me. One minor concern is that as recordings get bigger, both check_spike_times_without_nans and check_ascending_spike_times load the whole array into memory and will take some time to run.

Typical large recording today:

  • ~1,000–5,000 units (multi-probe Neuropixels)
  • mean rate ~5 Hz, session ~1–2 h (~3,600–7,200 s)
  • → 5,000 × 5 × 7,200 ≈ 1.8e8 spikes

Some labs are doing many-hour recordings, so on the order of 1e9 spike times is plausible.

With 1e9 spike times:

data[:] materializes 8 GB (float64) and np.isnan allocates another ~1 GB bool array, so ~9 GB transient. On a 16 GB laptop running the inspector that risks an OOM.

On my Mac M1 laptop, it would take about 35 seconds to read gzip-compressed data[:] + check, with a warm cache.

I think that's OK, but something to keep in mind as data gets bigger.

@bendichter
bendichter enabled auto-merge (squash) June 30, 2026 20:09
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.87%. Comparing base (13d0d67) to head (4e6efde).
⚠️ Report is 6 commits behind head on dev.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##              dev     #689      +/-   ##
==========================================
+ Coverage   79.44%   82.87%   +3.43%     
==========================================
  Files          48       48              
  Lines        1897     1904       +7     
==========================================
+ Hits         1507     1578      +71     
+ Misses        390      326      -64     
Files with missing lines Coverage Δ
src/nwbinspector/checks/__init__.py 100.00% <ø> (ø)
src/nwbinspector/checks/_ecephys.py 94.35% <100.00%> (+0.23%) ⬆️

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@bendichter
bendichter merged commit 85049d6 into dev Jun 30, 2026
25 checks passed
@bendichter
bendichter deleted the deal_with_nan_in_spike_times branch June 30, 2026 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants