Skip to content

Commit ce65555

Browse files
h-mayorquinh-mayorquin
committed
bette structure
1 parent 852fc67 commit ce65555

2 files changed

Lines changed: 107 additions & 102 deletions

File tree

‎.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ tests/*.h5
1111

1212
examples/*.prb
1313
examples/*.h5
14+
examples/*.json
1415

1516
ressources/*/*
1617
build/*
@@ -19,6 +20,7 @@ build/*
1920
dist/*
2021
doc/_*
2122
doc/examples/*
23+
doc/sg_execution_times.rst
2224

2325
dev_*
2426
.coverage

‎doc/neuropixels_readers.rst‎

Lines changed: 105 additions & 102 deletions
Original file line numberDiff line numberDiff line change
@@ -4,37 +4,84 @@ The Neuropixels catalogue pattern
44
.. currentmodule:: probeinterface
55

66

7-
The catalogue: :py:func:`build_neuropixels_probe`
8-
-------------------------------------------------
9-
10-
The foundation of every Neuropixels reader in probeinterface is
11-
:py:func:`build_neuropixels_probe`. Given a probe part number (a specific
12-
stock-keeping unit identifier such as ``"NP1000"``, ``"NP2000"``, ``"NP2014"``),
13-
it returns a :py:class:`Probe` carrying the full silicon geometry for that
14-
part number: every catalogue contact (960 for Neuropixels 1.0, 1280 per shank
15-
for Neuropixels 2.0), the planar contour of the shanks, the contact shapes and
16-
sizes, the analog-to-digital converter (ADC) multiplexer (MUX) routing table,
17-
and the probe-level annotations (``manufacturer``, ``model_name``,
18-
``part_number``, ``description``).
19-
20-
The numbers behind that geometry come from the
7+
Two kinds of Neuropixels probe
8+
------------------------------
9+
10+
Probeinterface distinguishes two kinds of Neuropixels probe.
11+
12+
**Catalogue probe.** The probe as it appears in the IMEC catalogue, with
13+
every contact on the silicon present (960 on Neuropixels 1.0, 1280 per
14+
shank on Neuropixels 2.0). Built via :py:func:`build_neuropixels_probe(part_number)
15+
<build_neuropixels_probe>`. It carries:
16+
17+
* contact positions
18+
* shank contour and dimensions
19+
* contact shapes and sizes
20+
* the analog-to-digital converter (ADC) multiplexer (MUX) routing on the
21+
silicon
22+
* identity metadata (manufacturer, model name, part number, description)
23+
24+
A catalogue probe is pure geometry, the same for every recording made with
25+
that variant. Use it to plot the probe layout, compute distances between
26+
contacts, or run any analysis that does not depend on a specific recording.
27+
28+
**Recording-setup probe.** The catalogue probe specialised for one
29+
recording session: only the contacts actually recorded are present
30+
(typically 384 of the 960 or more catalogue contacts), and per-contact
31+
recording state is attached:
32+
33+
* per-contact analog band (AP) and local field potential (LFP) gains
34+
* reference configuration
35+
* per-contact sampling order
36+
* probe wiring (the mapping from each contact to the recording channel
37+
that captured its data)
38+
39+
Use a recording-setup probe to hand the recording to SpikeInterface so
40+
spike sorters see both the geometry and the correct channel mapping, to
41+
convert raw samples to microvolts via the per-contact gains, or to plot
42+
the recorded contacts alongside the recorded traces.
43+
44+
45+
How readers connect the two
46+
---------------------------
47+
48+
A format reader turns a Neuropixels recording into a recording-setup probe
49+
in three steps:
50+
51+
1. **Fetch the catalogue probe.** Look up the probe part number (SKU) in
52+
the recording's metadata, then call
53+
:py:func:`build_neuropixels_probe(part_number) <build_neuropixels_probe>`.
54+
2. **Identify the active electrodes.** Read the recording's channel
55+
configuration to find which catalogue contacts were actually recorded,
56+
and slice the catalogue probe down to that subset via
57+
:py:meth:`probe.get_slice(active_indices) <Probe.get_slice>`.
58+
3. **Attach the recording-setup metadata.** Add the per-contact gains,
59+
reference settings, and sampling order, and set the probe wiring.
60+
61+
The sections below cover where the part number comes from per reader
62+
(step 1) and where the catalogue data itself comes from.
63+
64+
65+
The catalogue
66+
-------------
67+
68+
A probe part number (SKU) is an IMEC identifier such as ``"NP1000"``,
69+
``"NP2000"``, or ``"NP2014"``. The part number determines the silicon
70+
geometry: 960 contacts on Neuropixels 1.0, 1280 per shank on Neuropixels
71+
2.0, plus all the per-variant pitch and shank dimensions.
72+
73+
The data behind the catalogue comes from the
2174
`ProbeTable <https://github.com/billkarsh/ProbeTable>`_ repository maintained
2275
by `Bill Karsh <https://github.com/billkarsh>`_ (author of SpikeGLX).
2376
ProbeTable is the canonical machine-readable inventory of IMEC Neuropixels
24-
probe specifications: contact positions, electrode dimensions, shank geometry,
25-
MUX routing, ADC configuration, all keyed by part number. Probeinterface
26-
mirrors a postprocessed snapshot of that data into the package via
27-
``resources/postprocess_neuropixels_probe_features.py``, which is re-run after
28-
each ProbeTable sync. Without ProbeTable, every reader would have to carry
29-
its own hand-written copy of the manufacturer specs, which is exactly the
30-
situation the catalogue pattern is designed to avoid.
77+
probe specifications, all keyed by part number.
3178

3279

33-
The format readers
34-
------------------
80+
The readers
81+
-----------
3582

36-
Four entry points read Neuropixels recordings (or recording configurations)
37-
and produce a probe ready to use with SpikeInterface:
83+
Four readers produce a probe from a Neuropixels recording. They differ in
84+
where they look up the part number:
3885

3986
.. list-table::
4087
:header-rows: 1
@@ -58,91 +105,47 @@ and produce a probe ready to use with SpikeInterface:
58105
for the format transition.
59106
* - :py:func:`read_spikegadgets_neuropixels`
60107
- SpikeGadgets ``.rec`` XML header
61-
- Not present in the file; the reader picks a geometry-equivalent stand-in
62-
based on ``(SpikeConfiguration.device, deviceSubType)``: ``NP1000`` for
63-
Neuropixels 1.0, ``NP2000`` for Neuropixels 2.0 single-shank, ``NP2014``
64-
for Neuropixels 2.0 4-shank
65-
66-
The first three readers read the part number directly from the recording
67-
metadata. SpikeGadgets is the exception: its ``.rec`` XML does not carry a
68-
part number field, so the reader cannot know which specific variant produced
69-
the recording. It picks one representative per geometry-equivalent family
70-
(all Neuropixels 1.0 staggered variants share contact positions; all
71-
Neuropixels 2.0 single-shank variants share contact positions; all
72-
Neuropixels 2.0 4-shank variants share contact positions) and clears the
73-
``model_name``, ``description``, and ``part_number`` annotations on the
74-
returned probe so downstream code does not read the stand-in as an
75-
attribution.
76-
77-
78-
From catalogue probe to probe in a recording setup
79-
--------------------------------------------------
80-
81-
The catalogue probe is pure geometry, divorced from any recording session. A real
82-
recording uses only a subset of those contacts: the Neuropixels headstage
83-
acquires 384 channels at a time, and the recording configuration selects
84-
which catalogue contacts those 384 are drawn from (384 of 960 on Neuropixels
85-
1.0, 384 of 1280 per shank on Neuropixels 2.0 single-shank, 384 of 5120 on
86-
Neuropixels 2.0 4-shank). The selection mechanism differs by recording
87-
format (an IMRO table for SpikeGLX, a channel map in ``settings.xml`` for
88-
Open Ephys, the ``SpikeNTrode`` list in SpikeGadgets's ``.rec`` XML); each
89-
reader's docstring covers the specifics for that format. On top of the
90-
selection, the recording adds session-specific state:
91-
per-contact analog band (AP) and local field potential (LFP) gains, ADC
92-
sample order, reference configuration, and the channel-to-file mapping that
93-
says where each contact's data lives in the saved binary. Probeinterface
94-
calls the result a probe in a recording setup, to distinguish it from the
95-
catalogue.
96-
97-
Each reader produces the recording-setup probe in the same three steps:
98-
99-
1. Build the catalogue probe by calling
100-
:py:func:`build_neuropixels_probe(part_number) <build_neuropixels_probe>`
101-
with the part number obtained from the recording metadata.
102-
2. Slice the catalogue probe to the active electrodes for this recording session via
103-
:py:meth:`probe.get_slice(active_indices) <Probe.get_slice>`. The slice
104-
drops the unrecorded contacts but preserves the probe-level annotations
105-
and the per-contact catalogue annotations (ADC group, sample order) on the
106-
contacts that survive.
107-
3. Attach the recording-specific state: per-contact AP/LFP gains, any
108-
reference annotations, and finally
109-
:py:meth:`probe.set_device_channel_indices(...) <Probe.set_device_channel_indices>`
110-
to record where each surviving contact's data lives in the saved file.
108+
- Not present in the file; the reader picks a geometry-equivalent
109+
stand-in based on ``(SpikeConfiguration.device, deviceSubType)``:
110+
``NP1000`` for Neuropixels 1.0, ``NP2000`` for Neuropixels 2.0
111+
single-shank, ``NP2014`` for Neuropixels 2.0 4-shank
112+
113+
The first three readers read the part number directly. SpikeGadgets is the
114+
exception (its ``.rec`` XML does not carry a part number) and falls back to
115+
a geometry-equivalent stand-in; the variants within each Neuropixels family
116+
share identical 2D contact geometry, so any representative produces correct
117+
positions.
111118

112119

113120
What the pattern solves
114121
-----------------------
115122

116-
Constructing geometry from scratch inside each format reader (the situation
117-
before the catalogue pattern) had three problems:
118-
119-
* **Geometry drift across readers.** Each reader carried its own copy of the
120-
manufacturer specs. A Neuropixels 2.0 4-shank probe loaded through SpikeGLX
121-
and through SpikeGadgets could return contact positions that disagreed in
122-
the third decimal because the two readers had been updated against
123-
different snapshots of the IMEC spec. Centralising the geometry in
124-
:py:func:`build_neuropixels_probe` and sourcing it from ProbeTable means
125-
every reader returns the same positions for the same part number.
126-
* **Conflated geometry and wiring bugs.** When a saved recording looked wrong
127-
on the probe, it was difficult to say whether the geometry was off
128-
(catalogue issue) or the channel-to-contact mapping was off (wiring issue).
129-
With the two phases separated, a geometry bug is a bug in
130-
:py:func:`build_neuropixels_probe`; a wiring bug is a bug in the reader's
131-
matching step. The two can be diagnosed and fixed independently.
132-
* **Hidden active-electrode selection.** Readers that built a 384-contact
133-
probe directly hid the fact that 576 catalogue contacts were silently
134-
dropped. The explicit ``probe.get_slice(active_indices)`` step makes the
135-
selection visible and inspectable: callers can ask "which catalogue
136-
contacts did this recording session record?" and get a direct answer.
137-
138-
The pattern also pays out on the upgrade path. When IMEC ships a new probe
139-
variant, the integration work is "add the part number to ProbeTable, re-run
140-
the postprocess script".
123+
Building geometry from scratch inside each reader (the situation before this
124+
pattern) caused three problems:
125+
126+
* **Drift across readers.** Each reader carried its own copy of the
127+
manufacturer specs, and the copies could disagree. Centralising the
128+
geometry in :py:func:`build_neuropixels_probe` and sourcing it from
129+
ProbeTable means every reader returns the same positions for the same
130+
part number.
131+
* **Confused bugs.** When a saved recording looked wrong on the probe, it
132+
was hard to tell whether the geometry was off or the channel-to-contact
133+
mapping was off. With the two phases separated, a geometry bug is a bug
134+
in :py:func:`build_neuropixels_probe`; a wiring bug is a bug in the
135+
reader's slicing or wiring step. The two can be diagnosed independently.
136+
* **Hidden electrode selection.** A reader that built a 384-contact probe
137+
directly hid the fact that 576 catalogue contacts were silently dropped.
138+
The explicit slice step makes the selection visible: callers can ask
139+
which catalogue contacts the recording captured and get a direct answer.
140+
141+
The pattern also helps on the upgrade path. When IMEC ships a new probe
142+
variant, the integration work is to add the part number to ProbeTable; the
143+
readers do not change.
141144

142145

143146
Discussion
144147
----------
145148

146149
This pattern was proposed and is tracked in issue
147-
`#405 <https://github.com/SpikeInterface/probeinterface/issues/405>`_; if you have
148-
any discussion point to add please re-open the issue so the maintainers can discuss.
150+
`#405 <https://github.com/SpikeInterface/probeinterface/issues/405>`_.
151+
Reopen the issue if you want to discuss changes.

0 commit comments

Comments
 (0)