Skip to content

Commit 868a733

Browse files
gereonelversgereonelvers
committed
LibriBrain100: SpeechDetection on word-only schemas + docs/examples
SpeechDetection previously returned None whenever events.tsv had no explicit silence rows, silently dropping every run. Sherlock has explicit silence rows; TIMIT, MOCHATIMIT, and TheMoth do not. Fix: when silence rows are absent, derive the speech mask from word intervals on a silence-by-default canvas (1 inside word spans, 0 in gaps). Sherlock keeps its existing behavior intact. Two unit tests lock the new path against a hand-crafted events.tsv and confirm no regression on the Sherlock schema. End-to-end on the VM, SpeechDetection on TheMoth ses-1 now produces 1492 samples with ~69% speech proportion (vs 0 samples before). Also fills the LibriBrain100 documentation gap from the previous commit: - docs/libribrain100.md per-dataset page (deep/broad split, selector vocabulary, validation rules, partial-upload note) - examples/load_libribrain100.py with both the task-based entry point and the subset selector - _toc.yml, datasets.md, api/index.md, README.md updated to surface LibriBrain100 alongside the other datasets
1 parent f46bc88 commit 868a733

12 files changed

Lines changed: 599 additions & 22 deletions

‎README.md‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,11 +62,29 @@ meg_masc = Gwilliams2022(
6262
)
6363
```
6464

65+
For the full LibriBrain release (deep sub-0 across 9 Sherlock books +
66+
TIMIT + MOCHA-TIMIT + 30 Moth podcasts, plus 32 broad subjects on
67+
Sherlock1 ses-11/ses-12), use `LibriBrain100`:
68+
69+
```python
70+
from pnpl.datasets import LibriBrain100
71+
from pnpl.tasks import SpeechDetection
72+
73+
ds = LibriBrain100(
74+
data_path="./data/LibriBrain100",
75+
task=SpeechDetection(tmin=0.0, tmax=0.5),
76+
partition="train",
77+
subjects="deep", # or "broad", "all", 0, [1, 2, 3], range(1, 33)
78+
corpus="sherlock", # or "timit", "mocha", "podcasts", "all"
79+
)
80+
```
81+
6582
## Included Datasets
6683

6784
| Class | Source | Auth |
6885
| --- | --- | --- |
6986
| `LibriBrain` (+ `LibriBrainSpeech`/`Phoneme`/`Word`/`Sentence`) | Hugging Face `pnpl/LibriBrain` | none |
87+
| `LibriBrain100` (+ `LibriBrain100Speech`/`Phoneme`/`Word`) | HF `pnpl/LibriBrain` ∪ `pnpl/LibriBrain2` (deep + broad release) | none |
7088
| `Gwilliams2022` (MEG-MASC) | OSF `ag3kj` | none |
7189
| `Armeni2022` | Radboud `DSC_3011085.05_995_v1` | Radboud credentials |
7290
| `Schoffelen2019` (MOUS) | Radboud `DSC_3011020.09_236_v1` | Radboud credentials |

‎docs/_toc.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ parts:
99
- caption: Datasets
1010
chapters:
1111
- file: libribrain
12+
- file: libribrain100
1213
- file: gwilliams2022
1314
- file: armeni2022
1415
- file: schoffelen2019

‎docs/api/generated/pnpl.datasets.libribrain100.compat.LibriBrain100Phoneme.rst‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
pnpl.datasets.libribrain100.compat.LibriBrain100Phoneme
2+
=======================================================
3+
4+
.. currentmodule:: pnpl.datasets.libribrain100.compat
5+
6+
.. autoclass:: LibriBrain100Phoneme
7+
8+
9+
.. automethod:: __init__
10+
11+
12+
.. rubric:: Methods
13+
14+
.. autosummary::
15+
16+
~LibriBrain100Phoneme.__init__
17+
~LibriBrain100Phoneme.calculate_standardization_params
18+
~LibriBrain100Phoneme.clip_sample
19+
~LibriBrain100Phoneme.close_h5_files
20+
~LibriBrain100Phoneme.ensure_file
21+
~LibriBrain100Phoneme.ensure_file_download
22+
~LibriBrain100Phoneme.get_bids_raw_path
23+
~LibriBrain100Phoneme.get_calibration_files
24+
~LibriBrain100Phoneme.get_derivatives_path
25+
~LibriBrain100Phoneme.get_events_path
26+
~LibriBrain100Phoneme.get_h5_dataset
27+
~LibriBrain100Phoneme.get_h5_path
28+
~LibriBrain100Phoneme.get_headpos_path
29+
~LibriBrain100Phoneme.get_preprocessed_path
30+
~LibriBrain100Phoneme.get_sfreq_from_h5
31+
~LibriBrain100Phoneme.init_continuous_h5
32+
~LibriBrain100Phoneme.load_continuous_window
33+
~LibriBrain100Phoneme.load_continuous_window_from_sample
34+
~LibriBrain100Phoneme.load_head_positions
35+
~LibriBrain100Phoneme.load_preprocessed_bids
36+
~LibriBrain100Phoneme.load_raw_bids
37+
~LibriBrain100Phoneme.prefetch_files
38+
~LibriBrain100Phoneme.raw_bids_exists
39+
~LibriBrain100Phoneme.setup_standardization
40+
~LibriBrain100Phoneme.standardize
41+
42+
43+
44+
45+
46+
.. rubric:: Attributes
47+
48+
.. autosummary::
49+
50+
~LibriBrain100Phoneme.HUGGINGFACE_FALLBACK_REPOS
51+
~LibriBrain100Phoneme.HUGGINGFACE_REPO
52+
~LibriBrain100Phoneme.broadcasted_means
53+
~LibriBrain100Phoneme.broadcasted_stds
54+
~LibriBrain100Phoneme.channel_means
55+
~LibriBrain100Phoneme.channel_stds
56+
~LibriBrain100Phoneme.label_info
57+
~LibriBrain100Phoneme.n_channels
58+
~LibriBrain100Phoneme.n_times
59+
~LibriBrain100Phoneme.records
60+
61+

‎docs/api/generated/pnpl.datasets.libribrain100.compat.LibriBrain100Speech.rst‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
pnpl.datasets.libribrain100.compat.LibriBrain100Speech
2+
======================================================
3+
4+
.. currentmodule:: pnpl.datasets.libribrain100.compat
5+
6+
.. autoclass:: LibriBrain100Speech
7+
8+
9+
.. automethod:: __init__
10+
11+
12+
.. rubric:: Methods
13+
14+
.. autosummary::
15+
16+
~LibriBrain100Speech.__init__
17+
~LibriBrain100Speech.calculate_standardization_params
18+
~LibriBrain100Speech.clip_sample
19+
~LibriBrain100Speech.close_h5_files
20+
~LibriBrain100Speech.ensure_file
21+
~LibriBrain100Speech.ensure_file_download
22+
~LibriBrain100Speech.get_bids_raw_path
23+
~LibriBrain100Speech.get_calibration_files
24+
~LibriBrain100Speech.get_derivatives_path
25+
~LibriBrain100Speech.get_events_path
26+
~LibriBrain100Speech.get_h5_dataset
27+
~LibriBrain100Speech.get_h5_path
28+
~LibriBrain100Speech.get_headpos_path
29+
~LibriBrain100Speech.get_preprocessed_path
30+
~LibriBrain100Speech.get_sfreq_from_h5
31+
~LibriBrain100Speech.init_continuous_h5
32+
~LibriBrain100Speech.load_continuous_window
33+
~LibriBrain100Speech.load_continuous_window_from_sample
34+
~LibriBrain100Speech.load_head_positions
35+
~LibriBrain100Speech.load_preprocessed_bids
36+
~LibriBrain100Speech.load_raw_bids
37+
~LibriBrain100Speech.prefetch_files
38+
~LibriBrain100Speech.raw_bids_exists
39+
~LibriBrain100Speech.setup_standardization
40+
~LibriBrain100Speech.standardize
41+
42+
43+
44+
45+
46+
.. rubric:: Attributes
47+
48+
.. autosummary::
49+
50+
~LibriBrain100Speech.HUGGINGFACE_FALLBACK_REPOS
51+
~LibriBrain100Speech.HUGGINGFACE_REPO
52+
~LibriBrain100Speech.broadcasted_means
53+
~LibriBrain100Speech.broadcasted_stds
54+
~LibriBrain100Speech.channel_means
55+
~LibriBrain100Speech.channel_stds
56+
~LibriBrain100Speech.label_info
57+
~LibriBrain100Speech.n_channels
58+
~LibriBrain100Speech.n_times
59+
~LibriBrain100Speech.records
60+
61+

‎docs/api/generated/pnpl.datasets.libribrain100.compat.LibriBrain100Word.rst‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
pnpl.datasets.libribrain100.compat.LibriBrain100Word
2+
====================================================
3+
4+
.. currentmodule:: pnpl.datasets.libribrain100.compat
5+
6+
.. autoclass:: LibriBrain100Word
7+
8+
9+
.. automethod:: __init__
10+
11+
12+
.. rubric:: Methods
13+
14+
.. autosummary::
15+
16+
~LibriBrain100Word.__init__
17+
~LibriBrain100Word.calculate_standardization_params
18+
~LibriBrain100Word.clip_sample
19+
~LibriBrain100Word.close_h5_files
20+
~LibriBrain100Word.ensure_file
21+
~LibriBrain100Word.ensure_file_download
22+
~LibriBrain100Word.get_bids_raw_path
23+
~LibriBrain100Word.get_calibration_files
24+
~LibriBrain100Word.get_derivatives_path
25+
~LibriBrain100Word.get_events_path
26+
~LibriBrain100Word.get_h5_dataset
27+
~LibriBrain100Word.get_h5_path
28+
~LibriBrain100Word.get_headpos_path
29+
~LibriBrain100Word.get_preprocessed_path
30+
~LibriBrain100Word.get_sfreq_from_h5
31+
~LibriBrain100Word.init_continuous_h5
32+
~LibriBrain100Word.load_continuous_window
33+
~LibriBrain100Word.load_continuous_window_from_sample
34+
~LibriBrain100Word.load_head_positions
35+
~LibriBrain100Word.load_preprocessed_bids
36+
~LibriBrain100Word.load_raw_bids
37+
~LibriBrain100Word.prefetch_files
38+
~LibriBrain100Word.raw_bids_exists
39+
~LibriBrain100Word.setup_standardization
40+
~LibriBrain100Word.standardize
41+
42+
43+
44+
45+
46+
.. rubric:: Attributes
47+
48+
.. autosummary::
49+
50+
~LibriBrain100Word.HUGGINGFACE_FALLBACK_REPOS
51+
~LibriBrain100Word.HUGGINGFACE_REPO
52+
~LibriBrain100Word.broadcasted_means
53+
~LibriBrain100Word.broadcasted_stds
54+
~LibriBrain100Word.channel_means
55+
~LibriBrain100Word.channel_stds
56+
~LibriBrain100Word.label_info
57+
~LibriBrain100Word.n_channels
58+
~LibriBrain100Word.n_times
59+
~LibriBrain100Word.records
60+
61+

‎docs/api/generated/pnpl.datasets.libribrain100.dataset.LibriBrain100.rst‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
pnpl.datasets.libribrain100.dataset.LibriBrain100
2+
=================================================
3+
4+
.. currentmodule:: pnpl.datasets.libribrain100.dataset
5+
6+
.. autoclass:: LibriBrain100
7+
8+
9+
.. automethod:: __init__
10+
11+
12+
.. rubric:: Methods
13+
14+
.. autosummary::
15+
16+
~LibriBrain100.__init__
17+
~LibriBrain100.calculate_standardization_params
18+
~LibriBrain100.clip_sample
19+
~LibriBrain100.close_h5_files
20+
~LibriBrain100.ensure_file
21+
~LibriBrain100.ensure_file_download
22+
~LibriBrain100.get_bids_raw_path
23+
~LibriBrain100.get_calibration_files
24+
~LibriBrain100.get_derivatives_path
25+
~LibriBrain100.get_events_path
26+
~LibriBrain100.get_h5_dataset
27+
~LibriBrain100.get_h5_path
28+
~LibriBrain100.get_headpos_path
29+
~LibriBrain100.get_preprocessed_path
30+
~LibriBrain100.get_sfreq_from_h5
31+
~LibriBrain100.init_continuous_h5
32+
~LibriBrain100.load_continuous_window
33+
~LibriBrain100.load_continuous_window_from_sample
34+
~LibriBrain100.load_head_positions
35+
~LibriBrain100.load_preprocessed_bids
36+
~LibriBrain100.load_raw_bids
37+
~LibriBrain100.prefetch_files
38+
~LibriBrain100.raw_bids_exists
39+
~LibriBrain100.setup_standardization
40+
~LibriBrain100.standardize
41+
42+
43+
44+
45+
46+
.. rubric:: Attributes
47+
48+
.. autosummary::
49+
50+
~LibriBrain100.HUGGINGFACE_FALLBACK_REPOS
51+
~LibriBrain100.HUGGINGFACE_REPO
52+
~LibriBrain100.broadcasted_means
53+
~LibriBrain100.broadcasted_stds
54+
~LibriBrain100.channel_means
55+
~LibriBrain100.channel_stds
56+
~LibriBrain100.label_info
57+
~LibriBrain100.n_channels
58+
~LibriBrain100.n_times
59+
~LibriBrain100.records
60+
61+

‎docs/api/index.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,10 @@ module.
2121
pnpl.datasets.libribrain2025.compat.LibriBrainPhoneme
2222
pnpl.datasets.libribrain2025.compat.LibriBrainWord
2323
pnpl.datasets.libribrain2025.sentence_dataset.LibriBrainSentence
24+
pnpl.datasets.libribrain100.dataset.LibriBrain100
25+
pnpl.datasets.libribrain100.compat.LibriBrain100Speech
26+
pnpl.datasets.libribrain100.compat.LibriBrain100Phoneme
27+
pnpl.datasets.libribrain100.compat.LibriBrain100Word
2428
pnpl.datasets.gwilliams2022.dataset.Gwilliams2022
2529
pnpl.datasets.armeni2022.dataset.Armeni2022
2630
pnpl.datasets.schoffelen2019.dataset.Schoffelen2019

‎docs/datasets.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ and the dataset materializes preprocessed H5 files on demand.
1212
| Class | Source | Auth | Format on disk |
1313
| --- | --- | --- | --- |
1414
| `LibriBrain` | Hugging Face `pnpl/LibriBrain` | none | preprocessed H5 |
15+
| `LibriBrain100` | HF `pnpl/LibriBrain` ∪ `pnpl/LibriBrain2` (deep + broad release) | none | preprocessed H5 |
1516
| `Gwilliams2022` | OSF `ag3kj` (MEG-MASC) | none | KIT `.con` → H5 |
1617
| `Armeni2022` | Radboud `DSC_3011085.05_995_v1` | Radboud login | CTF `.ds` → H5 |
1718
| `Schoffelen2019` | Radboud `DSC_3011020.09_236_v1` (MOUS) | Radboud login | CTF `.ds` → H5 |
@@ -23,6 +24,7 @@ Common imports:
2324
from pnpl.datasets import (
2425
# Task-based entry points
2526
LibriBrain,
27+
LibriBrain100,
2628
Gwilliams2022,
2729
Armeni2022,
2830
Schoffelen2019,
@@ -32,14 +34,18 @@ from pnpl.datasets import (
3234
LibriBrainPhoneme,
3335
LibriBrainWord,
3436
LibriBrainSentence,
37+
LibriBrain100Speech,
38+
LibriBrain100Phoneme,
39+
LibriBrain100Word,
3540
# Utilities
3641
GroupedDataset,
3742
)
3843
```
3944

4045
See the per-dataset pages for end-to-end examples:
4146

42-
- [LibriBrain](libribrain.md) — speech, phoneme, word, sentence tasks
47+
- [LibriBrain](libribrain.md) — original release (sub-0 × Sherlock1..7), speech / phoneme / word / sentence tasks
48+
- [LibriBrain100](libribrain100.md) — full release: sub-0 across 9 Sherlock books + TIMIT + MOCHA-TIMIT + 30 Moth podcasts, plus 32 broad subjects on Sherlock1 ses-11/ses-12
4349
- [Gwilliams 2022 (MEG-MASC)](gwilliams2022.md) — story listening, OSF
4450
- [Armeni 2022](armeni2022.md) — audiobook listening, Radboud
4551
- [Schöffelen 2019 (MOUS)](schoffelen2019.md) — sentence comprehension, Radboud

0 commit comments

Comments
 (0)