Skip to content

Commit 572550a

Browse files
torwagerclaude
andcommitted
FEATURE_MAP: variable counts per group, level table, and one-page .docx
- Add (N) variable counts to every feature-type entry that lacked one (image stats 12, acoustics 34, faces 7, saliency/depth 8, etc.). - New "Feature summary table" organizing groups by class x level (low / mid / high-level). - docs/feature_summary_table.docx: one-page landscape Word table for Google Docs import, generated by tools/build_feature_summary.py (counts pulled from schema/channel_template.json). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 8657f82 commit 572550a

3 files changed

Lines changed: 271 additions & 18 deletions

File tree

docs/FEATURE_MAP.md

Lines changed: 40 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,27 @@ The map is a plain, flat **SVG** built for editing:
2020
python3 tools/build_feature_map.py # reads schema/channel_template.json
2121
```
2222

23+
## Feature summary table
24+
25+
Feature groups organized by class (row) and by level of abstraction (column). Numbers in
26+
parentheses are the count of **variables** in that group — each scalar/flag counts as 1
27+
and each vector counts as its dimensionality (so an embedding contributes many). "Level"
28+
is the standard perception hierarchy: **low-level** = raw physical/perceptual signal,
29+
**mid-level** = perceptual organization/structure, **high-level** = semantic/conceptual.
30+
31+
| Feature class | Low-level | Mid-level | High-level |
32+
|---|---|---|---|
33+
| **Visual** | image statistics (12); optical-flow motion (3); shot cuts (2) | saliency & depth (8); faces & bodies (7) | SigLIP semantics — embedding + scene probes (784); DINOv2 embedding (384); action recognition (400) |
34+
| **Audio** | acoustics — loudness / pitch / timbre / MFCC / chroma (34) | speech presence & rate (2) | AudioSet sound tags (527); CLAP semantics — embedding + probes (524); vocal affect V/A/D (3) |
35+
| **Language** | word frequency & length (2) | lexical norms — valence / arousal / dominance, concreteness, AoA (5); syntactic complexity (5) | word surprisal & next-word entropy (2) |
36+
| **Social** || agent count & closest-pair distance (2) | interaction type & social dominance (2) |
37+
| **Situation** ||| scene description / setting / indoor–outdoor (3); event structure — boundaries & event id (2) |
38+
| **Affect** || facial affect — 8 expressions + valence + arousal (10) | EmoNet image emotion (20); text emotion & sentiment (32); VLM depicted emotion (3) |
39+
40+
> A one-page Word version of this table (for import into Google Docs) is at
41+
> [`docs/feature_summary_table.docx`](feature_summary_table.docx), regenerated by
42+
> `python3 tools/build_feature_summary.py`.
43+
2344
## The major feature types
2445

2546
What each model in the pipeline is, what it outputs, why it is useful (with an eye to
@@ -29,7 +50,7 @@ copies live in `PDFs/` (Dropbox-only).
2950

3051
### Visual — what the frame looks like
3152

32-
- **Low-level image statistics** — Classic per-frame pixel/color measures computed with
53+
- **Low-level image statistics (12)** — Classic per-frame pixel/color measures computed with
3354
[scikit-image](https://doi.org/10.7717/peerj.453): luminance, RMS contrast, mean red/
3455
green/blue and hue/saturation/brightness, edge density, image entropy, the slope of the
3556
spatial-frequency (Fourier) power spectrum, and perceptual
@@ -62,25 +83,25 @@ copies live in `PDFs/` (Dropbox-only).
6283
probability vector plus the single top action label per second — a semantic
6384
"what are people doing" regressor for action-observation and social/motor brain systems.
6485

65-
- **Motion (optical flow)**[RAFT](https://arxiv.org/abs/2003.12039) estimates dense
86+
- **Motion optical flow (3)**[RAFT](https://arxiv.org/abs/2003.12039) estimates dense
6687
per-pixel motion between consecutive frames. From it the pipeline derives overall
6788
optical-flow magnitude, a camera-motion estimate, and residual (object) motion. Low-
6889
level motion energy is a classic driver of MT/MST and dorsal-stream responses and a
6990
standard motion regressor in naturalistic fMRI.
7091

71-
- **Shot cuts & shot index** — A histogram-based cut detector (a lightweight stand-in for
92+
- **Shot cuts & shot index (2)** — A histogram-based cut detector (a lightweight stand-in for
7293
[TransNetV2](https://arxiv.org/abs/2008.04838)) flags edit boundaries and numbers the
7394
shots. Cuts are strong low-level visual-transient events and useful for modeling
7495
attention/event-onset responses.
7596

76-
- **Faces & bodies**[MTCNN](https://arxiv.org/abs/1604.02878) detects faces (count,
97+
- **Faces & bodies (7)**[MTCNN](https://arxiv.org/abs/1604.02878) detects faces (count,
7798
detection confidence, largest-face size, face-present flag) and
7899
[Keypoint R-CNN](https://arxiv.org/abs/1703.06870) (from the Mask R-CNN family) detects
79100
human body pose (person count, pose confidence, body-present flag). Face presence/size
80101
is a primary regressor for face-selective cortex (FFA/OFA/STS); body/person presence
81102
drives EBA and the person-perception network.
82103

83-
- **Saliency & depth** — A spectral-residual [saliency](https://ieeexplore.ieee.org/document/4270292)
104+
- **Saliency & depth (8)** — A spectral-residual [saliency](https://ieeexplore.ieee.org/document/4270292)
84105
model (Hou & Zhang 2007; stand-in for a deep video-saliency net) gives where the eye is
85106
likely drawn (mean/peak saliency, spread, salient-area fraction), and
86107
[Depth-Anything-V2](https://arxiv.org/abs/2406.09414) gives monocular depth (mean depth,
@@ -89,7 +110,7 @@ copies live in `PDFs/` (Dropbox-only).
89110

90111
### Audio — what the soundtrack sounds like
91112

92-
- **Low-level acoustics**[librosa](https://doi.org/10.25080/Majora-7b98e3ed-003)
113+
- **Low-level acoustics (34)**[librosa](https://doi.org/10.25080/Majora-7b98e3ed-003)
93114
computes loudness (RMS), spectral shape (centroid, bandwidth, rolloff, flatness),
94115
zero-crossing rate, onset strength, a 13-value **MFCC** timbre vector, and a 12-value
95116
**chroma** (musical pitch-class) vector; pitch/F0 uses
@@ -112,7 +133,7 @@ copies live in `PDFs/` (Dropbox-only).
112133
pipeline keeps the 512-d embedding and 12 interpretable sound probes. The embedding is a
113134
general auditory-semantic feature space for encoding models.
114135

115-
- **Speech**[faster-whisper](https://arxiv.org/abs/2212.04356) (Whisper) transcribes
136+
- **Speech (5)**[faster-whisper](https://arxiv.org/abs/2212.04356) (Whisper) transcribes
116137
speech and yields a speech-present flag and speaking rate (words/s);
117138
[audEERING wav2vec2](https://arxiv.org/abs/2203.07378) reads **vocal affect** directly
118139
from the voice as continuous valence, arousal, and dominance. Speech presence and
@@ -121,49 +142,50 @@ copies live in `PDFs/` (Dropbox-only).
121142

122143
### Language — what is said (from the transcript)
123144

124-
- **Lexical semantic norms** — per word, human-rated norms are looked up: emotional
145+
- **Lexical semantic norms (5)** — per word, human-rated norms are looked up: emotional
125146
[valence, arousal, dominance](https://doi.org/10.3758/s13428-012-0314-x) (Warriner 2013),
126147
[concreteness](https://doi.org/10.3758/s13428-013-0403-5) (Brysbaert 2014), and
127148
[age of acquisition](https://doi.org/10.3758/s13428-012-0210-4) (Kuperman 2012). These
128149
are the classic word-property regressors for language and semantic fMRI.
129150

130-
- **Word frequency & length**[wordfreq](https://doi.org/10.5281/zenodo.7199437) gives
151+
- **Word frequency & length (2)**[wordfreq](https://doi.org/10.5281/zenodo.7199437) gives
131152
each word's log frequency (Zipf scale) and its length. Frequency is a robust predictor
132153
of lexical-access difficulty and reading/listening time.
133154

134-
- **Surprisal & next-word entropy**[GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
155+
- **Surprisal & next-word entropy (2)**[GPT-2](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)
135156
provides each word's **surprisal** (how unexpected it was, in bits) and the entropy of
136157
the model's next-word prediction. Word-by-word surprisal from language models is one of
137158
the most reliable predictors of language-network and N400-type responses in
138159
naturalistic-comprehension fMRI/MEG.
139160

140-
- **Syntax**[spaCy](https://spacy.io) parses each utterance for content-word fraction,
161+
- **Syntax (5)**[spaCy](https://spacy.io) parses each utterance for content-word fraction,
141162
noun/verb fraction, parse-tree depth, and mean dependency distance — proxies for
142163
syntactic complexity and processing load.
143164

144165
### Social — who is present and how they relate
145166

146-
- Derived from the video by the [Qwen2.5-VL](https://arxiv.org/abs/2502.13923) vision-
147-
language model (setting, interaction type, social dominance) plus face/pose detectors
167+
- **Social relations (4)** — derived from the video by the [Qwen2.5-VL](https://arxiv.org/abs/2502.13923)
168+
vision-language model (interaction type, social dominance) plus face/pose detectors
148169
([MTCNN](https://arxiv.org/abs/1604.02878) agent count;
149170
[Keypoint R-CNN](https://arxiv.org/abs/1703.06870) closest-pair distance). These target
150171
the social-perception network (STS, TPJ, mentalizing regions) that tracks the number of
151172
agents, their proximity, and the type of interaction.
152173

153174
### Situation — where/when; event structure
154175

155-
- **Scene & setting** — the [Qwen2.5-VL](https://arxiv.org/abs/2502.13923) VLM produces a
176+
- **Scene & setting (3)** — the [Qwen2.5-VL](https://arxiv.org/abs/2502.13923) VLM produces a
156177
free-text scene description, an indoor/outdoor label, and a coarse setting label per
157178
second — high-level context for scene/place-selective cortex.
158-
- **Event boundaries**[GSBS](https://doi.org/10.1016/j.neuroimage.2021.118085)
179+
- **Event boundaries (2)**[GSBS](https://doi.org/10.1016/j.neuroimage.2021.118085)
159180
(Greedy State Boundary Search; Geerligs 2021) segments the annotation time series into
160181
discrete "events," emitting boundary events and a running event number. This directly
161182
operationalizes event-segmentation theory and models the boundary responses seen in
162183
hippocampus and default-mode regions during naturalistic viewing.
163184

164185
### Affect — depicted emotion across channels
165186

166-
Emotion is read from four independent channels so it can be cross-validated by modality:
187+
Emotion is read from several independent channels/modalities so it can be cross-validated
188+
(facial, textual, and holistic-scene here; vocal affect lives under Audio → Speech):
167189

168190
- **EmoNet — image emotion (20)**[EmoNet](https://www.science.org/doi/10.1126/sciadv.aaw4358)
169191
(Kragel et al. 2019, *Science Advances*) is an AlexNet-based CNN that maps a video frame
@@ -172,7 +194,7 @@ Emotion is read from four independent channels so it can be cross-validated by m
172194
human ratings and brain responses, making it a purpose-built regressor for visual
173195
emotion-category coding in the brain.
174196

175-
- **Facial affect (8 + valence/arousal)**
197+
- **Facial affect (10: 8 expressions + valence + arousal)**
176198
[HSEmotion](https://arxiv.org/abs/2403.11590) (Savchenko) reads faces detected in the
177199
video and classifies expression into **8 categories** (anger, contempt, disgust, fear,
178200
happiness, neutral, sadness, surprise) plus continuous facial valence and arousal —
@@ -184,7 +206,7 @@ Emotion is read from four independent channels so it can be cross-validated by m
184206
RoBERTa scores **3-way sentiment** (negative/neutral/positive) with a continuous polarity.
185207
These capture the emotional content of what is said.
186208

187-
- **VLM depicted emotion**[Qwen2.5-VL](https://arxiv.org/abs/2502.13923) also judges the
209+
- **VLM depicted emotion (3)**[Qwen2.5-VL](https://arxiv.org/abs/2502.13923) also judges the
188210
overall depicted emotion, valence, and arousal of the scene as a holistic cross-check on
189211
the specialized channels.
190212

docs/feature_summary_table.docx

36.9 KB
Binary file not shown.

0 commit comments

Comments
 (0)