@@ -20,6 +20,27 @@ The map is a plain, flat **SVG** built for editing:
2020 python3 tools/build_feature_map.py # reads schema/channel_template.json
2121 ```
2222
23+ ## Feature summary table
24+
25+ Feature groups organized by class (row) and by level of abstraction (column). Numbers in
26+ parentheses are the count of ** variables** in that group — each scalar/flag counts as 1
27+ and each vector counts as its dimensionality (so an embedding contributes many). "Level"
28+ is the standard perception hierarchy: ** low-level** = raw physical/perceptual signal,
29+ ** mid-level** = perceptual organization/structure, ** high-level** = semantic/conceptual.
30+
31+ | Feature class | Low-level | Mid-level | High-level |
32+ | ---| ---| ---| ---|
33+ | ** Visual** | image statistics (12); optical-flow motion (3); shot cuts (2) | saliency & depth (8); faces & bodies (7) | SigLIP semantics — embedding + scene probes (784); DINOv2 embedding (384); action recognition (400) |
34+ | ** Audio** | acoustics — loudness / pitch / timbre / MFCC / chroma (34) | speech presence & rate (2) | AudioSet sound tags (527); CLAP semantics — embedding + probes (524); vocal affect V/A/D (3) |
35+ | ** Language** | word frequency & length (2) | lexical norms — valence / arousal / dominance, concreteness, AoA (5); syntactic complexity (5) | word surprisal & next-word entropy (2) |
36+ | ** Social** | — | agent count & closest-pair distance (2) | interaction type & social dominance (2) |
37+ | ** Situation** | — | — | scene description / setting / indoor–outdoor (3); event structure — boundaries & event id (2) |
38+ | ** Affect** | — | facial affect — 8 expressions + valence + arousal (10) | EmoNet image emotion (20); text emotion & sentiment (32); VLM depicted emotion (3) |
39+
40+ > A one-page Word version of this table (for import into Google Docs) is at
41+ > [ ` docs/feature_summary_table.docx ` ] ( feature_summary_table.docx ) , regenerated by
42+ > ` python3 tools/build_feature_summary.py ` .
43+
2344## The major feature types
2445
2546What each model in the pipeline is, what it outputs, why it is useful (with an eye to
@@ -29,7 +50,7 @@ copies live in `PDFs/` (Dropbox-only).
2950
3051### Visual — what the frame looks like
3152
32- - ** Low-level image statistics** — Classic per-frame pixel/color measures computed with
53+ - ** Low-level image statistics (12) ** — Classic per-frame pixel/color measures computed with
3354 [ scikit-image] ( https://doi.org/10.7717/peerj.453 ) : luminance, RMS contrast, mean red/
3455 green/blue and hue/saturation/brightness, edge density, image entropy, the slope of the
3556 spatial-frequency (Fourier) power spectrum, and perceptual
@@ -62,25 +83,25 @@ copies live in `PDFs/` (Dropbox-only).
6283 probability vector plus the single top action label per second — a semantic
6384 "what are people doing" regressor for action-observation and social/motor brain systems.
6485
65- - ** Motion ( optical flow)** — [ RAFT] ( https://arxiv.org/abs/2003.12039 ) estimates dense
86+ - ** Motion — optical flow (3 )** — [ RAFT] ( https://arxiv.org/abs/2003.12039 ) estimates dense
6687 per-pixel motion between consecutive frames. From it the pipeline derives overall
6788 optical-flow magnitude, a camera-motion estimate, and residual (object) motion. Low-
6889 level motion energy is a classic driver of MT/MST and dorsal-stream responses and a
6990 standard motion regressor in naturalistic fMRI.
7091
71- - ** Shot cuts & shot index** — A histogram-based cut detector (a lightweight stand-in for
92+ - ** Shot cuts & shot index (2) ** — A histogram-based cut detector (a lightweight stand-in for
7293 [ TransNetV2] ( https://arxiv.org/abs/2008.04838 ) ) flags edit boundaries and numbers the
7394 shots. Cuts are strong low-level visual-transient events and useful for modeling
7495 attention/event-onset responses.
7596
76- - ** Faces & bodies** — [ MTCNN] ( https://arxiv.org/abs/1604.02878 ) detects faces (count,
97+ - ** Faces & bodies (7) ** — [ MTCNN] ( https://arxiv.org/abs/1604.02878 ) detects faces (count,
7798 detection confidence, largest-face size, face-present flag) and
7899 [ Keypoint R-CNN] ( https://arxiv.org/abs/1703.06870 ) (from the Mask R-CNN family) detects
79100 human body pose (person count, pose confidence, body-present flag). Face presence/size
80101 is a primary regressor for face-selective cortex (FFA/OFA/STS); body/person presence
81102 drives EBA and the person-perception network.
82103
83- - ** Saliency & depth** — A spectral-residual [ saliency] ( https://ieeexplore.ieee.org/document/4270292 )
104+ - ** Saliency & depth (8) ** — A spectral-residual [ saliency] ( https://ieeexplore.ieee.org/document/4270292 )
84105 model (Hou & Zhang 2007; stand-in for a deep video-saliency net) gives where the eye is
85106 likely drawn (mean/peak saliency, spread, salient-area fraction), and
86107 [ Depth-Anything-V2] ( https://arxiv.org/abs/2406.09414 ) gives monocular depth (mean depth,
@@ -89,7 +110,7 @@ copies live in `PDFs/` (Dropbox-only).
89110
90111### Audio — what the soundtrack sounds like
91112
92- - ** Low-level acoustics** — [ librosa] ( https://doi.org/10.25080/Majora-7b98e3ed-003 )
113+ - ** Low-level acoustics (34) ** — [ librosa] ( https://doi.org/10.25080/Majora-7b98e3ed-003 )
93114 computes loudness (RMS), spectral shape (centroid, bandwidth, rolloff, flatness),
94115 zero-crossing rate, onset strength, a 13-value ** MFCC** timbre vector, and a 12-value
95116 ** chroma** (musical pitch-class) vector; pitch/F0 uses
@@ -112,7 +133,7 @@ copies live in `PDFs/` (Dropbox-only).
112133 pipeline keeps the 512-d embedding and 12 interpretable sound probes. The embedding is a
113134 general auditory-semantic feature space for encoding models.
114135
115- - ** Speech** — [ faster-whisper] ( https://arxiv.org/abs/2212.04356 ) (Whisper) transcribes
136+ - ** Speech (5) ** — [ faster-whisper] ( https://arxiv.org/abs/2212.04356 ) (Whisper) transcribes
116137 speech and yields a speech-present flag and speaking rate (words/s);
117138 [ audEERING wav2vec2] ( https://arxiv.org/abs/2203.07378 ) reads ** vocal affect** directly
118139 from the voice as continuous valence, arousal, and dominance. Speech presence and
@@ -121,49 +142,50 @@ copies live in `PDFs/` (Dropbox-only).
121142
122143### Language — what is said (from the transcript)
123144
124- - ** Lexical semantic norms** — per word, human-rated norms are looked up: emotional
145+ - ** Lexical semantic norms (5) ** — per word, human-rated norms are looked up: emotional
125146 [ valence, arousal, dominance] ( https://doi.org/10.3758/s13428-012-0314-x ) (Warriner 2013),
126147 [ concreteness] ( https://doi.org/10.3758/s13428-013-0403-5 ) (Brysbaert 2014), and
127148 [ age of acquisition] ( https://doi.org/10.3758/s13428-012-0210-4 ) (Kuperman 2012). These
128149 are the classic word-property regressors for language and semantic fMRI.
129150
130- - ** Word frequency & length** — [ wordfreq] ( https://doi.org/10.5281/zenodo.7199437 ) gives
151+ - ** Word frequency & length (2) ** — [ wordfreq] ( https://doi.org/10.5281/zenodo.7199437 ) gives
131152 each word's log frequency (Zipf scale) and its length. Frequency is a robust predictor
132153 of lexical-access difficulty and reading/listening time.
133154
134- - ** Surprisal & next-word entropy** — [ GPT-2] ( https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf )
155+ - ** Surprisal & next-word entropy (2) ** — [ GPT-2] ( https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf )
135156 provides each word's ** surprisal** (how unexpected it was, in bits) and the entropy of
136157 the model's next-word prediction. Word-by-word surprisal from language models is one of
137158 the most reliable predictors of language-network and N400-type responses in
138159 naturalistic-comprehension fMRI/MEG.
139160
140- - ** Syntax** — [ spaCy] ( https://spacy.io ) parses each utterance for content-word fraction,
161+ - ** Syntax (5) ** — [ spaCy] ( https://spacy.io ) parses each utterance for content-word fraction,
141162 noun/verb fraction, parse-tree depth, and mean dependency distance — proxies for
142163 syntactic complexity and processing load.
143164
144165### Social — who is present and how they relate
145166
146- - Derived from the video by the [ Qwen2.5-VL] ( https://arxiv.org/abs/2502.13923 ) vision-
147- language model (setting, interaction type, social dominance) plus face/pose detectors
167+ - ** Social relations (4) ** — derived from the video by the [ Qwen2.5-VL] ( https://arxiv.org/abs/2502.13923 )
168+ vision- language model (interaction type, social dominance) plus face/pose detectors
148169 ([ MTCNN] ( https://arxiv.org/abs/1604.02878 ) agent count;
149170 [ Keypoint R-CNN] ( https://arxiv.org/abs/1703.06870 ) closest-pair distance). These target
150171 the social-perception network (STS, TPJ, mentalizing regions) that tracks the number of
151172 agents, their proximity, and the type of interaction.
152173
153174### Situation — where/when; event structure
154175
155- - ** Scene & setting** — the [ Qwen2.5-VL] ( https://arxiv.org/abs/2502.13923 ) VLM produces a
176+ - ** Scene & setting (3) ** — the [ Qwen2.5-VL] ( https://arxiv.org/abs/2502.13923 ) VLM produces a
156177 free-text scene description, an indoor/outdoor label, and a coarse setting label per
157178 second — high-level context for scene/place-selective cortex.
158- - ** Event boundaries** — [ GSBS] ( https://doi.org/10.1016/j.neuroimage.2021.118085 )
179+ - ** Event boundaries (2) ** — [ GSBS] ( https://doi.org/10.1016/j.neuroimage.2021.118085 )
159180 (Greedy State Boundary Search; Geerligs 2021) segments the annotation time series into
160181 discrete "events," emitting boundary events and a running event number. This directly
161182 operationalizes event-segmentation theory and models the boundary responses seen in
162183 hippocampus and default-mode regions during naturalistic viewing.
163184
164185### Affect — depicted emotion across channels
165186
166- Emotion is read from four independent channels so it can be cross-validated by modality:
187+ Emotion is read from several independent channels/modalities so it can be cross-validated
188+ (facial, textual, and holistic-scene here; vocal affect lives under Audio → Speech):
167189
168190- ** EmoNet — image emotion (20)** — [ EmoNet] ( https://www.science.org/doi/10.1126/sciadv.aaw4358 )
169191 (Kragel et al. 2019, * Science Advances* ) is an AlexNet-based CNN that maps a video frame
@@ -172,7 +194,7 @@ Emotion is read from four independent channels so it can be cross-validated by m
172194 human ratings and brain responses, making it a purpose-built regressor for visual
173195 emotion-category coding in the brain.
174196
175- - ** Facial affect (8 + valence/ arousal)** —
197+ - ** Facial affect (10: 8 expressions + valence + arousal)** —
176198 [ HSEmotion] ( https://arxiv.org/abs/2403.11590 ) (Savchenko) reads faces detected in the
177199 video and classifies expression into ** 8 categories** (anger, contempt, disgust, fear,
178200 happiness, neutral, sadness, surprise) plus continuous facial valence and arousal —
@@ -184,7 +206,7 @@ Emotion is read from four independent channels so it can be cross-validated by m
184206 RoBERTa scores ** 3-way sentiment** (negative/neutral/positive) with a continuous polarity.
185207 These capture the emotional content of what is said.
186208
187- - ** VLM depicted emotion** — [ Qwen2.5-VL] ( https://arxiv.org/abs/2502.13923 ) also judges the
209+ - ** VLM depicted emotion (3) ** — [ Qwen2.5-VL] ( https://arxiv.org/abs/2502.13923 ) also judges the
188210 overall depicted emotion, valence, and arousal of the scene as a holistic cross-check on
189211 the specialized channels.
190212
0 commit comments