SEST MultiMediate Reproduction Package

This package contains the code needed to train, validate, and generate test predictions for NoXi, Noxi-J, MPIIGroupInteraction, and PinSoRo. SEST uses frame-level audiovisual features, optional window-level Gemini embeddings, and cross-attention between a target participant and a partner context.

No raw data, checkpoints, extracted features, or private API keys are included. Please edit the placeholder paths in the YAML config files before running.

Directory Layout

SEST/
  NoXi-X/
    train_noxi_config.yml
    train_sest.py
    evaluate_test.py
    dataset.py
    ssi_stream_utils.py
    src/
      SEST.py
      noxi_gemini_data.py
      noxi_overlap_data.py
      noxi_train_config.py
      noxi_train_data.py
      noxi_train_runtime.py
      loss.py
      metric.py
      ...
  MPIIGI/
    train_mpiigi_config.yml
    train_sest.py
    evaluate_test.py
    dataset.py
  Pinsoro/
    train_pinsoro_config.yml
    train_sest.py
    evaluate_test.py
    dataset.py
  extract_features/
    extraction_config.yml
    audio_features.py
    gemini_window_features.py
  evaluation_utils.py
  shared_data.py
  environment.yml

Main entrypoints:

• NoXi-X/train_sest.py
• NoXi-X/evaluate_test.py
• MPIIGI/train_sest.py
• MPIIGI/evaluate_test.py
• Pinsoro/train_sest.py
• Pinsoro/evaluate_test.py
• extract_features/audio_features.py
• extract_features/gemini_window_features.py

Environment

cd SEST
conda env create -f environment.yml
conda activate mee2026

If PyTorch/CUDA is already provided by the cluster, install only the missing packages listed in environment.yml.

Data Layout

Expected root:

DATA_ROOT/
  NoXi/
    train/
    val/
    test-base/
    test-additional/
  Noxi-J/
    train/
    val/
    test/
  MPIIGroupInteraction/
    train_new/
    val_new/
    test/
  Pinsoro/
    train-cc/
    train-cr/
    val-cc/
    val-cr/
    test-cc/
    test-cr/

Each role folder should contain the configured frame-level features and labels:

expert.audio.egemapsv2.stream
expert.audio_whisper.npy
expert.clip.stream
expert.openface2.stream
expert.engagement.annotation.csv
novice.audio.egemapsv2.stream
novice.audio_whisper.npy
novice.clip.stream
novice.openface2.stream
novice.engagement.annotation.csv

For the full SEST setting, also provide:

expert.gemini_avt_288_stride288.npz
novice.gemini_avt_288_stride288.npz

Update these files before running:

• NoXi-X/train_noxi_config.yml
• MPIIGI/train_mpiigi_config.yml
• Pinsoro/train_pinsoro_config.yml
• extract_features/extraction_config.yml

The YAML files use placeholder paths such as /path/to/data/root. You can either edit those values directly or override them with environment variables, for example DATA_ROOT, CHECKPOINT_ROOT, and HF_MODEL_ROOT.

Feature Extraction

Run from SEST/extract_features.

Whisper audio features:

python audio_features.py --dataset noxi --split train --extractors whisper
python audio_features.py --dataset noxi --split val --extractors whisper
python audio_features.py --dataset noxi --split test-base --extractors whisper
python audio_features.py --dataset noxi --split test-additional --extractors whisper

python audio_features.py --dataset noxi_j --split train --extractors whisper
python audio_features.py --dataset noxi_j --split val --extractors whisper
python audio_features.py --dataset noxi_j --split test --extractors whisper

Gemini audio-video-text window features:

python gemini_window_features.py --dataset noxi --split train --modality avt --api-key YOUR_KEY
python gemini_window_features.py --dataset noxi --split val --modality avt --api-key YOUR_KEY
python gemini_window_features.py --dataset noxi --split test-base --modality avt --api-key YOUR_KEY
python gemini_window_features.py --dataset noxi --split test-additional --modality avt --api-key YOUR_KEY

python gemini_window_features.py --dataset noxi_j --split train --modality avt --api-key YOUR_KEY
python gemini_window_features.py --dataset noxi_j --split val --modality avt --api-key YOUR_KEY
python gemini_window_features.py --dataset noxi_j --split test --modality avt --api-key YOUR_KEY

python gemini_window_features.py --dataset mpiigi --split train_new --modality avt \
  --output-root /path/to/DATA_ROOT/MPIIGroupInteraction/train_new --api-key YOUR_KEY
python gemini_window_features.py --dataset mpiigi --split val_new --modality avt \
  --output-root /path/to/DATA_ROOT/MPIIGroupInteraction/val_new --api-key YOUR_KEY
python gemini_window_features.py --dataset mpiigi --split test --modality avt \
  --output-root /path/to/DATA_ROOT/MPIIGroupInteraction/test --api-key YOUR_KEY

The default Gemini file is *.gemini_avt_288_stride288.npz. Each embedding is indexed by core_start; the paired dense model window is [core_start - 96, core_start + 192].

Gemini features are optional. If they are unavailable, skip this extraction step and train/evaluate with --no-gemini.

Training

Run from SEST/NoXi-X.

python train_sest.py \
  --feature-preset Noxi_base_j-openpose \
  --train-dataset noxi+noxi-j \
  --val-dataset noxi-j \
  --checkpoint-file-prefix SEST_noxij \
  --window-stride 288 \
  --gemini-suffix .gemini_avt_288_stride288.npz \
  --gemini-missing nearest

To run the same training flow without Gemini feature files:

python train_sest.py \
  --feature-preset Noxi_base_j-openpose \
  --train-dataset noxi+noxi-j \
  --val-dataset noxi-j \
  --checkpoint-file-prefix SEST_noxij_no_gemini \
  --window-stride 288 \
  --no-gemini

To isolate the target participant while keeping its Gemini embedding, add:

--partner-mode no_partner

Useful runtime overrides:

--learning-rate 5e-5
--batch-size 32
--val-batch-size 256
--clip-norm
--normalization
--partner-mode role
--partner-mode no_partner

Checkpoints are written to:

CHECKPOINT_ROOT/<val_dataset>/<checkpoint_file_prefix>/

Validation And Test Prediction

Run from SEST/NoXi-X. Multiple checkpoint files can be passed and are averaged as an ensemble.

python evaluate_test.py \
  --dataset noxi-j \
  --data-dir /path/to/DATA_ROOT/Noxi-J/test \
  --validation-data-dir /path/to/DATA_ROOT/Noxi-J/val \
  --output-dir /path/to/prediction/noxi-j \
  --checkpoint-prefix /path/to/checkpoints/Noxi-j/SEST_noxij \
  --checkpoint 'SEST_*seed40*.pt' 'SEST_*seed41*.pt' 'SEST_*seed42*.pt' \
  --feature-preset Noxi_base_j-openpose \
  --window-stride 288 \
  --gemini-suffix .gemini_avt_288_stride288.npz \
  --gemini-missing nearest \
  --overwrite

For checkpoints trained without Gemini features, add --no-gemini. The evaluation script can also infer this from the checkpoint structure, but passing the flag makes the intended data flow explicit.

For checkpoints trained with Min-Max normalization, add:

--normalization

If the normalization file cannot be inferred from the checkpoint folder, pass:

--normalization-stats /path/to/global_min_max.npz

Prediction files follow the official NoXi layout:

noxi-base/
  002/expert.engagement.prediction.csv
  002/novice.engagement.prediction.csv
noxi-additional/
  015/expert.engagement.prediction.csv
  015/novice.engagement.prediction.csv
noxi-j/
  127/expert.engagement.prediction.csv
  127/novice.engagement.prediction.csv

Partner Modes

The same SEST encoder is used across datasets. The context branch is selected with --partner-mode:

• role: NoXi and PinSoRo use another participant through cross-attention. NoXi uses the other member of the expert/novice pair, while PinSoRo uses the other color role.
• mean: MPIIGI uses the frame-wise mean of all other subjects in the same session. Their Gemini embeddings are averaged in the same way.
• random: MPIIGI selects one other subject for each window. Traditional and Gemini features always come from the same selected subject. Selection changes by training epoch but is fixed during validation and test for reproducibility.
• env: PinSoRo only. Cross-attention uses the separately configured env.* features. Target and environment features may have different input dims, but both encoders produce 256-dimensional sequences before cross-attention.
• no_partner: target-only ablation. The partner encoder and cross-attention layers are not instantiated. If Gemini is enabled, only the target Gemini embedding is used.

MPIIGroupInteraction

MPIIGI uses regression labels and CCC. Gemini is enabled by default. All participant folders remain together; windows are never split across sessions. Validation roles without an annotation file are skipped as targets but can still contribute to the group partner context.

Train with the mean of all other subjects:

cd SEST/MPIIGI
python train_sest.py \
  --partner-mode mean \
  --checkpoint-file-prefix SEST_MPIIGI_mean

Train by randomly selecting one other subject, or run the target-only ablation:

python train_sest.py \
  --partner-mode random \
  --checkpoint-file-prefix SEST_MPIIGI_random

python train_sest.py \
  --partner-mode no_partner \
  --checkpoint-file-prefix SEST_MPIIGI_no_partner

Validate and generate test files. Multiple checkpoints are averaged:

python evaluate_test.py \
  --data-dir /path/to/DATA_ROOT/MPIIGroupInteraction/test \
  --validation-data-dir /path/to/DATA_ROOT/MPIIGroupInteraction/val_new \
  --output-dir /path/to/prediction/mpiigroupinteraction \
  --checkpoint-prefix /path/to/checkpoints/MPIIGI/SEST_MPIIGI_mean \
  --checkpoint 'SEST_MPIIGI_mean_gemini_*seed40*.pt' \
               'SEST_MPIIGI_mean_gemini_*seed41*.pt' \
  --partner-mode mean \
  --overwrite

For a random-partner checkpoint, use --partner-mode random. The evaluator uses --partner-seed 0 by default, so repeated validation and test runs choose the same subject for every window.

Output files use names such as:

mpiigroupinteraction/001/subjectPos1.engagement.prediction.csv

PinSoRo

PinSoRo is classification rather than regression. Social and task engagement must be trained separately with --task social and --task task. Gemini is not used for PinSoRo. Labels equal to -100 are excluded from the loss and metrics. Validation reports CC and CR separately and then their unweighted mean, matching the official split-level metric.

Train with the other role as partner:

cd SEST/Pinsoro
python train_sest.py \
  --task task \
  --partner-mode role \
  --checkpoint-file-prefix SEST_PinSoRo_task_role

Train with environment context or without any partner:

python train_sest.py \
  --task social \
  --partner-mode env \
  --env-feature-preset Pinsoro_env_base \
  --checkpoint-file-prefix SEST_PinSoRo_social_env

python train_sest.py \
  --task social \
  --partner-mode no_partner \
  --checkpoint-file-prefix SEST_PinSoRo_social_no_partner

For every seed, training saves the best CC checkpoint and the best CR checkpoint independently. Early stopping occurs only after both splits exceed their patience. Use the matching best-cc or best-cr checkpoint for test prediction:

python evaluate_test.py \
  --dataset pinsoro-cc \
  --task task \
  --data-dir /path/to/DATA_ROOT/Pinsoro/test-cc \
  --validation-data-dir /path/to/DATA_ROOT/Pinsoro/val-cc \
  --output-dir /path/to/prediction/pinsoro-cc \
  --checkpoint-prefix /path/to/checkpoints/PinSoRo/SEST_PinSoRo_task_role \
  --checkpoint 'SEST_PinSoRo_task_role_*best-cc_seed40*.pt' \
  --partner-mode role \
  --overwrite

For CR, change the dataset and paths to pinsoro-cr, test-cr, and val-cr, and select a best-cr checkpoint. Output files contain class names:

pinsoro-cc/007/purple.task_engagement.prediction.csv
pinsoro-cc/007/yellow.task_engagement.prediction.csv
pinsoro-cr/016/purple.task_engagement.prediction.csv

Smoke Checks

After installing the environment, run these commands before a long job:

python NoXi-X/train_sest.py --help
python MPIIGI/train_sest.py --help
python MPIIGI/evaluate_test.py --help
python Pinsoro/train_sest.py --help
python Pinsoro/evaluate_test.py --help