Skip to content

OPERATIONS.md

Latest commit

 

History

History
782 lines (640 loc) · 35.2 KB

OPERATIONS.md

File metadata and controls

782 lines (640 loc) · 35.2 KB
Raw

Pipeline operations

Live deployment

  • AWS region: us-east-1
  • CloudFormation stack: sam-audio-pipeline
  • GPU instance: i-0aed4af178083ce58 (p5.4xlarge, H100 80 GB)
  • Host: ec2-44-211-31-38.compute-1.amazonaws.com
  • Private artifact bucket: sam-audio-pipeline-artifactbucket-ndb3jfc3asyk
  • DynamoDB table: sam-audio-pipeline-PipelineTable-1C6R2H8D5C4TW
  • Remote application root: /home/ubuntu/sam-audio-deploy

The AWS names are identifiers, not credentials. The EC2 instance profile grants only the S3, DynamoDB, and SQS operations declared in CloudFormation.

Private access

The panel/API binds to remote localhost port 8080. Open a local tunnel:

ssh -fN -o ExitOnForwardFailure=yes \
  -L 18080:127.0.0.1:8080 \
  ubuntu@ec2-44-211-31-38.compute-1.amazonaws.com

Then open http://127.0.0.1:18080/. The model APIs remain private on remote ports 8000 (SAM) and 8001 (Audio Flamingo).

Services

ssh ubuntu@ec2-44-211-31-38.compute-1.amazonaws.com \
  'systemctl is-active sam-audio-api audio-flamingo-next \
   sam-pipeline-api sam-pipeline-ingest sam-pipeline-sam \
   sam-pipeline-flamingo'

The reconciliation timer repairs durable tasks whose SQS message was lost or expired. It can also be invoked safely on demand:

ssh ubuntu@ec2-44-211-31-38.compute-1.amazonaws.com \
  'sudo systemctl start sam-pipeline-reconcile.service'

Queue and job health

curl -fsS http://127.0.0.1:18080/v1/overview | jq
curl -fsS http://127.0.0.1:18080/v1/jobs/JOB_ID | jq
curl -fsS http://127.0.0.1:18080/v1/review/stats | jq
curl -fsS http://127.0.0.1:18080/v1/datasets/DATASET_ID/overview | jq

DynamoDB is authoritative. A successful HTTP upload confirmation only means the source tasks are durable and queued; callers should poll the job endpoint rather than keep an inference request open.

Stereo mapping

The pipeline accepts exactly two-channel stereo sources. Mono and multichannel uploads remain visible in the dataset but finish with skip_reason set to non_stereo_input; they create no chunks, stems, or model tasks.

For accepted sources, Audio Flamingo scene analysis runs before SAM. Its has_music and has_voices booleans select zero, one, or both SAM targets. A malformed scene response deliberately falls back to both targets. Judge/CLAP scores remain separation-quality evidence and are not used as presence scores.

New audible stems automatically keep both versions:

  • {stem}.wav is the untouched mono model output used by review and existing integrations.
  • {stem}.stereo.wav is the frequency-aware stereo/loudness reconstruction.
  • The stem record and chunk metadata contain the mapping settings and pan/loudness curve; a backfill also writes {chunk}.stereo.json.

In Split data, select a source and use Raw / Stereo mapped above the players. Stereo mapped is selected by default; Raw remains available. The original player never changes. The maps below the players show the smoothed trajectory; left is below the center line and right is above it. Version 2 applies frequency-specific panning but only broadband gain, preventing the mapper from making stems bass-heavy. A source with neither music nor voices is stored as a single SFX stem; Raw and Stereo mapped are PCM-identical and the map is labelled identity · no EQ.

Backfill a completed job after deploying the mapper. The command is idempotent and skips stems that already have a mapped object unless --force is supplied:

ssh ubuntu@ec2-44-211-31-38.compute-1.amazonaws.com \
  "sudo bash -c 'set -a; source /etc/sam-audio-pipeline.env; set +a; \
   cd /home/ubuntu/sam-audio-deploy/pipeline; \
   .venv/bin/sam-pipeline-stereo-backfill --job-id JOB_ID'"

Use --max-chunks N for a canary. Mapping is CPU-only and does not require a SAM or Audio Flamingo restart.

Joined reconstruction and similarity

After stereo mapping and the output sound gate, the pipeline sums only the stored stereo variants into {chunk}.joined.stereo.wav. After all chunks for a record are terminal, it places those joins back on the source timeline and overlap-averages them into source.joined.stereo.wav. The primary dataset score compares that stored source PCM directly with the normalized stereo original, without gain fitting or time alignment:

100 × max(0, 2 × dot(original, joined) / (energy(original) + energy(joined)))

This makes 100 an exact reconstruction; missing-output silence against an audible original, opposite polarity, or another non-positive match scores 0. The source record also retains waveform correlation, left/right scores, level delta, normalized RMSE, SNR, coverage, and final limiter gain. Chunk records retain the same diagnostics plus the included stem types so a low source score can be localized. In Split data, the dataset histogram shows the source-score distribution, the source detail plays Original beside Joined, and each chunk exposes its normalized original and diagnostic join.

Backfill completed jobs from existing stereo companions without rerunning SAM or Audio Flamingo:

ssh ubuntu@ec2-44-211-31-38.compute-1.amazonaws.com \
  "sudo bash -c 'set -a; source /etc/sam-audio-pipeline.env; set +a; \
   cd /home/ubuntu/sam-audio-deploy/pipeline; \
   .venv/bin/sam-pipeline-reconstruction-backfill --job-id JOB_ID'"

The command is idempotent; use --force to regenerate an existing joined artifact or --max-chunks N for a canary.

Original audio profiles

Ingestion probes the first audio stream before the stereo-only gate and stores the original channel count/layout, codec and container, sample rate, bit depth when the codec exposes it, bitrate, and lossless/lossy quality tier. Split data shows these facts in both the source list and source header, including for mono inputs that were skipped before model work.

Backfill existing sources directly from their durable originals:

ssh ubuntu@ec2-44-211-31-38.compute-1.amazonaws.com \
  "sudo bash -c 'set -a; source /etc/sam-audio-pipeline.env; set +a; \
   cd /home/ubuntu/sam-audio-deploy/pipeline; \
   .venv/bin/sam-pipeline-audio-profile-backfill --all'"

The command is idempotent; pass one or more --job-id JOB_ID arguments instead of --all, or use --force to refresh already-profiled sources.

Dashboard URLs

Dashboard state is encoded in clean paths and is safe to refresh or share:

/                                      pipeline and upload overview
/review                                manual review queue
/data/{dataset_id}                     dataset explorer
/data/{dataset_id}/jobs/{job_id}       job within a dataset
/data/{dataset_id}/jobs/{job_id}/sources/{source_id}
                                       exact sound detail

Selecting a tab, job, dataset, or sound updates browser history. Back and forward navigation restore the corresponding view and selected record.

General YouTube random datasets

This sampler searches YouTube directly and does not read AudioSet manifests or timestamps. Randomized, reproducible queries favor program audio containing voices, music, and environmental effects while excluding obvious music-only, sleep, ambience, and playlist results. Each video ID contributes at most one clip.

cd pipeline
uv run sam-pipeline-youtube-random \
  --output ~/Downloads/youtube-random-1000-20260714 \
  --total 1000 \
  --seed 20260714 \
  --query-count 400 \
  --results-per-query 15 \
  --search-workers 8 \
  --download-workers 8 \
  --candidate-multiplier 2 \
  --max-attempts 2500

Continuous 30-second cinematic dataset

sam-cinematic-continuous.service runs a permanent producer/consumer graph. Acquisition publishes quality-gated files atomically; resident M2D and ASR workers consume independent deterministic shards; the assembler writes the append-only accepted catalog; and the publisher releases an exhaustive, immutable S3 snapshot at every 2,500 accepted clips. Manual review is diagnostic feedback and never blocks acceptance.

The SQLite/WAL catalog is authoritative; the live JSON manifest is intentionally constant-size. Review queries the newest 5,000 accepted records directly from the catalog so the page stays responsive while the global count grows into the millions. Each S3 snapshot contains only its new 2,500-record sequence range (for example v2-00000001-00002500), avoiding quadratic cumulative manifests. After S3 publishes READY.json, the local snapshot staging directory is removed. Completed acquisition runs and terminal raw-audio copies are pruned continuously, and accepted audio older than the newest 5,000-record review window is removed only after its snapshot is durable in S3.

The live dataset is under /home/ubuntu/cinematic-continuous-30s. Review it at http://127.0.0.1:18081/ and inspect queues, worker health, rolling throughput, the 10,000-hour ETA, and S3 releases at http://127.0.0.1:18081/progress.

Pool sizes are independent settings in /etc/sam-cinematic-continuous.env:

SAM_CONTINUOUS_SEARCH_WORKERS=8
SAM_CONTINUOUS_DOWNLOAD_WORKERS=8
SAM_CONTINUOUS_M2D_WORKERS=1
SAM_CONTINUOUS_ASR_WORKERS=1
SAM_CONTINUOUS_UPLOAD_CONCURRENCY=10

CPU-heavy source preparation can run on a separate media host while the resident M2D scan remains on the GPU host. Both machines must expose the shared scratch path at the same absolute location:

SAM_MEDIA_WORKER_SSH_TARGET=ubuntu@MEDIA_PRIVATE_IP
SAM_MEDIA_WORKER_SSH_IDENTITY=/home/ubuntu/.ssh/sam-media-worker
SAM_MEDIA_WORKER_REMOTE_TASKS=download,ffmpeg,search,extract
SAM_MEDIA_WORKER_SHARED_TMP=/home/ubuntu/cinematic-continuous-30s/source-work/.staging
SAM_CONTINUOUS_SOURCE_SCAN_CPU_EXEMPT=true

The media host must also allow the independent worker pools to authenticate concurrently. Install deploy/pipeline/sam-media-worker-sshd.conf as /etc/ssh/sshd_config.d/99-sam-media-worker.conf, validate with sshd -t, and reload ssh.service. OpenSSH's default MaxStartups 10:30:100 randomly drops connection bursts well below the configured pipeline concurrency.

extract offloads clip normalization. A passing scan retains its already downloaded full source on the shared media volume, extraction reads the selected regions locally, and the source directory is deleted when extraction becomes terminal. This avoids a second provider transfer and its separate CDN token or anti-bot failure. Legacy scanned jobs whose source was already deleted fall back to selected-section retrieval. ffmpeg also offloads whole-source proxy creation and ASR probe cuts. The source autoscaler treats remote download and extraction concurrency independently from GPU-host CPU pressure, while scan concurrency remains CPU-aware because M2D and scan coordination are resident on the GPU host.

When source proxy FFmpeg is remote, SAM_CONTINUOUS_SOURCE_SCAN_CPU_EXEMPT=true allows additional source threads to keep remote proxy work in flight. It does not remove the scanner's own GPU guard: --inference-concurrency still bounds resident M2D inference independently. Do not enable this setting when proxy creation runs locally.

To run transfers independently from scan, extraction, M2D, and ASR pressure, enable the dedicated source-download pool:

SAM_CONTINUOUS_SOURCE_DOWNLOAD_INDEPENDENT=true
SAM_CONTINUOUS_SOURCE_DOWNLOAD_WORKERS=128
SAM_CONTINUOUS_DOWNLOADED_HIGH_WATER=8192
SAM_CONTINUOUS_DOWNLOADED_HIGH_WATER_BYTES=751619276800
SAM_CONTINUOUS_DOWNLOAD_MINIMUM_FREE_BYTES=214748364800

Independent mode ignores the source autoscaler's downstream concurrency file. Provider circuit breakers still isolate failing services, and downloading stops at the first external storage boundary: the queue item cap, queue byte cap, or 200 GiB free-space floor. This intentionally allows the downloaded queue to grow while later stages are tuned separately.

For the Webshare proxy pool, the live configuration uses a 64-consecutive-failure breaker threshold with a 60-second initial cooldown. This keeps circuit isolation while avoiding provider-wide shutdowns caused by a few bad proxy IPs inside the transfer pool.

The live service pins 11 discovery controllers to providers (two Dailymotion, four Vimeo, four Bilibili, and one guarded YouTube controller) instead of rotating every controller through the same list. This prevents long Bilibili hydration rounds from synchronizing the producers and starving the 240-transfer pool. Each controller owns its seed; the 5,000-source ready high-water bounds search work while a 10,000-source per-provider active limit prevents one provider's downstream backlog from blocking every other provider. Empty discovery batches back off exponentially from 30 seconds to five minutes.

ASR runs on the GPU host and consumes both final clips and prioritized source probes from the media host's shared NFS queue:

SAM_CONTINUOUS_SOURCE_ASR_SHARED_ROOT=/home/ubuntu/cinematic-continuous-30s/source-work/.source-asr
SAM_CONTINUOUS_ASR_DEVICE=cuda
SAM_CONTINUOUS_ASR_COMPUTE_TYPE=float16
SAM_CONTINUOUS_ASR_CONCURRENCY_MIN=4
SAM_CONTINUOUS_ASR_CONCURRENCY_MAX=8
SAM_CONTINUOUS_ASR_CUDA_LIBRARY_PATH=/home/ubuntu/whisper-venv/lib/python3.12/site-packages/nvidia/cublas/lib:/home/ubuntu/whisper-venv/lib/python3.12/site-packages/nvidia/cudnn/lib
SAM_CONTINUOUS_SOURCE_ASR_CPU_FINAL_ENABLED=false
SAM_CONTINUOUS_EXTERNAL_ASR=false

The media host writes probe JSON requests and audio under the shared root. The GPU ASR worker reads them before normal final-validation work and writes results back to the same root, so no explicit file copy or polling over SSH is needed. raw-audio, m2d-validation, and asr-validation resolve to the shared continuous directories through the existing workspace symlinks. The absolute offline model snapshot is recorded with the stable small label, preserving resume behavior across CPU/GPU placement changes. The CPU ASR service is a disabled fallback only; do not enable it while the GPU worker is active.

Restart sam-cinematic-continuous.service after changing a count. Filename hash sharding prevents two M2D or ASR processes from claiming the same clip, and the SQLite/WAL catalog makes replay and worker-count changes idempotent. Promoter, assembler, and snapshot publisher remain single lightweight coordinators.

Throughput uses a rolling 60-minute window. audio min/min is clip duration processed per wall-clock minute and has the same numeric value as audio hours/wall-clock hour. The 10,000-hour estimate uses accepted throughput, not download or model throughput, and is intentionally unavailable until at least one clip has passed every gate.

The frozen 10-second baseline is published at:

s3://sam-audio-pipeline-artifactbucket-ndb3jfc3asyk/cinematic-dialogue-dataset/snapshots/v1-00001000/

Audio is content-addressed under cinematic-dialogue-dataset/audio/{sha256}.wav. Snapshot metadata is published first and READY.json last; consumers must require the ready marker and validate manifest.sha256.

Acquisition is resumable. attempts.jsonl records every accepted, rejected, or unavailable candidate; metadata/candidates.json and metadata/search.json preserve search provenance; manifest.json contains only accepted records and hashes. audit.json is written only after reopening and verifying the complete dataset. The gate requires source audio at 44.1 kHz or better and at least 120 kbps, then verifies exact ten-second PCM16/48 kHz stereo output, real left/right difference, loudness, silent-frame/run limits, clipping, uniqueness, and SHA-256.

Re-run the same command to continue an interrupted build. Audit an existing set without downloading:

uv run sam-pipeline-youtube-random \
  --output ~/Downloads/youtube-random-1000-20260714 \
  --total 1000 --verify-only

Cinematic mixed-audio acquisition

The cinematic profile is based on the completed 216-clip human-review findings in HUMAN_REVIEW_FINDINGS_20260714.md. It targets raw movie/TV/animated scenes, game cutscenes, short films, and produced news packages while rejecting review/reaction/vlog/tutorial/AI-voice metadata. It may retain up to three non-overlapping excerpts from one promising source:

uv run sam-pipeline-youtube-random \
  --output /data/cinematic-raw-20260715 \
  --total 3500 \
  --seed 20260715 \
  --profile cinematic \
  --clips-per-video 3 \
  --query-count 1600 \
  --results-per-query 15 \
  --candidate-multiplier 1.7 \
  --max-attempts 6000

If YouTube rejects the downloader host's public address, put the proxy in a root- or service-user-readable yt-dlp config rather than an environment value or command-line argument. The config must not be committed:

install -d -m 700 /home/ubuntu/.config/sam-audio
install -m 600 /dev/null /home/ubuntu/.config/sam-audio/youtube-proxy.conf
# Write this line through your secret manager, with the actual credentials:
# --proxy http://USERNAME:PASSWORD@p.webshare.io:80/

Use it with the one-shot builder via --youtube-proxy-config. The independent source services use these non-secret environment settings:

SAM_CONTINUOUS_DISCOVERY_SOURCES=bilibili,dailymotion,vimeo,youtube
SAM_CONTINUOUS_DISCOVERY_PROVIDER_WORKERS=dailymotion=2,vimeo=4,bilibili=4,youtube=1
SAM_CONTINUOUS_DISCOVERY_PROVIDER_QUERY_COUNTS=dailymotion=4,vimeo=2,bilibili=25,youtube=2
SAM_CONTINUOUS_DISCOVERED_HIGH_WATER=5000
SAM_CONTINUOUS_PLATFORM_DISCOVERED_HIGH_WATER=10000
SAM_CONTINUOUS_PLATFORM_DISCOVERED_HIGH_WATERS=dailymotion=2500,vimeo=2500,bilibili=2000,youtube=100
SAM_PAID_SEARCH_ENABLED=true
SAM_PAID_SEARCH_DAILY_REQUEST_LIMIT=200
SAM_CONTINUOUS_SOURCE_PROVIDER_MAX_ACTIVE=youtube=8
SAM_CONTINUOUS_SOURCE_PROVIDER_MIN_CLAIM_INTERVAL_SECONDS=youtube=1
SAM_CONTINUOUS_PROVIDER_CIRCUIT_FAILURE_THRESHOLDS=youtube=4
SAM_YOUTUBE_PROXY_CONFIG=/home/ubuntu/.config/sam-audio/youtube-proxy.conf

The Dailymotion lane combines its public API, deeper accepted-channel/related traversal, and a small no-key indexed fallback. Vimeo traverses paginated video lists from already accepted uploaders and hydrates metadata through oEmbed; no paid web search is required. Bilibili retains native search and graph traversal. Every result still passes the same stereo, source-quality, M2D, ASR, and diversity gates.

YouTube is opportunistic: it may use at most 8 of the 240 transfer slots (3.3%), claims at most one new source per second globally, and four consecutive transport failures open only the YouTube circuit. The other provider lanes continue while its exponential recovery cooldown runs.

Indexed-search credentials belong in the root-owned mode-0600 file /etc/sam-source-discovery-secrets.env, which is read only by the discovery service. Do not add values to the repository environment file:

SAM_EXA_API_KEY=...
SAM_EXTERNAL_SEARCH_MIN_INTERVAL_SECONDS=1.05
SAM_EXA_FALLBACK_PERCENT=100

Paid search fails closed without both SAM_PAID_SEARCH_ENABLED=true and a positive daily request limit. A cross-process budget file makes the cap durable across worker restarts. The production lane uses the configured Exa credential only after native discovery is exhausted; proxy-rotated Brave HTML remains the no-key fallback when the daily budget is exhausted.

SAM_YOUTUBE_PROXY_CONFIG may name one config file or a directory containing one mode-0600 .conf file per direct proxy. The service deterministically pins all requests for a source to one config. A stage retry changes the affinity key and therefore fails over through the pool without putting credentials in the frontier or manifests. Search queries are pinned independently.

For a direct pool, create a mode-0700 directory, write one --proxy ... line to each mode-0600 .conf, and point SAM_YOUTUBE_PROXY_CONFIG at the directory.

The live configuration applies the pool to discovery hydration and transfer for all enabled providers. Worker logs and frontier errors redact URL user information, while child process arguments contain only a selected config path. Test the credential before switching discovery; an HTTP 407 response means Webshare rejected the proxy username/password, while Invalid download token means a proxy-list export URL must be regenerated in Webshare.

When one provider fails, its circuit opens independently rather than weakening the selection policy or using account cookies. A one-shot Dailymotion-only run remains available:

uv run sam-pipeline-youtube-random \
  --output /data/cinematic-raw-20260715 \
  --source dailymotion \
  --total 3500 \
  --seed 20260715 \
  --profile cinematic \
  --clips-per-video 12 \
  --query-count 500 \
  --results-per-query 100 \
  --candidate-multiplier 2 \
  --max-attempts 9000

The Dailymotion path uses the public search API for metadata and yt-dlp only for the selected time sections. The same title, uploader, description, and tag exclusions apply, including explicit India/Indian and Indian-language source metadata. It does not infer a speaker's ethnicity or nationality from audio.

Score with --require-cinematic-mix so dialogue, music, and non-music SFX are independent requirements. Materialize with the same flag and --accepted-limit 1000 to create an exact 1,000-record result. The final materialized set retains source URLs, exact timestamps, query provenance, and the human-feedback-derived policy version.

M2D dialogue/background validation

The technical YouTube gate does not prove that spoken dialogue and background activity occur in the selected ten seconds. Validate candidates with the official M2D AudioSet-fine-tuned tagger before using them. The validator scores overlapping two-second windows and requires speech, a non-human background class, and at least three windows in which both are active. Singing, choir, chant, rapping, humming, opera, a capella, vocal music, and song evidence may appear in at most one window, preventing sung vocals from satisfying the speech requirement. It separately records music-led, effects/ambience-led, and mixed instrumental backgrounds.

The model repository, checkpoint, AudioSet label CSV, and ontology are runtime inputs rather than files baked into this repository or its Docker image:

uv run sam-pipeline-m2d-validate score \
  --input-dir /data/youtube-random-1000/audio \
  --output /data/youtube-random-1000/m2d-validation.jsonl \
  --m2d-repo /models/m2d \
  --checkpoint /models/m2d/weights_ep69it3124-0.47929.pth \
  --class-labels /models/audioset/class_labels_indices.csv \
  --ontology /models/audioset/ontology.json \
  --m2d-commit 3d0c4de9447c404a8d3f9f37e04f53bc902e09b3

Create a separate listening-test folder containing only accepted audio. The command hard-links files when possible and never removes the source dataset:

uv run sam-pipeline-m2d-validate materialize \
  --input-dir ~/Downloads/youtube-random-1000-20260714/audio \
  --results ~/Downloads/m2d-validation.jsonl \
  --source-manifest ~/Downloads/youtube-random-1000-20260714/manifest.json \
  --output-dir ~/Downloads/youtube-dialogue-background-m2d-ok-20260714

The output includes manifest.json, audit.json, the complete M2D JSONL, and the accepted WAVs under audio/. balanced-audio/ is a deterministic listening subset with equal music-led and non-music-led counts. Every record keeps the per-window scores, ranks, top labels, temporal coverage, rejection reasons, exact M2D checkpoint hash, and validator policy version.

Policy v5 requires audible voice evidence in at least five 2-second windows: the M2D speech-family probability must be at least 0.10 and rank within the top five labels in each counted window. This strong gate is separate from the older low-confidence speech diagnostic and rejects clips tagged as speech only because of weak background evidence. Materializing an older M2D JSONL applies the current voice gate from its stored per-window scores without rerunning the model.

M2D also records foreground-speech subclass evidence, but it is diagnostic: on the cinematic pilot those sublabels were too sparse to use as a mandatory gate. Confirm audible foreground voice with faster-whisper instead:

PYTHONPATH=/app/pipeline/src /models/whisper-venv/bin/python \
  -m sam_audio_pipeline.m2d_validator asr-score \
  --input-dir /data/cinematic-raw-20260715/audio \
  --output /data/cinematic-raw-20260715/asr-validation.jsonl \
  --model small \
  --download-root /models/faster-whisper

The ASR policy requires at least 1.5 seconds of VAD-positive audio, at least two decoded words, best segment average log probability of -0.80 or better, and no-speech probability no higher than 0.50. Against all 216 completed reviews, it rejected 13 of 14 clips explicitly marked lacking_voice while retaining 86 of 110 Good/Perfect clips. The M2D strong speech check remains required as an independent guard.

The validator detects the spoken language rather than forcing an English decode. Final clips must be detected as English with probability at least 0.80. This is a language-content filter; it does not infer a speaker's nationality or ethnicity from their voice.

The same M2D pass rejects synthetic narration when Speech Synthesizer scores at least 0.20 within the top five labels in two or more windows. This threshold comes directly from the reviewed ai voice failure, where the class scored 0.24–0.72 in all nine windows.

Pass both validation files when creating the exact final set:

uv run sam-pipeline-m2d-validate materialize \
  --input-dir /data/cinematic-raw-20260715/audio \
  --results /data/cinematic-raw-20260715/m2d-validation.jsonl \
  --asr-results /data/cinematic-raw-20260715/asr-validation.jsonl \
  --source-manifest /data/cinematic-raw-20260715/manifest.json \
  --output-dir /data/cinematic-final-1000-20260715 \
  --require-cinematic-mix \
  --accepted-limit 1000

Manual listening review

Start the local review app on the balanced M2D subset:

cd pipeline
uv run sam-pipeline-review \
  --dataset-dir ~/Downloads/youtube-dialogue-background-m2d-ok-20260714 \
  --audio-directory balanced-audio \
  --port 18081

Open http://127.0.0.1:18081/. Each browser first asks for a reviewer name, then receives a random unreviewed clip. Each clip can be marked Good, Perfect, or Not OK. Not OK supports multiple reasons: lacking music, lacking background audio/SFX, singing or vocal music, speech that is not dialogue, low quality, low volume, distortion/clipping, wrong voice/background balance, or an Other reason with a required note. Keyboard shortcuts are shown in the app. Press X to open Not OK, 1–9/0 to toggle its rejection reasons, Enter to save, or Esc to cancel. This keeps multi-reason tagging keyboard-only.

Progress is written atomically after every decision to manual-review.json inside the dataset directory, so closing or refreshing the browser does not lose work. Each clip also has a refreshable /clip/{filename} URL. Export CSV downloads a flat table suitable for analysis and later policy tuning. Use --annotations /some/path.json to store annotations elsewhere, or --audio-directory audio to review all M2D-accepted clips instead of the balanced subset.

The app supports multiple reviewers against one annotation file. Assignment is an atomic random lease: the server never gives a live clip to two reviewers, rejects stale or conflicting submissions with HTTP 409, renews an open clip in the browser, and releases an abandoned clip after 10 minutes. Saving or skipping immediately assigns another random clip. Reviewer identity and attribution are included in JSON and CSV exports. Change the lease with --claim-seconds.

For a shared server, install deploy/pipeline/sam-pipeline-review.service, set SAM_REVIEW_DATASET_DIR in /etc/sam-audio-review.env, and install the Nginx template after replacing __REVIEW_PATH__ with a random path. Keep the Python service bound to 127.0.0.1; only Nginx should be public. The frontend derives its API and audio URLs from the Nginx path prefix, so refreshable clip links also work through the shared URL.

If the EC2 security group cannot accept port 80, run the optional sam-audio-review-tunnel.service. It exposes the Nginx origin through an outbound Cloudflare Quick Tunnel without changing inbound firewall rules. Quick Tunnel hostnames are temporary and can change when the tunnel restarts; use a named Cloudflare Tunnel and DNS hostname for a permanent production URL.

AudioSet validation batches

Acquire a reproducible random sample from the official AudioSet segment CSVs. Unavailable YouTube videos are retained as failed provenance entries and replacement candidates are tried until the requested successful count exists:

cd pipeline
uv run sam-pipeline-audioset \
  --output ~/Downloads/audioset-random-100 \
  --total 100 \
  --seed 20260713 \
  --s3-bucket sam-audio-pipeline-artifactbucket-ndb3jfc3asyk \
  --s3-prefix references/audioset/random-100-v1

Submit every successful manifest entry as one persistent dataset job while retaining its AudioSet labels and YouTube timestamps:

uv run sam-pipeline-submit \
  --api http://127.0.0.1:18080 \
  --manifest ~/Downloads/audioset-random-100/manifest.json \
  --dataset-name "AudioSet random 100 · seed 20260713"

The verified seeded run is available in the panel as dataset 991945c1f082446a9ff66d482838f964; its completed job is 3f0233f2442f4044a0f2967c68cb5a89. The private reference prefix contains the 100 timestamped WAVs plus manifest.json and manifest.sha256.

Review controls

The panel autoplays the next uncertain/failure stem. Keys are:

  • 1: assertion correct / pass
  • 2: assertion incorrect / fail
  • 3: leave pending for later verification
  • Space: play or pause

Human decisions append review history and change effective_status; they never erase automatic_status, Judge/CLAP evidence, timings, or adaptive-route data.

Deployment safety

The repository and both Docker build contexts exclude model checkpoints. Models are downloaded separately to /home/ubuntu/models and referenced at runtime. When syncing systemd launchers, preserve executable modes. Stop only the SAM queue consumer while restarting the SAM model API; queued messages remain safe in SQS and task state remains safe in DynamoDB.

Dialogue/background training snapshots

sam-training-dataset.service is a separate, resumable downstream consumer of the continuous catalog. It never starts, stops, or applies backpressure to source discovery, download, scan, extraction, or the existing 2,500-record snapshot publisher. It also watches its own inbox/ for ZIP files and folders, so older accepted packages can be added without changing acquisition.

For every input it creates an aligned 48 kHz stereo contract:

  • original.wav: deterministic two-pass EBU R128 normalization;
  • dialogue.wav: the stereo-mapped SAM human voices target;
  • background.wav: the final voice-stage residual, retaining music, ambience, and sound effects together;
  • scene_description.txt: an M2D-grounded Audio Flamingo Next caption of the background only;
  • dialogue_transcript.txt: the faster-whisper transcript; and
  • metadata.json: source provenance, timestamped M2D/ASR evidence, SAM judges, adaptive candidate metadata, stereo mapping, reconstruction similarity, artifact hashes, and the three-bucket quality decision.

The current scene-caption contract is revision 4. scene_description.txt is always rendered as an 80–140 word DESCRIPTION: section followed by a TIMELINE: of one to six contiguous intervals covering exactly 00:00–00:30. The publisher rejects model/process boilerplate, missing coverage, and sparse multi-interval placeholders before any immutable snapshot is written. Speech recognized in the background is removed from the text but retained as a review signal.

The publisher commits immutable 1,000-record snapshots. Every completed record is included, including success, review, and failure; records are never silently removed because of their quality result. Each record repeats its bucket and reasons, the manifest records exact bucket counts, and publication fails before upload if the row label and metadata label disagree. Training code should apply the manifest's default filter, quality.bucket == 'success', while review and failure records remain available for audit and policy improvement. After a snapshot's manifest and READY.json are durable in S3, its local work files are removed to bound disk use.

Install the independent worker without changing the acquisition target:

sudo install -m 0644 deploy/pipeline/sam-training-dataset.env \
  /etc/sam-training-dataset.env
sudo install -m 0644 deploy/pipeline/sam-training-dataset.service \
  /etc/systemd/system/sam-training-dataset.service
sudo systemctl daemon-reload
sudo systemctl enable --now sam-training-dataset.service

Inspect progress or add a package:

cd /home/ubuntu/sam-audio-deploy/pipeline
.venv/bin/python -m sam_audio_pipeline.training_dataset \
  --workspace /home/ubuntu/dialogue-background-training-v1 status
cp /path/to/accepted-clips.zip \
  /home/ubuntu/dialogue-background-training-v1/inbox/

Snapshot control objects are under s3://$SAM_TRAINING_S3_BUCKET/dialogue-background-training-v1/snapshots/. Record artifacts are partitioned by their explicit quality bucket beneath the same prefix.

The inbox accepts ZIPs and directories. ZIP extraction is atomic: an interrupted or unsafe archive never becomes a visible package root. Directory packages are incremental and content-addressed, so adding a new file or replacing a file with new bytes creates a new durable job instead of being hidden by the first scan.

The official SAM checkpoint is gated and is deliberately not part of the image or repository. On a replacement GPU host, authenticate an approved Hugging Face account and download it directly on that host:

/home/ubuntu/sam-audio-deploy/.venv/bin/hf auth login
/home/ubuntu/sam-audio-deploy/.venv/bin/hf download \
  facebook/sam-audio-small-tv \
  --local-dir /home/ubuntu/models/sam-audio-small-tv
sudo systemctl enable --now sam-audio-api.service

The training supervisor waits for /healthz before it starts separation, so a missing checkpoint does not consume job retries. Audio Flamingo is independent of SAM. Its API handles the checkpoint's encoder-decoder output without removing input-prompt tokens, requests at most 384 generated tokens, compacts M2D evidence to ten timeline samples, and normalizes valid, truncated, or native caption formats into the same background-only schema. Any speech content removed from a caption is retained as a quality signal and sends the record to review.

Prompt changes are evaluated against saved real stems before deployment:

PYTHONPATH=pipeline/src pipeline/.venv/bin/python \
  -m sam_audio_pipeline.caption_prompt_lab \
  --workspace /home/ubuntu/dialogue-background-training-v1 \
  --api-url http://127.0.0.1:8001 \
  --samples 8 \
  --output /home/ubuntu/dialogue-background-training-v1/logs/caption-prompt-lab.jsonl

Corrective caption streams use dialogue-background-training-vN/snapshots/vN-... and preserve membership of every older immutable revision. The pipeline dashboard reads caption_revision_state, so transformed, published, remaining, and success/review/failure totals automatically follow the newest revision.

Before an immutable snapshot is uploaded, the publisher proves that every record has a unique ID and sequence, all six artifacts, a matching S3 bucket partition and quality label, and counts that reconcile exactly with the manifest. The resulting audit is embedded as verification.status = passed; READY.json is not written if any invariant fails.