Skip to content

Commit e3aa0da

Browse files
committed
docs(archive): document resumable transcription pipeline and ffmpeg stdin bug
1 parent 12893a9 commit e3aa0da

1 file changed

Lines changed: 21 additions & 6 deletions

File tree

backlog/docs/architecture/transcript-import.md

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,28 +15,43 @@ id: doc-008
1515

1616
## Automated YouTube Transcription (Async)
1717

18-
For missing transcripts of videos hosted on YouTube, we use a local asynchronous pipeline powered by `zdots-ctx` and `whisper.cpp`.
18+
For missing transcripts of videos hosted on YouTube, we use a local asynchronous pipeline powered by `zdots-ctx` and `whisper.cpp`. This process is designed to be highly resilient and fully resumable.
1919

2020
1. **Enqueue Pending Videos:**
2121
Identify all pending `video_assets` with a YouTube ID and enqueue them to the local `zdots-ctx` worker.
2222
```bash
2323
./bin/batch_ztranscribe.rb
2424
```
25-
2. **Start the Background Worker:**
26-
Ensure the worker is running to process the queue asynchronously.
25+
2. **Start the Background Worker (Safely):**
26+
Ensure the worker is running to process the queue asynchronously.
27+
**CRITICAL:** When running in the background, you MUST redirect stdin from `/dev/null`. If you do not, the underlying `ffmpeg` process (used by `yt-transcribe` to convert audio) will attempt to read from the terminal, immediately suspending the entire worker process.
2728
```bash
28-
zdots-ctx worker --type transcription
29+
# Run in background safely:
30+
zdots-ctx worker --type transcription < /dev/null &
2931
```
30-
3. **Stage Completed Transcripts:**
32+
3. **Pausing and Resuming (Interruption Recovery):**
33+
If the process is interrupted (e.g., power loss, internet outage, or manual kill):
34+
- Clear any jobs that were stuck in the "running" state when the interruption occurred:
35+
```bash
36+
zdots-ctx clear-stale-jobs
37+
```
38+
- Restart the worker using the safe command from Step 2. It will automatically pick up where it left off.
39+
4. **Stage Completed Transcripts:**
3140
As the worker finishes downloading and transcribing to `~/Downloads/transcripts/`, map them back to their canonical `video_asset_id` and stage them for ingestion.
3241
```bash
3342
./bin/stage_completed_transcripts.rb
3443
```
35-
4. **Ingest Staged Transcripts:**
44+
5. **Ingest Staged Transcripts:**
3645
Use the standard pipeline to ingest the staged files into `_data/transcripts/`.
3746
```bash
3847
./bin/transcripts ingest --source-dir tmp/transcript-id-staging --min-confidence 0.9 --auto-commit
3948
```
49+
6. **Audit and Extract Insights:**
50+
Once ingested, prepare the newly available transcripts for the conversational audit to extract SEO metadata, separate speakers, and pull durable insights.
51+
```bash
52+
bundle exec rake audit:prepare_wave
53+
```
54+
Then activate the `transcript-conversational-audit` skill to process the generated prompts.
4055

4156
## Canonical Model
4257

0 commit comments

Comments
 (0)