Skip to content

feat(local): concurrent frame transcription for local models#302

Open
julio4 wants to merge 1 commit into
JerryZLiu:mainfrom
julio4:feat/local-llm-concurrency
Open

feat(local): concurrent frame transcription for local models#302
julio4 wants to merge 1 commit into
JerryZLiu:mainfrom
julio4:feat/local-llm-concurrency

Conversation

@julio4

@julio4 julio4 commented Jul 5, 2026

Copy link
Copy Markdown

Local batch processing describes screenshots one at a time; roughly 15 sequential describe_frame vision calls per batch, which dominates transcription time. This PR makes that concurrency configurable so users whose local server can handle parallel requests can run several frame descriptions in flight and cut transcription time.

Opt-in: the default is 1 (today's sequential behavior), so nothing changes unless the user raises it.

Changes

  • New setting llmLocalMaxConcurrency (Int, default 1, clamped to 1…16), read by OllamaProvider.maxConcurrency.
  • Parallel frame description: OllamaProvider.transcribeScreenshots now runs describe_frame calls through a bounded withTaskGroup (up to maxConcurrency in flight), re-sorting results by timestamp. The dependent segment → summary → title chain is untouched (it can't be parallelized).
  • Settings UI: a "Max concurrent requests" stepper (1–16) in Settings → Providers, shown only for the local provider, wired through ProvidersSettingsViewModel.
  • OllamaProvider marked @unchecked Sendable so the task-group closures can capture it (effectively immutable; let endpoint + UserDefaults reads), matching the existing convention (StorageManager, ScreenRecorder).

Why manual setting

The local server's concurrency limit isn't exposed over HTTP: LM Studio reports model state/quant/context via /api/v0/models but not "Max Concurrent Predictions", and Ollama's parallelism is the OLLAMA_NUM_PARALLEL env var. So Dayflow can't detect it; the user sets it to match their server (keep it ≤ the server's own limit; going higher just queues server-side).

Notes / expectations

  • Speedup is GPU-bound. N concurrent is not N× throughput; on a healthy setup transcription typically drops ~1.5–2.5×.
  • Only the independent describe_frame calls are parallelized.

Risk / scope

  • Opt-in; no behavior change unless the setting is raised above 1.
  • Contained to the local-provider transcription path + its settings UI.

Describe screenshots with up to llmLocalMaxConcurrency describe_frame calls
in flight against the local server (Ollama/LM Studio), instead of strictly
one at a time. New setting defaults to 1 (sequential); a Max concurrent
requests stepper is added to the local provider settings. The server's
concurrency limit isn't exposed over HTTP, so the value is user-set.
@julio4 julio4 changed the title Feat: concurrent frame transcription for local models feat(local): concurrent frame transcription for local models Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant