feat(local): concurrent frame transcription for local models#302
Open
julio4 wants to merge 1 commit into
Open
Conversation
Describe screenshots with up to llmLocalMaxConcurrency describe_frame calls in flight against the local server (Ollama/LM Studio), instead of strictly one at a time. New setting defaults to 1 (sequential); a Max concurrent requests stepper is added to the local provider settings. The server's concurrency limit isn't exposed over HTTP, so the value is user-set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Local batch processing describes screenshots one at a time; roughly 15 sequential
describe_framevision calls per batch, which dominates transcription time. This PR makes that concurrency configurable so users whose local server can handle parallel requests can run several frame descriptions in flight and cut transcription time.Opt-in: the default is
1(today's sequential behavior), so nothing changes unless the user raises it.Changes
llmLocalMaxConcurrency(Int, default1, clamped to1…16), read byOllamaProvider.maxConcurrency.OllamaProvider.transcribeScreenshotsnow runsdescribe_framecalls through a boundedwithTaskGroup(up tomaxConcurrencyin flight), re-sorting results by timestamp. The dependentsegment → summary → titlechain is untouched (it can't be parallelized).ProvidersSettingsViewModel.OllamaProvidermarked@unchecked Sendableso the task-group closures can capture it (effectively immutable;let endpoint+UserDefaultsreads), matching the existing convention (StorageManager,ScreenRecorder).Why manual setting
The local server's concurrency limit isn't exposed over HTTP: LM Studio reports model state/quant/context via
/api/v0/modelsbut not "Max Concurrent Predictions", and Ollama's parallelism is theOLLAMA_NUM_PARALLELenv var. So Dayflow can't detect it; the user sets it to match their server (keep it ≤ the server's own limit; going higher just queues server-side).Notes / expectations
describe_framecalls are parallelized.Risk / scope