Fix OOM in search-sync + schedule daily cron#313
Merged
Conversation
Replaces the r1-search-gsheet-sync cron entry with device-definitions-search-sync (daily 13:00 UTC) and drops the sync-r1-compatibilty subcommand and its registration. The runtime read path for r1_compatibility is untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
QueryDefinitionsCustom built its SQL with OFFSET set to the raw pageIndex instead of pageIndex*500, so each page overlapped the previous by 499 rows. The caller in sync-device-definitions-search also broke only when a page returned fewer than 50 rows (vs. the 500-row page size), so for any manufacturer with more than 50 definitions the loop kept advancing by one row while appending a near-duplicate page each iteration. The resulting documents slice grew by roughly N*(N-50)/2 per make and drove the job OOM. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ility Splits Execute() into a SearchIndexer interface (with a typesense-backed implementation) and two pure orchestration functions: buildManufacturerDocuments converts a single manufacturer's tableland definitions into search entries, and runSearchSync walks manufacturers and upserts one make at a time. The per-make flush bounds steady-state memory to one make's documents instead of retaining the entire catalog until the end, and switches uploads from per-document Upsert calls to the batched Documents().Import upsert action. Adds unit tests covering the year filter, field population, pagination termination, error propagation, and per-make flush behaviour. Tests use the existing gomock mocks for IdentityAPI and DeviceDefinitionOnChainService plus a generated MockSearchIndexer; no database is required. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
values-prod.yamlto runsync-device-definitions-searchdaily at 13:00 UTC (previouslysync-r1-compatibilty, now deprecated and removed).QueryDefinitionsCustom(OFFSET was set to the raw page index, notpageIndex * pageSize) and a mismatched termination threshold in the sync loop (broke at<50for a 500-row page size). Combined, every manufacturer with more than 50 definitions appended a near-duplicate 500-row page per iteration until the sliding window finally fell below 50 — thedocumentsslice grew by roughlyN*(N-50)/2per make and OOM'd the job.SearchIndexerinterface plus two pure orchestration functions (buildManufacturerDocuments,runSearchSync). Uploads are flushed per manufacturer so steady-state memory is bounded by a single make, and per-documentUpsertcalls are replaced by batchedDocuments().Import(... upsert).Test plan
go build ./...andgo vet ./...cleango test ./cmd/device-definitions-api/...— 9 new tests pass🤖 Generated with Claude Code