Skip to content

feat: add support for multimodal indexing#286

Draft
kyteinsky wants to merge 28 commits into
masterfrom
feat/noid/ocr-support
Draft

feat: add support for multimodal indexing#286
kyteinsky wants to merge 28 commits into
masterfrom
feat/noid/ocr-support

Conversation

@kyteinsky

@kyteinsky kyteinsky commented Mar 13, 2026

Copy link
Copy Markdown
Contributor

for images (OCR) and audio (Speech-to-text) files

some rough edges and optimisation left on the table for now, will be fixed eventually, it seems to be working for now.

kyteinsky added 17 commits March 5, 2026 16:22
@kyteinsky
feat: add kubernetes app role selection
8e53243 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
feat: add thread start and stop logic
c282f3d 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
wip: migrate the indexing process
ea5208a 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
wip: parallelize file parsing and processing based on cpu count
b3d0c22 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
ci: use the kubernetes branch of context_chat
a9b8c8f 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix typo
8867c1b 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
migrate the update process to be thread based
3c4d698 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix pydantic types
3f2d57d 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix: use a dedicated event to allow app halt without app being disabled
fc5ca61 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix fetch url and pydantic types
23d89b9 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix: use the correct file id
c8399b5 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix: wip: improve embeddings exception handling
17b32b6 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix(ci): update to the latest changes
759f2c2 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix(ci): use file to store stderr
dda28b2 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix(ci): add cron jobs
9806225 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
fix(ci): do a occ files scan before cron jobs
84f6896 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky
feat: add support for multimodal indexing
634db90 
for images (OCR) and audio (Speech-to-text) files

Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky kyteinsky force-pushed the feat/noid/rev-content-flow branch 2 times, most recently from 3459ba5 to 85d29f1 Compare March 26, 2026 20:26
@kyteinsky
feat: add ocr to images inside pdfs
3c3021a 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky kyteinsky force-pushed the feat/noid/rev-content-flow branch 2 times, most recently from 1dc89e7 to 073f9d0 Compare April 7, 2026 11:33
@marcelklehr marcelklehr force-pushed the feat/noid/rev-content-flow branch from 44867af to 7337d17 Compare April 15, 2026 10:53
Base automatically changed from feat/noid/rev-content-flow to master April 30, 2026 11:28
kyteinsky added 2 commits May 1, 2026 13:14
@kyteinsky
merge origin/master
205c345
@kyteinsky
fix(multimodal): smooth rough edges
50d5ed2 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky kyteinsky force-pushed the feat/noid/ocr-support branch 3 times, most recently from f59454c to fc00630 Compare May 2, 2026 15:00
@kyteinsky kyteinsky force-pushed the feat/noid/ocr-support branch 2 times, most recently from 591ab8e to 87d1057 Compare May 2, 2026 17:07
@kyteinsky
feat(ci): add ocr/stt integration test
64f311d 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky kyteinsky force-pushed the feat/noid/ocr-support branch from 87d1057 to 64f311d Compare May 3, 2026 08:28
@kyteinsky
fix(ci): add env vars in the integration-test itself
3b77c9f 
Signed-off-by: Anupam Kumar <kyteinsky@gmail.com>
@kyteinsky kyteinsky force-pushed the feat/noid/ocr-support branch from 59bf2bb to 3b77c9f Compare May 3, 2026 11:41
kyteinsky added 6 commits May 3, 2026 18:21
@kyteinsky
fix(ci): only include text files in the corpus
d28aa89 
Signed-off-by: kyteinsky <kyteinsky@gmail.com>
@kyteinsky
fix(injest): allow all supported files through pydantic checks
04322ef 
Signed-off-by: kyteinsky <kyteinsky@gmail.com>
@kyteinsky
fix: pipeline and CI fixes
aa8a89d 
- do not OCR images in PDF
- fetch task type availability in the fork process
- log as warning if subprocess emits stderr
- mark items are retryable if task type is absent
- other minor fixes

Signed-off-by: kyteinsky <kyteinsky@gmail.com>
@kyteinsky
fix(ci): 10 secs cooldowns between taskproc tasks
ce1b63b 
Signed-off-by: kyteinsky <kyteinsky@gmail.com>
@kyteinsky
fix(ci): 30 secs cooldown in stats and early stop if no queued doc left
5b6a0fb 
Signed-off-by: kyteinsky <kyteinsky@gmail.com>
@kyteinsky
fix(ci): delimiter fix in prompt runner
2fde706 
Signed-off-by: kyteinsky <kyteinsky@gmail.com>
@kyteinsky

kyteinsky commented May 5, 2026

Copy link
Copy Markdown
Contributor Author

maybe we can pick it up later again

@kyteinsky kyteinsky closed this May 5, 2026
@kyteinsky

kyteinsky commented May 13, 2026

Copy link
Copy Markdown
Contributor Author

OCR task type supports pdfs natively so they can be directly sent in task processing task with the NC file id without parsing it locally, just like the image files.
The pdf parser can still be a fallback if the OCR task type is absent or fails.
The calls to context_chat PHP app can be dropped too, from here and in context_chat's PR.

@kyteinsky kyteinsky reopened this Jun 29, 2026
@kyteinsky

kyteinsky commented Jun 29, 2026

Copy link
Copy Markdown
Contributor Author

investigate if sending the pdf as images is better even when the pdf is partilly or fully text
easy on the cpu/gpu, context lengths, etc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@marcelklehr marcelklehr marcelklehr approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kyteinsky @marcelklehr