[Sementic search] MangoDB migration & Incremental Index Refresh#1139
[Sementic search] MangoDB migration & Incremental Index Refresh#1139johngerving wants to merge 5 commits into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 605cb79b04
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| { courseId: 1, year: 1, semester: 1 }, | ||
| { unique: true } |
There was a problem hiding this comment.
The unique index on (courseId, year, semester) conflicts with how embeddings are currently written in apps/semantic-search/app/engine.py (it populates courseId from courseNumber, not the globally unique Class.courseId). Because courseNumber values repeat across departments (for example many subjects have the same catalog number in a term), refresh can hit duplicate-key BulkWriteError during upserts and leave the semantic index build failed or partial. Including subject in this unique key (or storing the true courseId) is needed to avoid cross-department collisions.
Useful? React with 👍 / 👎.
vaclisinc
left a comment
vaclisinc
left a comment
There was a problem hiding this comment.
So far great work! I will first deploy on dev and also test on local dev now, and do code review asap.
BTW, just a reminder, after you create the PR, remember also deploy to dev in the future!
vaclisinc
changed the title
|
|
||
| try: | ||
| raw_results = list(self._embeddings_col.aggregate(pipeline)) | ||
| except Exception as exc: | ||
| raise RuntimeError( | ||
| f"Vector search failed for {semester} {year}. " | ||
| "The search index may still be building. Please try again shortly." | ||
| ) from exc | ||
|
|
||
| scored = [ | ||
| (r["score"], r.get("subject"), r.get("courseId")) | ||
| for r in raw_results | ||
| if r.get("score", 0.0) >= threshold | ||
| ] |
There was a problem hiding this comment.
if the index is not ready, it shouldn't provide the option for ppl to use semantic search - need to deep into
2 participants
Summary
Migrates the semantic search service from Redis to MongoDB Atlas Vector Search and adds incremental refresh so re-indexing only re-encodes courses whose content has actually changed.
Changes
Storage backend: Redis → MongoDB (22e65e5)
redisvl/rediswithpymongoinapps/semantic-search/requirements.txtcourseEmbeddingscollection alongside the rest of the app's MongoDB data, queried via$vectorSearchagainst an Atlas Vector Search index (course_embeddings_vector_search)CourseEmbeddingModelMongoose schema inpackages/commonwith indexes on(year, semester)and a unique(courseId, year, semester)year/semester/subjectfilter fields) indocker/mongodb/init/01-create-search-indexes.jsdocker-compose.ymlto depend onmongodb(healthy) and swapREDIS_URI→MONGODB_URIDrop embedding-result caching (205cac3)
Incremental refresh (605cb79)
refresh()now hashes each course's text (SHA-256), compares againstcontentHashstored in MongoDB, and only re-encodes courses that are new or changedDeleteOnepassmax_termsare evicted from MongoDB on each refresh