Skip to content

feat(cli): Stop hardcode the reranker of choice. Implements #68#85

Merged
Davidyz merged 12 commits into
mainfrom
cli/extensible_reranker
Apr 22, 2025
Merged

feat(cli): Stop hardcode the reranker of choice. Implements #68#85
Davidyz merged 12 commits into
mainfrom
cli/extensible_reranker

Conversation

@Davidyz
Copy link
Copy Markdown
Owner

@Davidyz Davidyz commented Apr 20, 2025

This PR refactored the reranker module to follow a similar paradigm as the embedding functions in chromadb. Users will now use the reranker key in the JSON config to specify the class (backend) for the reranker, and use the reranker_params field to initialise their chosen reranker class.

The original syntax, where users set the reranker model with the reranker key, is still supported, but will throw a warning message in the CLI and be removed in 0.6.0.

@Davidyz Davidyz force-pushed the cli/extensible_reranker branch from 592c08f to d032c0a Compare April 20, 2025 08:10
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 20, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.93%. Comparing base (314cb01) to head (657bef4).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #85      +/-   ##
==========================================
+ Coverage   98.81%   98.93%   +0.11%     
==========================================
  Files          18       21       +3     
  Lines        1266     1312      +46     
==========================================
+ Hits         1251     1298      +47     
+ Misses         15       14       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Davidyz Davidyz marked this pull request as ready for review April 20, 2025 08:31
@Davidyz Davidyz added the enhancement New feature or request label Apr 20, 2025
@Davidyz Davidyz force-pushed the cli/extensible_reranker branch from cf49f8b to 119d329 Compare April 20, 2025 12:31
@Davidyz Davidyz force-pushed the cli/extensible_reranker branch from 119d329 to ed4b3bd Compare April 20, 2025 12:49
@Davidyz
Copy link
Copy Markdown
Owner Author

Davidyz commented Apr 20, 2025

Should be pretty much done here. Still got some clean-up to do, and might try to simplify the rerank method so that it's easier to add support for new reranking backends.

@Davidyz
Copy link
Copy Markdown
Owner Author

Davidyz commented Apr 21, 2025

Planning to add a method RerankerBase.compute_similarity(self, results: list[str], query_message: str) -> Sequence[float]. Concrete child classes would now implement this method that returns a list of similarity scores between the query message and each of the results, and RerankerBase.rerank, which is now a shared method for the base class, will assemble the final results. This saves tons of trouble when implementing a new reranker.

@Davidyz Davidyz force-pushed the cli/extensible_reranker branch from 77ad864 to d61a68a Compare April 21, 2025 03:34
@Davidyz Davidyz force-pushed the cli/extensible_reranker branch from d61a68a to 498ec61 Compare April 21, 2025 03:40
Copy link
Copy Markdown
Owner Author

@Davidyz Davidyz Apr 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ciekawy, any comments on my approach to the extensible rerankers? This base class will simplify the implementation of the reranker by taking care of building final scores for each document/chunk, and you'll only need to implement compute_similarity, which is a lot easier to work with.

EDIT: I should probably make this async, so that it'll work better with API-based reranker.

@Davidyz Davidyz merged commit 20cdaa9 into main Apr 22, 2025
12 checks passed
@Davidyz Davidyz deleted the cli/extensible_reranker branch April 22, 2025 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant