Megabyte-based cache eviction

[luis-gasparschroeder]

Description

The current vCache implementation evicts based on the number of embeddings in the cache. To enable a more fine-grained eviction, vCache should evict based on a MegaByte target.

Impact

• Who: End user
• What: Fine-grained eviction
• Why: Real-world applicability

Proposed Solution

In addition to the number of embedding eviction thresholds, a user should be able to enter a maximum MB amount. It's important that the MB calculation includes the metadata.

Acceptance Criteria

• Followed coding conventions
• Implemented or updated tests
• Support maximum MB eviction threshold
• MB calculation includes embedding metadata

Risks & Dependencies

--

Additional Context

--