Commit 8fab5b3
authored
Add pluggable backend architecture to cuvs-bench (rapidsai#1536)
## Summary
This PR introduces a pluggable backend architecture for `cuvs-bench`, enabling extensibility to support multiple benchmark backend types beyond the current C++ Google Benchmark executables.
## Motivation
Currently, `cuvs-bench` is tightly coupled to C++ Google Benchmark executables via subprocess calls. This limits the ability to:
- Benchmark Python-native libraries (e.g., hnswlib-python, faiss-python)
- Test network-based vector databases (e.g., Milvus, Qdrant, Elasticsearch)
- Allow users to add custom backends without modifying core cuvs-bench code
This PR lays the foundation for a plugin system that addresses these limitations.
## Changes
### Phase 1: Core Infrastructure
- **`backends/base.py`**: Defines the `BenchmarkBackend` abstract interface and data structures (`Dataset`, `BuildResult`, `SearchResult`)
- **`backends/registry.py`**: Implements `BackendRegistry` for centralized plugin management with support for:
- Dynamic loading from Python modules
- Configuration-based loading from YAML files
- Runtime plugin registration
- **`backends/__init__.py`**: Package initialization with auto-registration of built-in backends
### Phase 2: C++ Backend Wrapper
- **`backends/cpp_gbench.py`**: `CppGoogleBenchmarkBackend` wraps existing C++ executables to maintain **100% backward compatibility**
- Handles subprocess execution, temporary file management, and Google Benchmark JSON parsing
- Auto-registered on import as `"cpp_gbench"`
### Testing
- **`tests/test_registry.py`**: Comprehensive unit tests for registry and base infrastructure (20 tests)
- **`tests/test_cpp_gbench.py`**: Unit tests for C++ backend wrapper (8 tests)
- **All 28 tests passing** ✅
## Backward Compatibility
✅ Existing C++ benchmarks continue to work without modification
✅ No changes required to existing benchmark configurations
✅ Current CLI and orchestration remain functional
## Future Work
This PR enables future phases:
- **Phase 3**: Python in-process backend (hnswlib-python, numpy-based algorithms)
- **Phase 4**: Network VDB backend (Milvus, Qdrant, Elasticsearch)
- **Phase 5**: Migration of existing `cuvs_bench.run` orchestration to use the plugin system
## Testing Instructions
```bash
cd python/cuvs_bench
python -m pytest cuvs_bench/tests/test_registry.py cuvs_bench/tests/test_cpp_gbench.py -v
```
Closes rapidsai#1100
Authors:
- Joseph Nke (https://github.com/jnke2016)
- Corey J. Nolet (https://github.com/cjnolet)
Approvers:
- Bradley Dice (https://github.com/bdice)
- Dante Gama Dessavre (https://github.com/dantegd)
URL: rapidsai#15361 parent 88c738e commit 8fab5b3
24 files changed
Lines changed: 4440 additions & 14 deletions
File tree
- conda/environments
- python
- cuvs_bench
- cuvs_bench
- backends
- orchestrator
- run
- tests
- cuvs
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
49 | | - | |
| 49 | + | |
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
49 | | - | |
| 49 | + | |
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
50 | | - | |
| 50 | + | |
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
50 | | - | |
| 50 | + | |
51 | 51 | | |
52 | 52 | | |
53 | 53 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
| 44 | + | |
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
44 | 44 | | |
45 | 45 | | |
46 | 46 | | |
47 | | - | |
| 47 | + | |
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
45 | | - | |
| 45 | + | |
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
48 | | - | |
| 48 | + | |
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
450 | 450 | | |
451 | 451 | | |
452 | 452 | | |
453 | | - | |
| 453 | + | |
454 | 454 | | |
455 | 455 | | |
456 | 456 | | |
| |||
488 | 488 | | |
489 | 489 | | |
490 | 490 | | |
491 | | - | |
| 491 | + | |
492 | 492 | | |
493 | 493 | | |
494 | 494 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
41 | | - | |
| 41 | + | |
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| |||
0 commit comments