Skip to content

Commit 8fab5b3

Browse files
authored
Add pluggable backend architecture to cuvs-bench (rapidsai#1536)
## Summary This PR introduces a pluggable backend architecture for `cuvs-bench`, enabling extensibility to support multiple benchmark backend types beyond the current C++ Google Benchmark executables. ## Motivation Currently, `cuvs-bench` is tightly coupled to C++ Google Benchmark executables via subprocess calls. This limits the ability to: - Benchmark Python-native libraries (e.g., hnswlib-python, faiss-python) - Test network-based vector databases (e.g., Milvus, Qdrant, Elasticsearch) - Allow users to add custom backends without modifying core cuvs-bench code This PR lays the foundation for a plugin system that addresses these limitations. ## Changes ### Phase 1: Core Infrastructure - **`backends/base.py`**: Defines the `BenchmarkBackend` abstract interface and data structures (`Dataset`, `BuildResult`, `SearchResult`) - **`backends/registry.py`**: Implements `BackendRegistry` for centralized plugin management with support for: - Dynamic loading from Python modules - Configuration-based loading from YAML files - Runtime plugin registration - **`backends/__init__.py`**: Package initialization with auto-registration of built-in backends ### Phase 2: C++ Backend Wrapper - **`backends/cpp_gbench.py`**: `CppGoogleBenchmarkBackend` wraps existing C++ executables to maintain **100% backward compatibility** - Handles subprocess execution, temporary file management, and Google Benchmark JSON parsing - Auto-registered on import as `"cpp_gbench"` ### Testing - **`tests/test_registry.py`**: Comprehensive unit tests for registry and base infrastructure (20 tests) - **`tests/test_cpp_gbench.py`**: Unit tests for C++ backend wrapper (8 tests) - **All 28 tests passing** ✅ ## Backward Compatibility ✅ Existing C++ benchmarks continue to work without modification ✅ No changes required to existing benchmark configurations ✅ Current CLI and orchestration remain functional ## Future Work This PR enables future phases: - **Phase 3**: Python in-process backend (hnswlib-python, numpy-based algorithms) - **Phase 4**: Network VDB backend (Milvus, Qdrant, Elasticsearch) - **Phase 5**: Migration of existing `cuvs_bench.run` orchestration to use the plugin system ## Testing Instructions ```bash cd python/cuvs_bench python -m pytest cuvs_bench/tests/test_registry.py cuvs_bench/tests/test_cpp_gbench.py -v ``` Closes rapidsai#1100 Authors: - Joseph Nke (https://github.com/jnke2016) - Corey J. Nolet (https://github.com/cjnolet) Approvers: - Bradley Dice (https://github.com/bdice) - Dante Gama Dessavre (https://github.com/dantegd) URL: rapidsai#1536
1 parent 88c738e commit 8fab5b3

24 files changed

Lines changed: 4440 additions & 14 deletions

conda/environments/all_cuda-129_arch-aarch64.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ dependencies:
4646
- recommonmark
4747
- rust
4848
- scikit-build-core>=0.11.0
49-
- scikit-learn
49+
- scikit-learn>=1.5
5050
- sphinx-copybutton
5151
- sphinx-markdown-tables
5252
- sphinx>=8.0.0

conda/environments/all_cuda-129_arch-x86_64.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ dependencies:
4646
- recommonmark
4747
- rust
4848
- scikit-build-core>=0.11.0
49-
- scikit-learn
49+
- scikit-learn>=1.5
5050
- sphinx-copybutton
5151
- sphinx-markdown-tables
5252
- sphinx>=8.0.0

conda/environments/all_cuda-131_arch-aarch64.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ dependencies:
4747
- recommonmark
4848
- rust
4949
- scikit-build-core>=0.11.0
50-
- scikit-learn
50+
- scikit-learn>=1.5
5151
- sphinx-copybutton
5252
- sphinx-markdown-tables
5353
- sphinx>=8.0.0

conda/environments/all_cuda-131_arch-x86_64.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ dependencies:
4747
- recommonmark
4848
- rust
4949
- scikit-build-core>=0.11.0
50-
- scikit-learn
50+
- scikit-learn>=1.5
5151
- sphinx-copybutton
5252
- sphinx-markdown-tables
5353
- sphinx>=8.0.0

conda/environments/bench_ann_cuda-129_arch-aarch64.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ dependencies:
4141
- pyyaml
4242
- rapids-build-backend>=0.4.0,<0.5.0
4343
- requests
44-
- scikit-learn
44+
- scikit-learn>=1.5
4545
- setuptools>=77.0.0
4646
- sysroot_linux-aarch64==2.28
4747
- wheel

conda/environments/bench_ann_cuda-129_arch-x86_64.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -44,7 +44,7 @@ dependencies:
4444
- pyyaml
4545
- rapids-build-backend>=0.4.0,<0.5.0
4646
- requests
47-
- scikit-learn
47+
- scikit-learn>=1.5
4848
- setuptools>=77.0.0
4949
- sysroot_linux-64==2.28
5050
- wheel

conda/environments/bench_ann_cuda-131_arch-aarch64.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ dependencies:
4242
- pyyaml
4343
- rapids-build-backend>=0.4.0,<0.5.0
4444
- requests
45-
- scikit-learn
45+
- scikit-learn>=1.5
4646
- setuptools>=77.0.0
4747
- sysroot_linux-aarch64==2.28
4848
- wheel

conda/environments/bench_ann_cuda-131_arch-x86_64.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ dependencies:
4545
- pyyaml
4646
- rapids-build-backend>=0.4.0,<0.5.0
4747
- requests
48-
- scikit-learn
48+
- scikit-learn>=1.5
4949
- setuptools>=77.0.0
5050
- sysroot_linux-64==2.28
5151
- wheel

dependencies.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -450,7 +450,7 @@ dependencies:
450450
common:
451451
- output_types: [conda, requirements, pyproject]
452452
packages:
453-
- scikit-learn
453+
- scikit-learn>=1.5
454454
java:
455455
common:
456456
- output_types: conda
@@ -488,7 +488,7 @@ dependencies:
488488
- pandas
489489
- pyyaml
490490
- requests
491-
- scikit-learn
491+
- scikit-learn>=1.5
492492
- output_types: [conda]
493493
packages:
494494
- matplotlib-base>=3.9

python/cuvs/pyproject.toml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ test = [
3838
"cupy-cuda13x>=13.6.0",
3939
"pytest-cov",
4040
"pytest<9.0.0",
41-
"scikit-learn",
41+
"scikit-learn>=1.5",
4242
] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
4343

4444
[project.urls]

0 commit comments

Comments
 (0)