Proposal: Add GGML Tensor Backend Support for Cross-Platform GPU Acceleration

[oh-tarnished]

Hi team,

I’ve been exploring the architecture of Knowhere and had an idea that could potentially improve portability and hardware abstraction across different platforms.

Would the maintainers be open to introducing support for the GGML tensor library as a backend abstraction for GPU operations across indexing/search algorithms?

The motivation is:

• Enable cross-platform GPU acceleration with a lightweight tensor abstraction layer
• Reduce dependency coupling to vendor-specific compute paths
• Allow Milvus at the upper layer to remain more hardware/backend agnostic
• Improve support for heterogeneous environments (CUDA, Metal, Vulkan, CPU fallback, etc.)
• Potentially simplify deployment on edge/local AI systems and Apple Silicon environments

The idea would be to gradually move algorithmic compute paths toward tensor operations that can be mapped through GGML-compatible backends while preserving existing optimized implementations where needed.

I’d be happy to contribute and submit PRs for this effort if the direction aligns with the roadmap or maintainers’ interests.

Would love to hear thoughts or concerns around feasibility, architecture fit, or preferred integration strategy.

Thanks!

[alexanderguzhva]

@oh-tarnished Nice to meet you. Let's discuss!

[oh-tarnished]

Hello @alexanderguzhva, Sorry for getting back a bit late. Here is my proposal

[oh-tarnished]

Proposal: GPU Abstraction + Bazel Build System

Summary: Keep native cuVS for production (fast), add optional GGML for edge devices, switch to Bazel for easy cross-platform builds.

The Problem

Today's Limitations

graph LR
    A["Knowhere Today"]
    
    B["YES: NVIDIA GPU<br/>cuVS<br/>Fast"]
    
    C["NO: Apple Silicon<br/>CPU Only<br/>No GPU"]
    
    D["NO: Edge Devices<br/>Not Supported<br/>Too Heavy"]
    
    E["NO: Build for<br/>Multiple Platforms<br/>Separate Builds<br/>3x Compilation"]
    
    A --> B
    A --> C
    A --> D
    A --> E

Loading

Core Issues:

1. GPU support is NVIDIA-only - What about Apple Metal? AMD? Vulkan?
2. Edge/local AI blocked - Apple Silicon M-series, small devices can't run Knowhere GPU
3. Cross-compilation is painful - Building for x86_64, ARM64, macOS requires 3 separate builds

The Solution: Two Changes

Change 1: GPU Architecture - Hybrid Approach

Idea: Keep the fast NVIDIA path, but add a portable abstraction layer for other platforms.

graph TB
    A["Algorithm<br/>HNSW, IVF, etc."]
    
    B{"GPU?"}
    
    C["Native cuVS<br/>NVIDIA CUDA<br/>Production Standard<br/>No Changes<br/>Fastest"]
    
    D["GGML Abstraction<br/>Optional Layer<br/>Multi-Platform<br/>Compile-Time Only<br/>Future"]
    
    E["CPU SIMD<br/>Fallback<br/>Always Works"]
    
    A --> B
    B -->|"Default"| C
    B -->|"Optional<br/>New"| D
    B -->|"Fallback"| E
 

Loading

Why GGML and not something else?

Option	Why/Why Not
Keep cuVS only	Blocks edge devices, Apple Silicon
Replace cuVS with GGML	Performance hit, breaks production systems
Hybrid (GGML + cuVS)	Fast by default, portable optionally, zero breaking changes

Key point: GGML is vendored locally (copied into knowhere), not a runtime dependency. Choose at compile-time, not runtime.

What Does This Enable?

# Today (production)
make WITH_GPU=True                    # NVIDIA only

# Tomorrow (new use cases)
make WITH_GGML_GPU=True               # Any GPU (Metal, Vulkan, etc.)
make WITH_GGML_GPU=True GGML_BACKEND=metal  # Apple Silicon
make WITH_GPU=True WITH_GGML_GPU=True # Both, pick at build time

Why not break production? Simple: default stays WITH_GPU=True (cuVS). GGML is opt-in WITH_GGML_GPU=True.

Change 2: Build System - CMake to Bazel

Problem:

# Build for x86_64
make WITH_GPU=True
# libknowhere.so (x86_64)

# Build for ARM64 - need separate build!
rm -rf build && cmake ... && make
# libknowhere.so (ARM64)

# Build for macOS - ANOTHER build!
cmake ... && make
# libknowhere.dylib

# 3 platforms = 3 builds = 3x compilation time

Solution with Bazel:

# ONE command, ALL platforms
bazel build \
  --platforms=@knowhere//bazel/platforms:x86_64 \
  --platforms=@knowhere//bazel/platforms:arm64 \
  --platforms=@knowhere//bazel/platforms:macos_arm64 \
  --platforms=@knowhere//bazel/platforms:gpu_cuda \
  //src:knowhere

# libknowhere.so (x86_64)
# libknowhere.so (ARM64)
# libknowhere.dylib (macOS)
# libknowhere.so (GPU) - all in parallel!

Why Bazel?

Feature	CMake	Bazel
Multi-platform single command	No	Yes
Hermetic builds (reproducible)	Partial	Yes
Content-addressed cache	Partial	Yes
Easy cross-compilation	Complex	Simple

Why not stay with CMake? CMake is verbose for this. Bazel is literally designed for multi-platform hermetic builds.

Appendix: Why GGML Specifically?

GGML strengths:

• Multi-platform support (Metal, Vulkan, CUDA, CPU)
• Lightweight (perfect for vector search kernels)
• Active community
• Works on edge devices
• No external runtime dependency (vendored locally)

Alternatives considered:

• OneDNN: GPU-only, slower, heavier
• TVM: Complex, overkill for vector search
• Native implementations: 3x code, 3x maintenance
• SYCL: Good but less ecosystem

GGML is the sweet spot.

[alexanderguzhva]

@oh-tarnished thinking

[oh-tarnished]

Sure let me know that works. Happy to answer any doubts or questions.

[oh-tarnished]

@alexanderguzhva any thoughts to continue on this ?