KBCInfer - Inference Engine

Project Overview

KBCInfer is a lightweight inference engine designed to run Qwen family models locally or on servers. It contains operator implementations, CUDA kernels, model loading utilities, and tokenizer support. The codebase includes core inference logic, operators (op), kernels (kernel), sampling strategies (sampler), and a suite of tests and demos.

Key Features

• CUDA-accelerated kernel implementations for common primitives (GEMV, attention, normalization).
• Tokenizer and model parameter loading utilities.
• Example demos and tests for easy validation and debugging.

Repository Layout

• CMakeLists.txt: Top-level CMake build script.

• demo/: Demonstration programs and example entry points (e.g. mainQwen3.cpp).

• KBCInfer/: Core project sources.

  • include/: Public headers and interfaces.
    • base/: Core utilities (e.g. allocator, buffer, base types).
    • kernel/: Kernel interfaces and declarations.
    • model/: Model-related types and loading interfaces.
    • op/: Operator interfaces and high-level wrappers.
    • sampler/: Sampling strategies.
    • tokenizer/: Tokenizer and vocabulary support.
  • source/: Implementation files corresponding to the headers.

• test/: Tests and unit/integration tests.

  • testMain.cpp: Test runner entry.
  • testBase/: Tests for base utilities (e.g. testBuffer.cpp, testTensor.cpp).
  • testKernel/: CUDA kernel tests (*.cu files).
  • testOp/, testModel/: Operator and model tests.

• build/: CMake build output (local, typically not committed).

• tools/: Helper scripts and utilities (export, conversion scripts, etc.).

Dependencies

• A C++17-compatible compiler (g++ or clang++).
• CMake (recommended >= 3.18).
• CUDA toolkit (if building and running CUDA kernels).

Build & Run (example)

From the project root:

mkdir -p build && cd build
cmake ..
make -j$(nproc)

After a successful build, binaries will be located in the build/ directory (for example demoQwen3, testMain). Run a demo or tests:

./build/demoQwen3 $(qwen3WeightPath) $(tokenizerPath)
# or run tests
./build/testMain