Add testing guide with benchmark data and update CLAUDE.md

TimDettmers · claude · TimDettmers · commit c1666aa5ac83 · 2026-02-13T23:32:21.000-05:00
Documents optimal test parallelization (-n 4) based on benchmarks
across two machines (8-core/RTX 4090 and 32-core/RTX PRO 6000 Blackwell).
Includes CPU/GPU utilization analysis, known Blackwell-specific test
failures, and practical pytest recommendations.

Co-Authored-By: Claude Opus 4.6 &lt;noreply@anthropic.com&gt;
diff --git a/.gitignore b/.gitignore
@@ -156,3 +156,5 @@ dmypy.json
 dependencies
 cuda_build
 output/
+cuda-spec.md
+cuda-spec-additions.md
diff --git a/CLAUDE.md b/CLAUDE.md
@@ -0,0 +1,21 @@
+# Parallel sessions
+
+To work on multiple branches at once, use git worktrees:
+
+```bash
+git worktree add ../bitsandbytes-<branch-name> -b <branch-name>
+cd ../bitsandbytes-<branch-name>
+claude
+```
+
+Full guide: `agents/worktree_guide.md`
+
+# Testing
+
+Run the test suite with 4 parallel workers (optimal for any machine):
+
+```bash
+pytest tests/ -v --tb=short -n 4
+```
+
+Best practices, benchmark data, and known architecture-specific issues: `agents/testing_guide.md`
diff --git a/COMPILE_H100_L40.md b/COMPILE_H100_L40.md
@@ -0,0 +1,113 @@
+# Compiling bitsandbytes for H100 and L40 GPUs
+
+This guide shows how to compile bitsandbytes from source specifically optimized for NVIDIA H100 and L40 GPUs.
+
+## Prerequisites
+
+- CMake >= 3.22.1
+- Python >= 3.9
+- GCC (version 9+ recommended)
+- CUDA Toolkit (11.8+)
+- PyTorch with CUDA support
+
+Verify your system:
+```bash
+cmake --version
+python3 --version
+gcc --version
+nvcc --version
+```
+
+## GPU Compute Capabilities
+
+- **L40**: Compute Capability 8.9 (sm_89)
+- **H100**: Compute Capability 9.0 (sm_90)
+
+## Compilation Steps
+
+### 1. Clean any previous build configuration
+
+```bash
+cd /path/to/bitsandbytes
+rm -rf CMakeCache.txt CMakeFiles/ build/
+```
+
+### 2. Configure CMake for H100 and L40
+
+```bash
+cmake -DCOMPUTE_BACKEND=cuda -DCOMPUTE_CAPABILITY="89;90" -S .
+```
+
+This configures the build to target only compute capabilities 89 (L40) and 90 (H100), significantly reducing compilation time compared to building for all architectures.
+
+### 3. Compile the library
+
+```bash
+make -j$(nproc)
+```
+
+This will create `bitsandbytes/libbitsandbytes_cuda<VERSION>.so` where `<VERSION>` matches your CUDA Toolkit version (e.g., `cuda124` for CUDA 12.4).
+
+### 4. Install the package
+
+```bash
+pip install -e .
+```
+
+Use `-e` flag for editable/development install, or omit it for regular installation.
+
+### 5. Handle PyTorch CUDA version mismatch (if needed)
+
+If your PyTorch was compiled with a different CUDA version than your Toolkit, you may need to create a symlink:
+
+```bash
+# Example: PyTorch uses CUDA 12.8, but you compiled with CUDA 12.4
+ln -sf libbitsandbytes_cuda124.so bitsandbytes/libbitsandbytes_cuda128.so
+```
+
+Alternatively, set the environment variable:
+```bash
+export BNB_CUDA_VERSION=124  # Use your compiled CUDA version
+```
+
+### 6. Verify installation
+
+```bash
+python3 -c "import bitsandbytes as bnb; print(f'bitsandbytes version: {bnb.__version__}'); print('Success!')"
+```
+
+## Expected Output
+
+After compilation, you should see:
+- Binary file: `bitsandbytes/libbitsandbytes_cuda<VERSION>.so` (approximately 7MB when targeting only sm_89 and sm_90)
+- Successful import in Python with no errors
+
+## Compilation Time
+
+Building for only H100/L40 (2 architectures) takes approximately **1-2 minutes** compared to **5+ minutes** when building for all 14+ compute capabilities.
+
+## Troubleshooting
+
+### Warning messages during compilation
+Warnings like "variable declared but never referenced" are harmless and can be ignored.
+
+### Wrong CUDA binary error
+If you see `Configured CUDA binary not found`, check:
+1. The compiled `.so` file exists in `bitsandbytes/` directory
+2. The CUDA version matches or create a symlink as shown in step 5
+3. Use `BNB_CUDA_VERSION` environment variable to override
+
+### CUDA version check
+```bash
+# Check your CUDA Toolkit version
+nvcc --version
+
+# Check PyTorch CUDA version
+python3 -c "import torch; print(torch.version.cuda)"
+```
+
+## Notes
+
+- The compiled library will **only work on GPUs with compute capability 8.9 or 9.0** (L40 and H100)
+- For other GPUs, you'll need to recompile with appropriate compute capabilities
+- The `-DCOMPUTE_CAPABILITY` flag accepts a semicolon-separated list: e.g., `"75;80;89;90"` for T4, A100, L40, and H100
diff --git a/agents/testing_guide.md b/agents/testing_guide.md
@@ -0,0 +1,127 @@
+# Testing Guide for bitsandbytes
+
+## Quick Start
+
+Run the full test suite with optimal parallelization:
+
+```bash
+pytest tests/ -v --tb=short -n 4
+```
+
+`-n 4` (4 pytest-xdist workers) is the recommended default for any machine.
+
+## Why 4 Workers?
+
+Benchmarks across two machines with very different hardware show that `-n 4` is consistently the fastest configuration. Going higher provides no benefit and often makes things worse.
+
+### Benchmark Data
+
+**Machine A:** AMD Threadripper 1900X (8 cores / 16 threads), RTX 4090 (24 GB), CUDA 12.4
+
+| Workers | Wall Time | Speedup vs n=1 | Avg CPU | Avg GPU | Failures |
+|---------|-----------|-----------------|---------|---------|----------|
+| 1       | 1319s     | 1.00x           | 32.5%   | 3.4%    | 0        |
+| **4**   | **565s**  | **2.33x**       | 70.5%   | 12.9%   | 0        |
+| 6       | 588s      | 2.24x           | 74.8%   | 10.9%   | 7 (OOM)  |
+| 8       | 570s      | 2.31x           | 87.9%   | 12.5%   | 7 (OOM)  |
+
+**Machine B:** AMD Threadripper PRO 9975WX (32 cores / 64 threads), RTX PRO 6000 Blackwell (98 GB), CUDA 13.0
+
+| Workers | Wall Time | Speedup vs n=1 | Avg CPU | Avg GPU | Failures |
+|---------|-----------|-----------------|---------|---------|----------|
+| 1       | 428s      | 1.00x           | 13.4%   | 3.1%    | 25*      |
+| **4**   | **322s**  | **1.33x**       | 75.3%   | 5.7%    | 25*      |
+| 8       | 578s      | 0.74x (slower)  | 91.9%   | 3.5%    | 25*      |
+| 16      | 566s      | 0.76x (slower)  | 97.0%   | 6.2%    | 25*      |
+| 24      | 560s      | 0.76x (slower)  | 97.2%   | 6.2%    | 40       |
+
+\* Blackwell-specific failures unrelated to worker count (see Known Issues below).
+
+### Analysis
+
+- **GPU utilization stays very low** (3-13%) regardless of worker count. The tests are primarily CPU-bound: short GPU kernel bursts interleaved with Python/numpy work for test setup, tensor creation, and result validation.
+- **4 workers is the sweet spot** because it balances overlapping CPU prep with GPU execution across workers. Each worker can prepare data while another waits on a GPU kernel.
+- **Beyond 4 workers, overhead dominates.** Additional workers add pytest-xdist coordination costs and per-worker CUDA context overhead without meaningful GPU throughput gain. On Machine B, `-n 8` was nearly 2x slower than `-n 4` despite 75% idle CPU at `-n 4`.
+- **Per-core CPU speed matters more than core count.** Machine B is 3.1x faster single-threaded (Zen 5 vs Zen 1). Having 4x more cores provided no additional benefit at the optimal worker count.
+- **GPU memory affects reliability, not speed.** More free VRAM avoids OOM failures at higher worker counts but does not improve throughput.
+
+### What About More/Fewer Workers?
+
+| Situation | Recommendation |
+|-----------|---------------|
+| Default | `-n 4` |
+| Low GPU memory (<8 GB free) | `-n 2` to avoid OOM |
+| Running a subset of tests | `-n 4` still fine |
+| Single specific test | No `-n` flag needed |
+| CI environment | `-n 4` |
+
+## Useful pytest Options
+
+```bash
+# Full suite, optimal speed
+pytest tests/ -v --tb=short -n 4
+
+# With timing breakdown of slowest tests
+pytest tests/ -v --tb=short -n 4 --durations=20
+
+# Run a specific test file
+pytest tests/test_functional.py -v --tb=short -n 4
+
+# Run tests matching a keyword
+pytest tests/ -v --tb=short -n 4 -k "4bit"
+
+# Stop on first failure
+pytest tests/ -v --tb=short -n 4 -x
+
+# Single worker (debugging, deterministic output)
+pytest tests/ -v --tb=long
+```
+
+## Test Suite Characteristics
+
+The full suite has ~7500 parametrized tests. Most of the wall-clock time is consumed by a small number of test functions with many parametrizations:
+
+- **`test_gemv_4bit`** dominates (~70% of total time) with 1500+ combinations. CPU variants at dim=1024 take 16-20s each; CUDA variants finish in ~0.05s.
+- **`test_functional.py`** alone accounts for ~80% of total test time.
+- **CPU tests are the bottleneck**: 81% of total time despite being only 37% of test count.
+- **87% of individual tests finish under 1 second**, but the remaining 13% consume 80% of wall-clock time.
+
+## Known Issues by Architecture
+
+### Blackwell (sm_120, e.g. RTX PRO 6000)
+
+25 tests fail on Blackwell as of the `main` branch (Feb 2026):
+
+1. **Int8 batched matmul (`test_ibmm`) - 16 failures**: cuBLAS returns `CUBLAS_STATUS_NOT_SUPPORTED` (status 15) for the int8 batched GEMM path on Blackwell. The legacy cuBLAS int8 API is not supported on sm_120. These tests produce garbage output (100% element mismatch). A fix would require migrating to cublasLt or a different int8 GEMM implementation.
+
+2. **FP4 quantization at blocksize=256 - 9 failures**: Relative error is marginally above the threshold (e.g., 0.29091 vs limit of 0.2908). Only affects `fp4` at `blocksize=256` on CUDA across all dtypes (fp32, fp16, bf16). The `nf4` quant type and other blocksizes pass. This is a minor numerical difference in fp4 dequantization likely caused by different FP rounding behavior on Blackwell.
+
+### Ada Lovelace (sm_89, e.g. RTX 4090)
+
+No architecture-specific failures. All tests pass with `-n 4`.
+
+## Build Before Testing
+
+Tests require a compiled native library matching your GPU and CUDA toolkit. See `COMPILE_H100_L40.md` for build instructions. Quick version:
+
+```bash
+# Find your GPU's compute capability
+nvidia-smi --query-gpu=compute_cap --format=csv,noheader
+
+# Build (replace 89 with your compute capability, e.g. 120 for Blackwell)
+cmake -DCOMPUTE_BACKEND=cuda -DCOMPUTE_CAPABILITY="89" -S . -B build
+cmake --build build -j$(nproc)
+
+# If your CUDA toolkit version differs from PyTorch's CUDA version, create a symlink:
+# e.g., toolkit is 12.4 but PyTorch expects 12.8:
+ln -sf bitsandbytes/libbitsandbytes_cuda124.so bitsandbytes/libbitsandbytes_cuda128.so
+
+# Install in editable mode
+pip install -e .
+```
+
+## Test Dependencies
+
+```bash
+pip install einops lion-pytorch pytest pytest-xdist scipy transformers
+```
diff --git a/agents/worktree_guide.md b/agents/worktree_guide.md
