Commit 6506ad0

and

committed

bench: improve Document-Level RAG test prompts + Q4_K dequant investigation

Findings from Karpathy loop: 1. Q4_K_M GGUF on-the-fly dequant (TQ_NO_Q4) has a bug — garbage output. Root cause: likely in fused_dot_q4_k or Metal dispatch. Does NOT affect normal users (auto Q4 conversion is the default and works correctly). 2. Auto Q4 path (Q8_0 → Q4) works for fact extraction but with artifacts: "Santos" → "SanSannt", "Park" → "PPar". Semantically correct, but character-level precision is lost. 3. 1B model instruction-following is the real bottleneck for Document QA. The model enters "Step 1: Identify..." reasoning chains instead of directly answering. Better prompts help but don't solve it fully. 4. GGUF on-the-fly Q4_K bug confirmed: logits have -2.68 bias vs Q8_0's +3.98. Scale/min handling in fused_dot_q4_k suspected but unit test shows correct formula. Bug may be in per-tensor type dispatch for mixed Q4_K/Q6_K models (Q4_K_M uses both). Action: Document QA concept is valid, needs 3B+ Q8 models for reliable demonstration. The Q4_K on-the-fly bug is tracked for future fix. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

1 parent 57a2918 commit 6506ad0Copy full SHA for 6506ad0

206 files changed

.gitignore
bindings/python
- quant.h
build_metal
- CMakeCache.txt
- CMakeFiles
  - 3.28.1
    - CMakeCCompiler.cmake
    - CMakeCXXCompiler.cmake
    - CMakeDetermineCompilerABI_C.bin
    - CMakeDetermineCompilerABI_CXX.bin
    - CMakeDetermineCompilerABI_OBJC.bin
    - CMakeOBJCCompiler.cmake
    - CMakeSystem.cmake
    - CompilerIdCXX
      - CMakeCXXCompilerId.cpp
    - CompilerIdC
      - CMakeCCompilerId.c
    - CompilerIdOBJC
      - CMakeOBJCCompilerId.m
  - CMakeConfigureLog.yaml
  - CMakeDirectoryInformation.cmake
  - CMakeRuleHashes.txt
  - Makefile.cmake
  - Makefile2
  - TargetDirectories.txt
  - ab_test.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - depend.make
    - flags.make
    - link.txt
    - progress.make
  - benchmark_types.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - depend.make
    - flags.make
    - link.txt
    - progress.make
  - cmake.check_cache
  - debug_compare.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - depend.make
    - flags.make
    - link.txt
    - progress.make
  - demo_real_model.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - depend.make
    - flags.make
    - link.txt
    - progress.make
  - embed_chat.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - depend.make
    - flags.make
    - link.txt
    - progress.make
  - embed_kv_compare.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - depend.make
    - flags.make
    - link.txt
    - progress.make
  - embed_minimal.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - depend.make
    - flags.make
    - link.txt
    - progress.make
  - llamacpp_integration.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - depend.make
    - flags.make
    - link.txt
    - progress.make
  - minimal.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - depend.make
    - flags.make
    - link.txt
    - progress.make
  - progress.marks
  - quant.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - depend.make
    - flags.make
    - link.txt
    - progress.make
    - tools
      - quant.c.o.d
  - single_header_example.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - depend.make
    - flags.make
    - link.txt
    - progress.make
  - standalone.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - depend.make
    - flags.make
    - link.txt
    - progress.make
  - tq_convert.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - depend.make
    - flags.make
    - link.txt
    - progress.make
  - turboquant.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - cmake_clean_target.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - depend.make
    - flags.make
    - link.txt
    - progress.make
    - src
  - turboquant_metallib.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - progress.make
  - turboquant_shared.dir
    - DependInfo.cmake
    - build.make
    - cmake_clean.cmake
    - compiler_depend.make
    - compiler_depend.ts
    - depend.make
    - flags.make
    - link.txt
    - progress.make
- Makefile
- cmake_install.cmake
- tq_elementwise.air
- tq_fused_cache.air
- tq_matmul.air
- tq_moe_kernel.air
- tq_polar.air
- tq_qjl.air
- tq_turbo.air
- tq_value.air
- turboquant.metallib
docs/pr

`‎.gitignore‎`

-Original file line number
+Diff line change
 docs/assets/hero_backup.png
 build_nomt/
 build_nometal/
++
 +.env
++

Comments

(0)