gemm block quantization infrastructure for llm decoder style

[nihui]

• move int468 dequant code to x86/arm/... create_pipeline
• how to encode block_size and storage type ?
• try fp4 e2m1/e3 type ?
• comp table ?
• expand to more general gemm ?
• port union hack to platform-independent style
• gemm test++
• doc++

./ncnnllm2int468 qwen3_decoder.ncnn.param qwen3_decoder.ncnn.bin qwen3_decoder-int6.ncnn.param qwen3_decoder-int6.ncnn.bin

[tencent-adm]

Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov-commenter]

Codecov Report

❌ Patch coverage is 84.15033% with 97 lines in your changes missing coverage. Please review.
✅ Project coverage is 94.67%. Comparing base (13b6d53) to head (d3d079e).

Files with missing lines	Patch %	Lines
src/layer/multiheadattention.cpp	86.78%	46 Missing ⚠️
src/layer/gemm.cpp	79.67%	38 Missing ⚠️
src/layer/vulkan/multiheadattention_vulkan.cpp	36.36%	7 Missing ⚠️
src/layer/vulkan/gemm_vulkan.cpp	50.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6439      +/-   ##
==========================================
- Coverage   94.70%   94.67%   -0.04%     
==========================================
  Files         947      947              
  Lines      415846   416607     +761     
==========================================
+ Hits       393844   394437     +593     
- Misses      22002    22170     +168     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

The binary size change of libncnn.so (bytes)

architecture	base size	pr size	difference
x86_64	15316400	15324592	+8192 ⚠️
armhf	6229892	6234020	+4128 ⚠️
aarch64	9527616	9527536	-80 😘

[Copilot]

Pull request overview

This WIP pull request implements block quantization for GEMM layers to support 4-bit, 6-bit, and 8-bit quantization for LLM decoder-style models. The changes introduce a new quantization tool and corresponding dequantization logic in the GEMM layer implementation.

Key changes:

• New ncnnllm2int468 tool for quantizing GEMM weight matrices with configurable block sizes and bit widths
• Block-based quantization scheme using per-block scaling factors stored in B_data_quantize_scales
• Dequantization logic in gemm.cpp that converts quantized weights back to fp32 during model loading

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
tools/quantize/ncnnllm2int468.cpp	New quantization tool implementing 4/6/8-bit block quantization with custom bit-packed storage formats
tools/quantize/CMakeLists.txt	Build configuration adding the new ncnnllm2int468 executable
tools/modelwriter.h	Extended serialization to save block quantization scales for int8_scale_term values 4/5/6
src/layer/gemm.h	Added B_data_quantize_scales member to store per-block scaling factors
src/layer/gemm.cpp	Implemented loading and dequantization logic for 4/6/8-bit block-quantized weights

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.