Skip to content

Commit 564af66

Browse files
Amrita H Samritahs-ibm
authored andcommitted
power/bgemm: add BFloat16-in, BFloat16-out GEMM kernel for POWER10
Add BGEMM (BF16 input → BF16 output) for POWER10 by reusing the existing SBGEMM kernel infrastructure. A -DBGEMM compile flag switches only the store path; the xvbf16ger2pp MMA instruction and BF16 packing routines are shared with SBGEMM unchanged. Changes ------- kernel/power/KERNEL.POWER10 - Register BGEMM kernel and copy-routine targets, reusing the sbgemm_ncopy/tcopy sources (packing layout is identical). kernel/power/sbgemm_kernel_power10.c - Under BGEMM: force v4sf_t to float so accumulators stay in float32; add STORE4_BF16/STORE2_BF16 macros (read BF16 C, widen, apply alpha*acc, convert back via xvcvspbf16, store); add f32_to_bf16_scalar() for m&1/n&1 tails; add BGEMM variants of all SAVE_ACC macros covering the full m/n tile hierarchy. param.h - Add BGEMM_DEFAULT_UNROLL_M=16, UNROLL_N=8, P/Q/R blocking parameters for POWER10, matching the 16x8 kernel tile. Unit test (utest/test_extensions/test_bgemm.c) - Uses SBGEMM as a trusted reference. 21 test cases cover all four transpose combinations, all m/n remainder paths, odd-k, alpha=0, beta=0/1, and the m>=32 fast path. Tolerance 0.01 to account for the one extra BF16 rounding on the BGEMM store. Performance (POWER10, single-threaded, Transa=N Transb=N) ---------------------------------------------------------- Step=1 (sizes 1–200): - Sizes 1–80: baseline ~12–131 MFlops (scalar fallback); patch 200–46,000 MFlops (up to ~350,000x faster). - Sizes 81–200: patch 1x–35x faster with 16x8 MMA tile engaged. Step=8 (sizes 8–1024): - Baseline peaks at ~5,200 MFlops; patch sustains 40,000–760,000 MFlops for sizes 384–960. Average speedup ~100x. Signed-off-by: Amrita H S <amritahs@linux.vnet.ibm.com>
1 parent 0b54dd1 commit 564af66

5 files changed

Lines changed: 644 additions & 25 deletions

File tree

kernel/power/KERNEL.POWER10

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,22 @@
33
#CGEMM_BETA = ../generic/zgemm_beta.c
44
#ZGEMM_BETA = ../generic/zgemm_beta.c
55

6+
# BGEMM (BFloat16-in, BFloat16-out) for POWER10 using plain MMA (xvbf16ger2pp).
7+
# The kernel source is shared with SBGEMM; -DBGEMM is injected by Makefile.L3
8+
# to switch the SAVE_ACC macros to convert float32 accumulators back to BF16.
9+
# UNROLL_M=16 (A-panel, packed by ncopy_16), UNROLL_N=8 (B-panel, packed by
10+
# ncopy_8) — matches the 16x8 primary tile in sbgemm_kernel_power10.c.
11+
BGEMM_BETA = ../generic/gemm_beta.c
12+
BGEMMKERNEL = sbgemm_kernel_power10.c
13+
BGEMMINCOPY = sbgemm_ncopy_16_power10.c
14+
BGEMMITCOPY = sbgemm_tcopy_16_power10.c
15+
BGEMMONCOPY = sbgemm_ncopy_8_power10.c
16+
BGEMMOTCOPY = sbgemm_tcopy_8_power10.c
17+
BGEMMINCOPYOBJ = bgemm_incopy$(TSUFFIX).$(SUFFIX)
18+
BGEMMITCOPYOBJ = bgemm_itcopy$(TSUFFIX).$(SUFFIX)
19+
BGEMMONCOPYOBJ = bgemm_oncopy$(TSUFFIX).$(SUFFIX)
20+
BGEMMOTCOPYOBJ = bgemm_otcopy$(TSUFFIX).$(SUFFIX)
21+
622
SBGEMM_BETA = ../generic/gemm_beta.c
723
SBGEMMKERNEL = sbgemm_kernel_power10.c
824
SBGEMMINCOPY = sbgemm_ncopy_16_power10.c

0 commit comments

Comments
 (0)