Commit 9381251
committed
popcnt: add AVX2 SIMD popcount for amd64 and revive the dispatch
What
----
Adds an AVX2 implementation of the five slice population-count routines
(popcntSlice and the And/Or/Xor/Mask variants) in popcnt_avx2_amd64.s,
selected at runtime by a hand-rolled CPUID/XGETBV feature check
(_hasAVX2 -> useAVX2). popcnt_generic.go is retargeted to cover only
the non-AVX2 paths, and the dead pre-Go-1.9 scalar assembly
(popcnt_asm.go, popcnt_amd64.s) is removed.
Why
---
The old amd64 assembly was guarded by the build tag "!go1.9", so on
every modern toolchain it was never compiled. All population counting
fell through to popcnt_generic.go -> popcnt_slices.go, a scalar loop
calling math/bits.OnesCount64 one 64-bit word at a time. In other words
amd64 had no SIMD acceleration at all. These routines are hot: a bitmap
container is a fixed 1024-word (8 KB) array, and a popcnt*Slice runs
after essentially every bitmap/bitmap AND/OR/XOR/ANDNOT to recompute
cardinality, plus in rank(), getCardinalityInRange() and
computeCardinality().
How / kernel
------------
The kernel is the Mula/Lemire VPSHUFB nibble-lookup popcount: each
256-bit lane is split into low/high nibbles, counted via a 16-entry
table lookup, summed byte-wise, then folded to quadwords with VPSADBW
and accumulated. Loads are unaligned (VMOVDQU) because the bitmap slices
are only 8-byte aligned; VZEROUPPER precedes every return; a scalar
POPCNTQ tail handles the final len%4 words so the routines are correct
for any length. Detection checks OSXSAVE+AVX (CPUID leaf 1), YMM OS
support (XGETBV/XCR0) and AVX2 (CPUID leaf 7) before enabling.
Fallback behavior is preserved exactly: amd64 CPUs without AVX2 take the
scalar path at runtime, and appengine / non-amd64 builds compile
popcnt_generic.go. A differential test checks the AVX2 output against
the Go reference across edge lengths {0,1,2,3,4,5,7,8,15,16,17,31,63,
64,65,1023,1024,1025}, and a dispatch test forces useAVX2=false to
exercise the fallback on amd64 too.
Speed
-----
On a full 1024-word container (AMD Ryzen AI 7 PRO 350):
popcntSlice 599 ns -> 331 ns (~1.8x)
popcntAndSlice 1108 ns -> 408 ns (~2.7x)1 parent 9e2afa4 commit 9381251
6 files changed
Lines changed: 485 additions & 173 deletions
This file was deleted.
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
0 commit comments