Skip to content

Commit 9381251

Browse files
committed
popcnt: add AVX2 SIMD popcount for amd64 and revive the dispatch
What ---- Adds an AVX2 implementation of the five slice population-count routines (popcntSlice and the And/Or/Xor/Mask variants) in popcnt_avx2_amd64.s, selected at runtime by a hand-rolled CPUID/XGETBV feature check (_hasAVX2 -> useAVX2). popcnt_generic.go is retargeted to cover only the non-AVX2 paths, and the dead pre-Go-1.9 scalar assembly (popcnt_asm.go, popcnt_amd64.s) is removed. Why --- The old amd64 assembly was guarded by the build tag "!go1.9", so on every modern toolchain it was never compiled. All population counting fell through to popcnt_generic.go -> popcnt_slices.go, a scalar loop calling math/bits.OnesCount64 one 64-bit word at a time. In other words amd64 had no SIMD acceleration at all. These routines are hot: a bitmap container is a fixed 1024-word (8 KB) array, and a popcnt*Slice runs after essentially every bitmap/bitmap AND/OR/XOR/ANDNOT to recompute cardinality, plus in rank(), getCardinalityInRange() and computeCardinality(). How / kernel ------------ The kernel is the Mula/Lemire VPSHUFB nibble-lookup popcount: each 256-bit lane is split into low/high nibbles, counted via a 16-entry table lookup, summed byte-wise, then folded to quadwords with VPSADBW and accumulated. Loads are unaligned (VMOVDQU) because the bitmap slices are only 8-byte aligned; VZEROUPPER precedes every return; a scalar POPCNTQ tail handles the final len%4 words so the routines are correct for any length. Detection checks OSXSAVE+AVX (CPUID leaf 1), YMM OS support (XGETBV/XCR0) and AVX2 (CPUID leaf 7) before enabling. Fallback behavior is preserved exactly: amd64 CPUs without AVX2 take the scalar path at runtime, and appengine / non-amd64 builds compile popcnt_generic.go. A differential test checks the AVX2 output against the Go reference across edge lengths {0,1,2,3,4,5,7,8,15,16,17,31,63, 64,65,1023,1024,1025}, and a dispatch test forces useAVX2=false to exercise the fallback on amd64 too. Speed ----- On a full 1024-word container (AMD Ryzen AI 7 PRO 350): popcntSlice 599 ns -> 331 ns (~1.8x) popcntAndSlice 1108 ns -> 408 ns (~2.7x)
1 parent 9e2afa4 commit 9381251

6 files changed

Lines changed: 485 additions & 173 deletions

File tree

popcnt_amd64.s

Lines changed: 0 additions & 103 deletions
This file was deleted.

popcnt_asm.go

Lines changed: 0 additions & 68 deletions
This file was deleted.

popcnt_avx2_amd64.go

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
//go:build amd64 && !appengine
2+
// +build amd64,!appengine
3+
4+
package roaring
5+
6+
// The functions below are implemented in popcnt_avx2_amd64.s using AVX2.
7+
// They are only used when the CPU supports AVX2 (see useAVX2); otherwise the
8+
// pure-Go fallbacks in popcnt_slices.go are used. This keeps behavior identical
9+
// on every target: appengine and non-amd64 builds compile popcnt_generic.go
10+
// instead, and amd64 CPUs without AVX2 take the scalar path at runtime.
11+
12+
//go:noescape
13+
func _hasAVX2() bool
14+
15+
//go:noescape
16+
func _popcntSliceAVX2(s []uint64) uint64
17+
18+
//go:noescape
19+
func _popcntMaskSliceAVX2(s, m []uint64) uint64
20+
21+
//go:noescape
22+
func _popcntAndSliceAVX2(s, m []uint64) uint64
23+
24+
//go:noescape
25+
func _popcntOrSliceAVX2(s, m []uint64) uint64
26+
27+
//go:noescape
28+
func _popcntXorSliceAVX2(s, m []uint64) uint64
29+
30+
// useAVX2 selects the AVX2 assembly implementations when the running CPU
31+
// supports AVX2. It is evaluated once at package initialization.
32+
var useAVX2 = _hasAVX2()
33+
34+
func popcntSlice(s []uint64) uint64 {
35+
if useAVX2 {
36+
return _popcntSliceAVX2(s)
37+
}
38+
return popcntSliceGo(s)
39+
}
40+
41+
func popcntMaskSlice(s, m []uint64) uint64 {
42+
if useAVX2 {
43+
return _popcntMaskSliceAVX2(s, m)
44+
}
45+
return popcntMaskSliceGo(s, m)
46+
}
47+
48+
func popcntAndSlice(s, m []uint64) uint64 {
49+
if useAVX2 {
50+
return _popcntAndSliceAVX2(s, m)
51+
}
52+
return popcntAndSliceGo(s, m)
53+
}
54+
55+
func popcntOrSlice(s, m []uint64) uint64 {
56+
if useAVX2 {
57+
return _popcntOrSliceAVX2(s, m)
58+
}
59+
return popcntOrSliceGo(s, m)
60+
}
61+
62+
func popcntXorSlice(s, m []uint64) uint64 {
63+
if useAVX2 {
64+
return _popcntXorSliceAVX2(s, m)
65+
}
66+
return popcntXorSliceGo(s, m)
67+
}

0 commit comments

Comments
 (0)