add bulk off-heap scoring for uint8 quantized vectors using panama vector api#16203
add bulk off-heap scoring for uint8 quantized vectors using panama vector api#16203iprithv wants to merge 1 commit into
Conversation
iprithv
force-pushed
the
bulk-offheap-scoring
branch
4 times, most recently
from
2336b8b to
94e8a44
Compare
iprithv
changed the title
iprithv
force-pushed
the
bulk-offheap-scoring
branch
3 times, most recently
from
c47bf55 to
8a127b8
Compare
iprithv
force-pushed
the
bulk-offheap-scoring
branch
from
8a127b8 to
fdab794
Compare
|
@rmuir could you please help review this? thanks! |
| IntVector acc1 = IntVector.zero(INT_SPECIES); | ||
| IntVector acc2 = IntVector.zero(INT_SPECIES); | ||
| IntVector acc3 = IntVector.zero(INT_SPECIES); | ||
| IntVector acc4 = IntVector.zero(INT_SPECIES); |
There was a problem hiding this comment.
I don't think this unrolling for integer-based math should be necessary. It should only be needed for the floating point due to causing change in result. I see some existing code in this file does it where I think it shouldn't, but I'm not convinced and let's avoid the trend. Can we try to remove the unrolling?
There was a problem hiding this comment.
acc1-acc4 aren't loop unrolling actually, they're the 4 document vectors d1-d4. each accumulates a separate result (q·d1, q·d2,...), same as sv1-sv4 in the float bulk methods right above. can't merge them into one. should I rename them maybe?
There was a problem hiding this comment.
it isn't "just like the float bulk methods above". it isn't necessary for integer math.
There was a problem hiding this comment.
sorry, but these are 4 separate output values, not 4 accumulators for the same result. acc1 goes to scores[0] (q·d1), acc2 goes to scores[1] (q·d2).... we need all 4 because each one is a different dot product against a different document vector. If we used a single accumulator we'd lose 3 of the 4 results. maybe I'm misunderstanding?
There was a problem hiding this comment.
ah, now I understand what you mean.. for integer math the bulk 4 at a time doesn't really buy us anything since there's no FP dependency chain to break, and the qv load would stay in cache anyway across separate calls..right?
I can try to remove the uint8 bulk methods and have the caller just call the single pair dot product 4 times instead..?
There was a problem hiding this comment.
sorry about the confusion earlier. removed the uint8 bulk kernels and now bulkVectorOp just calls the single pair :)
There was a problem hiding this comment.
@rmuir wanted to touch base with you on this, in case it got buried. Thanks!
iprithv
force-pushed
the
bulk-offheap-scoring
branch
2 times, most recently
from
b69f01d to
a52caf7
Compare
iprithv
force-pushed
the
bulk-offheap-scoring
branch
from
a52caf7 to
6774c6d
Compare
|
moved changes entry to 10.6! |
3 participants
add off-heap bulk scoring support for uint8 quantized vectors (via MemorySegment direct access in the Lucene99 scorer). processes neighbor batches from HNSW with direct offset reads and falls back for int4 / non-contiguous cases.
benchmarks (amd ryzen 7 7800x3d, avx-512):
related: #15155 #15257 #14980
also I tried this for int4 bulk but saw ~2.3x slower on avx-512 due to nibble unpacking overhead..