Skip to content

Commit 5108b77

Browse files
michel2323claude
andcommitted
Add an opt-in per-submission synchronize for the LTS dropped-tail bug
The Aurora LTS NEO stack intermittently drops the tail of a command list, silently corrupting results. Add an off-by-default workaround (ONEAPI_SYNC_EACH_SUBMISSION=1) that synchronizes the queue after every command-list submission (lib/level-zero/cmdlist.jl, parsed in lib/level-zero/oneL0.jl __init__). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 2adadbc commit 5108b77

2 files changed

Lines changed: 16 additions & 2 deletions

File tree

lib/level-zero/cmdlist.jl

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,8 +47,21 @@ function ZeCommandList(f::Base.Callable, args...; kwargs...)
4747
return list
4848
end
4949

50-
execute!(queue::ZeCommandQueue, lists::Vector{ZeCommandList}, fence=nothing) =
51-
zeCommandQueueExecuteCommandLists(queue, length(lists), lists, something(fence, C_NULL))
50+
# Opt-in workaround for the Aurora LTS NEO stack (set ONEAPI_SYNC_EACH_SUBMISSION=1).
51+
# Under heavy multi-process oversubscription of a single tile, a whole-queue
52+
# `zeCommandQueueSynchronize` does not reliably retire the tail of an earlier,
53+
# separately-submitted command list — producing silent "dropped tail" corruption (the
54+
# last work-item of a kernel / last element of a copy is missing). See
55+
# ISSUE_dropped_tail.md. Synchronizing after *every* submission eliminates it, at a large
56+
# throughput cost (~3x), so it is off by default and only enabled when correctness under
57+
# oversubscription matters more than speed.
58+
const sync_each_submission = Ref{Bool}(false)
59+
60+
function execute!(queue::ZeCommandQueue, lists::Vector{ZeCommandList}, fence=nothing)
61+
r = zeCommandQueueExecuteCommandLists(queue, length(lists), lists, something(fence, C_NULL))
62+
sync_each_submission[] && synchronize(queue)
63+
return r
64+
end
5265

5366
"""
5467
execute!(queue::ZeCommandQueue, ...) do list

lib/level-zero/oneL0.jl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,7 @@ function __init__()
201201

202202
validation_layer[] = parse(Bool, get(ENV, "ZE_ENABLE_VALIDATION_LAYER", "false"))
203203
parameter_validation[] = parse(Bool, get(ENV, "ZE_ENABLE_PARAMETER_VALIDATION", "false"))
204+
sync_each_submission[] = lowercase(get(ENV, "ONEAPI_SYNC_EACH_SUBMISSION", "")) in ("1", "true", "yes")
204205
end
205206

206207
end

0 commit comments

Comments
 (0)