Skip to content

an extension that adds a device side abort function #808

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pjaaskel wants to merge 3 commits into
base: main
Choose a base branch
from
+766 −0
Open

an extension that adds a device side abort function #808

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
68406ef
cl_pocl_device_side_abort rev 1
pjaaskel Apr 4, 2022
a05784e
Device side abort updates and clarifications
pjaaskel Apr 25, 2022
152333e
Change the device side abort to 'ext' extension
pjaaskel May 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 125 additions & 0 deletions extensions/cl_ext_device_side_abort.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
cl_ext_device_side_abort
=========================

// This section needs to be after the document title.
:doctype: book
:toc2:
:toc: left
:encoding: utf-8
:lang: en
:blank: pass:[ +]
:data-uri:
:icons: font
include::../config/attribs.txt[]
:source-highlighter: coderay

== Name Strings

+cl_ext_device_side_abort+

== Contact

Pekka Jääskeläinen, Parmance (pekka /at/ parmance /dot/ com)

== Contributors

Pekka Jääskeläinen, Parmance +
Henry Linjamäki, Parmance +
Brice Videau, Argonne National Laboratory

== Notice

Copyright (c) 2021-2022 Parmance and Argonne National Laboratory.

== Status

Draft

== Version

Built On: {docdate} +
Revision: 3

== Dependencies

This extension is written against the OpenCL Specification Version 1.2, Revision 19.

This extension requires OpenCL 1.2.

== Overview

With the device side abort extension, a work-item (WI) can return from the kernel
execution at any point and cause abnormal unrecoverable termination of the host
process.

This extension differs from the cl_arm_controlled_kernel_termination extension
in that the abort in the device side is expected to behave like a call to the
POSIX abort() in the host side, terminating also the host process immediately.

== New OpenCL C Types and Functions

[source,c]
----
void abort_platform(void);
----

== Modifications to the OpenCL C Specification

=== Add a new Section 6.12.X - "Device-side invoked platform-wide program abort"

A new function: ::
+
--

[source,c]
----
void abort_platform(void);
----

Stops the kernel execution and causes an abnormal unrecoverable termination of
the host process, thus causing abortion of execution of the whole program at
the platform level.

Calling the device side abort also causes the printf buffer to be flushed on the
host program's standard output before aborting the host process. This allows to
implement device side assertion functionality by using the OpenCL C printf
function and this extension.

The host should abort the process at latest when the kernel command that called
the function has finished execution, which means that all the WIs will
either call abort_platform() or return from the kernel command otherwise.

The host process is expected to call the standard abort function of the host
operating system. In POSIX/C99-compliant systems, this must be implemented by
a call to abort() which implies that the SIGABRT signal is raised and can be
caught by defining a signal handler. This means that a minimal implementation
of a device-side abort with a CPU device can simply call the standard abort()
directly to implement compliant behavior.

The results of an aborted kernel command are undefined. Device side abort is

@pjaaskel pjaaskel Jul 28, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The results of an aborted kernel command are undefined. Device side abort is
The results produced by an aborted kernel command are undefined. Device side abort is

guaranteed to stop the kernel command as soon as possible, even in the presence
of deadlocks or partially reached barriers in the kernel command.

@karolherbst karolherbst Jul 24, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unsure to be honest. I think it's a good idea in principle, but it does change certain implementation details. Like if a thread calling abort_platform causes a dead-lock, it might lead to a kernel driver to timeout the GPU job and that alone is undefined behavior, not just "undefined results".

Some implementations might just not be able to terminate the running kernel without undefined behavior. But now this extension actually defines it: it needs to terminate at some point.

I think calling abort_platform in itself causes undefined behavior and runtimes can only guarantee anything that happens afterwards on a best effort basis.

@pjaaskel pjaaskel Jul 25, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The basis for this extension's semantics are the one of libc's abort() and the __trap() of CUDA. It would be useful to be able to implement both of those with this extension.

I'm not sure what CUDA's "generate an interrupt to the host CPU" should imply, but to me the natural semantics here is to abort the whole (host) process and bring down the GPU resources, similar to killing a threaded CPU-only host application with multiple threads running and one of them calling abort().

@karolherbst karolherbst Jul 25, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I'm not concerned about Nvidia being able to reliably implement such a functionality, I'm more concerned about giving too strong guarantees for all the other GPU hardware out there. Nvidia has a BPT instruction which can create a trap signal that is either handled by a GPU queue configured trap handler (used to implement a debugger, e.g. single-stepping) or raised to the OS. So the OS can just handle that signal and deal with it by e.g. forcing a termination of the running workload.

I'm not sure every other GPU vendor has something similar. Maybe it's good enough to make the behavior after abort_platform implementation defined, but urge every implementation to at least try to guarantee the things mandated by this extension?

@pjaaskel pjaaskel Jul 28, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather see well-defined behavior and if the device cannot support it, then it just doesn't advertise support for the extension. My guess is that both AMD and Intel have some sort of features that could be used to implement the "canceling" of the GPU workload when signaled by the device side program (e.g. via a system shared global variable), even in presence of kernel deadlocks.

@karolherbst karolherbst Jul 28, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AMD has a pretty aggressive way of dealing with GPU side timeouts, which often lead to full GPU resets impacting other parts of the system, e.g. your desktop session could also just die.

I'm sure Intel will manage, I have my doubts about AMD guaranteeing no abnormal behavior of the system after abort_platform causes dead-locks, invalid memory accesses, etc...

@pjaaskel pjaaskel Jul 28, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

chipStar implements (or workarounds the lack of) this extension by means of a host-shared variable which a host thread polls and if set to 1 by the device, calls abort(). Do you suggest that even if the process is aborted this way (thus invoking the OS resource cleanup) there is a risk of instability in AMD's case? HIP provides a device reset API. Something like that could be invoked from the host side when device abort is detected for a cleaner ramp-down.

@karolherbst karolherbst Jul 28, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From my experience with AMD, hipDeviceReset sounds like its indeed a full device reset, resetting it for every application running on the system :)

@karolherbst karolherbst Jul 28, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

chipStar implements (or workarounds the lack of) this extension by means of a host-shared variable which a host thread polls and if set to 1 by the device, calls abort(). Do you suggest that even if the process is aborted this way (thus invoking the OS resource cleanup) there is a risk of instability in AMD's case?

Only if the kernel itself is misbehaving as a result. If the kernel just terminates normally, even after one thread quits, there is no danger. The issue simply just starts once it times out, or causes invalid memory accesses, etc....


The control flow behavior within the work-group of a kernel command with
a WI that called the device side abort is the same as immediately
returning from the kernel for the abort-calling WI at the abort call site.
For possible WIs that do not call abort, the return time is undefined and can
adhere to the original control flow of the kernel as long as all the WIs
eventually finish the execution and allow the host runtime to abort the
process also in a single-threaded/non-interrupting implementation.

@karolherbst karolherbst Jul 24, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The phrasing here is a bit hard to grasp what's actually meant.

Assuming we have control flow that is well defined with all threads participating, but now a single thread calls abort_platform and quits. If that causes an infinite loop an implementation might be able to guarantee that "the return time is undefined and can adhere to the original control flow of the kernel", however, this extension specifies that an implementation is only allowed of doing so "as long as all the WIs eventually finish the execution and allow the host runtime to abort the process", but what if they can't, because the one thread is causing issues?

Is it legal for an implementation to time out the kernel invocation, which might even cause system level problems, like a full GPU reset or even worse, a system reboot, in which case you won't get to see any printf buffer's contents?

@pjaaskel pjaaskel Jul 25, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right. This paragraph is in a conflict with the ability to abort kernel commands that have deadlocked (partial barrier reach or other reason). I think it could be just removed while fulfilling the goal of being able to implement libc abort and CUDA trap.

Comment on lines +106 to +109

@pjaaskel pjaaskel Jul 28, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
For possible WIs that do not call abort, the return time is undefined and can
adhere to the original control flow of the kernel as long as all the WIs
eventually finish the execution and allow the host runtime to abort the
process also in a single-threaded/non-interrupting implementation.
For possible other WIs that do not call abort can run for an undefined and
adhere to the original control flow of the kernel until the host detects the abort
request and ramps down the execution. Work-items misbehaving (for example
by means of illegal memory accessess) after the abort has been called result in undefined behavior, as normally.

@karolherbst could something along this lines cover your concerns?

@karolherbst karolherbst Jul 28, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small typo: "Possible other WIs that do not call abort can run for an undefined time ..."

But yeah, I think this covers it pretty well.


--

== Version History

[cols="5,15,15,70"]
[grid="rows"]
[options="header"]
|====
| Version | Date | Author | Changes
| 3 | 2022-05-14 | Pekka Jääskeläinen, Brice Videau |
*Changed to an 'ext' extension to promote multi-vendor adoption.*
| 2 | 2022-04-25 | Pekka Jääskeläinen, Brice Videau, Henry Linjamäki |
State explicitly that a compliant CPU device implementation can call abort() directly. Clarified the control flow behavior.
| 1 | 2022-04-08 | Pekka Jääskeläinen | *Initial revision for comments*
|====
Loading