Skip to content

Commit 41cc96b

Browse files
jkasprzaCopilot
andauthored
[GPU] Add Level Zero - OpenCL interoperability (#36014)
### Details: - Detect if current system supports interoperability. - If check is passed, then create OCL remote context over L0 GPU runtime. - If check fails, then create L0 remote context over L0 GPU runtime. - Remote context and remote tensors query internal GPU plugin objects fur runtime specific handles. - OCL remote context will query for OCL handles. - L0 remote context will query for L0 handles. - Enable L0 internal GPU plugin objects to return converted OCL handles. - Add `resource_owner` template for gpu resource management. - Add `ze_resource` template that will hold Level Zero objects and optionally corresponding OpenCL objects. - Add operators for conversion between Level Zero objects and OpenCL objects. ### Benchmarks: Peak memory usage measured on benchmark_app running resnet-50 on BMG Linux | | VmPeak (kB) | MaxRSS (kB) | | --- | --- | --- | | This PR (runtime=L0, interop=ON) | 3,262,536 | 190,924 | | This PR (runtime=L0, interop=OFF) | 3,263,688 | 193,200 | | Master (runtime=L0) | 3,293,048 | 192,188 | | [REF] Master (runtime=OCL) | 3,281,412 | 193,196 | ### Tickets: - [CVS-185378](https://jira.devtools.intel.com/browse/CVS-185378) ### AI Assistance: - AI assistance used: yes - AI help was used to create tests --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
1 parent 4a18707 commit 41cc96b

55 files changed

Lines changed: 2086 additions & 606 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

src/inference/include/openvino/runtime/intel_gpu/remote_properties.hpp

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ using os_handle_param = void*;
3030
enum class ContextType {
3131
OCL = 0, //!< Pure OpenCL context
3232
VA_SHARED = 1, //!< Context shared with a video decoding device
33-
ZE = 2, //!< Pure Level0 context
33+
ZE = 2, //!< Pure Level Zero context
3434
};
3535

3636
/** @cond INTERNAL */
@@ -52,10 +52,10 @@ inline std::istream& operator>>(std::istream& is, ContextType& context_type) {
5252
is >> str;
5353
if (str == "OCL") {
5454
context_type = ContextType::OCL;
55-
} else if (str == "ZE") {
56-
context_type = ContextType::ZE;
5755
} else if (str == "VA_SHARED") {
5856
context_type = ContextType::VA_SHARED;
57+
} else if (str == "ZE") {
58+
context_type = ContextType::ZE;
5959
} else {
6060
OPENVINO_THROW("Unsupported context type: ", str);
6161
}
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Support for Level Zero - OpenCL interoperability: New in OpenVINO 2026.3
2+
3+
*By Jakub Kasprzak | June 23, 2026*
4+
5+
Some projects maintain their own OpenCL GPU pipeline and require features such as memory, context or queue sharing. For such users OpenVINO GPU plugin implements Remote Context and Remote Tensor interface. However, this OpenCL interoperability works only if OpenVINO GPU runtime is set to OpenCL and prevents OpenVINO from switching to Level Zero runtime.
6+
OpenVINO 2026.3 introduces Level Zero - OpenCL interoperability enabling users that depend on OpenVINO Remote Context/Tensor OpenCL API to swap OpenVINO GPU runtime to Level Zero without changing their OpenCL based integration logic. This post explains: how to enable this feature and provides OpenVINO maintainers with design description.
7+
8+
## How to enable Level Zero - OpenCL interoperability
9+
10+
OpenVINO GPU plugin with Level Zero runtime detects if interoperability is supported on current system and enables it automatically. User must ensure that currently installed Intel GPU driver supports and enables "Level Zero executing OpenCL" (LEO) feature.
11+
12+
## Level Zero - OpenCL interoperability design
13+
14+
*This section describes internal design of the feature and is intended for OpenVINO maintainers*
15+
16+
### Object lifetimes
17+
18+
This subsection describes part of the design responsible for managing resources. Main goal was to provide modular API that will minimize chances of memory leaks and other memory bugs.
19+
20+
Logic for managing all (Level Zero and OpenCL) GPU resources is provided by `resource_owner` template class. It contains `is_borrowed` flag to distinguish between resources that need to be released and resources that should not be released (borrowed resources, for example: external memory buffers provided by the user). Once `resource_owner` is destroyed it will release the resource or do nothing if resource was borrowed. Basically `resource_owner` with `is_borrowed=false` is a generic RAII wrapper which means that only one such object can exist for a given resource and that it can not be copied (only move semantics are allowed).
21+
22+
In order to work correctly `resource_owner` template must be provided with information about the type of resource it manages and how to release it. This information is defined for Level Zero and OpenCL resources in `ze_resource_info` and `ocl_resource_info` class template specializations respectively. For example `ocl_resource_info<ocl_resource_type::command_queue>` defines `handle_t` as `cl_command_queue` and `deleter_t` as a functor that calls `clReleaseCommandQueue`. Finally `ze_owner` and `ocl_owner` aliases are defined by combining `resource_owner` with `ze_resource_info` and `ocl_resource_info` respectively.
23+
24+
### Binding corresponding Level Zero - OpenCL handles
25+
26+
It is important to recognize that with interoperability some Level Zero handles will have corresponding OpenCL handles and vice versa. For example, after converting Level Zero immediate command list to OpenCL command queue we end up with two different handles referring to the same entity. `ze_ocl_owner_impl` is a template that is responsible for combining Level Zero handle with OpenCL handle and managing them as a single resource. Internally it contains `ze_owner` and tuple of optional `ocl_owner`s. By design `ze_ocl_owner_impl` always contains Level Zero resource and OpenCL resources can be attached to it. Once OpenCL resource of some given type is attached to `ze_ocl_owner_impl` attempt to attach another OpenCL resource of the same type will result in **exception** (this design is to prevent exporting single Level Zero resource multiple times).
27+
28+
Not all handle combinations are valid. For example in current context it does not make sense to attach/bind `cl_command_queue` handle to `ze_ocl_owner_impl` that holds `ze_image_handle_t`. To prevent this `ze_ocl_owner` template inherits only valid `ze_ocl_owner_impl` specializations. For example `ze_ocl_owner<ze_resource_type::command_list>` is `ze_ocl_owner_impl<ze_resource_type::command_list, ocl_resource_type::command_queue>` meaning it holds `ze_command_list_handle_t` and optionally `cl_command_queue` handles. If specialization is not defined explicitly it fall backs to default implementation where only Level Zero handle is allowed, for example `ze_ocl_owner<ze_resource_type::kernel>` holds only `ze_kernel_handle_t` and attempt to attach or access any type of OpenCL handle will result in **compilation error**.
29+
30+
### Shared resource ownership
31+
32+
Finally `ze_resource` template combines `std::shared_ptr` and `ze_ocl_owner` to implement shared ownership model for GPU resources. This is especially useful for handling lifetime of Level Zero events and event pools where `ze_resource<ze_resource_type::event_pool>` can be copied to all event objects created from given pool ensuring that the pool resource will be destroyed only after last event from the pool was destroyed (same scheme is used to manage lifetime of Level Zero modules and kernels).
33+
34+
Additionaly aliases in form of `ze_*_resource` such as `ze_event_pool_resource` are defined for developer convenience.
35+
36+
### Interoperability integration
37+
38+
Basic interoperability is implemented by a `ze_ocl_interop` singleton that detects interoperability support and provides direct (no abstraction) Level Zero - OpenCL handle to handle conversion interface. When `ze_ocl_interop` is accessed for the first time it will initialize GPU runtimes and map corresponding Level Zero and OpenCL devices.
39+
40+
Conversion interface provided by `ze_ocl_interop` is wrapped by `ze_import_*` and `ze_export_ocl` functions from `ze_resource_interop.hpp` in order to simplify conversion logic when used with `ze_resource` objects. For example calling `ze_export_ocl_command_queue` for given `ze_command_list_resource` will:
41+
1. Return early if `ze_command_list_resource` already has attached OpenCL command queue handle
42+
2. Create new OpenCL command queue based on present Level Zero handle
43+
3. Attach new OpenCL handle to `ze_command_list_resource`
44+
4. Guarantee strong exception safety
45+
46+
### Remote context
47+
48+
Type of remote context decides what kind of objects are exposed by the GPU plugin. During model compilation OpenVINO creates remote context and internal engine that by default match currently selected GPU runtime. For example, when GPU runtime is set to Level Zero this causes the creation of Level Zero remote context and `ze_engine`. Remote context will query internal GPU plugin objects for handles based on the remote context type. So if Level Zero remote context is used only Level Zero handles can be requested from internal objects and OpenCL interoperability is not required.
49+
50+
If user provides OpenCL context or queue or OpenVINO detects that current system supports Level Zero - OpenCL interoperability then remote context type will be changed to OpenCL but engine will still match currently selected GPU runtime. In result OpenCL remote context will request OpenCL handles from internal Level Zero GPU plugin objects like `ze_engine` forcing usage of interoperability features.

src/plugins/intel_gpu/include/intel_gpu/plugin/remote_context.hpp

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -95,11 +95,8 @@ class RemoteContextImpl : public ov::IRemoteContext {
9595
ov::intel_gpu::gpu_handle_param m_va_display = nullptr;
9696
ov::intel_gpu::gpu_handle_param m_external_queue = nullptr;
9797

98-
#ifdef OV_GPU_WITH_ZE_RT
99-
ContextType m_type = ContextType::ZE;
100-
#else
101-
ContextType m_type = ContextType::OCL;
102-
#endif
98+
99+
ContextType m_type;
103100
std::string m_device_name = "";
104101
static const size_t cache_capacity = 100;
105102
cldnn::LruCache<size_t, cldnn::memory::ptr> m_memory_cache = cldnn::LruCache<size_t, cldnn::memory::ptr>(cache_capacity);

src/plugins/intel_gpu/include/intel_gpu/runtime/device_info.hpp

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -122,8 +122,9 @@ struct device_info {
122122
bool supports_usm; ///< Does engine support unified shared memory.
123123
bool has_separate_cache; ///< Does the target hardware has separate cache for usm_device and usm_host
124124

125-
bool supports_cp_offload; ///< [ZE] Does the command queue support copy offload
126-
bool supports_counter_based_events; ///< [ZE] Does the target runtime support counter based events
125+
bool supports_cp_offload; ///< [L0] Does the command queue support copy offload
126+
bool supports_counter_based_events; ///< [L0] Does the target runtime support counter based events
127+
bool supports_leo; ///< [L0] Does the device support Level Zero - OpenCL interoperability (LEO)
127128

128129
std::vector<size_t> supported_simd_sizes; ///< List of SIMD sizes supported by current device and compiler
129130

src/plugins/intel_gpu/include/intel_gpu/runtime/device_query.hpp

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,6 @@ namespace cldnn {
1717
struct device_query {
1818
public:
1919
static int device_id;
20-
/// @brief Get default engine type
21-
static engine_types get_default_engine_type();
22-
23-
/// @brief Get default runtime type
24-
static runtime_types get_default_runtime_type();
2520

2621
explicit device_query(engine_types engine_type,
2722
runtime_types runtime_type,

src/plugins/intel_gpu/include/intel_gpu/runtime/engine.hpp

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -113,8 +113,8 @@ class engine {
113113
/// Returns device object associated with the engine
114114
const device::ptr get_device() const;
115115

116-
/// Returns user context handle which was used to create the engine
117-
virtual void* get_user_context() const = 0;
116+
/// Returns L0 or OpenCL user context handle which was used to create the engine.
117+
virtual void* get_user_context(runtime_types rt_type) const = 0;
118118

119119
/// Returns the total maximum amount of GPU memory allocated by engine in current process for all allocation types
120120
uint64_t get_max_used_device_memory() const;

src/plugins/intel_gpu/include/intel_gpu/runtime/engine_configuration.hpp

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,4 +64,10 @@ inline std::ostream& operator<<(std::ostream& os, const backend_types& type) {
6464
return os;
6565
}
6666

67+
/// @brief Get default engine type
68+
engine_types get_default_engine_type();
69+
70+
/// @brief Get default runtime type
71+
runtime_types get_default_runtime_type();
72+
6773
} // namespace cldnn

src/plugins/intel_gpu/include/intel_gpu/runtime/memory.hpp

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -58,7 +58,11 @@ struct memory {
5858

5959
size_t size() const { return _bytes_count; }
6060
size_t count() const { return _layout.count(); }
61-
virtual shared_mem_params get_internal_params() const = 0;
61+
/// @brief Return internal params for specified runtime type
62+
virtual shared_mem_params get_internal_params(runtime_types rt_type) const = 0;
63+
shared_mem_params get_internal_params() const {
64+
return get_internal_params(get_default_runtime_type());
65+
}
6266
virtual bool is_allocated_by(const engine& engine) const { return &engine == _engine && _type != allocation_type::unknown; }
6367
engine* get_engine() const { return _engine; }
6468
const layout& get_layout() const { return _layout; }
@@ -156,7 +160,7 @@ struct simple_attached_memory : memory {
156160
void unlock(const stream& /* stream */) override {}
157161
event::ptr fill(stream& /* stream */, unsigned char, const std::vector<event::ptr>&, bool) override { return nullptr; }
158162
event::ptr fill(stream& /* stream */, const std::vector<event::ptr>&, bool) override { return nullptr; }
159-
shared_mem_params get_internal_params() const override { return { shared_mem_type::shared_mem_empty, nullptr, nullptr, nullptr,
163+
shared_mem_params get_internal_params(runtime_types rt_type) const override { return { shared_mem_type::shared_mem_empty, nullptr, nullptr, nullptr,
160164
#ifdef _WIN32
161165
nullptr,
162166
#else

src/plugins/intel_gpu/include/intel_gpu/runtime/memory_caps.hpp

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,7 @@ using shared_surface = uint32_t;
110110
/// @brief Low-level API handles required for using cldnn memory objects in external API calls.
111111
struct shared_mem_params {
112112
shared_mem_type mem_type; ///< shared buffer type
113-
shared_handle context; ///< OpenCL context for external operations
113+
shared_handle context; ///< OpenCL or Level Zero context for external operations
114114
shared_handle user_device; ///< DX/VA device for external operations
115115
shared_handle mem; ///< memory object handle
116116
#ifdef _WIN32

src/plugins/intel_gpu/src/plugin/remote_context.cpp

Lines changed: 29 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -23,12 +23,29 @@ Type extract_object(const ov::AnyMap& params, const ov::Property<Type>& p) {
2323
return res.as<Type>();
2424
}
2525

26+
ContextType get_default_context_type() {
27+
#ifdef OV_GPU_WITH_ZE_RT
28+
return ContextType::ZE;
29+
#elif defined(OV_GPU_WITH_OCL_RT)
30+
return ContextType::OCL;
31+
#else
32+
#error "Expected OpenVINO GPU runtime macros to be defined"
33+
#endif
34+
}
35+
2636
} // namespace
2737

2838
RemoteContextImpl::RemoteContextImpl(const std::string& device_name, std::vector<cldnn::device::ptr> devices, bool initialize_ctx)
2939
: m_device_name(device_name) {
3040
OPENVINO_ASSERT(devices.size() == 1, "[GPU] Currently context can be created for single device only");
3141
m_device = devices.front();
42+
m_type = get_default_context_type();
43+
const auto &info = m_device->get_info();
44+
if (m_type == ContextType::ZE && info.supports_leo) {
45+
m_type = ContextType::OCL;
46+
GPU_DEBUG_INFO << "Enabled Level Zero - OpenCL interoperability for "
47+
<< m_device_name << " (" << info.dev_name << ")" << std::endl;
48+
}
3249

3350
if (initialize_ctx) {
3451
initialize();
@@ -39,6 +56,7 @@ RemoteContextImpl::RemoteContextImpl(const std::map<std::string, RemoteContextIm
3956
gpu_handle_param context_id = nullptr;
4057
int ctx_device_id = 0;
4158
int target_tile_id = -1;
59+
m_type = get_default_context_type();
4260

4361
if (params.size()) {
4462
auto ctx_type = extract_object(params, ov::intel_gpu::context_type);
@@ -56,20 +74,23 @@ RemoteContextImpl::RemoteContextImpl(const std::map<std::string, RemoteContextIm
5674
} else if (ctx_type == ov::intel_gpu::ContextType::VA_SHARED) {
5775
m_va_display = extract_object(params, ov::intel_gpu::va_device);
5876
OPENVINO_ASSERT(m_va_display != nullptr, "[GPU] Can't create shared VA/DX context as user handle is nullptr! Params:\n", params);
59-
m_type = ContextType::VA_SHARED;
77+
} else if (ctx_type == ov::intel_gpu::ContextType::ZE) {
78+
OPENVINO_THROW("Level Zero interoperability is not supported");
6079
} else {
6180
OPENVINO_THROW("Invalid execution context type", ctx_type);
6281
}
82+
m_type = ctx_type;
6383
if (params.find(ov::intel_gpu::tile_id.name()) != params.end()) {
6484
target_tile_id = extract_object(params, ov::intel_gpu::tile_id);
6585
}
6686
}
6787

6888
const auto initialize_devices = true;
6989

70-
cldnn::device_query device_query(context_id, m_va_display, ctx_device_id, target_tile_id, initialize_devices);
71-
auto device_map = device_query.get_available_devices();
72-
90+
// Always use OCL for device query
91+
// Query will return devices compatible with default runtime
92+
cldnn::device_query ocl_device_query(cldnn::engine_types::ocl, cldnn::runtime_types::ocl, context_id, m_va_display, ctx_device_id, target_tile_id, initialize_devices);
93+
auto device_map = ocl_device_query.get_available_devices();
7394
OPENVINO_ASSERT(device_map.size() == 1, "[GPU] Exactly one device expected in case of context sharing, but ", device_map.size(), " found");
7495

7596
m_device = device_map.begin()->second;
@@ -89,19 +110,20 @@ const cldnn::engine& RemoteContextImpl::get_engine() const {
89110
}
90111

91112
void RemoteContextImpl::init_properties() {
92-
properties = { ov::intel_gpu::ocl_context(m_engine->get_user_context()) };
93-
94113
switch (m_type) {
95114
case ContextType::OCL:
96115
properties.insert(ov::intel_gpu::context_type(ov::intel_gpu::ContextType::OCL));
116+
properties.insert(ov::intel_gpu::ocl_context(m_engine->get_user_context(cldnn::runtime_types::ocl)));
97117
properties.insert(ov::intel_gpu::ocl_queue(m_external_queue));
98118
break;
99119
case ContextType::VA_SHARED:
100120
properties.insert(ov::intel_gpu::context_type(ov::intel_gpu::ContextType::VA_SHARED));
121+
properties.insert(ov::intel_gpu::ocl_context(m_engine->get_user_context(cldnn::runtime_types::ocl)));
101122
properties.insert(ov::intel_gpu::va_device(m_va_display));
102123
break;
103124
case ContextType::ZE:
104125
properties.insert(ov::intel_gpu::context_type(ov::intel_gpu::ContextType::ZE));
126+
properties.insert(ov::intel_gpu::ocl_context(m_engine->get_user_context(cldnn::runtime_types::ze)));
105127
break;
106128
default:
107129
OPENVINO_THROW("[GPU] Unsupported shared context type ", m_type);
@@ -256,7 +278,7 @@ void RemoteContextImpl::initialize() {
256278

257279
m_device->initialize(); // Initialize associated device before use
258280
m_engine = cldnn::engine::create(
259-
cldnn::device_query::get_default_engine_type(), cldnn::device_query::get_default_runtime_type(), m_device);
281+
cldnn::get_default_engine_type(), cldnn::get_default_runtime_type(), m_device);
260282

261283
init_properties();
262284

0 commit comments

Comments
 (0)