[GPU] Support CPU pointers on remote tensor by michal-miotk · Pull Request #36539 · openvinotoolkit/openvino

michal-miotk · 2026-06-23T16:04:42Z

Details:

supports pointers created by aligned_alloc and mmap
support only on OCL side

Tickets:

189076

maxnick

LGTM

jkasprza · 2026-07-08T07:33:36Z

+    auto tracker = std::make_shared<MemoryTracker>(this,
+                                                   cpu_address,
+                                                   data_size,
+                                                   allocation);


Should this memory tracker be created? I guess there is no additional allocation when CL_MEM_FORCE_HOST_MEMORY_INTEL is used?

tracker is needed in some places in code, constructed without engine not add allocation to statistic

should be same pattern made in case of NTHandle ?

Yes, I think we should only track OpenVINO allocations and do not count memory allocated by the user.

jkoniusz

Maybe this is gonna be of some use to you:
The checks are failing because (I checked build_win_vs2022_release log):
"
[2026-07-08T09:07:41.198Z] C:\Jenkins\workspace\private-ci\ie\build-windows-vs2022\b\repos\openvino\src\plugins\intel_gpu\tests\functional\remote_tensor_tests\dx11_nthandle.cpp(252): error C3861: 'SharedBufferHandle': identifier not found
[2026-07-08T09:07:41.199Z] C:\Jenkins\workspace\private-ci\ie\build-windows-vs2022\b\repos\openvino\src\plugins\intel_gpu\tests\functional\remote_tensor_tests\dx11_nthandle.cpp(250): error C2668: 'ov::intel_gpu::ocl::ClContext::create_tensor': ambiguous call to overloaded function
"
Theres compile error

Allegedly now dx11_nthandle.cpp / dx12_nthandle.cpp still reference a SharedBufferHandle symbol.

SharedBufferHandle isn't a valid/visible name in those files — either it was never namespace-qualified (ov::intel_gpu::ocl::SharedBufferHandle), or it got renamed/removed when you refactored API.

Because that argument doesn't resolve, the create_tensor(...) call can't pick an overload → error C2668: ambiguous call.

@michal-miotk @maxnick @jkasprza

michal-miotk · 2026-07-08T10:36:12Z

Maybe this is gonna be of some use to you: The checks are failing because (I checked build_win_vs2022_release log): " [2026-07-08T09:07:41.198Z] C:\Jenkins\workspace\private-ci\ie\build-windows-vs2022\b\repos\openvino\src\plugins\intel_gpu\tests\functional\remote_tensor_tests\dx11_nthandle.cpp(252): error C3861: 'SharedBufferHandle': identifier not found [2026-07-08T09:07:41.199Z] C:\Jenkins\workspace\private-ci\ie\build-windows-vs2022\b\repos\openvino\src\plugins\intel_gpu\tests\functional\remote_tensor_tests\dx11_nthandle.cpp(250): error C2668: 'ov::intel_gpu::ocl::ClContext::create_tensor': ambiguous call to overloaded function " Theres compile error

Allegedly now dx11_nthandle.cpp / dx12_nthandle.cpp still reference a SharedBufferHandle symbol.

SharedBufferHandle isn't a valid/visible name in those files — either it was never namespace-qualified (ov::intel_gpu::ocl::SharedBufferHandle), or it got renamed/removed when you refactored API.

Because that argument doesn't resolve, the create_tensor(...) call can't pick an overload → error C2668: ambiguous call.

@michal-miotk @maxnick @jkasprza

probably need add ov::intel_gpu:: before SharedBufferHandle

Copilot

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 4 comments.

    createHeap(byte_size);

-    auto remote_tensor = context.create_tensor(ov::element::f32, tensor.get_shape(), shared_mem, ov::intel_gpu::MemType::SHARED_BUF);
+    auto remote_tensor = context.create_tensor(ov::element::f32, tensor.get_shape(), SharedBufferHandle(shared_mem));


michal-miotk · 2026-07-08T12:15:13Z

+#ifdef CL_MEM_FORCE_HOST_MEMORY_INTEL
+    const size_t minimal_alignment = static_cast<size_t>(get_device_info().cacheline_size);
+    OPENVINO_ASSERT((reinterpret_cast<std::uintptr_t>(cpu_address) % minimal_alignment) == 0,
+                    "[GPU] shared buffer pointer must be ", minimal_alignment, "-byte aligned");
+    OPENVINO_ASSERT((data_size % minimal_alignment) == 0,
+                    "[GPU] shared buffer size must be a multiple of ", minimal_alignment, " bytes");
+    flags |= CL_MEM_FORCE_HOST_MEMORY_INTEL;
+#endif


@jkasprza should I add this ?

I would prefer 1 additional check over risking undefined behavior. Even if we expect that cacheline_size is more than 0.

michal-miotk · 2026-07-08T12:14:47Z

+    std::shared_ptr<MemoryTracker> tracker = nullptr;
+#ifdef CL_MEM_FORCE_HOST_MEMORY_INTEL
+    tracker = std::make_shared<MemoryTracker>(nullptr,
+                                              cpu_address,
+                                              data_size,
+                                              allocation);
+#else
+    tracker = std::make_shared<MemoryTracker>(this,
+                                              cpu_address,
+                                              data_size,
+                                              allocation);
+#endif


this is my intention to not count this in statistic

michal-miotk · 2026-07-08T13:58:19Z

+    uint32_t cacheline_size = core.get_property(target_device, ov::intel_gpu::cacheline_size);
+    const size_t float_size = sizeof(float);
+    const ov::Shape shape{cacheline_size/float_size};
+    const size_t element_count = ov::shape_size(shape);
+    const size_t byte_size = element_count * sizeof(float);
+    auto ctx = core.get_default_context(target_device).as<ov::intel_gpu::ocl::ClContext>();
+    void* input_ptr = ov::util::aligned_alloc(byte_size, byte_size);
+    void* output_ptr = ov::util::aligned_alloc(byte_size, byte_size);


partially applied, proposed change: const ov::Shape shape{static_cast<size_t>(cacheline_size)}; not check bound case

@isanghao

Typo was fixed. So, I dismiss review to respect @isanghao time zone. I will put my approve

#### Details: Description of the issue (symptom, root-cause, how it was resolved) - Zero-copy handling for `usm_host` and `usm_shared` was already implemented in earlier PRs (openvinotoolkit#34494, openvinotoolkit#36539); this PR focuses on closing the `usm_device` gap. - Previously, for `usm_device` allocations, a separate GPU `usm_device` buffer was allocated and filled from mapped host buffer data during cache load, causing duplicate allocation/copy overhead and increasing inference memory footprint. - This change enables zero-copy subbuffer usage for supported XE2+ iGPUs in the `usm_device` flow, removing extra staging/copy in applicable cases. - Blob offset alignment handling in deserialization is kept correct for subbuffer creation. Implementation changes: - Extend zero-copy eligibility in `data.hpp` to include `usm_device` while preserving existing `usm_host`/`usm_shared` behavior. - Add/use helper check for zero-copy-capable device (XE2+ integrated GPU). - Use `seek_current_ptr()` for padding skip during deserialization alignment. - Refine zero-copy condition naming/readability in load path. Changed files: src/plugins/intel_gpu/include/intel_gpu/graph/serialization/binary_buffer.hpp src/plugins/intel_gpu/include/intel_gpu/primitives/data.hpp src/plugins/intel_gpu/include/intel_gpu/runtime/device_info.hpp src/plugins/intel_gpu/src/graph/program.cpp src/plugins/intel_gpu/src/runtime/ocl/ocl_device.cpp src/plugins/intel_gpu/src/runtime/ocl/ocl_engine.cpp src/plugins/intel_gpu/tests/unit/test_cases/cache_serialization_test.cpp Reproduction step and snapshot (if applicable. Do not attach for customer model) NA Problematic graph NA #### Checklist - [x] Is it a proper fix? Yes (removes duplicate `usm_device` staging/copy in supported zero-copy cases) - [x] Did you include test case for this fix, if necessary? NA - [x] Did you review existing test that can be extended to cover this scenario? NA #### Tickets: - CVS-176160 #### AI Assistance: - AI assistance used: yes - AI was used for wording refinement and commit message cleanup.

michal-miotk added 6 commits June 15, 2026 14:01

wip

dd1c19f

wip2

d1f2e3d

wwip3

51aba8b

wip

b5b2202

wip

32055d8

delete test

acd8238

michal-miotk requested review from a team as code owners June 23, 2026 16:04

michal-miotk requested review from mfhansen and removed request for a team June 23, 2026 16:04

github-actions Bot added category: inference OpenVINO Runtime library - Inference category: GPU OpenVINO GPU plugin category: build OpenVINO cmake script / infra category: CPP API OpenVINO CPP API bindings category: docs_snippets OpenVINO docs snippets (docs/snippets) labels Jun 23, 2026

michal-miotk changed the title ~~Mmap~~ [GPU] Support CPU pointers on remote tensor Jun 23, 2026

michal-miotk added 5 commits June 23, 2026 16:15

wip

0163d80

fix

6d607f4

Merge branch 'master' into mmap

8169026

cleaning

39867e7

better name

c59a8da

github-actions Bot removed the category: build OpenVINO cmake script / infra label Jun 23, 2026

michal-miotk added 3 commits June 23, 2026 20:30

cleaning

0395b7f

better tensor name, fix code style

fe30d2e

added missing include

85ea425

p-durandin added this to the 2026.3 milestone Jun 24, 2026

p-durandin assigned jkasprza and maxnick Jun 24, 2026

maxnick approved these changes Jul 7, 2026

View reviewed changes

Update ocl.hpp code style

c93f145

isanghao previously requested changes Jul 8, 2026

View reviewed changes

Comment thread src/plugins/intel_gpu/include/intel_gpu/plugin/remote_tensor.hpp Outdated

p-durandin added 3 commits July 8, 2026 09:13

Update remote_properties.hpp naming fix

6dc84d4

Update remote_properties.hpp

ecadba8

Update remote_properties.hpp

2104785

jkasprza requested changes Jul 8, 2026

View reviewed changes

jkoniusz reviewed Jul 8, 2026

View reviewed changes

fix some typo and move hash functions and change alloc type

720064b

tracker alive but without engine not add memory

64415ca

michal-miotk requested a review from Copilot July 8, 2026 12:01

Copilot started reviewing on behalf of michal-miotk July 8, 2026 12:02 View session

Copilot AI reviewed Jul 8, 2026

View reviewed changes

michal-miotk added 5 commits July 8, 2026 12:06

added missing empty line

9444fb1

added part of copilot review

6715948

check, change not reporting memory in same way for HANDLE

483e8fe

fix tests

5d97af5

added check for cacheline_size to test

e2d0c24

jkasprza approved these changes Jul 8, 2026

View reviewed changes

p-durandin approved these changes Jul 8, 2026

View reviewed changes

p-durandin enabled auto-merge July 8, 2026 14:48

fix

d24d24e

p-durandin added this pull request to the merge queue Jul 9, 2026

Merged via the queue into openvinotoolkit:master with commit d64394f Jul 9, 2026
206 of 207 checks passed

Prithviraj-R mentioned this pull request Jul 15, 2026

[GPU] Enable zero-copy subbuffers for usm_device on iGPUs #36645

Open

3 tasks

Uh oh!

Conversation

michal-miotk commented Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details:

Tickets:

Uh oh!

maxnick left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jkoniusz left a comment

Choose a reason for hiding this comment

Uh oh!

michal-miotk commented Jul 8, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

michal-miotk commented Jun 23, 2026 •

edited

Loading