[POC] [Core][NPU][GPU] Weights sharing between GPU and NPU#36941
Open
sivanov-work wants to merge 1 commit into
Open
[POC] [Core][NPU][GPU] Weights sharing between GPU and NPU#36941sivanov-work wants to merge 1 commit into
sivanov-work wants to merge 1 commit into
Conversation
github-actions
Bot
added
category: inference
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces a proof-of-concept implementation for sharing (relocated) Constant weights across GPU and NPU by backing them with a shared aligned host allocation that is exposed to both plugins via remote tensors. It adds the necessary buffer/descriptor plumbing on the NPUW side and teaches the GPU plugin to recognize and import these shared constants to avoid extra allocations/copies.
Changes:
- Added
ov::npuw::SharedContextBufferto allocate aligned host memory and bind it into GPU/NPU remote tensors, plus aSharedContextBufferDescriptorto expose per-context remote tensors to consumers. - Updated NPUW LLM compilation to optionally relocate large constants into shared buffers based on a
SHARED_WEIGHTSproperty. - Updated Intel GPU constant handling to detect shared-weight constants and reuse the underlying GPU-imported memory; added unit + functional tests covering the new sharing path.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/plugins/intel_npu/src/plugin/npuw/shared_context_buffer.hpp | Declares shared aligned buffer wrapper for cross-context remote tensor binding. |
| src/plugins/intel_npu/src/plugin/npuw/shared_context_buffer.cpp | Implements allocation + remote-tensor binding logic for GPU/NPU contexts. |
| src/inference/dev_api/openvino/runtime/shared_context_buffer_descriptor.hpp | Adds a runtime-visible descriptor type to expose remote tensors per context. |
| src/inference/src/dev/shared_context_buffer_descriptor.cpp | Implements the descriptor’s basic accessors. |
| src/plugins/intel_npu/src/plugin/npuw/llm_compiled_model.hpp | Adds shared-weight assignment hook and stores shared-weight context. |
| src/plugins/intel_npu/src/plugin/npuw/llm_compiled_model.cpp | Implements constant relocation into shared buffers when enabled by properties. |
| src/plugins/intel_gpu/src/plugin/ops/constant.cpp | Adds GPU-side detection/import path for shared-weight constants. |
| src/plugins/intel_npu/tests/unit/npuw/shared_context_buffer_test.cpp | Adds unit coverage for buffer descriptor + per-context remote tensor retrieval. |
| src/plugins/intel_npu/tests/functional/behavior/npuw/cross_context_tests.hpp | Adds functional test fixtures for cross-context sharing scenarios. |
| src/plugins/intel_npu/tests/functional/behavior/npuw/cross_context_tests.cpp | Adds functional tests validating shared I/O buffers and shared weights behavior. |
| src/plugins/intel_npu/tests/unit/CMakeLists.txt | Wires shared_context_buffer sources and the new unit test into the unit test target. |
| src/plugins/intel_npu/tests/functional/CMakeLists.txt | Wires shared_context_buffer sources and include path into functional tests. |
Comment on lines
+5
to
+9
| #include <algorithm> | ||
| #include <atomic> | ||
| #include <functional> | ||
| #include <memory> | ||
| #include <numeric> |
Comment on lines
+50
to
+55
| ov::AnyMap params = { | ||
| {ov::intel_npu::mem_type.name(), ov::intel_npu::MemType::CPU_VA}, | ||
| {ov::intel_npu::mem_handle.name(), buffer.get_ptr()}, | ||
| {ov::intel_npu::tensor_type.name(), ov::intel_npu::TensorType::INPUT}, | ||
| }; | ||
| return remote_ctx->create_tensor(ov::element::u8, ov::Shape{aligned_size}, params); |
Contributor
There was a problem hiding this comment.
@sivanov-work Do you see this usable for other cases besides as weights storage? I guess name of the file implies it's not only read only, so probably make sense to change.
| if (gpu_remote_tensor) { | ||
| GPU_DEBUG_LOG << "[" << initialconstPrimID << ": constant] layout " | ||
| << constLayout.to_short_string() << ", weight_sharing(" << static_cast<const void*>(data) << ", " << constLayout.bytes_count() <<" bytes)" << std::endl; | ||
| auto iremote_tensor = std::dynamic_pointer_cast<ov::intel_gpu::RemoteTensorImpl>(gpu_remote_tensor); |
Comment on lines
+46
to
+50
| auto remote_gpu_tensor_input = shared_context_input_output->get_remote_tensors_if_exist(remote_context_gpu); | ||
| assert(remote_gpu_tensor_input != nullptr && "Failed to get input remote tensor for GPU context"); | ||
| auto remote_npu_tensor_output = shared_context_input_output->get_remote_tensors_if_exist(remote_context_npu); | ||
| assert(remote_npu_tensor_output != nullptr && "Failed to get output remote tensor for NPU context"); | ||
| auto gpu_tensor_input = ov::make_tensor(remote_gpu_tensor_input); |
Comment on lines
+678
to
+681
| std::ostringstream oss; | ||
| oss << "TODO print me" << std::endl; | ||
| (void)shared_ctx; | ||
| return oss.str(); |
sshumihi
reviewed
Jul 17, 2026
| std::shared_ptr<IRemoteTensor> gpu_remote_tensor; | ||
| for (const auto& [remote_context, remote_tensor_ptr] : remote_contexts) { | ||
| GPU_DEBUG_LOG << "[" << initialconstPrimID << ": constant] check whether weight from the remote context: " << remote_context->get_device_name() << " is available on device: " << p.get_engine().get_device_info().dev_name << std::endl; | ||
| if (remote_context->get_device_name().find("GPU") != std::string::npos) { |
Contributor
There was a problem hiding this comment.
@michal-miotk Do you know what might be the right way to check for GPU device?
My concerns are:
- dGPU vs iGPU
- Can there be multiple contexts for one device?
Contributor
There was a problem hiding this comment.
- maybe get_engine().get_device_info().dev_type == device_type::integrated_gpu
sshumihi
changed the title
sshumihi
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Details:
Added SharedContextBuffer class private for NPUW context, which creates RemoteTensors using shared aligned memory, which is said to be available for read/write among GPU,NPU and CPU
Bind SharedContextBuffer memory objects as new ov::Constants for nodes of ov::Model preprocessed on the NPUW side
Teach GPU Plugin to perceive these constants as shared constants and import relevant RemoteTensors created by NPUW using the GPU remote context as new shared handle to avoid additional memory allocation for such constants and data copying.
Introduce a new SharedContextBufferDescriptor to allow to distinguish instances of AlignedBuffer and SharedContextBuffer between each other, which also provides RemoteTensors getters for consumers of remote contexts
Added unit tests on SharedContextBuffer in the scope of NPUW plugin
Added behavioral tests on SharedContextBuffer with RemoteTensors as well as shared weights in the scope on NPU plugin
Tickets:
-E###225081 Weights sharing between GPU and NPU POC
AI Assistance: