[pyapi, tests] Fix async inference queue in pythons API#36554
Open
praasz wants to merge 19 commits into
Open
Conversation
github-actions
Bot
added
category: Python API
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes a Python API lifetime bug in AsyncInferQueue.start_async(..., share_inputs=True) that could lead to use-after-free of NumPy-backed input buffers during async inference, and it adds/updates tests plus safeguards against unbounded blocking in sample test utilities.
Changes:
- Fix
AsyncInferQueue.start_async()to retain Python references to shared input data for the lifetime of the async request. - Add regression tests for
share_inputs=Trueuse-after-free scenarios (including a platform-independent lifetime test). - Add timeouts / bounded waits to sample test helpers to prevent indefinite hangs.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| tests/samples_tests/smoke_tests/common/samples_common_test_class.py | Adds request timeouts and adjusts download lock handling to avoid indefinite blocking. |
| tests/samples_tests/smoke_tests/common/common_utils.py | Adds a timeout to the shell() helper to bound command execution time. |
| src/bindings/python/tests/test_runtime/test_async_infer_request.py | Adds regression tests for async queue shared-input lifetime/data-integrity. |
| src/bindings/python/src/openvino/_ov_api.py | Fixes shared-input lifetime by storing per-handle references on the queue instance. |
Comment on lines
66
to
70
| for _ in range(9999): # Give up after about 3 hours | ||
| lock_acquired = False | ||
| with contextlib.suppress(FileExistsError, PermissionError): | ||
| with lock_path.open('bx'): | ||
| if not file_path.exists(): |
Comment on lines
+571
to
+575
| if share_inputs: | ||
| if not hasattr(self, "_inputs_data"): | ||
| self._inputs_data = {} | ||
| self._inputs_data[request_id] = getattr(request, "_inputs_data", None) | ||
| super().start_async(dispatched, userdata) |
almilosz
reviewed
Jun 24, 2026
| del img | ||
| gc.collect() | ||
| # Allocate a same-sized array with negative values. | ||
| # On CPython this lands at the exact same address the allocator just freed (100% reproducible). |
Contributor
There was a problem hiding this comment.
Where is this behavior documented?
| # relu(0.0) = 0.0 ≠ EXPECTED — if inference reads this the callback will NOT signal the event. | ||
| poison = np.zeros(shape, dtype=np.float32) # noqa: F841 – must stay alive | ||
|
|
||
| event_was_set = correct_output_seen.wait(timeout=hang_timeout) |
Contributor
There was a problem hiding this comment.
Let's assume for a moment this test runs without fix.
How setting hang_timeout on correct_output_seen prevents hang of the whole test suite? Is there a possibility that the whole test suite hangs before python interpreter gets to line 467?
Contributor
Author
There was a problem hiding this comment.
Why it should hang to this line?
all calls are non blocking
Contributor
There was a problem hiding this comment.
I am not sure what this test checks.
It simulates a hang but does not prevent the test from hanging for other reasons
…th share_inputs=False
The Copilot-suggested fix (pop _inputs_data[handle] when share_inputs=False) introduces a use-after-free: update_tensor (non-shared path) copies new data INTO the existing ov::Tensor buffer via tensor.data[:] = inputs[:]. That buffer IS the old numpy array's memory; releasing the reference before or after the copy leaves the tensor with a dangling pointer, causing a segfault when inference reads the tensor. The old array must remain alive until the NEXT share_inputs=True call creates a fresh zero-copy tensor and overwrites _inputs_data[handle]. New tests verify the correct ownership semantics: - _inputs_data not created when share_inputs always False - False->True establishes ownership - True->True releases old, holds new - True->False->True: buffer kept alive during non-shared reuse, released only on next shared call that replaces the tensor - Multi-handle: _inputs_data[A] and _inputs_data[B] are independent - Many consecutive shared cycles: each overwrite releases the previous array
github-actions
Bot
added
category: CI
…imeout handling in shell command
…ure timely process termination
…async-infer-request
… deadlock When AsyncInferQueue is destroyed, m_requests.clear() triggers a deep C++ destructor chain (InferRequest -> AUTO plugin CompiledModel -> AutoSchedule -> executor thread joins). If any of those threads attempt to acquire the GIL (e.g., via py::gil_scoped_acquire in callbacks), a deadlock occurs because the destructor runs with the GIL held. Fix: release the GIL in ~AsyncInferQueue() before clearing requests. The custom deleters in wrap_pyobject_to_sp and wrap_pyfunction will safely re-acquire the GIL when needed to destroy Python objects. The remaining member variables (m_user_ids, m_errors) are destroyed after the dtor body exits and the GIL is re-acquired, so they remain safe. Also add diagnostic logging in AutoSchedule::~AutoSchedule() to help identify compile-context model reset ordering in the destructor chain.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Details:
Bug
AsyncInferQueue.start_async(share_inputs=Trueuse-after-free (introduced July 2023, PR [PyOV] Memory flow control with share_inputs and share_outputs #18275)_inputs_dataon theAsyncInferQueueinstance itself, keyed by idle-handle index, beforesuper().start_async()is called. The entry is replaced only when the handle is next reused — which can only happen after the request's callback has fired and returned the handle to the idle queue — guaranteeing the numpy buffer outlives the inference.Bug: Three unbounded blocking calls in the test helpers:
samples_common_test_class.pyhad no timeout, now set 10s for TCP connection and 30s idle between chunkssamples_common_test_class.py— download lock loop could spin forever on Windows.common_utils.py—shell()had no execution timeout, now set default for 1200s (20 min)Tickets:
AI Assistance: