WIP: PEP 688 demonstration#5665
Closed
blowekamp wants to merge 1 commit intoInsightSoftwareConsortium:mainfrom
Closed
WIP: PEP 688 demonstration#5665blowekamp wants to merge 1 commit intoInsightSoftwareConsortium:mainfrom
blowekamp wants to merge 1 commit intoInsightSoftwareConsortium:mainfrom
Conversation
github-actions
bot
added
area:Python wrapping
Member
Author
|
@thewtex I don' have any intention of taking this change further. I am just sharing the code in case it's useful for exploring this new python feature in ITK Python. It has recently been implemented in SimpleITK. This PR can be closed without merging. |
Experiment with implementing PEP 688. Replace __array__ with __buffer__. Convert memoryview to correctly specified n-d view. PEP 688 automatically maintains a a reference to the provider of the buffer. Add incomplete wrapping for the import image container, and buffer interface.
github-actions
bot
removed
the
area:Filtering
Member
|
@blowekamp wow! Awesome! Thank you so much for contributing this! I'll push it forward |
7 tasks
Member
Author
|
There may additional useful information related to this here: SimpleITK/SimpleITK#2447 |
This was referenced Apr 7, 2026
Member
|
Superseded by #6020 which builds on this exploration. The new PR adds PEP 688 |
hjmjohnson
added a commit
to hjmjohnson/ITK
that referenced
this pull request
Apr 9, 2026
Add zero-copy data export to all wrapped itk.Image types via two
protocols, ensuring the exported array remains valid even after
the source image is deleted:
image = itk.imread("brain.nii.gz")
arr = np.asarray(image)
del image
print(arr[1,1,1]) # safe -- no crash
Protocol dispatch by Python version:
3.12+: np.asarray -> __buffer__ (PEP 688, zero-copy, memoryview
pins self via NDArrayITKBase intermediary)
3.10-11: np.asarray -> __array__ -> array_view_from_image (zero-copy,
NDArrayITKBase.itk_base holds reference to image)
Changes to pyBase.i:
- Add __buffer__() implementing PEP 688 buffer export with shaped
memoryview. Uses NDArrayITKBase as intermediary to hold a Python
reference to the image, preventing GC while any derived
memoryview/array exists.
- Simplify __array__() to always return zero-copy view via
array_view_from_image(). Supports NumPy 2.0 copy= parameter.
- Remove __array_interface__ (returned raw pointer with no reference
holder -- use-after-free on del image, confirmed by test).
- Remove SIMULATE_PEP688 / SIMULATE_PEP688_DEBUG (confusing,
contradictory behaviors between __buffer__ and __array__ paths).
Changes to PyBuffer.i.init:
- Add _get_buffer_formatstring() with module-level _BUFFER_FORMAT_MAP
for struct format lookup (UC->B, SS->h, F->f, D->d, etc.)
- Add _get_numpy_pixelid() with module-level _NUMPY_PIXELID_MAP
- Remove LD (long double) mapping -- sizeof(long double) varies by
platform, struct format "d" is always 8 bytes (silent corruption)
Supersedes InsightSoftwareConsortium#6020, InsightSoftwareConsortium#6018, InsightSoftwareConsortium#5673, InsightSoftwareConsortium#5665. Key improvements over each:
- InsightSoftwareConsortium#6020: Fixed __buffer__ lifetime (memoryview didn't pin image),
removed unsafe __array_interface__, removed SIMULATE_PEP688
- InsightSoftwareConsortium#6018: Closed in favor of InsightSoftwareConsortium#6020
- InsightSoftwareConsortium#5673: Added __array_interface__ (now removed) and __buffer__
- InsightSoftwareConsortium#5665: Original PEP 688 implementation by blowekamp
Addresses review concerns from @thewtex (del image crash),
@blowekamp (reference pinning at buffer owner level).
Tested: 121 assertions across 3 test suites, Python 3.13 and 3.14,
with NumPy 2.4.3, PyTorch 2.11.0, Dask 2026.3.0. All 32 lifetime
tests pass (del image safe on every export path).
Co-Authored-By: Hans J. Johnson <hans-johnson@uiowa.edu>
11 tasks
hjmjohnson
added a commit
to hjmjohnson/ITK
that referenced
this pull request
Apr 9, 2026
Add zero-copy data export to all wrapped itk.Image types via two
protocols, ensuring the exported array remains valid even after
the source image is deleted:
image = itk.imread("brain.nii.gz")
arr = np.asarray(image)
del image
print(arr[1,1,1]) # safe -- no crash
Protocol dispatch by Python version:
3.12+: np.asarray -> __buffer__ (PEP 688, zero-copy, memoryview
pins self via NDArrayITKBase intermediary)
3.10-11: np.asarray -> __array__ -> array_view_from_image (zero-copy,
NDArrayITKBase.itk_base holds reference to image)
Changes to pyBase.i:
- Add __buffer__() implementing PEP 688 buffer export with shaped
memoryview. Uses NDArrayITKBase as intermediary to hold a Python
reference to the image, preventing GC while any derived
memoryview/array exists.
- Simplify __array__() to always return zero-copy view via
array_view_from_image(). Supports NumPy 2.0 copy= parameter.
- Remove __array_interface__ (returned raw pointer with no reference
holder -- use-after-free on del image, confirmed by test).
- Remove SIMULATE_PEP688 / SIMULATE_PEP688_DEBUG (confusing,
contradictory behaviors between __buffer__ and __array__ paths).
Changes to PyBuffer.i.init:
- Add _get_buffer_formatstring() with module-level _BUFFER_FORMAT_MAP
for struct format lookup (UC->B, SS->h, F->f, D->d, etc.)
- Add _get_numpy_pixelid() with module-level _NUMPY_PIXELID_MAP
- Remove LD (long double) mapping -- sizeof(long double) varies by
platform, struct format "d" is always 8 bytes (silent corruption)
Supersedes InsightSoftwareConsortium#6020, InsightSoftwareConsortium#6018, InsightSoftwareConsortium#5673, InsightSoftwareConsortium#5665. Key improvements over each:
- InsightSoftwareConsortium#6020: Fixed __buffer__ lifetime (memoryview didn't pin image),
removed unsafe __array_interface__, removed SIMULATE_PEP688
- InsightSoftwareConsortium#6018: Closed in favor of InsightSoftwareConsortium#6020
- InsightSoftwareConsortium#5673: Added __array_interface__ (now removed) and __buffer__
- InsightSoftwareConsortium#5665: Original PEP 688 implementation by blowekamp
Addresses review concerns from @thewtex (del image crash),
@blowekamp (reference pinning at buffer owner level).
Tested: 121 assertions across 3 test suites, Python 3.13 and 3.14,
with NumPy 2.4.3, PyTorch 2.11.0, Dask 2026.3.0. All 32 lifetime
tests pass (del image safe on every export path).
Co-Authored-By: Hans J. Johnson <hans-johnson@uiowa.edu>
hjmjohnson
added a commit
to hjmjohnson/ITK
that referenced
this pull request
Apr 9, 2026
Add zero-copy data export to all wrapped itk.Image types via two
protocols, ensuring the exported array remains valid even after
the source image is deleted:
image = itk.imread("brain.nii.gz")
arr = np.asarray(image)
del image
print(arr[1,1,1]) # safe -- no crash
Protocol dispatch by Python version:
3.12+: np.asarray -> __buffer__ (PEP 688, zero-copy, memoryview
pins self via NDArrayITKBase intermediary)
3.10-11: np.asarray -> __array__ -> array_view_from_image (zero-copy,
NDArrayITKBase.itk_base holds reference to image)
Changes to pyBase.i:
- Add __buffer__() implementing PEP 688 buffer export with shaped
memoryview. Uses NDArrayITKBase as intermediary to hold a Python
reference to the image, preventing GC while any derived
memoryview/array exists.
- Simplify __array__() to always return zero-copy view via
array_view_from_image(). Supports NumPy 2.0 copy= parameter.
copy=True returns a plain ndarray (not NDArrayITKBase) so the
image can be GCd immediately.
- Remove __array_interface__ (returned raw pointer with no reference
holder -- use-after-free on del image, confirmed by test).
- Remove SIMULATE_PEP688 / SIMULATE_PEP688_DEBUG (confusing,
contradictory behaviors between __buffer__ and __array__ paths).
Changes to PyBuffer.i.init:
- Add _get_buffer_formatstring() with module-level _BUFFER_FORMAT_MAP
- Add _get_numpy_pixelid() with module-level _NUMPY_PIXELID_MAP
- Remove LD (long double) mapping (silent corruption)
Test suites added (121 assertions):
itkImageTest.py (29): __buffer__, memoryview, np.asarray, __array__
itkImageInteropTest.py (60): NumPy, PyTorch, Dask clinical sizes
itkImageLifetimeTest.py (32): del image on every export path
Supersedes InsightSoftwareConsortium#6020, InsightSoftwareConsortium#6018, InsightSoftwareConsortium#5673, InsightSoftwareConsortium#5665.
Co-Authored-By: Hans J. Johnson <hans-johnson@uiowa.edu>
hjmjohnson
added a commit
to hjmjohnson/ITK
that referenced
this pull request
Apr 9, 2026
Add zero-copy data export to all wrapped itk.Image types via two
protocols, ensuring the exported array remains valid even after
the source image is deleted:
image = itk.imread("brain.nii.gz")
arr = np.asarray(image)
del image
print(arr[1,1,1]) # safe -- no crash
Protocol dispatch by Python version:
3.12+: np.asarray -> __buffer__ (PEP 688, zero-copy, memoryview
pins self via NDArrayITKBase intermediary)
3.10-11: np.asarray -> __array__ -> array_view_from_image (zero-copy,
NDArrayITKBase.itk_base holds reference to image)
Changes to pyBase.i:
- Add __buffer__() implementing PEP 688 buffer export with shaped
memoryview. Uses NDArrayITKBase as intermediary to hold a Python
reference to the image, preventing GC while any derived
memoryview/array exists.
- Simplify __array__() to always return zero-copy view via
array_view_from_image(). Supports NumPy 2.0 copy= parameter.
copy=True returns a plain ndarray (not NDArrayITKBase) so the
image can be GCd immediately.
- Remove __array_interface__ (returned raw pointer with no reference
holder -- use-after-free on del image, confirmed by test).
- Remove SIMULATE_PEP688 / SIMULATE_PEP688_DEBUG (confusing,
contradictory behaviors between __buffer__ and __array__ paths).
Changes to PyBuffer.i.init:
- Add _get_buffer_formatstring() with module-level _BUFFER_FORMAT_MAP
- Add _get_numpy_pixelid() with module-level _NUMPY_PIXELID_MAP
- Remove LD (long double) mapping (silent corruption)
Test suites (145 assertions total, 0 failures on Python 3.13 + 3.14):
itkImageTest.py (29): __buffer__, memoryview, np.asarray, __array__
itkImageInteropTest.py (60): NumPy, PyTorch 2.11, Dask 2026.3
itkImageLifetimeTest.py (56): lifetime safety on every export path,
weak-ref GC verification, refcount checks, RSS growth detection,
circular reference detection -- all inside a function scope with
explicit cleanup to verify no memory leaks
Supersedes InsightSoftwareConsortium#6020, InsightSoftwareConsortium#6018, InsightSoftwareConsortium#5673, InsightSoftwareConsortium#5665. Key improvements:
- InsightSoftwareConsortium#6020: Fixed __buffer__ lifetime, removed unsafe __array_interface__
- InsightSoftwareConsortium#5665/InsightSoftwareConsortium#5673: Original PEP 688 by blowekamp, __array_interface__
Addresses review from @thewtex (del image crash), @blowekamp (ref pinning).
Co-Authored-By: Hans J. Johnson <hans-johnson@uiowa.edu>
hjmjohnson
added a commit
to hjmjohnson/ITK
that referenced
this pull request
Apr 9, 2026
Add zero-copy data export to all wrapped itk.Image types via two
protocols, ensuring the exported array remains valid even after
the source image is deleted:
image = itk.imread("brain.nii.gz")
arr = np.asarray(image)
del image
print(arr[1,1,1]) # safe -- no crash
Protocol dispatch by Python version:
3.12+: np.asarray -> __buffer__ (PEP 688, zero-copy, memoryview
pins self via NDArrayITKBase intermediary)
3.10-11: np.asarray -> __array__ -> array_view_from_image (zero-copy,
NDArrayITKBase.itk_base holds reference to image)
Changes to pyBase.i:
- Add __buffer__() implementing PEP 688 buffer export with shaped
memoryview. Uses NDArrayITKBase as intermediary to hold a Python
reference to the image, preventing GC while any derived
memoryview/array exists.
- Simplify __array__() to always return zero-copy view via
array_view_from_image(). Supports NumPy 2.0 copy= parameter.
copy=True returns a plain ndarray (not NDArrayITKBase) so the
image can be GCd immediately.
- Remove __array_interface__ (returned raw pointer with no reference
holder -- use-after-free on del image, confirmed by test).
- Remove SIMULATE_PEP688 / SIMULATE_PEP688_DEBUG (confusing,
contradictory behaviors between __buffer__ and __array__ paths).
Changes to PyBuffer.i.init:
- Add _get_buffer_formatstring() with module-level _BUFFER_FORMAT_MAP
- Add _get_numpy_pixelid() with module-level _NUMPY_PIXELID_MAP
- Remove LD (long double) mapping (silent corruption)
Test suites added (121 assertions):
itkImageTest.py (29): __buffer__, memoryview, np.asarray, __array__
itkImageInteropTest.py (60): NumPy, PyTorch, Dask clinical sizes
itkImageLifetimeTest.py (32): del image on every export path
Supersedes InsightSoftwareConsortium#6020, InsightSoftwareConsortium#6018, InsightSoftwareConsortium#5673, InsightSoftwareConsortium#5665.
Co-Authored-By: Hans J. Johnson <hans-johnson@uiowa.edu>
hjmjohnson
added a commit
to hjmjohnson/ITK
that referenced
this pull request
Apr 9, 2026
Add zero-copy data export to all wrapped itk.Image types via two
protocols, ensuring the exported array remains valid even after
the source image is deleted:
image = itk.imread("brain.nii.gz")
arr = np.asarray(image)
del image
print(arr[1,1,1]) # safe -- no crash
Protocol dispatch by Python version:
3.12+: np.asarray -> __buffer__ (PEP 688, zero-copy, memoryview
pins self via NDArrayITKBase intermediary)
3.10-11: np.asarray -> __array__ -> array_view_from_image (zero-copy,
NDArrayITKBase.itk_base holds reference to image)
Changes to pyBase.i:
- Add __buffer__() implementing PEP 688 buffer export with shaped
memoryview. Uses NDArrayITKBase as intermediary to hold a Python
reference to the image, preventing GC while any derived
memoryview/array exists.
- Simplify __array__() to always return zero-copy view via
array_view_from_image(). Supports NumPy 2.0 copy= parameter.
copy=True returns a plain ndarray (not NDArrayITKBase) so the
image can be GCd immediately.
- Remove __array_interface__ (returned raw pointer with no reference
holder -- use-after-free on del image, confirmed by test).
- Remove SIMULATE_PEP688 / SIMULATE_PEP688_DEBUG (confusing,
contradictory behaviors between __buffer__ and __array__ paths).
Changes to PyBuffer.i.init:
- Add _get_buffer_formatstring() with module-level _BUFFER_FORMAT_MAP
- Add _get_numpy_pixelid() with module-level _NUMPY_PIXELID_MAP
- Remove LD (long double) mapping (silent corruption)
Test suites added (121 assertions):
itkImageTest.py (29): __buffer__, memoryview, np.asarray, __array__
itkImageInteropTest.py (60): NumPy, PyTorch, Dask clinical sizes
itkImageLifetimeTest.py (32): del image on every export path
Supersedes InsightSoftwareConsortium#6020, InsightSoftwareConsortium#6018, InsightSoftwareConsortium#5673, InsightSoftwareConsortium#5665.
Co-Authored-By: Hans J. Johnson <hans-johnson@uiowa.edu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sharing code I wrote to play around with PEP 688 in ITK Python.