Add support for Python 3.11#7851
Open
nblomerus wants to merge 1 commit into
Open
Conversation
Closes tensorflow#7743. == Packaging == - Bump requires-python upper bound from <3.11 to <3.12 in pyproject.toml, package_build/tfx/pyproject.toml, and package_build/ml-pipelines-sdk/pyproject.toml. - Add Programming Language :: Python :: 3.11 classifier in each. == CI matrix == - Add '3.11' to ci-test.yml matrix and 'cp311' to wheels.yml matrix. - Disable strategy.fail-fast so individual job failures don't cancel the whole matrix (useful regardless of this change). == Dependency pins (test_constraints.txt + nightly_test_constraints.txt) == TFX 1.17 cp311 wheels require newer versions of several deps than the cp39/cp310 wheels. Use PEP 508 environment markers to pin per-Python-version: - apache-beam: 2.50.0 on <3.11, 2.53.0 on >=3.11 (tfx-bsl/tfdv cp311 wheels require beam >=2.53) - protobuf: 4.21.12 on <3.11, 4.25.5 on >=3.11 (tfx-bsl/tfdv/ml-metadata cp311 wheels require protobuf >=4.25.2; beam 2.53 requires !=4.21.*) - pyarrow: pin to 10.0.1 on >=3.11 (10.0.0 has no cp311 wheel and fails to build from sdist under setuptools 70) - tensorflow-io-gcs-filesystem: 0.24.0 on <3.11, 0.37.1 on >=3.11 (0.24.0 has no cp311 wheel) - grpcio-status: 1.62.3 (was 1.48.2); google-api-core requires grpcio-status>=1.49.1 on >=3.11, while grpcio-status 1.63+ would require protobuf 5 which conflicts with TFX's protobuf<5 == Test fixes == Two test files had patterns that broke on Python 3.11: - tfx/components/infra_validator/request_builder_test.py: mock.patch.object(..., wraps=Cls) returns _SentinelObject for .return_value before the wrapped class is called. Switch to autospec=True. autospec is strict about kwargs so pass examples= by keyword. - tfx/components/trainer/rewriting/tflite_rewriter_test.py: mock.patch('tensorflow.lite.TFLiteConverter.from_saved_model') fails on 3.11 because mock's string-path resolution doesn't trigger TF's lazy tf.lite module wrapper. Switch to mock.patch.object( tf.lite.TFLiteConverter, ...) for direct object reference. All 12 CI jobs (3 Python versions x not-e2e/e2e x DEFAULT/NIGHTLY) pass on this branch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Python 3.11 to TFX's supported and tested matrix. Closes #7743.
All 12 CI jobs pass on the contributor's fork (3 Python versions × not-e2e/e2e × DEFAULT/NIGHTLY dependency selectors): https://github.com/nblomerus/tfx/actions/runs/26000991135
Changes
Packaging
requires-pythonupper bound bumped from<3.11to<3.12in the three pyproject.toml files (top-level +package_build/tfx+package_build/ml-pipelines-sdk).Programming Language :: Python :: 3.11classifier added in each.CI matrix
'3.11'added toci-test.yml;'cp311'added towheels.yml.strategy.fail-fast: falseadded so individual job failures don't cancel the rest of the matrix, and was essential for debugging this PR. Happy to split this into a separate PR if preferred.Dependency pins (test_constraints.txt + nightly_test_constraints.txt)
The TFX 1.17 cp311 wheels require newer versions of several deps than the cp39/cp310 wheels, so this uses PEP 508 environment markers to pin per-Python-version. Each
==pin is conditional on Python version where the existing pin would conflict with the cp311 wheels' ownRequires-Dist:<3.11>=3.11apache-beam2.50.02.53.0>=2.53protobuf4.21.124.25.5>=4.25.2; beam 2.53 requires!=4.21.*pyarrow10.0.1tensorflow-io-gcs-filesystem0.24.00.37.1Plus one unconditional bump:
grpcio-status1.48.2→1.62.3.google-api-corerequiresgrpcio-status>=1.49.1on Python 3.11+;grpcio-status1.63+ would require protobuf 5, conflicting with TFX'sprotobuf<5. 1.62.3 is the last release using protobuf 4.Test fixes
Two test files had patterns that broke on Python 3.11:
tfx/components/infra_validator/request_builder_test.py—mock.patch.object(..., wraps=Cls)returns a_SentinelObjectfor.return_valueuntil the wrapped class is called (Python 3.11 mock behaviour change). Switched toautospec=True.autospecis strict about kwargs so theexamples=arg is passed by keyword.tfx/components/trainer/rewriting/tflite_rewriter_test.py—mock.patch('tensorflow.lite.TFLiteConverter.from_saved_model')fails on 3.11 because mock's string-path resolution doesn't trigger TF's lazytf.litemodule wrapper. Switched tomock.patch.object(tf.lite.TFLiteConverter, ...), which references the object directly. This keeps TFLite functional — a deliberately different approach than Adding python 3.11 support and Removing TF-Lite Dependency #7764, which gutted TFLite to side-step the issue.Scope choices
==pins to ranges (>=) and letting pip's resolver pick. But the current constraints file is a frozen snapshot for reproducibility, and the maintainers can tell me if they'd prefer the range approach.Test plan
request_builder_test.py::testBuildRequests_TFServing,::testBuildRequests_DefaultArgument,tflite_rewriter_test.py::testInvokeTFLiteRewriterQuantizationFullIntegerFailsNoData,::testInvokeTFLiteRewriterWithSignatureKey) passes on 3.9, 3.10, 3.11.