Skip to content

Add support for Python 3.11#7851

Open
nblomerus wants to merge 1 commit into
tensorflow:masterfrom
nblomerus:py311-support
Open

Add support for Python 3.11#7851
nblomerus wants to merge 1 commit into
tensorflow:masterfrom
nblomerus:py311-support

Conversation

@nblomerus
Copy link
Copy Markdown

@nblomerus nblomerus commented May 18, 2026

Summary

Adds Python 3.11 to TFX's supported and tested matrix. Closes #7743.

All 12 CI jobs pass on the contributor's fork (3 Python versions × not-e2e/e2e × DEFAULT/NIGHTLY dependency selectors): https://github.com/nblomerus/tfx/actions/runs/26000991135

Changes

Packaging

  • requires-python upper bound bumped from <3.11 to <3.12 in the three pyproject.toml files (top-level + package_build/tfx + package_build/ml-pipelines-sdk).
  • Programming Language :: Python :: 3.11 classifier added in each.

CI matrix

  • '3.11' added to ci-test.yml; 'cp311' added to wheels.yml.
  • strategy.fail-fast: false added so individual job failures don't cancel the rest of the matrix, and was essential for debugging this PR. Happy to split this into a separate PR if preferred.

Dependency pins (test_constraints.txt + nightly_test_constraints.txt)

The TFX 1.17 cp311 wheels require newer versions of several deps than the cp39/cp310 wheels, so this uses PEP 508 environment markers to pin per-Python-version. Each == pin is conditional on Python version where the existing pin would conflict with the cp311 wheels' own Requires-Dist:

Package <3.11 >=3.11 Why
apache-beam 2.50.0 2.53.0 tfx-bsl/tfdv cp311 wheels require beam >=2.53
protobuf 4.21.12 4.25.5 tfx-bsl/tfdv/ml-metadata cp311 wheels require >=4.25.2; beam 2.53 requires !=4.21.*
pyarrow (unconstrained) 10.0.1 10.0.0 has no cp311 wheel and fails sdist build under setuptools 70
tensorflow-io-gcs-filesystem 0.24.0 0.37.1 0.24.0 has no cp311 wheel

Plus one unconditional bump: grpcio-status 1.48.21.62.3. google-api-core requires grpcio-status>=1.49.1 on Python 3.11+; grpcio-status 1.63+ would require protobuf 5, conflicting with TFX's protobuf<5. 1.62.3 is the last release using protobuf 4.

Test fixes

Two test files had patterns that broke on Python 3.11:

  • tfx/components/infra_validator/request_builder_test.pymock.patch.object(..., wraps=Cls) returns a _SentinelObject for .return_value until the wrapped class is called (Python 3.11 mock behaviour change). Switched to autospec=True. autospec is strict about kwargs so the examples= arg is passed by keyword.
  • tfx/components/trainer/rewriting/tflite_rewriter_test.pymock.patch('tensorflow.lite.TFLiteConverter.from_saved_model') fails on 3.11 because mock's string-path resolution doesn't trigger TF's lazy tf.lite module wrapper. Switched to mock.patch.object(tf.lite.TFLiteConverter, ...), which references the object directly. This keeps TFLite functional — a deliberately different approach than Adding python 3.11 support and Removing TF-Lite Dependency #7764, which gutted TFLite to side-step the issue.

Scope choices

  • 3.11 only, not 3.12. The TFX-ecosystem cp312 wheels (tfdv, tfx-bsl, ml-metadata) aren't published on PyPI or the nightly index yet, so 3.12 support is blocked on Google releasing those. build: enable Python 3.12 support and relax core dependency bounds #7821 covers the broader 3.12 work (via local wheel builds).
  • Conditional pins, not range widening. I considered loosening == pins to ranges (>=) and letting pip's resolver pick. But the current constraints file is a frozen snapshot for reproducibility, and the maintainers can tell me if they'd prefer the range approach.

Test plan

  • All 12 CI jobs green on contributor fork.
  • Each modified test (request_builder_test.py::testBuildRequests_TFServing, ::testBuildRequests_DefaultArgument, tflite_rewriter_test.py::testInvokeTFLiteRewriterQuantizationFullIntegerFailsNoData, ::testInvokeTFLiteRewriterWithSignatureKey) passes on 3.9, 3.10, 3.11.
  • No tests skipped or xfailed as a result of this change.

Closes tensorflow#7743.

== Packaging ==
- Bump requires-python upper bound from <3.11 to <3.12 in
  pyproject.toml, package_build/tfx/pyproject.toml, and
  package_build/ml-pipelines-sdk/pyproject.toml.
- Add Programming Language :: Python :: 3.11 classifier in each.

== CI matrix ==
- Add '3.11' to ci-test.yml matrix and 'cp311' to wheels.yml matrix.
- Disable strategy.fail-fast so individual job failures don't cancel
  the whole matrix (useful regardless of this change).

== Dependency pins (test_constraints.txt + nightly_test_constraints.txt) ==
TFX 1.17 cp311 wheels require newer versions of several deps than
the cp39/cp310 wheels. Use PEP 508 environment markers to pin
per-Python-version:

- apache-beam: 2.50.0 on <3.11, 2.53.0 on >=3.11 (tfx-bsl/tfdv cp311
  wheels require beam >=2.53)
- protobuf: 4.21.12 on <3.11, 4.25.5 on >=3.11 (tfx-bsl/tfdv/ml-metadata
  cp311 wheels require protobuf >=4.25.2; beam 2.53 requires !=4.21.*)
- pyarrow: pin to 10.0.1 on >=3.11 (10.0.0 has no cp311 wheel and
  fails to build from sdist under setuptools 70)
- tensorflow-io-gcs-filesystem: 0.24.0 on <3.11, 0.37.1 on >=3.11
  (0.24.0 has no cp311 wheel)
- grpcio-status: 1.62.3 (was 1.48.2); google-api-core requires
  grpcio-status>=1.49.1 on >=3.11, while grpcio-status 1.63+ would
  require protobuf 5 which conflicts with TFX's protobuf<5

== Test fixes ==
Two test files had patterns that broke on Python 3.11:

- tfx/components/infra_validator/request_builder_test.py:
  mock.patch.object(..., wraps=Cls) returns _SentinelObject for
  .return_value before the wrapped class is called. Switch to
  autospec=True. autospec is strict about kwargs so pass examples=
  by keyword.

- tfx/components/trainer/rewriting/tflite_rewriter_test.py:
  mock.patch('tensorflow.lite.TFLiteConverter.from_saved_model')
  fails on 3.11 because mock's string-path resolution doesn't trigger
  TF's lazy tf.lite module wrapper. Switch to mock.patch.object(
  tf.lite.TFLiteConverter, ...) for direct object reference.

All 12 CI jobs (3 Python versions x not-e2e/e2e x DEFAULT/NIGHTLY)
pass on this branch.
@keerthanakadiri keerthanakadiri self-assigned this May 18, 2026
@keerthanakadiri keerthanakadiri requested review from aktech and nikelite and removed request for aktech May 18, 2026 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support Python 3.11

2 participants