Upgrade Struct2Tensor to TensorFlow 2.21.0#59
Conversation
| - h5py==3.14.0 | ||
| - idna==3.11 | ||
| - keras==3.13.2 | ||
| - keras>=3.0.0 |
There was a problem hiding this comment.
Why is this lowered? The vulnerabilities ask for this to be 3.12.1+.
There was a problem hiding this comment.
Keras is lowered because 3.13.2 dosn't support python 3.10 version, since we have python 3.10 workflows, Instead of removing it, I loosen the dependency to satisfy the requirement.
| 'protobuf>=4.25.2,<6.0.0;python_version>="3.11"', | ||
| 'protobuf>=4.21.6,<6.0.0;python_version<"3.11"', | ||
| 'tensorflow>=2.17,<2.18', | ||
| 'protobuf>=4.25.2,<7.0.0', |
There was a problem hiding this comment.
In environment.yml you set protobuf to 6.31. Why isn't this set to that level?
There was a problem hiding this comment.
I've updated the protobuf version range to >=6.0.0,<7.0.0 in setup.py. Since 6.31.1 falls within this range, it should satisfy the requirement. As we generally avoid pinning exact versions in setup.py, I believe maintaining version ranges is better. If we pinned an exact version, anyone building S2T who requires a different protobuf version than 6.31.1 might encounter issues.
| - python>=3.10,<3.13 | ||
|
|
||
| - pip | ||
| - pip: |
There was a problem hiding this comment.
should this file include explicit specifications of component versions needed to address vulnerabilities? I don't see components like cryptography, jaraco-context, ipython, pyopenssl.
Also, I see some that have values that don't meet the "fix" level for the vulnerabilities
- requests needs to be 2.33 (not 2.32.5)
- pygments needs to be at 2.20 (not 2.19.2)
There was a problem hiding this comment.
I've also updated the below dependencies to fix vulnerabilities, meanwhile these are transitive dependency's [cryptography, jaraco-context, ipython, pyopenssl.] that might be coming from diff components, I think we can't include them here..!
- requests to 2.33
- pygments to 2.20
…ore valid tfmd.patch spaces
…mmodate pre-commit hook
…5 wheel stamping, and Python 3.13 support
…rt drop to Deprecations section
2 participants
This PR upgrades
struct2tensorto support TensorFlow 2.21.0 and resolves various build system incompatibilities, dependency conflicts, and compilation errors encountered during the process.Key Changes
1. TensorFlow 2.21.0 Dependency Resolution Conflicts
.bazelversionto7.7.0to align with TensorFlow 2.21.0 requirements.rules_java,local_config_cuda,local_config_tensorrt,llvm-raw, etc.).struct2tensor.WORKSPACEfile to satisfy these unresolved references without adding unnecessary overhead. We also applied custom patches to TensorFlow’s source files to remove Java, CUDA, ROCm, and TensorRT dependencies that were causing failures.2. Python 3.12, 3.13 Support and Python 3.9 Drop
setup.pyandenvironment.ymlto declare support for Python 3.12, 3.13 dropped Python 3.9, and pinned compatible versions for critical libraries likepyarrow(>=14),h5py, andml-dtypes.3. Protobuf Version Mismatch (Compiler vs. Headers)
field_mask.pb.h) were generated by a newer version ofprotocthan the Protocol Buffer headers being included in compilation.WORKSPACE, causing it to fetch mismatched Protobuf compilers and headers..bazelrcusingcommon --noenable_bzlmodand moved theregister_toolchainscall to the very top of theWORKSPACEfile to enforce correct resolution priority. We also applied a specialized patch (protobuf_tensorflow.patch) to fix macro and namespace issues.4. Test Sharding Validation Errors
test --noincompatible_check_sharding_supportto.bazelrcto allow the tests to skip this check and execute successfully.5. CI Assembler Error (
--gsframe)as: option '--gsframe' doesn't allow an argument.--gsframe=nowas added to fixed local build issues on GCC 15, but the CI runner uses an older assembler that does not support this flag.--config=sframe_fixin.bazelrc. This allows default builds (including CI) to run without it, while local users with GCC 15 can usebazel build --config=sframe_fixto avoid linker errors.6. Added CI Pre-Commit workflow
pre-commitworkflow that check lint Issue on each PR workflow run to main branch7. Simplified Dependencies in setup.py
protobufbased on Python version were present insetup.py.protobuf>=4.25.2,<7.0.0which satisfies Python 3.10, 3.11, and 3.12.8. Updated CI Workflow for Multi-Version Testing
Python version (3.12)and fixedKeras version (3.13.2)which conflicted with matrix versions or Python 3.10 compatibility.python-version matrix ["3.10", "3.11", "3.12"]to build.yml. Updated environment.ymlto use a
Python version range (>=3.10,<3.13)and relaxedKerasconstraint tokeras>=3.0.09. Docker TF-Serving Build Infrastructure Refactoring
rules_appleandrules_ccunder Bazel 7.3.9to3.13.rules_apple-3.13.0,rules_swift-1.18.0, andapple_support-1.15.1—directly into theWORKSPACEduring container build to resolve platform translation load failures.docker-serving-buildin .github/workflows/build.yml to build, compile, and verify the serving image on every pull request and push.10. Modern Release Wheel Compilation (
manylinux_2_28&manylinux_2_35)manylinux2014(glibc 2.17) environment, which is incompatible with modern high-performance runtime standards liketfx-bsl.Dockerfile.manylinux_2_28utilizing devtoolsets (/dt10/usr/bin/gcc) and upgraded nativeauditwheelto6.6.0.AUDITWHEEL_PLAT_TAG).auditwheel repairtargetingmanylinux_2_35due to modern symbol requirements inside the compiled C++ operator files (e.g.,_parquet_dataset_op.so), producingstruct2tensor-0.49.0.dev0-cp312-cp312-manylinux_2_35_x86_64.whl.manylinux_2_28services natively to docker-compose.yml.Verification (Additional Results)
struct2tensor-serving:latest._parquet_dataset_op.soand_decode_proto_map_op.soC++ operator assets.Verification
All tests passed successfully locally:
bazel test //... --noincompatible_check_sharding_support ...