Skip to content

Commit 169c8bf

Browse files
vkarampudivkarampudi
authored
Upgrade Struct2Tensor to TensorFlow 2.21.0 (#59)
* Upgrade to TensorFlow 2.21.0 and fix build issues * Remove local absolute paths from .bazelrc * Disable Bzlmod, upgrade Bazel to 7.7.0, and patch Protobuf to fix version mismatch. * Remove --gsframe=no from .bazelrc to fix PR build failure. * Remove Bazel version overwrite from configure.sh * Update environment.yml for TF 2.21.0 * Downgrade h5py in environment.yml to fix conflict * Upgrade ml-dtypes in environment.yml to fix conflict * Pin pyarrow to 23.0.1 in environment.yml * Update RELEASE.md for version 0.49.0 * Revert header in RELEASE.md to Current Version * Add Python 3.12 support and drop Python 3.9 * Add Python 3.12 support to RELEASE.md and disable SFrame in .bazelrc * Remove local environment lines from .bazelrc * Add pre-commit workflow * Apply pre-commit auto-fixes * Fix Protobuf version mismatch by disabling Bzlmod and moving toolchain registration * Fix trailing whitespace and end-of-file issues identified by pre-commit * Delete unused protobuf_retain_options.patch * Move --gsframe=no to a separate sframe_fix config in .bazelrc to fix CI failure * Simplify protobuf dependency in setup.py to a single line * Add python-version matrix to build.yml to test on 3.10, 3.11, 3.12 * Fix YAML indentation in build.yml * Remove fixed python version from environment.yml in CI before setup-miniconda * Use python version range in environment.yml and remove CI removal step * Relax Keras constraint in environment.yml to >=3.0.0 * Port Protobuf constraint update and vulnerability fixes to testing1 * Remove empty lines in setup.py * Add Python 3.13 support to CI and setup.py * Unpin numpy and tensorflow in environment.yml * Unpin tensorflow-io-gcs-filesystem in environment.yml * Remove tensorflow-io-gcs-filesystem from environment.yml * Update TFMD dependency to latest master commit * Configure setup-miniconda to use Mamba to resolve 403 errors * Pin Bazel version to 7.7.0 globally in CI and update WORKSPACE check * Remove TFMD patch application to test unpatched compatibility * Apply reconstructed TFMD patch to resolve proto build issues * Fix trailing whitespace in TFMD patch to satisfy pre-commit * Exclude patch files from pre-commit trailing whitespace hook and restore valid tfmd.patch spaces * Simplify TFMD patch to avoid Starlark patch apply errors * Apply programmatically generated perfect TFMD patch * Update Dockerfile to TF-Serving r2.19, Bazel 7.7.0, and Python 3.13 * Add automated docker-serving-build CI job and refine Bazel 7 serving overrides * Style: Fix extra trailing empty newline in GHA Build workflow to accommodate pre-commit hook * Add support for modern manylinux_2_28 container builds, manylinux_2_35 wheel stamping, and Python 3.13 support * RELEASE.md: Combine Python support versions and move Python 3.9 support drop to Deprecations section * docker-compose.yml: Trim trailing whitespace at end of file to satisfy pre-commit
1 parent 53fcdd7 commit 169c8bf

139 files changed

Lines changed: 830 additions & 658 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.bazelrc‎

Lines changed: 43 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,15 +35,54 @@ build:linux --copt=-w
3535
build:macos --copt=-w
3636

3737
# By default, build TF in C++ 17 mode.
38-
build:linux --cxxopt=-std=c++17
38+
build --cxxopt=-std=c++17
39+
build --host_cxxopt=-std=c++17
3940
build:linux --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=1
40-
build:linux --host_cxxopt=-std=c++17
41-
build:macos --cxxopt=-std=c++17
4241
build:macos --cxxopt=-D_GLIBCXX_USE_CXX11_ABI=1
43-
build:macos --host_cxxopt=-std=c++17
4442

4543

4644
# Suppress all warning messages.
45+
4746
build:short_logs --output_filter=DONT_MATCH_ANYTHING
4847

48+
# Config to disable sframe in assembler to avoid linker errors with gcc 15
49+
build:sframe_fix --copt=-Wa,--gsframe=no
50+
4951
build:macos --macos_minimum_os=10.12
52+
53+
common --noenable_bzlmod
54+
55+
test --noincompatible_check_sharding_support
56+
build --action_env TF_HEADER_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow/include"
57+
build --action_env TF_SHARED_LIBRARY_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow"
58+
build --action_env TF_SHARED_LIBRARY_NAME="libtensorflow_framework.so.2"
59+
build --action_env TF_HEADER_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow/include"
60+
build --action_env TF_SHARED_LIBRARY_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow"
61+
build --action_env TF_SHARED_LIBRARY_NAME="libtensorflow_framework.so.2"
62+
build --action_env TF_HEADER_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow/include"
63+
build --action_env TF_SHARED_LIBRARY_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow"
64+
build --action_env TF_SHARED_LIBRARY_NAME="libtensorflow_framework.so.2"
65+
build --action_env TF_HEADER_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow/include"
66+
build --action_env TF_SHARED_LIBRARY_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow"
67+
build --action_env TF_SHARED_LIBRARY_NAME="libtensorflow_framework.so.2"
68+
build --action_env TF_HEADER_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow/include"
69+
build --action_env TF_SHARED_LIBRARY_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow"
70+
build --action_env TF_SHARED_LIBRARY_NAME="libtensorflow_framework.so.2"
71+
build --action_env TF_HEADER_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow/include"
72+
build --action_env TF_SHARED_LIBRARY_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow"
73+
build --action_env TF_SHARED_LIBRARY_NAME="libtensorflow_framework.so.2"
74+
build --action_env TF_HEADER_DIR="/usr/local/google/home/vkarampudi/anaconda3/lib/python3.12/site-packages/tensorflow/include"
75+
build --action_env TF_SHARED_LIBRARY_DIR="/usr/local/google/home/vkarampudi/anaconda3/lib/python3.12/site-packages/tensorflow"
76+
build --action_env TF_SHARED_LIBRARY_NAME="libtensorflow_framework.so.2"
77+
build --action_env TF_HEADER_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow/include"
78+
build --action_env TF_SHARED_LIBRARY_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow"
79+
build --action_env TF_SHARED_LIBRARY_NAME="libtensorflow_framework.so.2"
80+
build --action_env TF_HEADER_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow/include"
81+
build --action_env TF_SHARED_LIBRARY_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow"
82+
build --action_env TF_SHARED_LIBRARY_NAME="libtensorflow_framework.so.2"
83+
build --action_env TF_HEADER_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow/include"
84+
build --action_env TF_SHARED_LIBRARY_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow"
85+
build --action_env TF_SHARED_LIBRARY_NAME="libtensorflow_framework.so.2"
86+
build --action_env TF_HEADER_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow/include"
87+
build --action_env TF_SHARED_LIBRARY_DIR="/usr/local/lib/python3.12/dist-packages/tensorflow"
88+
build --action_env TF_SHARED_LIBRARY_NAME="libtensorflow_framework.so.2"

‎.bazelversion‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
7.7.0

‎.github/workflows/build.yml‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,17 @@ on:
66
pull_request:
77
branches: [ master ]
88

9+
env:
10+
USE_BAZEL_VERSION: 7.7.0
11+
912
jobs:
1013
build:
11-
name: Build (${{ matrix.os }})
14+
name: Build (${{ matrix.os }}, Python ${{ matrix.python-version }})
1215
runs-on: ${{ matrix.os }}
1316
strategy:
1417
matrix:
1518
os: [ubuntu-latest, macos-latest]
19+
python-version: ["3.10", "3.11", "3.12", "3.13"]
1620

1721
steps:
1822
- uses: actions/checkout@v4
@@ -23,6 +27,8 @@ jobs:
2327
auto-activate-base: false
2428
activate-environment: s2t-env
2529
environment-file: environment.yml
30+
python-version: ${{ matrix.python-version }}
31+
use-mamba: true
2632

2733
- name: Install Bazel
2834
shell: bash -l {0}
@@ -55,3 +61,13 @@ jobs:
5561
if: runner.os == 'Linux'
5662
shell: bash -l {0}
5763
run: bazel build //...
64+
65+
docker-serving-build:
66+
name: Build Docker Serving Image
67+
runs-on: ubuntu-latest
68+
steps:
69+
- uses: actions/checkout@v4
70+
- name: Build TF-Serving Docker Image
71+
run: |
72+
cd struct2tensor/tools/tf_serving_docker
73+
docker build -t struct2tensor-serving:latest .

‎.github/workflows/pre-commit.yml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
name: pre-commit
2+
3+
on:
4+
pull_request:
5+
push:
6+
branches: [master]
7+
8+
jobs:
9+
pre-commit:
10+
runs-on: ubuntu-latest
11+
steps:
12+
- uses: actions/checkout@v4.1.7
13+
with:
14+
# Ensure the full history is fetched
15+
# This is required to run pre-commit on a specific set of commits
16+
# TODO: Remove this when all the pre-commit issues are fixed
17+
fetch-depth: 0
18+
- uses: actions/setup-python@v5.1.1
19+
with:
20+
python-version: 3.13
21+
- uses: pre-commit/action@v3.0.1

‎.pre-commit-config.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
repos:
2+
- repo: https://github.com/pre-commit/pre-commit-hooks
3+
rev: v4.6.0
4+
hooks:
5+
- id: trailing-whitespace
6+
exclude: \.patch$
7+
- id: end-of-file-fixer
8+
- id: check-yaml
9+
- id: check-added-large-files

‎LICENSE‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -198,4 +198,4 @@
198198
distributed under the License is distributed on an "AS IS" BASIS,
199199
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200200
See the License for the specific language governing permissions and
201-
limitations under the License.
201+
limitations under the License.

‎RELEASE.md‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,12 +4,31 @@
44

55
## Major Features and Improvements
66

7+
* N/A
8+
79
## Bug Fixes and Other Changes
810

11+
* Depends on `tensorflow 2.21.0`.
12+
* Depends on `protobuf==6.31.1`.
13+
* Depends on `pyarrow==23.0.1`.
14+
* Enforced C++17 in `.bazelrc`.
15+
* Disabled Bzlmod in `.bazelrc` to resolve protobuf conflicts.
16+
* Added dummy repositories in `WORKSPACE` to bypass circular dependencies with TensorFlow.
17+
* Fixed missing `#include <cstdint>` in various files to support compilation with `gcc 15`.
18+
* Added support for Python 3.12 and 3.13.
19+
* Upgraded Bazel global pin to `7.7.0`.
20+
* Upgraded release build container configurations from legacy `manylinux2014` to modern `manylinux_2_28`.
21+
* Stamps dynamic release wheels to match `manylinux_2_35` dynamic system dependencies.
22+
923
## Breaking Changes
1024

25+
* N/A
26+
1127
## Deprecations
1228

29+
* Dropped support for Python 3.9.
30+
31+
1332
# Version 0.48.1
1433

1534
## Major Features and Improvements

‎WORKSPACE‎

Lines changed: 130 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,58 @@
1616

1717
workspace(name = "struct2tensor")
1818

19+
local_repository(
20+
name = "python_version_repo",
21+
path = "third_party/python_version_repo",
22+
)
23+
24+
local_repository(
25+
name = "python_3_11_host",
26+
path = "third_party/python_3_11_host",
27+
)
28+
29+
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
30+
load("@bazel_tools//tools/build_defs/repo:utils.bzl", "maybe")
31+
32+
local_repository(
33+
name = "rules_java",
34+
path = "third_party/rules_java",
35+
)
36+
37+
local_repository(
38+
name = "local_config_cuda",
39+
path = "third_party/local_config_cuda",
40+
)
41+
42+
local_repository(
43+
name = "local_config_tensorrt",
44+
path = "third_party/local_config_tensorrt",
45+
)
46+
47+
local_repository(
48+
name = "local_config_rocm",
49+
path = "third_party/local_config_rocm",
50+
)
51+
52+
local_repository(
53+
name = "local_config_sycl",
54+
path = "third_party/local_config_sycl",
55+
)
56+
57+
local_repository(
58+
name = "tf_wheel_version_suffix",
59+
path = "third_party/tf_wheel_version_suffix",
60+
)
61+
62+
maybe(
63+
http_archive,
64+
name = "platforms",
65+
urls = [
66+
"https://github.com/bazelbuild/platforms/releases/download/0.0.11/platforms-0.0.11.tar.gz",
67+
],
68+
sha256 = "29742e87275809b5e598dc2f04d86960cc7a55b3067d97221c9abbc9926bff0f",
69+
)
70+
1971
load("//tf:tf_configure.bzl", "tf_configure")
2072

2173
load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")
@@ -26,23 +78,53 @@ tf_configure(name = "local_config_tf")
2678

2779
#####################################################################################
2880

81+
82+
83+
# ===== Abseil dependency =====
2984
http_archive(
30-
name = "zlib",
31-
build_file = "@com_google_protobuf//:third_party/zlib.BUILD",
32-
sha256 = "17e88863f3600672ab49182f217281b6fc4d3c762bde361935e436a95214d05c",
33-
strip_prefix = "zlib-1.3.1",
34-
urls = ["https://github.com/madler/zlib/archive/v1.3.1.tar.gz"],
85+
name = "com_google_absl",
86+
sha256 = "d1abe9da2003e6cbbd7619b0ced3e52047422f4f4ac6c66a9bef5d2e99fea837",
87+
strip_prefix = "abseil-cpp-d38452e1ee03523a208362186fd42248ff2609f6",
88+
urls = [
89+
"https://github.com/abseil/abseil-cpp/archive/d38452e1ee03523a208362186fd42248ff2609f6.tar.gz",
90+
],
91+
patches = [
92+
"//third_party:abseil_visibility.patch",
93+
],
94+
patch_args = ["-p0"],
95+
)
96+
97+
http_archive(
98+
name = "abseil-cpp",
99+
sha256 = "d1abe9da2003e6cbbd7619b0ced3e52047422f4f4ac6c66a9bef5d2e99fea837",
100+
strip_prefix = "abseil-cpp-d38452e1ee03523a208362186fd42248ff2609f6",
101+
urls = [
102+
"https://github.com/abseil/abseil-cpp/archive/d38452e1ee03523a208362186fd42248ff2609f6.tar.gz",
103+
],
104+
patches = [
105+
"//third_party:abseil_visibility.patch",
106+
],
107+
patch_args = ["-p0"],
35108
)
36109

37-
# ===== Protobuf 4.25.6 dependency =====
110+
111+
112+
113+
# ===== Protobuf 6.31.1 dependency =====
38114
# Must be declared BEFORE TensorFlow's workspaces to override the version they pull
39115
http_archive(
40116
name = "com_google_protobuf",
41-
sha256 = "4e6727bc5d23177edefa3ad86fd2f5a92cd324151636212fd1f7f13aef3fd2b7",
42-
strip_prefix = "protobuf-4.25.6",
117+
sha256 = "6e09bbc950ba60c3a7b30280210cd285af8d7d8ed5e0a6ed101c72aff22e8d88",
118+
strip_prefix = "protobuf-6.31.1",
43119
urls = [
44-
"https://github.com/protocolbuffers/protobuf/archive/v4.25.6.tar.gz",
120+
"https://github.com/protocolbuffers/protobuf/archive/refs/tags/v6.31.1.zip",
45121
],
122+
patches = ["//third_party:protobuf_tensorflow.patch"],
123+
patch_args = ["-p1"],
124+
)
125+
126+
register_toolchains(
127+
"@com_google_protobuf//bazel/private/toolchains:cc_source_toolchain_bazel7",
46128
)
47129

48130
# ===== TensorFlow dependency =====
@@ -60,23 +142,55 @@ http_archive(
60142
# 3. Request the new archive to be mirrored on mirror.bazel.build for more
61143
# reliable downloads.
62144

63-
_TENSORFLOW_GIT_COMMIT = "3c92ac03cab816044f7b18a86eb86aa01a294d95" # tf 2.17.1
64-
_TENSORFLOW_ARCHIVE_SHA256 = "317dd95c4830a408b14f3e802698eb68d70d81c7c7cfcd3d28b0ba023fe84a68"
145+
_TENSORFLOW_GIT_COMMIT = "2.21.0" # tf 2.21.0
146+
_TENSORFLOW_ARCHIVE_SHA256 = "ef3568bb4865d6c1b2564fb5689c19b6b9a5311572cd1f2ff9198636a8520921"
65147

66148
http_archive(
67149
name = "org_tensorflow",
68150
sha256 = _TENSORFLOW_ARCHIVE_SHA256,
69151
urls = [
70-
"https://github.com/tensorflow/tensorflow/archive/%s.tar.gz" % _TENSORFLOW_GIT_COMMIT,
152+
"https://github.com/tensorflow/tensorflow/archive/v%s.tar.gz" % _TENSORFLOW_GIT_COMMIT,
71153
],
72154
strip_prefix = "tensorflow-%s" % _TENSORFLOW_GIT_COMMIT,
73155
patches = ["//third_party:tensorflow.patch"],
74156
patch_args = ["-p1"],
157+
repo_mapping = {
158+
"@abseil-cpp": "@com_google_absl",
159+
},
160+
)
161+
162+
http_archive(
163+
name = "llvm-raw",
164+
sha256 = "3f986184ee126677dbd77edb16d6b82c057ec869fefd7a9871979941e52e837a",
165+
strip_prefix = "llvm-project-909041e4802c4b9a2223ca04099f35bf1dbbd460",
166+
urls = [
167+
"https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/909041e4802c4b9a2223ca04099f35bf1dbbd460.tar.gz",
168+
"https://github.com/llvm/llvm-project/archive/909041e4802c4b9a2223ca04099f35bf1dbbd460.tar.gz",
169+
],
170+
build_file = "@xla//third_party/llvm:llvm.BUILD",
171+
patches = [
172+
"@xla//third_party/llvm:generated.patch",
173+
"@xla//third_party/llvm:build.patch",
174+
"@xla//third_party/llvm:mathextras.patch",
175+
"@xla//third_party/llvm:toolchains.patch",
176+
"@xla//third_party/llvm:zstd.patch",
177+
"@xla//third_party/llvm:lit_test.patch",
178+
"//third_party:llvm_configure.patch",
179+
],
180+
patch_args = ["-p1"],
181+
)
182+
183+
http_archive(
184+
name = "zlib",
185+
build_file = "@com_google_protobuf//:third_party/zlib.BUILD",
186+
sha256 = "17e88863f3600672ab49182f217281b6fc4d3c762bde361935e436a95214d05c",
187+
strip_prefix = "zlib-1.3.1",
188+
urls = ["https://github.com/madler/zlib/archive/v1.3.1.tar.gz"],
75189
)
76190

77-
load("//third_party:python_configure.bzl", "local_python_configure")
78-
local_python_configure(name = "local_config_python")
79-
local_python_configure(name = "local_execution_config_python")
191+
# load("//third_party:python_configure.bzl", "local_python_configure")
192+
# local_python_configure(name = "local_config_python")
193+
# local_python_configure(name = "local_execution_config_python")
80194

81195

82196
# Please add all new struct2tensor dependencies in workspace.bzl.
@@ -113,4 +227,4 @@ rules_pkg_dependencies()
113227

114228
# Specify the minimum required bazel version.
115229
load("@bazel_skylib//lib:versions.bzl", "versions")
116-
versions.check("6.5.0")
230+
versions.check("7.7.0")

‎build_common.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ function install_tensorflow() {
2323

2424
#TODO(b/329181965): As TFX lags for TensorFlow version, let's pass
2525
#the ceiling TF Version to the installation script.
26-
TF_MAX_VERSION = $4
26+
TF_MAX_VERSION=$4
2727

2828
if [[ ("$1" == NIGHTLY_TF) || ("$1" == NIGHTLY_TF_2) ]]; then
2929
TF_PIP_PACKAGE="tf-nightly"

‎configure.sh‎

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -53,12 +53,7 @@ else
5353
fi
5454
fi
5555

56-
# Set Bazel version based on OS
57-
if [[ "$(uname)" == "Darwin" ]]; then
58-
echo "7.4.1" > .bazelversion
59-
else
60-
echo "6.5.0" > .bazelversion
61-
fi
56+
# Bazel version is controlled by .bazelversion file.
6257

6358
ensure_tensorflow
6459
TF_CFLAGS=( $(${PYTHON_BIN_PATH} -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))') )

0 commit comments

Comments
 (0)