Isolate py314 sdk snapshots

.github/actions/setup-k8s-access/action.yml

@@ -68,7 +68,7 @@ runs:
     run: |
       kubectl config set-context --current --namespace=${{ steps.replace_namespace.outputs.TEST_NAMESPACE }}
   - name: Post cleanup
-    uses: pyTooling/Actions/with-post-step@v4.2.2
+    uses: pytooling/actions/with-post-step@42e17fae05f224e5ac3d79d021a4e3577878efe5 # v4.2.2
     with:
       main: echo "Post Cleanup"
       post: |

.github/trigger_files/beam_PostCommit_Python_ValidatesContainer_Dataflow.json

@@ -1,4 +1,4 @@
 {
     "comment": "Modify this file in a trivial way to cause this test suite to run",
-    "modification": 4
+    "modification": 5
 }

.github/trigger_files/beam_PreCommit_Flink_Container.json

@@ -1,4 +1,4 @@
 {
     "comment": "Modify this file in a trivial way to cause this test suite to run",
-    "modification": 3
+    "modification": 4
 }

.github/workflows/README.md

@@ -58,6 +58,7 @@ On top of normal Actions workflow steps, all new CI workflows (excluding release
 4) A set of GitHub token permissions
 5) Concurrency Groups
 6) Comment Triggering Support
+7) Allowed Actions (See below)
 
 Each of these is described in more detail below.
 ## Name and Phrase
@@ -165,6 +166,14 @@ In order to make it easier for non-committers to interact with workflows, workfl
 
 **Note:** Comment triggering is found not scalable ([#28909](https://github.com/apache/beam/issues/28909)) and is currently limited to a subset of suites. For more information see the [Running Workflows Manually](#running-workflows-manually) section.
 
+## Allowed Actions
+
+GitHub Actions under the trusted namespaces `actions/*`, `github/*`, and `apache/*` are automatically approved. Any third-party actions outside of these namespaces must be explicitly approved and added to the ASF-wide allowlist before use.
+
+Additionally, third-party actions must be pinned to a specific Git commit SHA (e.g., `zizmorcore/zizmor-action@6599ee8b7a49aef6a770f63d261d214911a7ce02`) instead of a version tag or branch name to comply with ASF security policy.
+
+For details on the allowlist and how to request new actions, see the [ASF Infrastructure Actions Allowlist Check Documentation](https://github.com/apache/infrastructure-actions/blob/main/allowlist-check/README.md).
+
 # Testing new workflows or workflow updates
 
 ## Testing New Workflows

.github/workflows/beam_Infrastructure_UsersPermissions.yml

@@ -52,7 +52,7 @@ jobs:
       - name: Setup gcloud
         uses: google-github-actions/setup-gcloud@aa5489c8933f4cc7a4f7d45035b3b1440c9c10db
       - name: Install Terraform
-        uses: hashicorp/setup-terraform@v4
+        uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
         with:
           terraform_version: 1.12.2
       - name: Initialize Terraform

.github/workflows/beam_PostCommit_Java_IO_Performance_Tests.yml

@@ -64,7 +64,7 @@ jobs:
       matrix:
         job_name: ["beam_PostCommit_Java_IO_Performance_Tests"]
         job_phrase: ["Run Java PostCommit IO Performance Tests"]
-        test_case: ["GCSPerformanceTest", "BigTablePerformanceTest", "BigQueryStorageApiStreamingPerformanceTest"]
+        test_case: ["IcebergPerformanceTest", "GCSPerformanceTest", "BigTablePerformanceTest", "BigQueryStorageApiStreamingPerformanceTest"]
     steps:
     - uses: actions/checkout@v7
       with:

.github/workflows/beam_PostCommit_Python_Arm.yml

@@ -54,7 +54,8 @@ jobs:
   beam_PostCommit_Python_Arm:
     name: ${{ matrix.job_name }} ${{ matrix.python_version }}
     runs-on: [self-hosted, ubuntu-24.04, main]
-    timeout-minutes: 480
+    # Python 3.14 multiarch container builds take much longer than other versions.
+    timeout-minutes: ${{ matrix.python_version == '3.14' && 600 || 480 }}
     strategy:
       fail-fast: false
       matrix:

.github/workflows/beam_PreCommit_GHA.yml

@@ -80,15 +80,19 @@ jobs:
         comment_phrase: ${{ matrix.job_phrase }}
         github_token: ${{ secrets.GITHUB_TOKEN }}
         github_job: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
-    - name: Run zizmor
-      uses: zizmorcore/zizmor-action@6599ee8b7a49aef6a770f63d261d214911a7ce02 # v0.6.0
-      with:
-        advanced-security: true
     - name: Setup environment
       uses: ./.github/actions/setup-environment-action
       with:
         java-version: default
         go-version: default
+        python-version: default
+    # For details on the ASF GHA allowlist, see: https://github.com/apache/infrastructure-actions/blob/main/allowlist-check/README.md
+    - name: Validate GHA Allowlist
+      uses: apache/infrastructure-actions/allowlist-check@main # zizmor: ignore[unpinned-uses]
+    - name: Run zizmor
+      uses: zizmorcore/zizmor-action@6599ee8b7a49aef6a770f63d261d214911a7ce02 # v0.6.0
+      with:
+        advanced-security: true
     - name: run GHA PreCommit script
       uses: ./.github/actions/gradle-command-self-hosted-action
       with:

.github/workflows/beam_Publish_Beam_SDK_Snapshots.yml

@@ -62,12 +62,10 @@ jobs:
           - "python:container:py311:docker"
           - "python:container:py312:docker"
           - "python:container:py313:docker"
-          - "python:container:py314:docker"
           - "python:container:distroless:py310:docker"
           - "python:container:distroless:py311:docker"
           - "python:container:distroless:py312:docker"
           - "python:container:distroless:py313:docker"
-          - "python:container:distroless:py314:docker"
           - "python:container:ml:py310:docker"
           - "python:container:ml:py311:docker"
           - "python:container:ml:py312:docker"
@@ -90,8 +88,16 @@ jobs:
         # This is needed to run pipelines that use the default environment at HEAD, for example, when a
         # pipeline uses an expansion service built from HEAD.
         run: |
-          BEAM_VERSION_LINE=$(cat gradle.properties | grep "sdk_version")
-          echo "BEAM_VERSION=${BEAM_VERSION_LINE#*sdk_version=}" >> $GITHUB_ENV
+          BEAM_VERSION=$(grep -E '^sdk_version=' gradle.properties | cut -d= -f2-)
+          if [[ ! "${BEAM_VERSION}" =~ ^[0-9A-Za-z._-]+$ ]]; then
+            echo "Invalid sdk_version in gradle.properties: ${BEAM_VERSION}"
+            exit 1
+          fi
+          {
+            echo "BEAM_VERSION<<EOF"
+            echo "${BEAM_VERSION}"
+            echo "EOF"
+          } >> "$GITHUB_ENV"
       - name: Set latest tag only on master branch
         if: github.ref == 'refs/heads/master'
         run: echo "LATEST_TAG=,latest" >> $GITHUB_ENV
@@ -127,3 +133,75 @@ jobs:
             -Pcontainer-architecture-list=arm64,amd64 \
             -Ppush-containers \
             -Pdocker-pull-licenses
+
+  beam_Publish_Beam_SDK_Snapshots_py314:
+    needs: beam_Publish_Beam_SDK_Snapshots
+    if: |
+      always() &&
+      !cancelled() &&
+      (github.event_name == 'workflow_dispatch' ||
+      (github.event_name == 'schedule' && github.repository == 'apache/beam'))
+    runs-on: [self-hosted, ubuntu-24.04, main]
+    timeout-minutes: 300
+    name: ${{ matrix.job_name }} (${{ matrix.container_task }})
+    strategy:
+      fail-fast: false
+      matrix:
+        job_name: ["beam_Publish_Beam_SDK_Snapshots"]
+        job_phrase: ["N/A"]
+        container_task:
+          - "python:container:py314:docker"
+          - "python:container:distroless:py314:docker"
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          persist-credentials: false
+      - name: Free Disk Space (Ubuntu)
+        uses: jlumbroso/free-disk-space@v1.3.1
+      - name: Setup repository
+        uses: ./.github/actions/setup-action
+        with:
+          comment_phrase: ${{ matrix.job_phrase }}
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          github_job: ${{ matrix.job_name }} (${{ matrix.container_task }})
+      - name: Find Beam Version
+        run: |
+          BEAM_VERSION=$(grep -E '^sdk_version=' gradle.properties | cut -d= -f2-)
+          if [[ ! "${BEAM_VERSION}" =~ ^[0-9A-Za-z._-]+$ ]]; then
+            echo "Invalid sdk_version in gradle.properties: ${BEAM_VERSION}"
+            exit 1
+          fi
+          {
+            echo "BEAM_VERSION<<EOF"
+            echo "${BEAM_VERSION}"
+            echo "EOF"
+          } >> "$GITHUB_ENV"
+      - name: Set latest tag only on master branch
+        if: github.ref == 'refs/heads/master'
+        run: echo "LATEST_TAG=,latest" >> $GITHUB_ENV
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c
+      - name: Authenticate on GCP
+        uses: google-github-actions/auth@7c6bc770dae815cd3e89ee6cdf493a5fab2cc093
+        with:
+          service_account: ${{ secrets.GCP_SA_EMAIL }}
+          credentials_json: ${{ secrets.GCP_SA_KEY }}
+      - name: Set up Cloud SDK
+        uses: google-github-actions/setup-gcloud@aa5489c8933f4cc7a4f7d45035b3b1440c9c10db
+      - name: GCloud Docker credential helper
+        run: |
+          gcloud auth configure-docker ${{ env.docker_registry }}
+      - name: Setup Python environment
+        uses: ./.github/actions/setup-environment-action
+        with:
+          python-version: default
+      - name: run Publish Beam SDK Snapshots script
+        uses: ./.github/actions/gradle-command-self-hosted-action
+        with:
+          gradle-command: :sdks:${{ matrix.container_task }}
+          arguments: |
+            -Pdocker-repository-root=gcr.io/apache-beam-testing/beam-sdk \
+            -Pdocker-tag-list=${{ github.sha }},${BEAM_VERSION}${LATEST_TAG} \
+            -Pcontainer-architecture-list=arm64,amd64 \
+            -Ppush-containers \
+            -Pdocker-pull-licenses

.github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml

@@ -62,6 +62,8 @@ jobs:
       startsWith(github.event.comment.body, 'Run Python ValidatesContainer Dataflow ARM')
 
     runs-on: ubuntu-22.04
+    # Python 3.14 multiarch container builds compile many deps from source on arm64.
+    timeout-minutes: ${{ matrix.python_version == '3.14' && 300 || 120 }}
     steps:
       - uses: actions/checkout@v7
         with:

.github/workflows/build_wheels.yml

@@ -37,7 +37,8 @@ concurrency:
   cancel-in-progress: true
 
 env:
-  GCP_PATH: "gs://${{ secrets.GCP_PYTHON_WHEELS_BUCKET }}/${GITHUB_REF##*/}/${GITHUB_SHA}-${GITHUB_RUN_ID}/"
+  # Use github.* context; workflow env is not bash-evaluated.
+  GCP_PATH: "gs://${{ secrets.GCP_PYTHON_WHEELS_BUCKET }}/${{ github.ref_name }}/${{ github.sha }}-${{ github.run_id }}/"
 
 
 jobs:

.github/workflows/codeql.yml

@@ -36,7 +36,7 @@ jobs:
       matrix: ${{ steps.set-matrix.outputs.matrix }}
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v4
+        uses: actions/checkout@v7
       - uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d
         id: filter
         with:

.github/workflows/update_python_dependencies.yml

@@ -51,6 +51,7 @@ jobs:
     runs-on: [self-hosted, ubuntu-24.04, main]
     needs: set-properties
     name: Update Python Dependencies
+    timeout-minutes: 180
     steps:
       - name: Checkout code
         uses: actions/checkout@v7

.test-infra/tools/refresh_looker_metrics.py

@@ -34,6 +34,8 @@
     ("33", ["21", "70", "116", "69", "115"]),  # BigTableIO_Write
     ("34", ["22", "56", "96", "55", "95"]),    # TextIO_Read
     ("35", ["23", "64", "110", "63", "109"]),  # TextIO_Write
+    ("113", ["386", "388", "390", "392", "394"]),    # IcebergIO_Read
+    ("114", ["387", "389", "391", "393", "395"]),  # IcebergIO_Write
     ("75", ["258", "259", "260", "261", "262"]),  # TensorFlow MNIST
     ("76", ["233", "234", "235", "236", "237"]),  # PyTorch BERT base uncased
     ("77", ["238", "239", "240", "241", "242"]),  # PyTorch BERT large uncased

CHANGES.md

@@ -75,6 +75,7 @@
 ## Breaking Changes
 
 * (Python) Removed `google-perftools` from the SDK container images. Users who wish to use `--profiler_agent=tcmalloc` should install google-perftools APT package in their custom container images separately ([#39323](https://github.com/apache/beam/issues/39323)).
+* (Java) Added `DRAINING` and `DRAINED` states to `PipelineResult`, including runner state mappings and Dataflow update handling ([#39020](https://github.com/apache/beam/issues/39020)).
 
 ## Deprecations
 
@@ -84,7 +85,7 @@
 
 * Fixed unbounded checkpoint state growth for splittable DoFns that self-checkpoint on the portable Flink runner (Java) ([#27648](https://github.com/apache/beam/issues/27648)).
 * Improved Java pipeline performance by avoiding repeated `DoFn` type descriptor resolution when creating cached invokers ([#39309](https://github.com/apache/beam/issues/39309)).
-* Fixed X (Java/Python) ([#X](https://github.com/apache/beam/issues/X)).
+* (Python) Fixed a memory leak in Python SDK caused by storing exceptions with potentially large stack frames in a cache ([#39406](https://github.com/apache/beam/issues/39406)).
 
 ## Security Fixes
 
@@ -138,6 +139,7 @@
 ## Known Issues
 
 * (Java) Projects using the Flink runner with Flink 2.1 or later alongside libraries requiring `org.lz4:lz4-java` (e.g., Kafka clients) may encounter a Gradle capability conflict, because Flink 2.1+ ships `at.yawk.lz4:lz4-java` which declares the same capability. To resolve, add a `capabilitiesResolution` rule to your `build.gradle` that selects `at.yawk.lz4:lz4-java` ([#38947](https://github.com/apache/beam/issues/38947)).
+* (Python) Long-running Python pipelines might experience memory growth and periodic OOMs ([#39406](https://github.com/apache/beam/issues/39406)).
 
 # [2.74.0] - 2026-06-02
 
@@ -179,6 +181,10 @@
 * Fixed BigQueryEnrichmentHandler batch mode dropping earlier requests when multiple requests share the same enrichment key (Python) ([#38035](https://github.com/apache/beam/issues/38035)).
 * Added `max_batch_duration_secs` passthrough support in Python Enrichment BigQuery and CloudSQL handlers so batching duration can be forwarded to `BatchElements` ([#38243](https://github.com/apache/beam/issues/38243)).
 
+## Known Issues
+
+* (Python) Long-running Python pipelines might experience memory growth and periodic OOMs ([#39406](https://github.com/apache/beam/issues/39406)).
+
 # [2.73.0] - 2026-04-29
 
 ## Highlights
@@ -215,6 +221,10 @@
 * Fixed [CVE-2023-46604](https://www.cve.org/CVERecord?id=CVE-2023-46604) (CVSS 10.0) and [CVE-2022-41678](https://www.cve.org/CVERecord?id=CVE-2022-41678) by upgrading ActiveMQ from 5.14.5 to 5.19.2 (Java) ([#37943](https://github.com/apache/beam/issues/37943)).
 * Fixed [CVE-2024-1597](https://www.cve.org/CVERecord?id=CVE-2024-1597), [CVE-2022-31197](https://www.cve.org/CVERecord?id=CVE-2022-31197), and [CVE-2022-21724](https://www.cve.org/CVERecord?id=CVE-2022-21724) by upgrading PostgreSQL JDBC Driver from 42.2.16 to 42.6.2 (Java) ([#37942](https://github.com/apache/beam/issues/37942)).
 
+## Known Issues
+
+* (Python) Long-running Python pipelines might experience memory growth and periodic OOMs ([#39406](https://github.com/apache/beam/issues/39406)).
+
 # [2.72.0] - 2026-03-30
 
 ## Highlights
@@ -248,6 +258,10 @@
 
 * Fixed [CVE-2024-28397](https://www.cve.org/CVERecord?id=CVE-2024-28397) by switching from js2py to pythonmonkey (Yaml) ([#37560](https://github.com/apache/beam/issues/37560)).
 
+## Known Issues
+
+* (Python) Long-running Python pipelines might experience memory growth and periodic OOMs ([#39406](https://github.com/apache/beam/issues/39406)).
+
 # [2.71.0] - 2026-01-22
 
 ## I/Os
@@ -269,6 +283,7 @@
 
 ## Known Issues
 
+* (Python) Long-running Python pipelines might experience memory growth and periodic OOMs ([#39406](https://github.com/apache/beam/issues/39406)).
 
 # [2.70.0] - 2025-12-16
 
@@ -291,6 +306,10 @@ Now Beam has full support for Milvus integration including Milvus enrichment and
 
 * (Python) Python 3.9 reached EOL in October 2025 and support for the language version has been removed. ([#36665](https://github.com/apache/beam/issues/36665)).
 
+## Known Issues
+
+* (Python) Long-running Python pipelines might experience memory growth and periodic OOMs ([#39406](https://github.com/apache/beam/issues/39406)).
+
 # [2.69.0] - 2025-10-28
 
 ## Highlights
@@ -350,6 +369,10 @@ Now Beam has full support for Milvus integration including Milvus enrichment and
   ([#36141](https://github.com/apache/beam/issues/36141)).
 * Fixed Spanner Change Stream reading stuck issue due to watermark of partition moving backwards ([#36470](https://github.com/apache/beam/issues/36470)).
 
+## Known Issues
+
+* (Python) Long-running Python pipelines might experience memory growth and periodic OOMs ([#39406](https://github.com/apache/beam/issues/39406)).
+
 # [2.68.0] - 2025-09-22
 
 ## Highlights
@@ -404,6 +427,7 @@ Now Beam has full support for Milvus integration including Milvus enrichment and
 ## Known Issues
 
 * ([#36470](https://github.com/apache/beam/issues/36470)). Spanner Change Stream reading stuck issue due to watermark of partition moving backwards. This issue exists in 2.67.0 and 2.68.0. To mitigate the issue, either use old version 2.66.0 or go to 2.69.0.
+* (Python) Long-running Python pipelines might experience memory growth and periodic OOMs ([#39406](https://github.com/apache/beam/issues/39406)).
 
 # [2.67.0] - 2025-08-12
 
@@ -452,6 +476,7 @@ Now Beam has full support for Milvus integration including Milvus enrichment and
 
 * ([#35666](https://github.com/apache/beam/issues/35666)). YAML Flatten incorrectly drops fields when input PCollections' schema are different. This issue exists for all versions since 2.52.0.
 * ([#36470](https://github.com/apache/beam/issues/36470)). Spanner Change Stream reading stuck issue due to watermark of partition moving backwards. This issue exists in 2.67.0 and 2.68.0. To mitigate the issue, either use old version 2.66.0 or go to 2.69.0.
+* (Python) Long-running Python pipelines might experience memory growth and periodic OOMs ([#39406](https://github.com/apache/beam/issues/39406)).
 
 # [2.66.0] - 2025-07-01