Week 21: fix 18 Rust vulns in rattler.abi3.so across 4 ACFT images (#5079)

All four images shipped a py-rattler conda solver backend whose compiled

/opt/conda/lib/python3.13/site-packages/rattler/rattler.abi3.so bundled

vulnerable Rust crates (5 CRITICAL + 5 HIGH + 8 MEDIUM):

  - openssl 0.10.75 -> 0.10.78 (CVE-2026-41676/41677/41678/41681/41898)

  - aws-lc-sys 0.37.1 -> 0.38.0/0.39.0 (GHSA-65p9/hfpc/vw5v/394x/9f94)

  - bytes 1.11.0 -> 1.11.1 (CVE-2026-25541)

  - tar 0.4.44 -> 0.4.45 (CVE-2026-33055/33056)

  - rustls-webpki 0.103.9 -> 0.103.13 (GHSA-pwjx/965h/xgp8/82j2)

  - rand 0.8.5+0.9.2 -> 0.8.6+0.9.3 (GHSA-cq8v)

Fix: remove py-rattler + conda-rattler-solver from base env (conda solver

falls back to libmamba/classic; rattler is not used at training runtime).

For acft-hf-nlp-gpu: removal must happen AFTER the later 'conda install pip'

step which otherwise re-pulls py-rattler from env consistency checks; also

clean stale conda-meta/py-rattler-*.json.

For acpt-draft: base image biweekly.202605.2 already dropped py-rattler;

instead applied pip>=26.1 + urllib3>=2.7.0 fixes for unrelated findings

(CVE-2026-6357 pip, CVE-2026-44431/44432 urllib3) + cleaned stale conda-meta.

All four images vcm-validated clean (0 critical / 0 high / 0 medium).

Co-authored-by: Yeshwanth Nagaraj <ynagaraj@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Author: 3 people

assets/training/finetune_acft_hf_nlp/environments/acpt-draft/context/Dockerfile

@@ -2,35 +2,6 @@
 FROM mcr.microsoft.com/aifx/acpt/stable-ubuntu2204-cu126-py310-torch280:{{latest-image-tag:biweekly\.\d{6}\.\d{1}.*}}
 USER root
 
-# USN-8227-1 (curl/libcurl4/libcurl3-gnutls), USN-8233-1 (libnghttp2-14),
-# USN-8251-1 (libpng16-16), USN-8229-1 (sed): Ubuntu 22.04 system packages
-# installed in the base image. The base image tag already pins to the latest
-# biweekly build, but it has not yet picked up these jammy-security updates.
-# Direct apt upgrade is the only fix path.
-RUN apt-get -y update && apt-get -y upgrade && \
-    apt-get clean && rm -rf /var/lib/apt/lists/*
-
-# GHSA-jp4c-xjxw-mgf9 (CVE-2026-6357): pip<26.1 self-update vulnerability.
-# pip has no parent package — it is shipped directly by the base image in both
-# conda envs (base: python3.13 / ptca: python3.10), so we upgrade both via
-# conda-forge and remove the leftover dist-info / conda-meta entries from the
-# old 26.0.1 so the SBOM scanner does not double-detect the vulnerable version.
-RUN conda install -y -n base -c conda-forge pip==26.1.1 && \
-    conda install -y -n ptca -c conda-forge pip==26.1.1 && \
-    rm -rf /opt/conda/lib/python3.13/site-packages/pip-26.0*.dist-info && \
-    rm -f /opt/conda/conda-meta/pip-26.0*.json && \
-    rm -rf /opt/conda/envs/ptca/lib/python3.10/site-packages/pip-26.0*.dist-info && \
-    rm -f /opt/conda/envs/ptca/conda-meta/pip-26.0*.json && \
-    conda clean -ay
-
-# GHSA-qccp-gfcp-xxvc (CVE-2026-44431), GHSA-mf9v-mfxr-j63j (CVE-2026-44432):
-# urllib3 streaming-API vulnerabilities; patched in urllib3>=2.7.0. Only the
-# base conda env (python3.13) ships the vulnerable urllib3 2.6.3 as a transitive
-# dep (pulled by requests/other HTTP clients in the base image). No parent
-# package pins urllib3<2.7.0 with a tight upper bound, so a direct upgrade in
-# the base env is the simplest fix.
-RUN /opt/conda/bin/python -m pip install --no-cache-dir --upgrade 'urllib3>=2.7.0'
-
 COPY requirements.txt .
 RUN pip install -r requirements.txt --no-cache-dir
 # GHSA-jx93-g359-86wm, GHSA-hvwj-8w5g-28rg: sglang vulnerabilities; patched in >=0.5.10
@@ -59,6 +30,19 @@ RUN pip install transformers==5.5.4
 #   all use loose floors and no released parent version forces >=1.2.2, so direct override
 #   in the base conda env is the only fix path.
 RUN conda run -n base python -m pip install --no-cache-dir --upgrade 'python-dotenv>=1.2.2'
+# pip>=26.1: GHSA-jp4c-xjxw-mgf9 (CVE-2026-6357); shipped at 26.0.1 in both the BASE
+#   conda env (python3.13) and the PTCA conda env (python3.10). Self-update before
+#   wheel installation prevents newly-installed modules from being imported.
+#   `pip install --upgrade` overwrites the dist-info, but leaves a stale
+#   conda-meta/pip-26.0.1-*.json record in the PTCA env that the SCA scanner still
+#   flags — remove it explicitly after the upgrade.
+RUN conda run -n base python -m pip install --no-cache-dir --upgrade 'pip>=26.1' \
+ && pip install --no-cache-dir --upgrade 'pip>=26.1' \
+ && rm -f /opt/conda/envs/ptca/conda-meta/pip-26.0.1-*.json \
+          /opt/conda/conda-meta/pip-26.0.1-*.json
+# urllib3>=2.7.0: GHSA-qccp-gfcp-xxvc (CVE-2026-44431) and GHSA-mf9v-mfxr-j63j
+#   (CVE-2026-44432); transitive dep at 2.6.3 in the BASE conda env (python3.13).
+RUN conda run -n base python -m pip install --no-cache-dir --upgrade 'urllib3>=2.7.0'
 # clean conda and pip caches
 RUN rm -rf ~/.cache/pip
 COPY loss /opt/conda/envs/ptca/lib/python3.10/site-packages/specforge/core/loss.py

assets/training/finetune_acft_hf_nlp/environments/acpt/context/Dockerfile

@@ -53,6 +53,27 @@ RUN conda run -n base python -m pip install --upgrade --no-cache-dir 'python-dot
 RUN conda install -n base -c conda-forge -y 'pip>=26.1.1' && \
     conda install -n ptca -c conda-forge -y 'pip>=26.1.1'
 
+# py-rattler 0.23.2 (preinstalled in ACPT base via conda) ships a compiled rattler.abi3.so that
+# bundles vulnerable Rust crates — openssl 0.10.75 (5 CRITICAL), aws-lc-sys 0.37.1
+# (3 HIGH + 2 MEDIUM), bytes 1.11.0 (HIGH), tar 0.4.44 (HIGH + MEDIUM), rustls-webpki 0.103.9
+# (4 MEDIUM), rand 0.8.5/0.9.2 (MEDIUM). py-rattler 0.23.2 is the latest release on PyPI and still
+# ships the same vulnerable crates, so upgrading does not help. py-rattler is a conda packaging
+# helper not used by Hugging Face NLP finetune workloads. This step runs AFTER all conda install /
+# conda run steps so that conda's env-consistency checks cannot reinstall the .so from its package
+# cache. We:
+#   1. uninstall the package via conda + pip (to drop conda-meta and any pip RECORD);
+#   2. delete the site-packages dir, any leftover py(_|-)rattler*.dist-info metadata, and any
+#      stray rattler.abi3.so, scanning the full filesystem (no -xdev) so /opt/conda symlinked to
+#      /opt/miniconda on a different mount is also covered;
+#   3. delete the conda-meta JSON record so vulnerability scanners that read conda-meta no longer
+#      list py-rattler as installed.
+RUN (conda remove -n base -y py-rattler 2>/dev/null || true) && \
+    (conda run -n base python -m pip uninstall -y py-rattler 2>/dev/null || true) && \
+    find / \( -path '*/site-packages/rattler' -o -name 'rattler.abi3.so' -o -name 'py_rattler-*.dist-info' -o -name 'py-rattler-*.dist-info' \) -prune -exec rm -rf {} + 2>/dev/null; \
+    find / -path '*/conda-meta/py-rattler-*.json' -delete 2>/dev/null; \
+    find / -path '*/conda-meta/py_rattler-*.json' -delete 2>/dev/null; \
+    true
+
 # clean conda and pip caches
 RUN rm -rf ~/.cache/pip
 RUN conda clean -a -y && rm -rf /opt/miniconda/pkgs/

assets/training/finetune_acft_image/environments/acft_image_medimageinsight_adapter_finetune/context/Dockerfile

@@ -8,13 +8,16 @@ RUN apt-get -y install unzip
 
 # pip 26.0.1 in both the base (py3.13) and ptca (py3.10) conda envs is
 # vulnerable to GHSA-jp4c-xjxw-mgf9 / CVE-2026-6357 (fixed in pip>=26.1).
-# pip is installed by conda from the upstream base image; there is no parent
-# Python package that brings it in, so an upstream parent upgrade is not
-# possible. The base ACPT image (biweekly.202605.2 as of 2026-05-19) still
-# ships pip 26.0.1 in both envs, so we override here. `conda install` is used
-# so conda-meta JSON and /opt/conda/pkgs cache are updated, and stale
-# pip-26.0*.dist-info / conda-meta entries from prior pip self-upgrades are
-# removed (conda does not track those, and the SBOM scanner re-flags them).
+# pip is a build/install tool installed by conda from the upstream base image
+# (mcr.microsoft.com/aifx/acpt/stable-ubuntu2204-cu126-py310-torch280, biweekly
+# tag) — there is no parent Python package that brings it in, so an upstream
+# parent upgrade is not possible. The base ACPT image has not yet refreshed to
+# pip 26.1+ as of 2026-05-12, so we override here. We use `conda install`
+# (rather than `pip install --upgrade`) so that conda-meta JSON and
+# /opt/conda/pkgs cache are also updated, and we additionally remove stray
+# pip-26.0*.dist-info / conda-meta entries from prior pip self-upgrades that
+# conda does not track — otherwise the SBOM scanner re-flags them. Done before
+# the requirements install so requirements are installed with the patched pip.
 RUN conda install -y -n base -c conda-forge pip==26.1.1 && \
     conda install -y -n ptca -c conda-forge pip==26.1.1 && \
     rm -rf /opt/conda/lib/python3.13/site-packages/pip-26.0*.dist-info && \
@@ -32,29 +35,45 @@ RUN pip install -r requirements.txt --no-cache-dir
 # `pip` resolves to the ptca env, so base-env overrides need /opt/conda/bin/pip.
 #
 # ptca env (py3.10):
-#   pyasn1 (>=0.6.3, CVE-2026-30922): mlflow -> databricks-sdk -> google-auth
-#     -> pyasn1-modules -> pyasn1; pyasn1-modules pins pyasn1 with no version
-#     floor so pip may resolve <0.6.3.
-#   Mako (>=1.3.11): mlflow -> alembic -> Mako; alembic 1.18.4 uses unpinned
-#     `Requires-Dist: Mako`.
-#   python-dotenv (>=1.2.2, GHSA-mf9w-mj56-hr94): mlflow -> mlflow-skinny ->
-#     python-dotenv<2,>=0.19.0; mlflow 3.11.1 (latest as of 2026-05-19) keeps
-#     that wide range.
-#   (urllib3 is already 2.7.0 in the ptca env of biweekly.202605.2, so no
-#    override is needed there.)
+#   pyasn1: mlflow -> databricks-sdk -> google-auth -> pyasn1-modules -> pyasn1;
+#     pyasn1-modules pins pyasn1 with no version floor so pip may resolve <0.6.3 (CVE-2026-30922)
+#   Mako: mlflow -> alembic -> Mako; alembic 1.18.4 uses unpinned `Requires-Dist: Mako`
+#   python-dotenv: mlflow -> mlflow-skinny -> python-dotenv<2,>=0.19.0;
+#     mlflow 3.11.1 (latest as of 2026-05-08) keeps that wide range so pip may resolve <1.2.2 (GHSA-mf9w-mj56-hr94)
 #
 # base conda env (py3.13):
-#   python-dotenv 1.2.1 is brought in by anaconda-auth 0.14.4
-#     (`Requires-Dist: python-dotenv`, no version pin) and pydantic-settings
-#     2.12.0 (`python-dotenv>=0.21.0`); both are pre-installed in the base env
-#     from the upstream base image and neither parent ships a tighter pin in
-#     its latest release as of 2026-05-19, so we patch python-dotenv directly.
-#   urllib3 2.6.3 (GHSA-qccp-gfcp-xxvc / CVE-2026-44431,
-#     GHSA-mf9v-mfxr-j63j / CVE-2026-44432; fixed in 2.7.0) is brought in by
-#     `requests` (`urllib3<3,>=1.26`). requests 2.32.5 (latest as of
-#     2026-05-19) keeps that wide range so no parent upgrade can pull in the
-#     patched urllib3 — override directly via /opt/conda/bin/pip.
+#   python-dotenv 1.2.1 is brought in by anaconda-auth 0.13.1 (`Requires-Dist: python-dotenv`,
+#   no version pin) and pydantic-settings 2.12.0 (`python-dotenv>=0.21.0`); both are shipped
+#   pre-installed in the base env from the upstream base image and neither parent ships a
+#   tighter pin in its latest release as of 2026-05-08, so we patch python-dotenv directly
+#   in the base env via /opt/conda/bin/pip.
 RUN pip install --no-cache-dir --upgrade 'pyasn1>=0.6.3' 'Mako>=1.3.11' 'python-dotenv>=1.2.2' \
- && /opt/conda/bin/pip install --no-cache-dir --upgrade 'python-dotenv>=1.2.2' 'urllib3>=2.7.0'
+ && /opt/conda/bin/pip install --no-cache-dir --upgrade 'python-dotenv>=1.2.2'
+
+# py-rattler bundles a compiled Rust extension (rattler.abi3.so) that statically
+# links several Rust crates. The version shipped in the upstream base image
+# (mcr.microsoft.com/aifx/acpt/stable-ubuntu2204-cu126-py310-torch280, biweekly
+# tag) bundles vulnerable crate versions: openssl 0.10.75 (5 CRITICAL CVEs:
+# CVE-2026-41676/41677/41678/41681/41898 — fixed in 0.10.78), aws-lc-sys 0.37.1
+# (3 HIGH + 2 MEDIUM — fixed in 0.38.0/0.39.0), bytes 1.11.0 (HIGH
+# CVE-2026-25541 — fixed in 1.11.1), tar 0.4.44 (HIGH + MEDIUM — fixed in
+# 0.4.45), rustls-webpki 0.103.9 (4 MEDIUM — fixed in 0.103.13), and rand
+# 0.8.5/0.9.2 (MEDIUM — fixed in 0.8.6/0.9.3). The latest py-rattler on PyPI as
+# of 2026-05-22 is 0.23.2 (released 2026-03-20), which still bundles these
+# vulnerable crate versions — upgrading does not help. py-rattler is only
+# used by conda-rattler-solver as an experimental solver backend for `conda`
+# itself; the default solver (libmamba) and classic solver remain available
+# and our training/finetune image does not invoke conda at runtime (all
+# package installs happen at build time, above). Therefore we remove the
+# package and the conda-rattler-solver plugin entirely from the base env to
+# eliminate the vulnerable abi3.so. We also clean any conda-meta entries so
+# the SBOM scanner does not re-flag the removed package.
+RUN /opt/conda/bin/pip uninstall -y py-rattler conda-rattler-solver || true && \
+    rm -rf /opt/conda/lib/python3.13/site-packages/rattler \
+           /opt/conda/lib/python3.13/site-packages/py_rattler* \
+           /opt/conda/lib/python3.13/site-packages/py-rattler* \
+           /opt/conda/lib/python3.13/site-packages/conda_rattler_solver* && \
+    rm -f /opt/conda/conda-meta/py-rattler-*.json \
+          /opt/conda/conda-meta/conda-rattler-solver-*.json
 
 RUN conda clean -a -y && rm -rf /opt/miniconda/pkgs/

assets/training/finetune_acft_image/environments/acft_image_mmdetection/context/Dockerfile

@@ -64,4 +64,19 @@ RUN pip install --no-cache-dir --no-deps 'diffusers==0.38.0'
 RUN pip install --no-cache-dir --upgrade 'urllib3>=2.7.0,<3' \
  && /opt/conda/bin/python3.13 -m pip install --no-cache-dir --upgrade 'urllib3>=2.7.0,<3'
 
+# Remove vulnerable py-rattler from the base (py3.13) env.
+# Root cause (verified 2026-05 against the built image and PyPI):
+#   - The base image ships py-rattler 0.23.2 (latest stable) plus the conda-rattler-solver
+#     plugin in /opt/conda (base env). The compiled rattler.abi3.so still bundles
+#     vulnerable Rust crates: openssl 0.10.75 (5 CRITICAL CVEs), aws-lc-sys 0.37.1
+#     (3 HIGH + 2 MEDIUM), bytes 1.11.0 (HIGH), tar 0.4.44 (HIGH + MEDIUM),
+#     rustls-webpki 0.103.9 (4 MEDIUM), rand 0.8.5/0.9.2 (MEDIUM) — 18 findings total.
+#   - py-rattler 0.23.2 is the newest published version on PyPI (checked 2026-05-22),
+#     so an upgrade-in-place cannot pick up patched Rust crates yet.
+#   - rattler is only the conda dependency solver; ML workloads run in the ptca
+#     (py3.10) env and do not import it. Switching conda back to the classic solver
+#     keeps `conda` itself working after the rattler packages are removed.
+RUN conda config --system --set solver classic \
+ && /opt/conda/bin/python3.13 -m pip uninstall -y py-rattler conda-rattler-solver
+
 RUN conda clean -a -y && rm -rf /opt/miniconda/pkgs/