Patch onedir Lib/ssl.py instead of dropping sitecustomize for cpython#104135

After #69490 landed, the `ci-test-onedir` job got past Install Deps, but the
next session - `pre-archive-cleanup` (noxfile.py:1397-1402) - failed at the
same step with the same `[ASN1: NOT_ENOUGH_DATA]`. That session also uses
the onedir Python but creates its virtualenv WITHOUT `--system-site-packages`,
so the `sitecustomize.py` we dropped in `artifacts/salt/Lib/site-packages/`
isn't loaded by it and the buggy `_load_windows_store_certs` is back.

Patch `Lib/ssl.py` in the onedir itself instead: virtualenv's venvs always
read stdlib from the base Python's `Lib/` (with or without
`--system-site-packages`), so a `Lib/ssl.py` append covers every venv launched
from the onedir, plus the onedir Python directly.

Verified on a local relenv 0.22.14 build (Python 3.10.20, OpenSSL 3.5.7):
both a `--system-site-packages` venv and an isolated venv now report
`_salt_safe_load_windows_store_certs` as the qualname for
`ssl.SSLContext._load_windows_store_certs`, and a synthetic truncated cert
injected via a monkey-patched `_ssl.enum_certificates` no longer aborts
`load_default_certs()`.

Drops `cicd/windows-ssl-104135-sitecustomize.py` (now redundant) and adds
`cicd/windows-ssl-104135-patch.py` (idempotent in-place patcher; takes the
ssl.py path as argv). The CI step swaps `cp ... -> sitecustomize.py` for
`python3 cicd/windows-ssl-104135-patch.py artifacts/salt/Lib/ssl.py`.

Cross-references in `salt/__init__.py` and `salt/ext/tornado/netutil.py`
updated to point at the new filename so the trio still drops together when
this branch's onedir Python moves off 3.10.

Author: dwoz

.github/workflows/build-deps-ci-action.yml

@@ -292,20 +292,21 @@ jobs:
           cd artifacts
           tar xvf ${{ inputs.package-name }}-${{ inputs.salt-version }}-onedir-windows-${{ matrix.arch }}.tar.xz
 
-      - name: Apply cpython#104135 sitecustomize to onedir Python
-        # The ci-test-onedir nox session creates a virtualenv from the onedir
-        # Python (3.10 on this branch) and runs pip *inside it* before salt is
-        # importable; Python 3.10's _load_windows_store_certs feeds the whole
-        # Windows root store to load_verify_locations as one blob, which
-        # OpenSSL 3.5.x (relenv >= 0.22.13) rejects on a single bad cert.
-        # Dropping the sitecustomize at the onedir site-packages root means
-        # the venv inherits it via --system-site-packages and the iter-and-
-        # skip patch fires at every interpreter start. Self-disables on
-        # Python 3.11+; remove with the rest of the cpython#104135 trio.
+      - name: Patch cpython#104135 in onedir Lib/ssl.py
+        # The build-deps-ci nox sessions (ci-test-onedir, pre-archive-cleanup)
+        # create virtualenvs from the onedir Python (3.10 on this branch) and
+        # run pip *inside them* before salt is importable; Python 3.10's
+        # _load_windows_store_certs feeds the whole Windows root store to
+        # load_verify_locations as one blob, which OpenSSL 3.5.x (relenv >=
+        # 0.22.13) rejects on a single bad cert. Patching Lib/ssl.py in the
+        # onedir reaches every venv created from it (the venv's python.exe
+        # always reads stdlib from the base Python's Lib/), whether or not
+        # nox passes --system-site-packages. Self-disables on Python 3.11+;
+        # remove with the rest of the cpython#104135 trio.
         if: steps.nox-dependencies-cache.outputs.cache-hit != 'true'
         shell: bash
         run: |
-          cp cicd/windows-ssl-104135-sitecustomize.py artifacts/salt/Lib/site-packages/sitecustomize.py
+          python3 cicd/windows-ssl-104135-patch.py artifacts/salt/Lib/ssl.py
 
       - name: Set up Python ${{ inputs.ci-python-version }}
         if: steps.nox-dependencies-cache.outputs.cache-hit != 'true'

cicd/windows-ssl-104135-patch.py

@@ -0,0 +1,97 @@
+"""
+Patch the salt onedir Python's ``Lib/ssl.py`` in place to apply
+cpython#104135's iterate-and-skip variant of
+``ssl.SSLContext._load_windows_store_certs``.
+
+Usage: ``python windows-ssl-104135-patch.py <path-to-ssl.py>``
+
+Background
+----------
+
+The build-deps-ci Windows job extracts the salt onedir, then runs
+``nox --install-only -e ci-test-onedir`` and ``nox -e pre-archive-cleanup``.
+Both sessions target the onedir's relenv-bundled Python (3.10.20 on 3006.x)
+and create virtualenvs from it. The ``ci-test-onedir`` session uses
+``--system-site-packages``; ``pre-archive-cleanup`` does not. Either way,
+both venvs share the onedir's ``Lib/ssl.py`` because virtualenv leaves the
+base Python's stdlib on ``sys.path``.
+
+Python 3.10's stdlib ``ssl._load_windows_store_certs`` concatenates every
+cert from the Windows root store and feeds them to
+``load_verify_locations(cadata=...)`` as one blob. OpenSSL 3.5.x (shipped by
+relenv >= 0.22.13) rejects the whole blob on a single ASN.1-malformed cert,
+so pip's TLS to pypi.org dies with ``[ASN1: NOT_ENOUGH_DATA]`` before any
+deps land. See cpython#104135.
+
+Appending the patch to ``Lib/ssl.py`` means every Python invocation using
+the onedir picks up the fix - the ``ci-test-onedir`` venv, the
+``pre-archive-cleanup`` venv, the raw onedir python itself, anything.
+
+A ``sitecustomize.py`` would only reach the venv that turns on
+``--system-site-packages``; a ``Lib/ssl.py`` append covers both.
+
+Python-3.10-only: self-disables on Python 3.11+ (whose stdlib already has
+the upstream fix). Delete this file together with the salt/__init__.py
+block and the salt/ext/tornado/netutil.py ``sys.platform == 'win32'``
+special-case on any branch whose onedir Python is >= 3.11.
+"""
+
+import sys
+
+MARKER = "# >>> cpython#104135 patch (windows-ssl-104135-patch.py) <<<"
+
+PATCH_BODY = f"""
+
+{MARKER}
+# Replace the pre-fix _load_windows_store_certs with the iterate-and-skip
+# variant cpython merged for the 3.11 branch. See cpython#104135.
+# Self-disable on Python 3.11+ (where Lib/ssl.py already has the upstream
+# fix) and on non-Windows.
+import sys as _patch_sys
+
+if _patch_sys.platform == "win32" and _patch_sys.version_info < (3, 11):
+
+    def _salt_safe_load_windows_store_certs(
+        self, storename, purpose, _SSLError=SSLError
+    ):
+        try:
+            from _ssl import enum_certificates
+        except ImportError:
+            return
+        try:
+            for cert, encoding, trust in enum_certificates(storename):
+                if encoding != "x509_asn":
+                    continue
+                if trust is True or purpose.oid in trust:
+                    try:
+                        self.load_verify_locations(cadata=cert)
+                    except _SSLError:
+                        pass
+        except PermissionError:
+            pass
+
+    SSLContext._load_windows_store_certs = _salt_safe_load_windows_store_certs
+"""
+
+
+def main(argv: list[str]) -> int:
+    if len(argv) != 2:
+        print(
+            f"usage: {argv[0]} <path-to-ssl.py>",
+            file=sys.stderr,
+        )
+        return 2
+    path = argv[1]
+    with open(path, encoding="utf-8") as fh:
+        contents = fh.read()
+    if MARKER in contents:
+        print(f"{path}: already patched, leaving alone")
+        return 0
+    with open(path, "a", encoding="utf-8") as fh:
+        fh.write(PATCH_BODY)
+    print(f"{path}: appended cpython#104135 work-around")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main(sys.argv))

cicd/windows-ssl-104135-sitecustomize.py

@@ -25,8 +25,8 @@
 #
 # Companion work-arounds (delete together with this block):
 #   - salt/ext/tornado/netutil.py: certifi.where() pin on Windows
-#   - cicd/windows-ssl-104135-sitecustomize.py + the Apply-sitecustomize step
-#     in .github/workflows/build-deps-ci-action.yml's Windows job, which
+#   - cicd/windows-ssl-104135-patch.py + the Patch-Lib/ssl.py step in
+#     .github/workflows/build-deps-ci-action.yml's Windows job, which
 #     re-applies this same patch to the onedir Python *before* salt is
 #     importable (covers pip during the CI-Deps step).
 if sys.platform == "win32":

salt/__init__.py

@@ -289,8 +289,8 @@ def match_hostname(cert, hostname):
         #
         # Companion work-arounds (delete together with this block):
         #   - salt/__init__.py: _load_windows_store_certs monkey-patch
-        #   - cicd/windows-ssl-104135-sitecustomize.py + the Apply-
-        #     sitecustomize step in build-deps-ci-action.yml's Windows job.
+        #   - cicd/windows-ssl-104135-patch.py + the Patch-Lib/ssl.py step
+        #     in build-deps-ci-action.yml's Windows job.
         if sys.platform == 'win32' and certifi is not None:
             _client_ssl_defaults = ssl.create_default_context(
                 ssl.Purpose.SERVER_AUTH, cafile=certifi.where())