Skip to content

Commit 1e41626

Browse files
kshitij-microsoftscbedd
kshitij-microsoft
and
scbedd
authored
[TEST] Test pypy311 CI issue (#46916)
* test pypy311 CI issue * probe: enable PYTHONFAULTHANDLER in dispatch_checks subprocesses to capture C-level traceback for PyPy SIGABRT * probe: invoke pytest via python -X faulthandler -X dev to capture C-level traceback for PyPy SIGABRT * probe: add -X importtime -u + PYTHONUNBUFFERED to identify crashing import on PyPy * cleanup: untrack unrelated files committed by mistake * probe: wrap pytest in bash to tee stderr to file and cat after, ensuring faulthandler/importtime output survives SIGABRT * probe: incremental import bisect to find which import triggers PyPy SIGABRT (interpreter starts but emits zero output before abort) * ci(pypy): pin PyPy 3.11 to v7.3.22 to work around RPython AssertionError in 7.3.21 astcompiler PyPy 7.3.21 (currently in the Azure hosted toolcache for Python 3.11.15) crashes during AST compilation of azure.ai.ml with: Fatal RPython error: AssertionError should be unreachable! please report a bug about the astcompiler _archive_or_restore ...PythonCodeMaker_duplicate_exits_without_lineno PyPy 7.3.22 (same Python 3.11.15 language version) does not exhibit the bug. This change overlays the 7.3.22 build into the toolcache before UsePythonVersion@0 resolves pypy3.11 in CI. Also reverts the temporary diagnostic changes to dispatch_checks.py and azpysdk/install_and_test.py that were used to bisect the failure, and adds sdk/ml/azure-ai-ml/pypy_astcompiler_repro.py describing the bug and reproduction steps for an upstream PyPy issue. * ci(pypy): bootstrap pip via ensurepip after overlaying PyPy 7.3.22 tarball * ci(pypy): gate PyPy 7.3.22 overlay behind OverridePyPyVersion variable; enable it only for ubuntu2404_pypy311 matrix entry so non-pypy linux jobs are unaffected * ci(pypy): make PyPy override step matrix-driven via PyPyBuildVersion / PyPyLangVersion variables instead of hardcoding 7.3.22 / 3.11.15 in the step * cleanup: revert seed README change and remove pypy reproducer file * limit blast radius. this is going to hit issues with CFSClean most likely regardless though. It'll unblock main for the ml team though --------- Co-authored-by: Scott Beddall <scbedd@microsoft.com>
1 parent 3dad2eb commit 1e41626

3 files changed

Lines changed: 135 additions & 0 deletions

File tree

‎eng/pipelines/templates/steps/use-python-version.yml‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,60 @@ steps:
1212
}
1313
displayName: 'Use Public Feed'
1414
15+
# Optional override for the hosted toolcache PyPy build. Activated when the
16+
# matrix entry sets:
17+
# OverridePyPyVersion: "true"
18+
# PyPyBuildVersion: "v7.3.22" (or any release from downloads.python.org/pypy)
19+
# PyPyLangVersion: "3.11.15" (the Python language version directory in the toolcache)
20+
# Rationale: PyPy 7.3.21 (currently in the hosted toolcache for Python 3.11.15)
21+
# crashes with `Fatal RPython error: AssertionError` from
22+
# `PythonCodeMaker_duplicate_exits_without_lineno` while compiling
23+
# `azure.ai.ml`. PyPy 7.3.22+ does not exhibit this bug.
24+
- bash: |
25+
set -euo pipefail
26+
PYPY_BUILD="${PYPY_BUILD_VERSION:?PyPyBuildVersion must be set in the matrix when OverridePyPyVersion=true}"
27+
PY_LANG="${PYPY_LANG_VERSION:?PyPyLangVersion must be set in the matrix when OverridePyPyVersion=true}"
28+
TARGET="/opt/hostedtoolcache/PyPy/${PY_LANG}/x64"
29+
WANT_MICRO="${PYPY_BUILD#v}"
30+
if [ -x "${TARGET}/bin/python3" ]; then
31+
CURRENT=$("${TARGET}/bin/python3" -c 'import sys; v=sys.pypy_version_info; print(f"{v.major}.{v.minor}.{v.micro}")') || CURRENT="unknown"
32+
echo "Existing toolcache PyPy: ${CURRENT}"
33+
if [ "$CURRENT" = "$WANT_MICRO" ]; then
34+
echo "PyPy ${WANT_MICRO} already installed; nothing to do."
35+
exit 0
36+
fi
37+
fi
38+
echo "Installing PyPy ${PYPY_BUILD} for Python ${PY_LANG} into ${TARGET}..."
39+
TARBALL="pypy3.11-${PYPY_BUILD}-linux64.tar.bz2"
40+
cd /tmp
41+
curl -sSLO "https://downloads.python.org/pypy/${TARBALL}"
42+
tar xjf "${TARBALL}"
43+
sudo rm -rf "${TARGET}"
44+
sudo mkdir -p "$(dirname "${TARGET}")"
45+
sudo mv "pypy3.11-${PYPY_BUILD}-linux64" "${TARGET}"
46+
# Recreate the python3/python symlinks that the upstream toolcache layout provides.
47+
if [ ! -e "${TARGET}/bin/python3" ]; then
48+
sudo ln -sf "${TARGET}/bin/pypy3.11" "${TARGET}/bin/python3"
49+
fi
50+
if [ ! -e "${TARGET}/bin/python" ]; then
51+
sudo ln -sf "${TARGET}/bin/pypy3.11" "${TARGET}/bin/python"
52+
fi
53+
sudo touch "${TARGET}.complete"
54+
"${TARGET}/bin/python3" --version
55+
# The plain pypy tarball does not include pip; the Azure hosted
56+
# toolcache build does. Bootstrap pip via ensurepip so downstream
57+
# steps (e.g. `python -m pip install uv`) keep working.
58+
if ! "${TARGET}/bin/python3" -c "import pip" >/dev/null 2>&1; then
59+
echo "Bootstrapping pip via ensurepip..."
60+
sudo "${TARGET}/bin/python3" -m ensurepip --upgrade
61+
fi
62+
"${TARGET}/bin/python3" -m pip --version
63+
env:
64+
PYPY_BUILD_VERSION: $(PyPyBuildVersion)
65+
PYPY_LANG_VERSION: $(PyPyLangVersion)
66+
displayName: 'Override hosted toolcache PyPy build'
67+
condition: and(succeeded(), eq(variables['Agent.OS'], 'Linux'), eq(variables['OverridePyPyVersion'], 'true'))
68+
1569
- task: UsePythonVersion@0
1670
displayName: "Use Python ${{ parameters.versionSpec }}"
1771
inputs:

‎sdk/ml/ci.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,11 @@ extends:
3232
ServiceDirectory: ml
3333
TestTimeoutInMinutes: 75
3434
TestProxy: true
35+
MatrixConfigs:
36+
- Name: Python_ml_ci_matrix
37+
Path: sdk/ml/platform-matrix.json
38+
Selection: sparse
39+
GenerateVMJobs: true
3540
# This is a short term solution to create API review for python azure-ml package only when running pipeline manually
3641
# Long term solution should be to have different versions on main branch and release branch for python package so APIView can have different revisions for each version.
3742
# Tracking issue: https://github.com/Azure/azure-sdk-for-python/issues/29196

‎sdk/ml/platform-matrix.json‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
{
2+
"displayNames": {
3+
"--disablecov": "",
4+
"false": "",
5+
"true": ""
6+
},
7+
"matrix": {
8+
"Agent": {
9+
"ubuntu-24.04": { "OSVmImage": "env:LINUXVMIMAGE", "Pool": "env:LINUXPOOL" },
10+
"windows-2022": { "OSVmImage": "env:WINDOWSVMIMAGE", "Pool": "env:WINDOWSPOOL" }
11+
},
12+
"PythonVersion": [ "3.10", "3.12" ],
13+
"CoverageArg": "--disablecov",
14+
"TestSamples": "false"
15+
},
16+
"include": [
17+
{
18+
"MacTestConfig": {
19+
"macos311": {
20+
"OSVmImage": "env:MACVMIMAGE",
21+
"Pool": "env:MACPOOL",
22+
"PythonVersion": "3.11",
23+
"CoverageArg": "--disablecov",
24+
"TestSamples": "false"
25+
}
26+
}
27+
},
28+
{
29+
"PyPyConfig": {
30+
"ubuntu2404_pypy311": {
31+
"OSVmImage": "env:LINUXVMIMAGE",
32+
"Pool": "env:LINUXPOOL",
33+
"PythonVersion": "pypy3.11",
34+
"CoverageArg": "--disablecov",
35+
"TestSamples": "false",
36+
"OverridePyPyVersion": "true",
37+
"PyPyBuildVersion": "v7.3.22",
38+
"PyPyLangVersion": "3.11.15"
39+
}
40+
}
41+
},
42+
{
43+
"CoverageConfig": {
44+
"ubuntu2404_310_coverage": {
45+
"OSVmImage": "env:LINUXVMIMAGE",
46+
"Pool": "env:LINUXPOOL",
47+
"PythonVersion": "3.10",
48+
"CoverageArg": "",
49+
"TestSamples": "false"
50+
}
51+
}
52+
},
53+
{
54+
"Config": {
55+
"Ubuntu2404_313": {
56+
"OSVmImage": "env:LINUXVMIMAGE",
57+
"Pool": "env:LINUXPOOL",
58+
"PythonVersion": "3.13",
59+
"CoverageArg": "--disablecov",
60+
"TestSamples": "false"
61+
}
62+
}
63+
},
64+
{
65+
"Config": {
66+
"Ubuntu2404_314": {
67+
"OSVmImage": "env:LINUXVMIMAGE",
68+
"Pool": "env:LINUXPOOL",
69+
"PythonVersion": "3.14",
70+
"CoverageArg": "--disablecov",
71+
"TestSamples": "false"
72+
}
73+
}
74+
}
75+
]
76+
}

0 commit comments

Comments
 (0)