Skip to content

[Packaging] Bump Python to 3.12 on RHEL and CentOS Stream #31264

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bebound merged 3 commits into from
Apr 24, 2025
Merged

[Packaging] Bump Python to 3.12 on RHEL and CentOS Stream #31264

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
5251ae2
Init
bebound Apr 14, 2025
a64cea1
Install pip
bebound Apr 18, 2025
8ae6d82
Remove python3.12-devel, gcc
bebound Apr 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -816,13 +816,13 @@ jobs:
dockerfile: ubi
image: registry.access.redhat.com/ubi8/ubi:8.4
artifact: rpm-ubi8-${{ arch.value }}
python_package: python39
python_package: python3.12
pool: ${{ arch.pool }}
Red Hat Universal Base Image 9 ${{ arch.name }}:
dockerfile: ubi
image: registry.access.redhat.com/ubi9/ubi:9.0.0
artifact: rpm-ubi9-${{ arch.value }}
python_package: python3.9
python_package: python3.12
pool: ${{ arch.pool }}
steps:
- bash: ./scripts/ci/install_docker.sh
Expand Down Expand Up @@ -864,17 +864,17 @@ jobs:
artifact: rpm-ubi8-${{ arch.value }}
distro: el8
image: registry.access.redhat.com/ubi8/ubi:8.4
python_package: python39
python_cmd: python3.9
pip_cmd: pip3.9
python_package: python3.12
python_cmd: python3.12
pip_cmd: pip3.12
pool: ${{ arch.pool }}
Red Hat Universal Base Image 9 ${{ arch.name }}:
artifact: rpm-ubi9-${{ arch.value }}
distro: el9
image: registry.access.redhat.com/ubi9/ubi:9.0.0
python_package: python3.9
python_cmd: python3.9
pip_cmd: pip3.9
python_package: python3.12
python_cmd: python3.12
pip_cmd: pip3.12
pool: ${{ arch.pool }}
steps:
- task: DownloadPipelineArtifact@1
Expand Down
1 change: 1 addition & 0 deletions scripts/release/rpm/azure-cli.spec
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ deactivate

# Fix up %{buildroot} appearing in some files...
for d in %{buildroot}%{cli_lib_dir}/bin/*; do perl -p -i -e "s#%{buildroot}##g" $d; done;
rm %{buildroot}%{cli_lib_dir}/pyvenv.cfg
Copy link
Copy Markdown
Contributor Author

@bebound bebound Apr 18, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el9.x86_64/usr/lib64/az/pyvenv.cfg content is

home = /usr/bin
include-system-site-packages = false
version = 3.12.5
executable = /usr/bin/python3.12
command = /usr/bin/python3.12 -m venv /root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el9.x86_64/usr/lib64/az

It raises this error during build.

#10 83.39 RPM build errors:
#10 83.39 error: Bad exit status from /var/tmp/rpm-tmp.ZBlRUB (%install)
#10 83.39     Bad exit status from /var/tmp/rpm-tmp.ZBlRUB (%install)
#10 ERROR: process "/bin/sh -c export PIP_INDEX_URL=$(cat /run/secrets/PIP_INDEX_URL) &&     dos2unix ./scripts/release/rpm/azure-cli.spec &&     REPO_PATH=$(pwd) CLI_VERSION=$cli_version PYTHON_PACKAGE=$python_package PYTHON_CMD=$python_package     rpmbuild -v -bb --clean scripts/release/rpm/azure-cli.spec &&     cp /root/rpmbuild/RPMS/*/azure-cli-${cli_version}-1.*.rpm /azure-cli-dev.rpm &&     mkdir /out && cp /root/rpmbuild/RPMS/*/azure-cli-${cli_version}-1.*.rpm /out/" did not complete successfully: exit code: 1
------
 > [build-env 6/6] RUN --mount=type=secret,id=PIP_INDEX_URL export PIP_INDEX_URL=$(cat /run/secrets/PIP_INDEX_URL) &&     dos2unix ./scripts/release/rpm/azure-cli.spec &&     REPO_PATH=$(pwd) CLI_VERSION=2.71.0 PYTHON_PACKAGE=python3.12 PYTHON_CMD=python3.12     rpmbuild -v -bb --clean scripts/release/rpm/azure-cli.spec &&     cp /root/rpmbuild/RPMS/*/azure-cli-2.71.0-1.*.rpm /azure-cli-dev.rpm &&     mkdir /out && cp /root/rpmbuild/RPMS/*/azure-cli-2.71.0-1.*.rpm /out/:
82.82 + mkdir -p /root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el8.x86_64/etc/bash_completion.d/
82.82 + cat /azure-cli/az.completion
82.82 + /usr/lib/rpm/check-buildroot
83.38 /root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el8.x86_64/usr/lib64/az/pyvenv.cfg:command = /usr/bin/python3.12 -m venv /root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el8.x86_64/usr/lib64/az
83.38 Found '/root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el8.x86_64' in installed files; aborting
83.39 
83.39 
83.39 RPM build errors:
83.39 error: Bad exit status from /var/tmp/rpm-tmp.ZBlRUB (%install)
83.39     Bad exit status from /var/tmp/rpm-tmp.ZBlRUB (%install)

In 3.9, pyvenv.cfg does not include rpmbuild root:

home = /usr/bin
include-system-site-packages = false
version = 3.9.20

Copy link
Copy Markdown
Member

@jiasli jiasli Apr 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/usr/lib/rpm/check-buildroot wants to make sure the RPM build path /root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el9.x86_64 does not exist in any of the installed file, but pyvenv.cfg contains command = /usr/bin/python3.12 -m venv /root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el9.x86_64/usr/lib64/az which makes /usr/lib/rpm/check-buildroot fail.

  1. Will deleting pyvenv.cfg cause any side effect?
  2. Can we use sed or perl (L60) to only remove command = /usr/bin/python3.12 -m venv /root/rpmbuild/BUILDROOT/azure-cli-2.71.0-1.el9.x86_64/usr/lib64/az?

Copy link
Copy Markdown
Contributor Author

@bebound bebound Apr 24, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pyvenv.cfg contains some metadata of the venv. We only need the site-pacakges folder in the venv.
This is the RPM entry script:

azure-cli/scripts/release/rpm/azure-cli.spec

Line 79 in 81de68b

AZ_INSTALLER=RPM PYTHONPATH=\"\$bin_dir/../lib64/az/lib/${python_version}/site-packages\" \$python_cmd -sm azure.cli \"\$@\"

I thinks it's fine to remove it. As a proof, we have explicitly excluded the bin folder created by venv:

azure-cli/scripts/release/rpm/azure-cli.spec

Line 88 in 81de68b

%exclude %{cli_lib_dir}/bin/

PS: Although {cli_lib_dir}/bin/ is excluded in RPM, it still checks whether this path contains hard-coded buildroot during the build. Line 60 is still necessary.


# Create executable (entry script)

Expand Down
2 changes: 1 addition & 1 deletion scripts/release/rpm/test_rpm_in_docker.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ export USERNAME=azureuser

dnf --nogpgcheck install /mnt/rpm/$RPM_NAME -y

dnf install git gcc $PYTHON_PACKAGE-devel findutils -y
dnf install git findutils $PYTHON_PACKAGE-pip -y

ln -s -f /usr/bin/$PYTHON_CMD /usr/bin/python
ln -s -f /usr/bin/$PIP_CMD /usr/bin/pip
Expand Down
4 changes: 2 additions & 2 deletions scripts/release/rpm/ubi.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ ARG image=registry.access.redhat.com/ubi8/ubi:8.4

FROM ${image} AS build-env
ARG cli_version=dev
ARG python_package=python39
ARG python_package=python3.12

RUN yum update -y
RUN yum install -y wget rpm-build gcc libffi-devel ${python_package}-devel openssl-devel make bash diffutils patch dos2unix perl
Expand All @@ -18,7 +18,7 @@ COPY . .
# We have to explicitly specify 'python39' to install Python 3.9.
RUN --mount=type=secret,id=PIP_INDEX_URL export PIP_INDEX_URL=$(cat /run/secrets/PIP_INDEX_URL) && \
dos2unix ./scripts/release/rpm/azure-cli.spec && \
REPO_PATH=$(pwd) CLI_VERSION=$cli_version PYTHON_PACKAGE=$python_package PYTHON_CMD=python3.9 \
REPO_PATH=$(pwd) CLI_VERSION=$cli_version PYTHON_PACKAGE=$python_package PYTHON_CMD=$python_package \
Copy link
Copy Markdown
Contributor Author

@bebound bebound Apr 23, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In RHEL8, the PYTHON_PACKAGE is python39, PYTHON_CMD is python3.9.
In RHEL9, both are unified as python3.9. Ref: https://github.com/Azure/azure-cli/pull/23556/files#r952101959

In python3.12, both distros use python3.12.

I'm unsure if these values will differ in the future; keep these variables for now.

rpmbuild -v -bb --clean scripts/release/rpm/azure-cli.spec && \
cp /root/rpmbuild/RPMS/*/azure-cli-${cli_version}-1.*.rpm /azure-cli-dev.rpm && \
mkdir /out && cp /root/rpmbuild/RPMS/*/azure-cli-${cli_version}-1.*.rpm /out/
Expand Down