Skip to content

Commit f753fca

Browse files
MarkCallowMarkCallow
authored
Restore rotted bits: update vcpkg caching and pyktx to latest Python (#1033)
The previously recommended way of caching vcpkg packages has been withdrawn thanks to GitHub replacing the API vcpkg was using with an API that does not provide the previous functionality. This PR updates the CI runs to use the new recommended way, which is via NuGet. It also changes the Linux CI to use vcpkg for dependencies. Some older versions of Python will soon be removed from the CI runners so this PR updates the builds to use the latest which is Python 3.13. Making this work necessitated updating the version of CFFI, which is a dependency for building pyktx, to 1.17.1. Changes the Default and Clang jobs in the Linux CI to use `ubuntu-latest` (24.04) so that gets some testing. Release packages are still built using 22.04 runners. Making 24.04 builds work required changing the packages need for libgl to drop a removed package. Exposes the PY_USE_VENV build option in the .yml files so it can be set should the runner's Python require it. Only the macOS runner has it ON. While working on this, the VS 2022 MSVC compiler was updated to 14.44 which introduces an incompatible operands warning when different enums are the operands of, e.g, an OR. In order to get a green build prior to merge it was necessary to fix the 3 lines of code in `lib/texture2.c` that were doing this and include the fix here. Fixes a poorly formatted error message that was seen in `texturetests.cc` while fixing the enum issue.
1 parent b2f4da2 commit f753fca

10 files changed

Lines changed: 239 additions & 60 deletions

File tree

‎.github/workflows/linux.yml‎

Lines changed: 74 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ jobs:
4848
matrix:
4949
include:
5050
- name: Default
51-
os: ubuntu-22.04
51+
os: ubuntu-latest
5252
generator: Ninja
5353
arch: x86_64
5454
compiler: gcc
@@ -59,7 +59,7 @@ jobs:
5959
sse: ON, opencl: OFF
6060
}
6161
- name: Clang
62-
os: ubuntu-22.04
62+
os: ubuntu-latest
6363
generator: Ninja Multi-Config
6464
arch: x86_64
6565
compiler: clang
@@ -86,6 +86,7 @@ jobs:
8686
arch: aarch64
8787
compiler: gcc
8888
vk_sdk_ver: '1.3.290'
89+
#vcpkg_install_options: --debug
8990
options: {
9091
config: Release,
9192
doc: ON, jni: ON, loadtests: OpenGL, py: ON, tools: ON, tools_cts: ON,
@@ -149,12 +150,15 @@ jobs:
149150
PACKAGE: ${{ matrix.options.package && matrix.options.package || 'NO' }}
150151
SUPPORT_OPENCL: ${{ matrix.options.opencl }}
151152
SUPPORT_SSE: ${{ matrix.options.sse }}
153+
PY_USE_VENV: ${{ matrix.options.py.use_venv && matrix.options.py_use_venv || 'OFF' }}
152154
WERROR: ${{ matrix.options.werror && matrix.options.werror || 'ON' }}
153155
# CC Handled by job step
154156
# CXX Handled by job step
155157

156158
PYTHON_DIST_DIR: interface/python_binding/dist
157159

160+
VCPKG_INSTALL_OPTIONS: ${{ matrix.vcpkg_install_options && matrix.vcpkg_install_options || '' }}
161+
158162
VULKAN_SDK_VER: ${{ matrix.vk_sdk_ver }}
159163

160164
steps:
@@ -191,11 +195,29 @@ jobs:
191195
echo "JAVA_HOME=$JAVA_HOME_17_X64" >> $GITHUB_ENV
192196
./scripts/install_linux.sh
193197
194-
- name: Set up Python 3.11
195-
if: matrix.options.py == 'ON'
196-
uses: actions/setup-python@v5
197-
with:
198-
python-version: '3.11.4'
198+
- name: Install libldtl and mono
199+
if: matrix.options.loadtests != 'OFF' && (matrix.os == 'ubuntu-24.04' || matrix.os == 'ubuntu-latest')
200+
# These packages were omitted from ubuntu-24.04 for some reason.
201+
# libldtl is needed by libxcrypt which is needed by vcpkg.
202+
# mono is needed for adding NuGet sources for caching.
203+
run: sudo apt-get install mono-complete libltdl-dev
204+
205+
# Should a specific version of Python ever be wanted, use this.
206+
# - name: Set up Python
207+
# if: matrix.options.py == 'ON' && matrix.check_mkvk != 'ONLY'
208+
# uses: actions/setup-python@v5
209+
# with:
210+
# python-version: '3.13'
211+
212+
# If the selected Python does not have virtualenv installed, use this.
213+
#
214+
# Note that the builds, via sdist and wheel, require a virtual env.
215+
# regardless of whether the in-use Python is externally managed.
216+
# Installing the explicit dependencies of pyktx only needs virtualenv
217+
# and PY_USE_VENV: ON when Python is externally managed.
218+
# - name: Install Python virtualenv
219+
# if: matrix.options.py == 'ON' && matrix.check_mkvk != 'ONLY'
220+
# run: pip install virtualenv
199221

200222
- name: before_script
201223
run: |
@@ -206,7 +228,52 @@ jobs:
206228
./install-gitconfig.sh
207229
scripts/smudge_date.sh
208230
231+
- name: Export environment variables for vcpkg binary caching
232+
if: matrix.options.loadtests != 'OFF'
233+
# actions/github-script is used primarily because it is the only
234+
# way to access VCPKG_INSTALLATION_ROOT outside of the runner's shell.
235+
# Maintaining cross-platform access is desirable so the code can be
236+
# easily copied to a multi-platform workflow. NUGET_FEED_URL is set
237+
# here to keep the vcpkg cache environment variables together.
238+
uses: actions/github-script@v7
239+
with:
240+
# JSON in use here. #<comments> in the script will break it.
241+
# Pre-installed vcpkg location is indicated by a non-standard env.
242+
# var.
243+
script: |
244+
core.exportVariable('VCPKG_ROOT', process.env.VCPKG_INSTALLATION_ROOT || '');
245+
core.exportVariable('VCPKG_EXE', process.env.VCPKG_INSTALLATION_ROOT + '/vcpkg' || '');
246+
core.exportVariable('NUGET_FEED_URL', 'https://nuget.pkg.github.com/KhronosGroup/index.json' || '');
247+
248+
- name: Overwrite /usr/bin/mono
249+
# nuget downloaded by vcpkg fetch on aarch64 is a shell script. This
250+
# prevents vcpkg from trying and failing, to run it in mono. This is
251+
# my ugliest hack in 45 years of software development.
252+
if: matrix.options.loadtests != 'OFF' && matrix.arch == 'aarch64'
253+
run: sudo sh -c "mv /usr/bin/mono /usr/bin/mono-real && echo '#! /usr/bin/env bash\nif [ \"\$1\" = \"/usr/bin/nuget\" ]; then \$*; else /usr/bin/mono-real \$*; fi' > /usr/bin/mono && chmod 755 /usr/bin/mono"
254+
255+
- name: Add NuGet sources for vcpkg binary caching
256+
if: matrix.options.loadtests != 'OFF'
257+
env:
258+
USERNAME: KhronosGroup
259+
shell: bash
260+
# `vcpkg fetch nuget` fetches nuget and outputs the location of
261+
# the fetched command. It is a .exe that runs in the mono VM.
262+
run: |
263+
mono `${{ env.VCPKG_EXE }} fetch nuget | tail -n 1` \
264+
sources add \
265+
-Source "${{ env.NUGET_FEED_URL }}" \
266+
-StorePasswordInClearText \
267+
-Name GitHubPackages \
268+
-UserName "${{ env.USERNAME }}" \
269+
-Password "${{ secrets.GH_PACKAGES_TOKEN }}"
270+
mono `${{ env.VCPKG_EXE }} fetch nuget | tail -n 1` \
271+
setapikey "${{ secrets.GH_PACKAGES_TOKEN }}" \
272+
-Source "${{ env.NUGET_FEED_URL }}"
273+
209274
- name: script
275+
env:
276+
VCPKG_BINARY_SOURCES: clear;nuget,${{ env.NUGET_FEED_URL }},readwrite
210277
run: |
211278
./scripts/build_linux.sh
212279

‎.github/workflows/macos.yml‎

Lines changed: 58 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -147,12 +147,16 @@ jobs:
147147
SUPPORT_OPENCL: ${{ matrix.options.opencl }}
148148
SUPPORT_SSE: ${{ matrix.options.sse }}
149149
WASM_BUILD: ${{ matrix.options.wasm && matrix.options.wasm || 'OFF' }}
150+
# Virtual env (ON) is needed because pre-installed python is "externally managed."
151+
PY_USE_VENV: ${{ matrix.options.py.use_venv && matrix.options.py_use_venv || 'ON' }}
150152
WERROR: ${{ matrix.options.werror && matrix.options.werror || 'ON' }}
151153
# CC Handled by job step
152154
# CXX Handled by job step
153155

154156
PYTHON_DIST_DIR: interface/python_binding/dist
155157

158+
VCPKG_INSTALL_OPTIONS: ${{ matrix.vcpkg_install_options && matrix.vcpkg_install_options || '' }}
159+
156160
VULKAN_SDK_VER: ${{ matrix.vk_sdk_ver }}
157161
# This is just to tell the Vulkan install script where to install.
158162
# NOTE: Hardcoded $HOME expansion, due to 'env' context being unavailble in env:
@@ -180,23 +184,6 @@ jobs:
180184
echo -n "CMake version on the runner is " && cmake --version
181185
echo -e "machine github.com\n login $GITHUB_TOKEN" >> ~/.netrc # Prevent rate limiting on Git LFS.
182186
cat ~/.netrc
183-
if [[ -n "$FEATURE_LOADTESTS" && "$FEATURE_LOADTESTS" != "OFF" ]]; then
184-
# Install vcpkg.
185-
pushd $HOME
186-
git clone https://github.com/microsoft/vcpkg
187-
cd vcpkg && ./bootstrap-vcpkg.sh -disableMetrics && echo "VCPKG_ROOT=`pwd`" >> $GITHUB_ENV
188-
popd
189-
fi
190-
191-
- name: Export GitHub Actions cache and vcpkg environment variables
192-
if: matrix.options.loadtests != 'OFF'
193-
uses: actions/github-script@v7
194-
with:
195-
# Caution. JSON in use. "#core" breaks the build.
196-
script: |
197-
core.exportVariable('ACTIONS_CACHE_URL', process.env.ACTIONS_CACHE_URL || '');
198-
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
199-
core.exportVariable('VCPKG_BINARY_SOURCES', "clear;x-gha,readwrite");
200187
201188
- name: Install Doxygen
202189
if: matrix.options.doc == 'ON'
@@ -209,11 +196,59 @@ jobs:
209196
- name: Run install script
210197
run: ./scripts/install_macos.sh
211198

212-
- name: Set up Python 3.11
213-
if: matrix.options.py == 'ON'
214-
uses: actions/setup-python@v5
199+
# Should a specific version of Python ever be wanted, use this.
200+
# - name: Set up Python
201+
# if: matrix.options.py == 'ON' && matrix.check_mkvk != 'ONLY'
202+
# uses: actions/setup-python@v5
203+
# with:
204+
# python-version: '3.13'
205+
206+
# If the selected Python does not have virtualenv installed, use this.
207+
# It is included currently in the runner's pre-installed Pythons.
208+
#
209+
# Note that the builds, via sdist and wheel, require a virtual env.
210+
# regardless of whether the in-use Python is externally managed.
211+
# Installing the explicit dependencies of pyktx only needs virtualenv
212+
# and PY_USE_VENV: ON when Python is externally managed.
213+
# - name: Install Python virtualenv
214+
# if: matrix.options.py == 'ON' && matrix.check_mkvk != 'ONLY'
215+
# run: pip install virtualenv
216+
217+
- name: Export environment variables for vcpkg binary caching
218+
if: matrix.options.loadtests != 'OFF'
219+
# actions/github-script is used primarily because it is the only
220+
# way to access VCPKG_INSTALLATION_ROOT outside of the runner's shell.
221+
# Maintaining cross-platform access is desirable so the code can be
222+
# easily copied to a multi-platform workflow. NUGET_FEED_URL is set
223+
# here to keep the vcpkg cache environment variables together.
224+
uses: actions/github-script@v7
215225
with:
216-
python-version: '3.11.4'
226+
# JSON in use here. #<comments> in the script will break it.
227+
# Pre-installed vcpkg location is indicated by a non-standard env.
228+
# var.
229+
script: |
230+
core.exportVariable('VCPKG_ROOT', process.env.VCPKG_INSTALLATION_ROOT || '');
231+
core.exportVariable('VCPKG_EXE', process.env.VCPKG_INSTALLATION_ROOT + '/vcpkg' || '');
232+
core.exportVariable('NUGET_FEED_URL', 'https://nuget.pkg.github.com/KhronosGroup/index.json' || '');
233+
234+
- name: Add NuGet sources for vcpkg binary caching
235+
if: matrix.options.loadtests != 'OFF'
236+
env:
237+
USERNAME: KhronosGroup
238+
shell: bash
239+
# `vcpkg fetch nuget` fetches nuget and outputs the location of
240+
# the fetched command. It is a .exe that runs in the mono VM.
241+
run: |
242+
mono `${{ env.VCPKG_EXE }} fetch nuget | tail -n 1` \
243+
sources add \
244+
-Source "${{ env.NUGET_FEED_URL }}" \
245+
-StorePasswordInClearText \
246+
-Name GitHubPackages \
247+
-UserName "${{ env.USERNAME }}" \
248+
-Password "${{ secrets.GH_PACKAGES_TOKEN }}"
249+
mono `${{ env.VCPKG_EXE }} fetch nuget | tail -n 1` \
250+
setapikey "${{ secrets.GH_PACKAGES_TOKEN }}" \
251+
-Source "${{ env.NUGET_FEED_URL }}"
217252
218253
- name: before_script
219254
run: |
@@ -225,6 +260,8 @@ jobs:
225260
scripts/smudge_date.sh
226261
227262
- name: script
263+
env:
264+
VCPKG_BINARY_SOURCES: clear;nuget,${{ env.NUGET_FEED_URL }},readwrite
228265
run: |
229266
set -e # Instead of trying to chain everything together with &&
230267
# otherwise subsequent commands will swallow bad exit status.

‎.github/workflows/windows.yml‎

Lines changed: 60 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@ jobs:
8484
generator: 'Visual Studio 17 2022'
8585
toolset: CLangCL
8686
arch: arm64
87+
#vcpkg_install_options: --debug
8788
options: {
8889
config: 'Debug,Release',
8990
doc: ON, jni: ON, loadtests: OpenGL+Vulkan, py: OFF, tests: ON, tools: ON, tools_cts: ON,
@@ -114,8 +115,15 @@ jobs:
114115
PACKAGE: ${{ matrix.options.package }}
115116
SUPPORT_OPENCL: ${{ matrix.options.opencl }}
116117
SUPPORT_SSE: ${{ matrix.options.sse }}
118+
PY_USE_VENV: ${{ matrix.options.py.use_venv && matrix.options.py_use_venv || 'OFF' }}
117119
WERROR: ON
118120

121+
VCPKG_INSTALL_OPTIONS: ${{ matrix.vcpkg_install_options && matrix.vcpkg_install_options || '' }}
122+
# This is set to avoid the compiler running out of heap space
123+
# when vcpkg is compiling glew. Default is logical cores + 1.
124+
# Runners have 1 thread per core and 2 cores.
125+
VCPKG_MAX_CONCURRENCY: 2
126+
119127
CODE_SIGN_TIMESTAMP_URL: "http://timestamp.digicert.com"
120128
AZURE_KEY_VAULT_URL: ${{ secrets.AZURE_KEY_VAULT_URL }}
121129
AZURE_KEY_VAULT_CERTIFICATE: ${{ secrets.AZURE_KEY_VAULT_CERTIFICATE }}
@@ -128,12 +136,7 @@ jobs:
128136
OPENGL_ES_EMULATOR: C:/Imagination/Windows_x86_64
129137
OPENGL_ES_EMULATOR_WIN: C:\Imagination\Windows_x86_64
130138
PVR_SDK_HOME: https://github.com/powervr-graphics/Native_SDK/raw/master/lib/Windows_x86_64/
131-
PYTHON: C:\hostedtoolcache\windows\Python\3.11.4\x64\python.exe
132139
PYTHON_DIST_DIR: interface/python_binding/dist
133-
# This is set to avoid the compiler running out of heap space
134-
# when vcpkg is compiling glew. Default is logical cores + 1.
135-
# Runners have 1 thread per core and 2 cores.
136-
VCPKG_MAX_CONCURRENCY: 2
137140
VULKAN_SDK_VER: 1.3.290.0
138141

139142
steps:
@@ -188,16 +191,41 @@ jobs:
188191
echo "CODE_SIGN_KEY_VAULT=Azure" >> $env:GITHUB_ENV
189192
}
190193
191-
- name: Export GitHub Actions cache and vcpkg environment variables
194+
- name: Export environment variables for vcpkg binary caching
192195
if: matrix.options.loadtests != 'OFF'
196+
# actions/github-script is used primarily because it is the only
197+
# way to access VCPKG_INSTALLATION_ROOT outside of the runner's shell.
198+
# Maintaining cross-platform access is desirable so the code can be
199+
# easily copied to a multi-platform workflow. NUGET_FEED_URL is set
200+
# here to keep the vcpkg cache environment variables together.
193201
uses: actions/github-script@v7
194202
with:
195-
# Caution. JSON in use. "#core" breaks the build.
203+
# JSON in use here. #<comments> in the script will break it.
204+
# Pre-installed vcpkg location is indicated by a non-standard env.
205+
# var.
196206
script: |
197-
core.exportVariable('ACTIONS_CACHE_URL', process.env.ACTIONS_CACHE_URL || '');
198-
core.exportVariable('ACTIONS_RUNTIME_TOKEN', process.env.ACTIONS_RUNTIME_TOKEN || '');
199-
core.exportVariable('VCPKG_BINARY_SOURCES', "clear;x-gha,readwrite");
200207
core.exportVariable('VCPKG_ROOT', process.env.VCPKG_INSTALLATION_ROOT || '');
208+
core.exportVariable('VCPKG_EXE', process.env.VCPKG_INSTALLATION_ROOT + '/vcpkg' || '');
209+
core.exportVariable('NUGET_FEED_URL', 'https://nuget.pkg.github.com/KhronosGroup/index.json' || '');
210+
211+
- name: Add NuGet sources for vcpkg binary caching
212+
if: matrix.options.loadtests != 'OFF'
213+
env:
214+
USERNAME: KhronosGroup
215+
shell: pwsh
216+
# `vcpkg fetch nuget` fetches nuget and outputs the location of
217+
# the fetched command.
218+
run: |
219+
.$(${{ env.VCPKG_EXE }} fetch nuget) `
220+
sources add `
221+
-Source "${{ env.NUGET_FEED_URL }}" `
222+
-StorePasswordInClearText `
223+
-Name GitHubPackages `
224+
-UserName "${{ env.USERNAME }}" `
225+
-Password "${{ secrets.GH_PACKAGES_TOKEN }}"
226+
.$(${{ env.VCPKG_EXE }} fetch nuget) `
227+
setapikey "${{ secrets.GH_PACKAGES_TOKEN }}" `
228+
-Source "${{ env.NUGET_FEED_URL }}"
201229
202230
- name: Install Dependencies
203231
if: matrix.check_mkvk != 'ONLY'
@@ -211,18 +239,31 @@ jobs:
211239
distribution: 'temurin'
212240
java-version: '17'
213241

214-
- name: Set up Python 3.11
242+
# Pre-installed Python in Windows Server 2022 is 3.9. However
243+
# CMake finds the latest on the runner (3.13.5 now). Should a
244+
# a specific version of Python ever be wanted, use this.
245+
# A bug in 3.13.4 for Windows
246+
# (https://github.com/python/cpython/issues/135151) means it looks
247+
# like a free-threaded (whatever that means) version to CPython leading
248+
# to an error that the "limited API" (whatever that is) is not supported
249+
# by free-threaded Python. This is fixed in 3.13.5. Not all the runners
250+
# have been updated (see https://github.com/actions/runner-images/issues/12377).
251+
# Install a version that works.
252+
- name: Set up Python 3.12
215253
if: matrix.options.py == 'ON' && matrix.check_mkvk != 'ONLY'
216254
uses: actions/setup-python@v5
217255
with:
218-
python-version: '3.11.4'
256+
python-version: '3.12'
219257

220258
# virtualenv is no longer included as of 20240929.1 runner so
221259
# install it ourselves. See
222260
# https://github.com/actions/runner-images/issues/10749.
223261
#
224-
# Note that it is the builds of sdist and wheel that require virtualenv
225-
# not the explicit dependencies of pyktx.
262+
# Note that the builds, via sdist and wheel, require a virtual env.
263+
# regardless of whether the in-use Python is externally managed.
264+
# Installing the explicit dependencies of pyktx only needs virtualenv
265+
# and PY_USE_VENV: ON when Python is externally managed. The packages
266+
# on the Windows runners are not externally managed.
226267
- name: Install Python virtualenv
227268
if: matrix.options.py == 'ON' && matrix.check_mkvk != 'ONLY'
228269
run: pip install virtualenv
@@ -240,6 +281,8 @@ jobs:
240281
241282
- name: Build Windows
242283
if: matrix.check_mkvk != 'ONLY'
284+
env:
285+
VCPKG_BINARY_SOURCES: clear;nuget,${{ env.NUGET_FEED_URL }},readwrite
243286
# The installers run as part of "Install Dependencies" add
244287
# environment variables to the registry and augment $Path there.
245288
# Although each step gets a new Powershell instance, that instance
@@ -269,9 +312,9 @@ jobs:
269312
if: matrix.arch == 'x64' && matrix.options.tests == 'ON'
270313
run: ctest --output-on-failure --test-dir $env:BUILD_DIR -C Release
271314

272-
- name: Upload test log
273-
if: ${{ failure() }}
274-
run: scripts/on_failure.ps1
315+
# - name: Upload test log
316+
# if: ${{ failure() }}
317+
# run: scripts/on_failure.ps1
275318

276319
- name: Get KTX version
277320
if: matrix.options.package == 'YES'

‎interface/python_binding/requirements.txt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# SPDX-License-Identifier: Apache-2.0
33

44
build==1.0.3
5-
cffi==1.15.1
5+
cffi==1.17.1
66
setuptools==78.1.1
77
wheel>=0.38.4
88
sphinx==6.2.1

0 commit comments

Comments
 (0)