GH-148112: Add configure.py, a configure.ac replacement#146034
Draft
nascheme wants to merge 98 commits intopython:mainfrom
Draft
GH-148112: Add configure.py, a configure.ac replacement#146034nascheme wants to merge 98 commits intopython:mainfrom
nascheme wants to merge 98 commits intopython:mainfrom
Conversation
Assisted-by: claude-opus-4-6
Keep srcdir as an absolute path if it was originally. Match host_prefix setting done by configure.ac. Fix AWK implementations of pyconf_env_var(), pyconf_check_header() and pyconf_check_decl().
Match the autoconf macro behavior: set per-package variables (ZLIB_CFLAGS, ZLIB_LIBS, etc.) only when the system is Emscripten and the user hasn't already provided values.
nascheme
changed the title
… pymalloc, checking/result. - Fix module docstring: remove incorrect PROFILE_TASK mention (that lives in conf_optimization.py); expand to list all options the file handles. - Fix setup_assertions: process_bool() emitted checking+result immediately, then the pydebug-implied branch emitted a second checking+result. Replace with a single manual checking()/result() call, matching configure.ac flow. - Fix setup_sanitizers: add v.with_pymalloc = "no" when asan or msan is enabled, matching configure.ac (so setup_pymalloc honours the override). - Fix check_tail_call_interp: add missing checking()/result() calls, including the "no value specified" case when neither --with nor --without is given. - Fix check_remote_debug: add missing checking()/result() calls. Assisted-by: Claude
…alls. pyconf.error() already prepends "configure: error: " so passing messages that start with "error: " produced "configure: error: error: ..." output. Removed the redundant prefix from all seven error calls. Assisted-by: Claude
…alls. - Correct module docstring: ensurepip, d_type, shm_open, and test-modules are handled by other modules; list what this file actually does. - Add pyconf.checking/result for --with-openssl-rpath (matches configure.ac AC_MSG_CHECKING/AC_MSG_RESULT at lines 7544/7575). - Add pyconf.checking/result for unsupported static OpenSSL build (matches configure.ac lines 7582/7595). - Add pyconf.checking/result for --with-ssl-default-suites (matches configure.ac lines 7663/7690). - Add pyconf.checking/result for --with-builtin-hashlib-hashes (matches configure.ac lines 7696/7710). Assisted-by: Claude
Call the pyconf.checking() function explicitly. Remove implicit "checking" output from other pyconf functions. This matches more closely the configure.ac code, even though it is more verbose.
- Fix pyconf.run_check(), run_check_with_cc_flag(), and compile_link_check()
to automatically call checking()/result() when a description is passed,
matching the existing AWK pyconf_run_check/pyconf_compile_link_check
behaviour. Previously configure.py silently omitted "checking whether
pthreads are available without options...", "checking whether CC accepts
-pthread...", and related lines.
- Fix setup_pthreads() fallback: check _POSIX_THREADS in unistd.h before
calling define("_REENTRANT"), matching configure.ac order.
- Fix module docstring typo ("Checked" → "Checks").
- Add docstring to check_posix_semaphores().
- Regenerate configure.awk.
Assisted-by: Claude
…e, and wchar.h double-define.
- check_math_library: move __fpu_control check before --with-libm/--with-libc
to match configure.ac order (AC_CHECK_FUNC at line 6134, before AC_SUBST LIBM)
- detect_libmpdec: have_ipa_pure_const_bug and have_glibc_memmove_bug CFLAGS
appends were inside the bundled-only block; configure.ac applies them
unconditionally regardless of with_system_libmpdec
- check_wchar: use autodefine=False for check_header("wchar.h") to avoid
double-defining HAVE_WCHAR_H (autodefine would define it with a generic
description, then the explicit define overwrote it)
Assisted-by: Claude
… does.
Command-line VAR=VALUE arguments (e.g. ./configure ac_cv_file__dev_ptmx=yes)
were stored only in os.environ, but vars.is_set() only checked vars.__dict__.
This caused two bugs when cross-compiling:
1. is_set("ac_cv_file__dev_ptmx") returned False even when the user had
explicitly set it, triggering a spurious CONFIG_SITE error.
2. A "no" value stayed as a string in os.environ; Python truthiness
(if v.ac_cv_file__dev_ptmx:) would evaluate it as True, incorrectly
defining HAVE_DEV_PTMX.
Fix: in init_args(), after storing in os.environ, also convert the value
via _cache_deserialize() (yes→True, no→False) and setattr on vars, matching
exactly what _load_config_site() does. For ac_cv_* vars, also populate
cache[]. This makes command-line VAR=VALUE args behave consistently with
CONFIG_SITE values, matching AWK where VAR=val populates V[].
Also clear vars.__dict__ user attributes in _reset() so test isolation is
complete. Add TestInitArgsVarValue tests to cover is_set() detection and
bool truthiness for command-line ac_cv_* args.
Found during review of conf_filesystem.py (check_device_files cross-compile
path uses is_set()).
Assisted-by: Claude
- check_network_libs: pass extra_libs=v.LIBS when checking for -lsocket, matching configure.ac's AC_CHECK_LIB 5th arg ($LIBS); on SVR4/Solaris, socket requires -lnsl in the link test. - check_getaddrinfo: replace v.enable_ipv6 != "" (always False, falls back to unset env var) with ENABLE_IPV6.given, which correctly detects whether --enable-ipv6 or --disable-ipv6 was explicitly passed. Assisted-by: Claude
- pyconf.py / pyconf.awk: check_funcs/pyconf_check_funcs now return True/1 if any function was found, matching AC_CHECK_FUNCS action-if-found semantics. Previously returned None/0, causing the backtrace/dladdr1 check in check_base_libraries to never set ac_cv_require_ldl, so -ldl was never added to LDFLAGS for faulthandler support. - conf_syslibs.py: fix module docstring (remove incorrect "pthreads setup", add missing check_stat_timestamps and setup_tzpath descriptions). - conf_syslibs.py: initialize r and libatomic_needed before with-blocks to avoid Pyright possibly-unbound warnings. Assisted-by: Claude
…parsing. Fix docstring (perf trampoline lives in conf_buildopts.py, not here). Add pyconf.is_digit() / pyconf_is_digit() so the deployment target version components can be validated before int() conversion, matching autoconf's silent-failure behaviour when the variable is empty or non-numeric. Register is_digit in the transpiler lint whitelist. Assisted-by: Claude
…ing call.
- setup_universalsdk: use cmd_status instead of cmd_output for xcodebuild
so that a missing or failing xcodebuild is silently ignored, matching
configure.ac's use of 2>/dev/null.
- _setup_darwin_flags: add missing pyconf.checking("which compiler should
be used") to match configure.ac's AC_MSG_CHECKING before the
MacOSX10.4u.sdk case.
Assisted-by: Claude
- _setup_emscripten_flags: add missing ERRNO_CODES to -sEXPORTED_RUNTIME_METHODS, matching configure.ac line 2368. - _setup_wasi_flags: also append -target wasm32-wasi-threads -pthread to CFLAGS (not just CFLAGS_NODIST); the comment explains this is needed so ac_compile/ac_link tests succeed during configure. - setup_wasm_flags: fix ac_cv_func_dlopen logic: use == "no" instead of "is False" so that explicit --enable-wasm-dynamic-linking=no suppresses dlopen (matching configure.ac), and "missing" (no option given) correctly leaves dlopen unprobed. Assisted-by: Claude
…save_env.
- Expand module docstring to list all covered areas (was missing UUID,
readline, compression, and curses).
- Fix Darwin libffi detection: replace path_is_file() with save_env() +
check_header("ffi.h") + check_lib("ffi", "ffi_call"), matching the
WITH_SAVE_ENV + AC_CHECK_HEADER + AC_CHECK_LIB pattern in configure.ac.
The old code skipped the link test entirely.
- Fix detect_uuid pkg-config fallback: replace manual save_CPPFLAGS/save_LIBS
pattern with with pyconf.save_env() for consistency.
Assisted-by: Claude
…otices. - Use pyconf.warn() for AC_MSG_WARN (pkg-config, support tier) instead of warnings.warn(); use pyconf.notice() for AC_MSG_NOTICE (stdatomic.h) instead of print(). - Add missing pyconf.notice() calls for "creating Modules/Setup.local" and "creating Makefile" to match configure.ac lines 8365 and 8371. - Add docstring to generate_output(); remove unused import warnings. Assisted-by: Claude
…blank lines.
Three issues found during cross-platform testing (Ubuntu, FreeBSD, OpenBSD,
NetBSD, WASI):
1. CONFIG_ARGS missing precious env vars and build/host aliases (WASI):
- pyconf.py init_args(): reorder built-in precious vars to match
configure.ac's ac_precious_vars order (PKG_CONFIG before CC); remove
CXX/CXXFLAGS which configure.ac does not declare as precious.
- pyconf.py canonical_host(): insert build_alias/host_alias before the
first NAME=VALUE env-var entry, matching autoconf's arg ordering.
- pyconf.awk pyconf_env_var(): accumulate env vars in _pyconf_env_args
(separate from command-line flags) so ordering can be enforced.
- pyconf.awk pyconf_parse_args(): register built-in precious vars after
the arg loop (matching Python's init_args behaviour); use ordered split
to match ac_precious_vars sequence.
- pyconf.awk _pyconf_resolve_exports(): combine flags + aliases + env vars
in the correct order when building CONFIG_ARGS.
- pyconf.awk pyconf_canonical_host(): record build_alias/host_alias in a
separate _pyconf_alias_args accumulator.
- configure.awk: regenerated.
2. OpenSSL fallback paths wrong (NetBSD):
- conf_security.py: replace ad-hoc include-dir search with the ssldirs
list from AX_CHECK_OPENSSL (/usr/local/ssl, /usr/lib/ssl, /usr/ssl,
/usr/pkg, /usr/local, /usr) and set OPENSSL_INCLUDES/LDFLAGS properly.
3. Spurious blank lines in Makefile.pre (OpenBSD, diff -B unavailable):
- compare-conf.sh: strip blank lines from Makefile.pre before diffing to
work around the m4/_AS_QUOTE expansion artifact that adds extra newlines
to MODULE_BLOCK entries when values are shell variable references.
Assisted-by: Claude
Avoid restoring variables we just set.
nascheme
added
the
🔨 test-with-buildbots
|
🤖 New build scheduled with the buildbot fleet by @nascheme for commit 9d1c39f 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F146034%2Fmerge If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again. |
bedevere-bot
removed
the
🔨 test-with-buildbots
If "awk" is gawk, try using python3 if available and new enough. Otherwise, give an error.
This should be applied in 'main'.
conf_extlibs.py was over 1100 lines. Move check_readline, check_curses, and _check_curses (plus the WITH_READLINE arg) into conf_terminal.py, grouping terminal/console I/O library detection together. Split conf_probes.py from conf_platform.py. Assisted-by: Claude
nascheme
added
the
🔨 test-with-buildbots
|
🤖 New build scheduled with the buildbot fleet by @nascheme for commit 9fa1a36 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F146034%2Fmerge If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again. |
bedevere-bot
removed
the
🔨 test-with-buildbots
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note this is WIP, not complete and subject to design changes. There will be a PEP for this change and it will be discussed on DPO, using the regular PEP process. For now, this is a draft PR so it can be tested with the CPython buildbots.
CPython's build configuration is driven by
configure.ac(8,344 lines, ~271KB), which uses autoconf/M4 macros to generate a 946 KB shell script
(
configure). This system is:This replaces it with a Python-based build configuration system where
Tools/configure/configure.pyis a real Python script that imports apyconfmodule. The
pyconffunctions generally match the autoconf behaviour, so thattranslation from the configure.ac file is mostly direct and mechanical.
There is also a transpiler that converts configure.py into POSIX AWK (wrapped
in a small shell stub). The transpiler lives in
Tools/configure/transpiler/and the pipeline is: Python AST → pysh_ast → awk_ast → AWK text. The sh and
AWK code needs to be compatible with those tools on various Unix-like operating
systems.