Improve ctypes startup time

[jonathandung]

Feature or enhancement

Proposal:

In the ctypes source, sysconfig is loaded as early as line 5.

"""create and manipulate C data types in Python"""

import os as _os
import sys as _sys
import sysconfig as _sysconfig
import types as _types

However, scanning through the file, it is only used once, and in a conditional branch too:

if _os.name == "nt":
    pythonapi = PyDLL("python dll", None, _sys.dllhandle)
elif _sys.platform in ["android", "cygwin"]:
    # These are Unix-like platforms which use a dynamically-linked libpython.
    pythonapi = PyDLL(_sysconfig.get_config_var("LDLIBRARY"))
else:
    pythonapi = PyDLL(None)

The reason this is inefficient is that sysconfig imports threading instead of _thread, which cannot be helped since it has to use a top-level RLock. Correct me if I'm wrong, but this means at least 50 ms of import time.

I propose to move the import into this if branch. This may require a backport to 3.13 and 3.14, which have similar inefficiencies. (The import was not present prior.)

Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

Links to previous discussion of this feature:

No response

Linked PRs

• gh-146547: Use lazy imports in ctypes, including for sysconfig, shutil, subprocess and warnings modules #146548

[skirpichev]

Correct me if I'm wrong, but this means at least 50 ms of import time.

Have you tried to benchmark this?

[jonathandung]

I just did, and I was wrong. It took around 3 ms. However, that is still significantly slower than the _thread module (3 μs).

Command used:

python -I -S -E -B -m timeit -n 1 -r 1 "import threading"

[jonathandung]

Using the same command structure, I see that sysconfig takes around 6 ms to load, and ctypes around 12 ms.

[jonathandung]

I just did, and I was wrong. It took around 3 ms. However, that is still significantly slower than the _thread module (3 μs).

Command used:

python -I -S -E -B -m timeit -n 1 -r 1 "import threading"

I was mistaken. I though the -B flag asked the interpreter not to read .pyc files, but it actually asks the interpreter not to write them. Therefore, I removed the relevant pycache entries (_weakrefset and threading) and tried again. This time, threading took 69.7 ms to load.

Of course, it is unrealistic that the standard library modules wouldn't be cached. However, my point stands that importing sysconfig only when necessary would decrease ctypes import time overhead by a great deal.

[picnixz]

What is the rationale for improving ctypes startup time? If necessary, we can use a lazy import now. But otherwise I don't think it really matters. In addition, don't just use timeit for timing import times. Please also use -X importtime.

[jonathandung]

pyinstaller uses win32ctypes, which falls back to using ctypes as a backend if the cffi module is missing. Since it is a somewhat popular command-line tool, improving the startup time on most platforms would improve user experience.

[jonathandung]

I assume there are many other examples, such as device interaction libraries that access hardware API through the C FFI and would benefit from faster import of ctypes, especially since they are unlikely to use sysconfig, but these examples are difficult to come across.