Skip to content
87 changes: 73 additions & 14 deletions peps/pep-0788.rst
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,8 @@ Prior to this PEP, deprecating daemon threads was discussed
`extensively <https://discuss.python.org/t/68836>`_. Daemon threads technically
cause many of the issues outlined in this proposal, so removing daemon threads
could be seen as a potential solution. The main argument for removing daemon
threads is that they're a large cause of problems in the interpreter:
threads is that they're a large cause of problems in the interpreter
`[1] <https://discuss.python.org/t/68836/6>`_.

Except that daemon threads don’t actually work reliably. They’re attempting
to run and use Python interpreter resources after the runtime has been shut
Expand All @@ -190,7 +191,8 @@ threads is that they're a large cause of problems in the interpreter:

However, in practice, daemon threads are useful for simplifying many threading
applications in Python, and since the program is about to close in most cases,
it's not worth the added complexity to try and gracefully shut down a thread.
it's not worth the added complexity to try and gracefully shut down a thread
`[2] <https://discuss.python.org/t/68836/3>`_.

When I’ve needed daemon threads, it’s usually been the case of “Long-running,
uninterruptible, third-party task” in terms of the examples in the linked issue.
Expand All @@ -205,7 +207,8 @@ As noted by this PEP, extension modules are free to create their own threads
and attach thread states for them. Similar to daemon threads, Python doesn't
try and join them during finalization, so trying to remove daemon threads
as a whole would involve trying to remove them from the C API, which would
require a much more massive API change.
require a much more massive API change than what is currently being proposed
`[3] <https://discuss.python.org/t/68836/7>`_.

Realize however that even if we get rid of daemon threads, extension
module code can and does spawn its own threads that are not tracked by
Expand Down Expand Up @@ -379,20 +382,23 @@ Weak References

This proposal also comes with weak references to an interpreter that don't
prevent it from shutting down, but can be promoted to a strong reference when
the user decides that they want to call the C API. A weak reference will
typically live much longer than a strong reference. This is useful for many of
the asynchronous situations stated previously, where the thread itself
shouldn't prevent the desired interpreter from shutting down, but also allow
the thread to execute Python when needed.
the user decides that they want to call the C API. If an interpreter is
destroyed or past the point where it can create strong references, promotion
of a weak reference will fail.

A weak reference will typically live much longer than a strong reference.
This is useful for many of the asynchronous situations stated previously,
where the thread itself shouldn't prevent the desired interpreter from shutting
down, but also allow the thread to execute Python when needed.

For example, a (non-reentrant) event handler may store a weak interpreter
reference in its ``void *arg`` parameter, and then that weak reference will
be promoted to a strong reference when it's time to call Python code.

Deprecation of the GIL-state APIs
---------------------------------
Removing the outdated GIL-state APIs
------------------------------------

Due to the plethora of issues with ``PyGILState``, this PEP intends to do away
Due to the unfixable issues with ``PyGILState``, this PEP intends to do away
with them entirely. In today's C API, all ``PyGILState`` functions are
replaceable with ``PyThreadState`` counterparts that are compatibile with
subinterpreters:
Expand All @@ -402,6 +408,8 @@ subinterpreters:
- :c:func:`PyGILState_GetThisThreadState`: :c:func:`PyThreadState_Get` (roughly)
- :c:func:`PyGILState_Check`: ``PyThreadState_GetUnchecked() != NULL``

(See :ref:`pep-788-porting-guide` for more information.)

This PEP specifies a deprecation for these functions (while remaining
in the stable ABI), because :c:func:`PyThreadState_Ensure` and
:c:func:`PyThreadState_Release` will act as more-correct replacements for
Expand Down Expand Up @@ -507,6 +515,8 @@ Weak Interpreter References
The interpreter will *not* wait for the reference to be
released before shutting down.

This type is guaranteed to be pointer-sized.

.. c:function:: int PyInterpreterWeakRef_Get(PyInterpreterWeakRef *wref)

Acquire a weak reference to the current interpreter.
Expand Down Expand Up @@ -535,7 +545,8 @@ Weak Interpreter References
reference to the interpreter denoted by *wref*.

If the interpreter no longer exists or has already finished waiting
for its reference count to reach zero, then this function returns ``-1``.
for its reference count to reach zero, then this function returns ``-1``
without an exception set.

This function is not safe to call in a re-entrant signal handler.

Expand Down Expand Up @@ -631,6 +642,54 @@ in the C API documentation, ideally under the :ref:`python:gilstate` section.
The existing ``PyGILState`` documentation should be updated accordingly to point
to the new APIs.

.. _pep-788-porting-guide:

Porting Guide
-------------

Ensuring Python Can Be Called
*****************************

In all places where :c:func:`PyGILState_Ensure` and :c:func:`PyGILState_Release`
are used, provide access to a strong or weak interpreter reference in the code
path. If this is too difficult or impossible (such as in a public library interface
that uses ``PyGILState`` under the hood), consider using :c:func:`PyInterpreterRef_Main`
to get a reference for the main interpreter. Then, replace the legacy calls with
:c:func:`PyThreadState_Ensure` and :c:func:`PyThreadState_Release`, acquiring
and releasing interpreter references where necessary. A strong interpreter
reference should typically be released after calling :c:func:`PyThreadState_Release`.

Thread States
*************

There are two ``PyGILState`` APIs that act as an abstraction for
:term:`thread states <thread state>`. For some background, Python stores
Comment thread
encukou marked this conversation as resolved.
Outdated
an internal thread-local pointer known as the "gilstate". The gilstate points
to the thread state used by :c:func:`PyGILState_Ensure`, or more broadly, the
last thread state used by the thread (that is not destroyed).
:c:func:`PyGILState_GetThisThreadState` returns the gilstate pointer, *not*
the :term:`attached thread state`. The gilstate is not something that should
be accessed by users of the C API; instead, use :c:func:`PyThreadState_Get`
(or :c:func:`PyThreadState_GetUnchecked` if possibly expecting ``NULL``).

The other function, :c:func:`PyGILState_Check`, is an older function for
checking if the calling thread holds the :term:`GIL`. In newer versions,
:c:func:`PyGILState_Check` simply checks if the gilstate matches the
attached thread state (which is always true if the attached thread state
is non-``NULL``). However, :c:func:`PyGILState_Check` always returns ``1`` if
a subinterpreter was ever created in the current Python process. As such,
:c:func:`PyGILState_Check` is generally not a great API to use. Instead,
check if ``PyThreadState_GetUnchecked()`` returns ``NULL`` to determine
if the thread has an attached thread state.

As a band-aid solution, these macros can be added to avoid rewriting
code:

.. code-block:: c

#define PyGILState_GetThisThreadState PyThreadState_GetUnchecked
#define PyGILState_Check() (PyThreadState_GetUnchecked() != NULL)

Examples
--------

Expand Down Expand Up @@ -679,8 +738,8 @@ If you were to use :c:func:`PyGILState_Ensure` for this case, then your
thread would hang if the interpreter were to be finalizing at that time!

Additionally, the API supports subinterpreters. If you were to assume that
the main interpreter created the file object, then your library wouldn't be safe to use
with file objects created by a subinterpreter.
the main interpreter created the file object (via :c:func:`PyGILState_Ensure`),
then using file objects owned by a subinterpreter could possibly crash.

Example: A Single-threaded Ensure
*********************************
Expand Down