Skip to content

Expose Py_CriticalSection in Stable ABI #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
encukou wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Expose Py_CriticalSection in Stable ABI #39

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 46 additions & 9 deletions Doc/c-api/synchronization.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,11 +84,6 @@ there is no :c:type:`PyObject` -- for example, when working with a C type that
does not extend or wrap :c:type:`PyObject` but still needs to call into the C
API in a manner that might lead to deadlocks.

The functions and structs used by the macros are exposed for cases
where C macros are not available. They should only be used as in the
given macro expansions. Note that the sizes and contents of the structures may
change in future Python versions.

.. note::

Operations that need to lock two objects at once must use
Expand All @@ -114,12 +109,15 @@ section API avoids potential deadlocks due to reentrancy and lock ordering
by allowing the runtime to temporarily suspend the critical section if the
code triggered by the finalizer blocks and calls :c:func:`PyEval_SaveThread`.

.. _critical-section-macros:

.. c:macro:: Py_BEGIN_CRITICAL_SECTION(op)

Acquires the per-object lock for the object *op* and begins a
critical section.

In the free-threaded build, this macro expands to::
In the free-threaded build, and when building for the
:ref:`Stable ABI <stable-abi>`, this macro expands to::

{
PyCriticalSection _py_cs;
Expand Down Expand Up @@ -150,7 +148,8 @@ code triggered by the finalizer blocks and calls :c:func:`PyEval_SaveThread`.

Ends the critical section and releases the per-object lock.

In the free-threaded build, this macro expands to::
In the free-threaded build, and when building for the
:ref:`Stable ABI <stable-abi>`, this macro expands to::

PyCriticalSection_End(&_py_cs);
}
Expand Down Expand Up @@ -179,7 +178,8 @@ code triggered by the finalizer blocks and calls :c:func:`PyEval_SaveThread`.

Locks the mutexes *m1* and *m2* and begins a critical section.

In the free-threaded build, this macro expands to::
In the free-threaded build, and when building for the
:ref:`Stable ABI <stable-abi>`, this macro expands to::

{
PyCriticalSection2 _py_cs2;
Expand All @@ -196,7 +196,8 @@ code triggered by the finalizer blocks and calls :c:func:`PyEval_SaveThread`.

Ends the critical section and releases the per-object locks.

In the free-threaded build, this macro expands to::
In the free-threaded build, and when building for the
:ref:`Stable ABI <stable-abi>`, this macro expands to::

PyCriticalSection2_End(&_py_cs2);
}
Expand All @@ -205,6 +206,42 @@ code triggered by the finalizer blocks and calls :c:func:`PyEval_SaveThread`.

.. versionadded:: 3.13

Low-level critical section API
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The following functions and structs are exposed for cases where C macros
are not available.

.. c:function:: void PyCriticalSection_Begin(PyCriticalSection *c, PyObject *op)
void PyCriticalSection_End(PyCriticalSection *c)
void PyCriticalSection2_Begin(PyCriticalSection2 *c, PyObject *a, PyObject *b)
void PyCriticalSection2_End(PyCriticalSection2 *c);

To be used only as in the macro expansions
listed :ref:`earlier in this section <critical-section-macros>`.

.. versionadded:: 3.13

.. c:type:: PyCriticalSection
PyCriticalSection2

To be used only as in the macro expansions
listed :ref:`earlier in this section <critical-section-macros>`.
Note that the contents of the structures are private and their meaning may
change in future Python versions.

.. versionadded:: 3.13

.. c:function:: void PyCriticalSection_BeginMutex(PyCriticalSection *c, PyMutex *m);
void PyCriticalSection2_BeginMutex(PyCriticalSection2 *c, PyMutex *m1, PyMutex *m2);

.. These need to be in a separate section without a Stable ABI anotation.

To be used only as in the macro expansions
listed :ref:`earlier in this section <critical-section-macros>`.

.. versionadded:: 3.14


Legacy locking APIs
-------------------
Expand Down
10 changes: 10 additions & 0 deletions Doc/data/stable_abi.dat
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

93 changes: 7 additions & 86 deletions Include/cpython/critical_section.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,6 @@
# error "this header file must not be included directly"
#endif

// Python critical sections
//
// Conceptually, critical sections are a deadlock avoidance layer on top of
// per-object locks. These helpers, in combination with those locks, replace
// our usage of the global interpreter lock to provide thread-safety for
// otherwise thread-unsafe objects, such as dict.
//
// NOTE: These APIs are no-ops in non-free-threaded builds.
//
// Straightforward per-object locking could introduce deadlocks that were not
// present when running with the GIL. Threads may hold locks for multiple
// objects simultaneously because Python operations can nest. If threads were
Expand Down Expand Up @@ -43,52 +34,19 @@
// `_PyThreadState_Attach()`, it resumes the top-most (i.e., most recent)
// critical section by reacquiring the associated lock or locks. See
// `_PyCriticalSection_Resume()`.
//
// NOTE: Only the top-most critical section is guaranteed to be active.
// Operations that need to lock two objects at once must use
// `Py_BEGIN_CRITICAL_SECTION2()`. You *CANNOT* use nested critical sections
// to lock more than one object at once, because the inner critical section
// may suspend the outer critical sections. This API does not provide a way
// to lock more than two objects at once (though it could be added later
// if actually needed).
//
// NOTE: Critical sections implicitly behave like reentrant locks because
// attempting to acquire the same lock will suspend any outer (earlier)
// critical sections. However, they are less efficient for this use case than
// purposefully designed reentrant locks.
//
// Example usage:
// Py_BEGIN_CRITICAL_SECTION(op);
// ...
// Py_END_CRITICAL_SECTION();
//
// To lock two objects at once:
// Py_BEGIN_CRITICAL_SECTION2(op1, op2);
// ...
// Py_END_CRITICAL_SECTION2();

typedef struct PyCriticalSection PyCriticalSection;
typedef struct PyCriticalSection2 PyCriticalSection2;

PyAPI_FUNC(void)
PyCriticalSection_Begin(PyCriticalSection *c, PyObject *op);

PyAPI_FUNC(void)
PyCriticalSection_BeginMutex(PyCriticalSection *c, PyMutex *m);

PyAPI_FUNC(void)
PyCriticalSection_End(PyCriticalSection *c);

PyAPI_FUNC(void)
PyCriticalSection2_Begin(PyCriticalSection2 *c, PyObject *a, PyObject *b);

PyAPI_FUNC(void)
PyCriticalSection2_BeginMutex(PyCriticalSection2 *c, PyMutex *m1, PyMutex *m2);

PyAPI_FUNC(void)
PyCriticalSection2_End(PyCriticalSection2 *c);

#ifndef Py_GIL_DISABLED
#undef Py_BEGIN_CRITICAL_SECTION
#undef Py_END_CRITICAL_SECTION
#undef Py_BEGIN_CRITICAL_SECTION2
#undef Py_END_CRITICAL_SECTION2

# define Py_BEGIN_CRITICAL_SECTION(op) \
{
# define Py_BEGIN_CRITICAL_SECTION_MUTEX(mutex) \
Expand All @@ -101,54 +59,17 @@ PyCriticalSection2_End(PyCriticalSection2 *c);
{
# define Py_END_CRITICAL_SECTION2() \
}
#else /* !Py_GIL_DISABLED */

// NOTE: the contents of this struct are private and may change betweeen
// Python releases without a deprecation period.
struct PyCriticalSection {
// Tagged pointer to an outer active critical section (or 0).
uintptr_t _cs_prev;

// Mutex used to protect critical section
PyMutex *_cs_mutex;
};

// A critical section protected by two mutexes. Use
// Py_BEGIN_CRITICAL_SECTION2 and Py_END_CRITICAL_SECTION2.
// NOTE: the contents of this struct are private and may change betweeen
// Python releases without a deprecation period.
struct PyCriticalSection2 {
PyCriticalSection _cs_base;

PyMutex *_cs_mutex2;
};

# define Py_BEGIN_CRITICAL_SECTION(op) \
{ \
PyCriticalSection _py_cs; \
PyCriticalSection_Begin(&_py_cs, _PyObject_CAST(op))
#else /* !Py_GIL_DISABLED */

# define Py_BEGIN_CRITICAL_SECTION_MUTEX(mutex) \
{ \
PyCriticalSection _py_cs; \
PyCriticalSection_BeginMutex(&_py_cs, mutex)

# define Py_END_CRITICAL_SECTION() \
PyCriticalSection_End(&_py_cs); \
}

# define Py_BEGIN_CRITICAL_SECTION2(a, b) \
{ \
PyCriticalSection2 _py_cs2; \
PyCriticalSection2_Begin(&_py_cs2, _PyObject_CAST(a), _PyObject_CAST(b))

# define Py_BEGIN_CRITICAL_SECTION2_MUTEX(m1, m2) \
{ \
PyCriticalSection2 _py_cs2; \
PyCriticalSection2_BeginMutex(&_py_cs2, m1, m2)

# define Py_END_CRITICAL_SECTION2() \
PyCriticalSection2_End(&_py_cs2); \
}

#endif
#endif /* !Py_GIL_DISABLED */
85 changes: 85 additions & 0 deletions Include/critical_section.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,91 @@
extern "C" {
#endif

// Python critical sections
//
// Conceptually, critical sections are a deadlock avoidance layer on top of
// per-object locks. These helpers, in combination with those locks, replace
// our usage of the global interpreter lock to provide thread-safety for
// otherwise thread-unsafe objects, such as dict.
//
// NOTE: These APIs are no-ops in non-free-threaded builds.
//
// NOTE: Only the top-most critical section is guaranteed to be active.
// Operations that need to lock two objects at once must use
// `Py_BEGIN_CRITICAL_SECTION2()`. You *CANNOT* use nested critical sections
// to lock more than one object at once, because the inner critical section
// may suspend the outer critical sections. This API does not provide a way
// to lock more than two objects at once (though it could be added later
// if actually needed).
//
// NOTE: Critical sections implicitly behave like reentrant locks because
// attempting to acquire the same lock will suspend any outer (earlier)
// critical sections. However, they are less efficient for this use case than
// purposefully designed reentrant locks.
//
// Example usage:
// Py_BEGIN_CRITICAL_SECTION(op);
// ...
// Py_END_CRITICAL_SECTION();
//
// To lock two objects at once:
// Py_BEGIN_CRITICAL_SECTION2(op1, op2);
// ...
// Py_END_CRITICAL_SECTION2();

// NOTE: the contents of this struct are private and their meaning may
// change betweeen Python releases without a deprecation period.
typedef struct PyCriticalSection {
// Tagged pointer to an outer active critical section (or 0).
uintptr_t _cs_prev;

// Mutex used to protect critical section
struct PyMutex *_cs_mutex;

// Space for Stable ABI additions
void *_cs_reserved;
} PyCriticalSection;

// A critical section protected by two mutexes. Use
// Py_BEGIN_CRITICAL_SECTION2 and Py_END_CRITICAL_SECTION2.
// NOTE: the contents of this struct are private and may change betweeen
// Python releases without a deprecation period.
typedef struct PyCriticalSection2 {
PyCriticalSection _cs_base;

struct PyMutex *_cs_mutex2;
} PyCriticalSection2;

PyAPI_FUNC(void)
PyCriticalSection_Begin(PyCriticalSection *c, PyObject *op);

PyAPI_FUNC(void)
PyCriticalSection_End(PyCriticalSection *c);

PyAPI_FUNC(void)
PyCriticalSection2_Begin(PyCriticalSection2 *c, PyObject *a, PyObject *b);

PyAPI_FUNC(void)
PyCriticalSection2_End(PyCriticalSection2 *c);

# define Py_BEGIN_CRITICAL_SECTION(op) \
{ \
PyCriticalSection _py_cs; \
PyCriticalSection_Begin(&_py_cs, _PyObject_CAST(op))

# define Py_END_CRITICAL_SECTION() \
PyCriticalSection_End(&_py_cs); \
}

# define Py_BEGIN_CRITICAL_SECTION2(a, b) \
{ \
PyCriticalSection2 _py_cs2; \
PyCriticalSection2_Begin(&_py_cs2, _PyObject_CAST(a), _PyObject_CAST(b))

# define Py_END_CRITICAL_SECTION2() \
PyCriticalSection2_End(&_py_cs2); \
}

#ifndef Py_LIMITED_API
# define Py_CPYTHON_CRITICAL_SECTION_H
# include "cpython/critical_section.h"
Expand Down
4 changes: 4 additions & 0 deletions Lib/test/test_cext/extension.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,10 @@ _testcext_exec(PyObject *module)
obj = NULL;
Py_CLEAR(obj);

// Test that Py_BEGIN_CRITICAL_SECTION is available
Py_BEGIN_CRITICAL_SECTION(module);
Py_END_CRITICAL_SECTION();

return 0;
}

Expand Down
4 changes: 4 additions & 0 deletions Lib/test/test_stable_abi_ctypes.py
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading