Merge branch 'main' into branch-unique-reference-tracking

Author: reidenong

.github/CODEOWNERS

@@ -132,7 +132,9 @@ Tools/c-analyzer/             @ericsnowcurrently
 Tools/check-c-api-docs/       @ZeroIntensity
 
 # Fuzzing
-Modules/_xxtestfuzz/          @ammaraskar
+Modules/_xxtestfuzz/                    @python/fuzzers
+Lib/test/test_xxtestfuzz.py             @python/fuzzers
+.github/workflows/reusable-cifuzz.yml   @python/fuzzers
 
 # Limited C API & Stable ABI
 Doc/c-api/stable.rst          @encukou

Doc/c-api/memory.rst

@@ -204,8 +204,11 @@ The following function sets, modeled after the ANSI C standard, but specifying
 behavior when requesting zero bytes, are available for allocating and releasing
 memory from the Python heap.
 
-The :ref:`default memory allocator <default-memory-allocators>` uses the
-:ref:`pymalloc memory allocator <pymalloc>`.
+In the GIL-enabled build (default build) the
+:ref:`default memory allocator <default-memory-allocators>` uses the
+:ref:`pymalloc memory allocator <pymalloc>`, whereas in the
+:term:`free-threaded build`, the default is the
+:ref:`mimalloc memory allocator <mimalloc>` instead.
 
 .. warning::
 
@@ -215,6 +218,11 @@ The :ref:`default memory allocator <default-memory-allocators>` uses the
 
    The default allocator is now pymalloc instead of system :c:func:`malloc`.
 
+.. versionchanged:: 3.13
+
+   In the :term:`free-threaded <free threading>` build, the default allocator
+   is now :ref:`mimalloc <mimalloc>`.
+
 .. c:function:: void* PyMem_Malloc(size_t n)
 
    Allocates *n* bytes and returns a pointer of type :c:expr:`void*` to the
@@ -340,7 +348,9 @@ memory from the Python heap.
     the :ref:`Customize Memory Allocators <customize-memory-allocators>` section.
 
 The :ref:`default object allocator <default-memory-allocators>` uses the
-:ref:`pymalloc memory allocator <pymalloc>`.
+:ref:`pymalloc memory allocator <pymalloc>`.  In the
+:term:`free-threaded <free threading>` build, the default is the
+:ref:`mimalloc memory allocator <mimalloc>` instead.
 
 .. warning::
 
@@ -420,23 +430,24 @@ Default Memory Allocators
 
 Default memory allocators:
 
-===============================  ====================  ==================  =====================  ====================
-Configuration                    Name                  PyMem_RawMalloc     PyMem_Malloc           PyObject_Malloc
-===============================  ====================  ==================  =====================  ====================
-Release build                    ``"pymalloc"``        ``malloc``          ``pymalloc``           ``pymalloc``
-Debug build                      ``"pymalloc_debug"``  ``malloc`` + debug  ``pymalloc`` + debug   ``pymalloc`` + debug
-Release build, without pymalloc  ``"malloc"``          ``malloc``          ``malloc``             ``malloc``
-Debug build, without pymalloc    ``"malloc_debug"``    ``malloc`` + debug  ``malloc`` + debug     ``malloc`` + debug
-===============================  ====================  ==================  =====================  ====================
+===================================  =======================  ====================  ======================  ======================
+Configuration                        Name                     PyMem_RawMalloc       PyMem_Malloc            PyObject_Malloc
+===================================  =======================  ====================  ======================  ======================
+Release build                        ``"pymalloc"``           ``malloc``            ``pymalloc``            ``pymalloc``
+Debug build                          ``"pymalloc_debug"``     ``malloc`` + debug    ``pymalloc`` + debug    ``pymalloc`` + debug
+Release build, without pymalloc      ``"malloc"``             ``malloc``            ``malloc``              ``malloc``
+Debug build, without pymalloc        ``"malloc_debug"``       ``malloc`` + debug    ``malloc`` + debug      ``malloc`` + debug
+Free-threaded build                  ``"mimalloc"``           ``mimalloc``          ``mimalloc``            ``mimalloc``
+Free-threaded debug build            ``"mimalloc_debug"``     ``mimalloc`` + debug  ``mimalloc`` + debug    ``mimalloc`` + debug
+===================================  =======================  ====================  ======================  ======================
 
 Legend:
 
 * Name: value for :envvar:`PYTHONMALLOC` environment variable.
 * ``malloc``: system allocators from the standard C library, C functions:
   :c:func:`malloc`, :c:func:`calloc`, :c:func:`realloc` and :c:func:`free`.
 * ``pymalloc``: :ref:`pymalloc memory allocator <pymalloc>`.
-* ``mimalloc``: :ref:`mimalloc memory allocator <mimalloc>`.  The pymalloc
-  allocator will be used if mimalloc support isn't available.
+* ``mimalloc``: :ref:`mimalloc memory allocator <mimalloc>`.
 * "+ debug": with :ref:`debug hooks on the Python memory allocators
   <pymem-debug-hooks>`.
 * "Debug build": :ref:`Python build in debug mode <debug-build>`.
@@ -733,9 +744,27 @@ The mimalloc allocator
 
 .. versionadded:: 3.13
 
-Python supports the mimalloc allocator when the underlying platform support is available.
-mimalloc "is a general purpose allocator with excellent performance characteristics.
-Initially developed by Daan Leijen for the runtime systems of the Koka and Lean languages."
+Python supports the `mimalloc <https://github.com/microsoft/mimalloc/>`__
+allocator when the underlying platform support is available.
+mimalloc is a general purpose allocator with excellent performance
+characteristics, initially developed by Daan Leijen for the runtime systems
+of the Koka and Lean languages.
+
+Unlike :ref:`pymalloc <pymalloc>`, which is optimized for small objects (512
+bytes or fewer), mimalloc handles allocations of any size.
+
+In the :term:`free-threaded <free threading>` build, mimalloc is the default
+and **required** allocator for the :c:macro:`PYMEM_DOMAIN_MEM` and
+:c:macro:`PYMEM_DOMAIN_OBJ` domains.  It cannot be disabled in free-threaded
+builds.  The free-threaded build uses per-thread mimalloc heaps, which allows
+allocation and deallocation to proceed without locking in most cases.
+
+In the default (non-free-threaded) build, mimalloc is available but not the
+default allocator.  It can be selected at runtime using
+:envvar:`PYTHONMALLOC`\ ``=mimalloc`` (or ``mimalloc_debug`` to include
+:ref:`debug hooks <pymem-debug-hooks>`).  It can be disabled at build time
+using the :option:`--without-mimalloc` configure option, but this option
+cannot be combined with :option:`--disable-gil`.
 
 tracemalloc C API
 =================

Doc/using/cmdline.rst

@@ -1085,6 +1085,13 @@ conflict.
    * ``pymalloc_debug``: same as ``pymalloc`` but also install debug hooks.
    * ``mimalloc_debug``: same as ``mimalloc`` but also install debug hooks.
 
+   .. note::
+
+      In the :term:`free-threaded <free threading>` build, the ``malloc``,
+      ``malloc_debug``, ``pymalloc``, and ``pymalloc_debug`` values are not
+      supported.  Only ``default``, ``debug``, ``mimalloc``, and
+      ``mimalloc_debug`` are accepted.
+
    .. versionadded:: 3.6
 
    .. versionchanged:: 3.7
@@ -1094,12 +1101,13 @@ conflict.
 .. envvar:: PYTHONMALLOCSTATS
 
    If set to a non-empty string, Python will print statistics of the
-   :ref:`pymalloc memory allocator <pymalloc>` every time a new pymalloc object
-   arena is created, and on shutdown.
+   :ref:`pymalloc memory allocator <pymalloc>` or the
+   :ref:`mimalloc memory allocator <mimalloc>` (whichever is in use)
+   every time a new object arena is created, and on shutdown.
 
    This variable is ignored if the :envvar:`PYTHONMALLOC` environment variable
    is used to force the :c:func:`malloc` allocator of the C library, or if
-   Python is configured without ``pymalloc`` support.
+   Python is configured without both ``pymalloc`` and ``mimalloc`` support.
 
    .. versionchanged:: 3.6
       This variable can now also be used on Python compiled in release mode.

Doc/using/configure.rst

@@ -774,6 +774,9 @@ also be used to improve performance.
    Disable the fast :ref:`mimalloc <mimalloc>` allocator
    (enabled by default).
 
+   This option cannot be used together with :option:`--disable-gil`
+   because the :term:`free-threaded <free threading>` build requires mimalloc.
+
    See also :envvar:`PYTHONMALLOC` environment variable.
 
 .. option:: --without-pymalloc

Include/cpython/pystate.h

@@ -198,6 +198,7 @@ struct _ts {
     _PyStackChunk *datastack_chunk;
     PyObject **datastack_top;
     PyObject **datastack_limit;
+    _PyStackChunk *datastack_cached_chunk;
     /* XXX signal handlers should also be here */
 
     /* The following fields are here to avoid allocation during init.

Lib/tarfile.py

@@ -1276,6 +1276,20 @@ def _create_pax_generic_header(cls, pax_headers, type, encoding):
     @classmethod
     def frombuf(cls, buf, encoding, errors):
         """Construct a TarInfo object from a 512 byte bytes object.
+
+        To support the old v7 tar format AREGTYPE headers are
+        transformed to DIRTYPE headers if their name ends in '/'.
+        """
+        return cls._frombuf(buf, encoding, errors)
+
+    @classmethod
+    def _frombuf(cls, buf, encoding, errors, *, dircheck=True):
+        """Construct a TarInfo object from a 512 byte bytes object.
+
+        If ``dircheck`` is set to ``True`` then ``AREGTYPE`` headers will
+        be normalized to ``DIRTYPE`` if the name ends in a trailing slash.
+        ``dircheck`` must be set to ``False`` if this function is called
+        on a follow-up header such as ``GNUTYPE_LONGNAME``.
         """
         if len(buf) == 0:
             raise EmptyHeaderError("empty header")
@@ -1306,7 +1320,7 @@ def frombuf(cls, buf, encoding, errors):
 
         # Old V7 tar format represents a directory as a regular
         # file with a trailing slash.
-        if obj.type == AREGTYPE and obj.name.endswith("/"):
+        if dircheck and obj.type == AREGTYPE and obj.name.endswith("/"):
             obj.type = DIRTYPE
 
         # The old GNU sparse format occupies some of the unused
@@ -1341,8 +1355,15 @@ def fromtarfile(cls, tarfile):
         """Return the next TarInfo object from TarFile object
            tarfile.
         """
+        return cls._fromtarfile(tarfile)
+
+    @classmethod
+    def _fromtarfile(cls, tarfile, *, dircheck=True):
+        """
+        See dircheck documentation in _frombuf().
+        """
         buf = tarfile.fileobj.read(BLOCKSIZE)
-        obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors)
+        obj = cls._frombuf(buf, tarfile.encoding, tarfile.errors, dircheck=dircheck)
         obj.offset = tarfile.fileobj.tell() - BLOCKSIZE
         return obj._proc_member(tarfile)
 
@@ -1400,7 +1421,7 @@ def _proc_gnulong(self, tarfile):
 
         # Fetch the next header and process it.
         try:
-            next = self.fromtarfile(tarfile)
+            next = self._fromtarfile(tarfile, dircheck=False)
         except HeaderError as e:
             raise SubsequentHeaderError(str(e)) from None
 
@@ -1535,7 +1556,7 @@ def _proc_pax(self, tarfile):
 
         # Fetch the next header.
         try:
-            next = self.fromtarfile(tarfile)
+            next = self._fromtarfile(tarfile, dircheck=False)
         except HeaderError as e:
             raise SubsequentHeaderError(str(e)) from None
 

Lib/test/test_tarfile.py

@@ -1234,6 +1234,25 @@ def test_longname_directory(self):
                 self.assertIsNotNone(tar.getmember(longdir))
                 self.assertIsNotNone(tar.getmember(longdir.removesuffix('/')))
 
+    def test_longname_file_not_directory(self):
+        # Test reading a longname file and ensure it is not handled as a directory
+        # Issue #141707
+        buf = io.BytesIO()
+        with tarfile.open(mode='w', fileobj=buf, format=self.format) as tar:
+            ti = tarfile.TarInfo()
+            ti.type = tarfile.AREGTYPE
+            ti.name = ('a' * 99) + '/' + ('b' * 3)
+            tar.addfile(ti)
+
+            expected = {t.name: t.type for t in tar.getmembers()}
+
+        buf.seek(0)
+        with tarfile.open(mode='r', fileobj=buf) as tar:
+            actual = {t.name: t.type for t in tar.getmembers()}
+
+        self.assertEqual(expected, actual)
+
+
 class GNUReadTest(LongnameTest, ReadTest, unittest.TestCase):
 
     subdir = "gnu"

Lib/test/test_traceback.py

@@ -4213,6 +4213,27 @@ def method(self, name):
         self.assertIn("'._bluch'", self.get_suggestion(partial(B().method, '_luch')))
         self.assertIn("'._bluch'", self.get_suggestion(partial(B().method, 'bluch')))
 
+    def test_suggestions_with_custom___dir__(self):
+        class M(type):
+            def __dir__(cls):
+                return [None, "fox"]
+
+        class C0:
+            def __dir__(self):
+                return [..., "bluch"]
+
+        class C1(C0, metaclass=M):
+            pass
+
+        self.assertNotIn("'.bluch'", self.get_suggestion(C0, "blach"))
+        self.assertIn("'.bluch'", self.get_suggestion(C0(), "blach"))
+
+        self.assertIn("'.fox'", self.get_suggestion(C1, "foo"))
+        self.assertNotIn("'.fox'", self.get_suggestion(C1(), "foo"))
+
+        self.assertNotIn("'.bluch'", self.get_suggestion(C1, "blach"))
+        self.assertIn("'.bluch'", self.get_suggestion(C1(), "blach"))
+
 
     def test_do_not_trigger_for_long_attributes(self):
         class A:

Lib/traceback.py

@@ -1698,6 +1698,19 @@ def _check_for_nested_attribute(obj, wrong_name, attrs):
     return None
 
 
+def _get_safe___dir__(obj):
+    # Use obj.__dir__() to avoid a TypeError when calling dir(obj).
+    # See gh-131001 and gh-139933.
+    # Also filters out lazy imports to avoid triggering module loading.
+    try:
+        d = obj.__dir__()
+    except TypeError:  # when obj is a class
+        d = type(obj).__dir__(obj)
+    return sorted(
+        x for x in d if isinstance(x, str) and not _is_lazy_import(obj, x)
+    )
+
+
 def _compute_suggestion_error(exc_value, tb, wrong_name):
     if wrong_name is None or not isinstance(wrong_name, str):
         return None
@@ -1711,13 +1724,7 @@ def _compute_suggestion_error(exc_value, tb, wrong_name):
     if isinstance(exc_value, AttributeError):
         obj = exc_value.obj
         try:
-            try:
-                d = dir(obj)
-            except TypeError:  # Attributes are unsortable, e.g. int and str
-                d = list(obj.__class__.__dict__.keys()) + list(obj.__dict__.keys())
-            d = sorted([x for x in d if isinstance(x, str)])
-            # Filter out lazy imports to avoid triggering module loading
-            d = [x for x in d if not _is_lazy_import(obj, x)]
+            d = _get_safe___dir__(obj)
             hide_underscored = (wrong_name[:1] != '_')
             if hide_underscored and tb is not None:
                 while tb.tb_next is not None:
@@ -1744,13 +1751,7 @@ def _compute_suggestion_error(exc_value, tb, wrong_name):
     elif isinstance(exc_value, ImportError):
         try:
             mod = __import__(exc_value.name)
-            try:
-                d = dir(mod)
-            except TypeError:  # Attributes are unsortable, e.g. int and str
-                d = list(mod.__dict__.keys())
-            d = sorted([x for x in d if isinstance(x, str)])
-            # Filter out lazy imports to avoid triggering module loading
-            d = [x for x in d if not _is_lazy_import(mod, x)]
+            d = _get_safe___dir__(mod)
             if wrong_name[:1] != '_':
                 d = [x for x in d if x[:1] != '_']
         except Exception:

Misc/ACKS

@@ -1557,6 +1557,7 @@ Ashwin Ramaswami
 Jeff Ramnani
 Grant Ramsay
 Bayard Randel
+Eashwar Ranganathan
 Varpu Rantala
 Brodie Rao
 Rémi Rampin