Memory leak on macOS

[janbridley]

The readline module included with UV python builds leaks memory, resulting in unexpected behavior for all inheriting packages (including Pytest most notably).

To reproduce, create a uv venv on macOS with python versions 3.10, 3.11, 3.12, or 3.13. Then, run a leak checker on import readline

> sudo leaks --atExit -- python -c "import readline" | grep bytes

Process 5364: 1 leak for 16 total leaked bytes.
    1 (16 bytes) ROOT LEAK: <non-object from libedit.3.dylib 0x149805d80> [16]

Running the (roughly) equivalent command on linux appears not to leak memory:

> valgrind --leak-check=full python -c "import readline" 
==2484992== LEAK SUMMARY:
==2484992==    definitely lost: 0 bytes in 0 blocks
==2484992==    indirectly lost: 0 bytes in 0 blocks
==2484992==      possibly lost: 0 bytes in 0 blocks
==2484992==    still reachable: 500,336 bytes in 199 blocks
==2484992==         suppressed: 0 bytes in 0 blocks

This only seems to be the case in UV python distros - I can confirm that homebrew python versions do not leak memory.

See pytest-dev/pytest#13636 for the original reference.

[geofft]

I can reproduce this on my laptop. The relevant trace is

8   libpython3.13.dylib                   0x102146a50 _PyImport_RunModInitFunc + 84
7   libpython3.13.dylib                   0x10234b8bc PyInit_readline + 244
6   libpython3.13.dylib                   0x102284de4 setup_readline + 136
5   libedit.3.dylib                       0x1e4637498 add_history + 84
4   libedit.3.dylib                       0x1e4642464 history + 1768
3   libedit.3.dylib                       0x1e4641b04 0x1e462d000 + 84740
2   libsystem_c.dylib                     0x192098590 strdup + 44
1   libsystem_malloc.dylib                0x192023a78 _malloc + 88
0   libsystem_malloc.dylib                0x19203912c _malloc_zone_malloc_instrumented_or_legacy + 152

which the call to add_history("2") in this code in Modules/readline.c:

    /* Detect if libedit's readline emulation uses 0-based
     * indexing or 1-based indexing.
     */
    add_history("1");
    if (history_get(1) == NULL) {
        libedit_history_start = 0;
    } else {
        libedit_history_start = 1;
    }
    /* Some libedit implementations use 1 based indexing on
     * replace_history_entry where libreadline uses 0 based.
     * The API our module presents is supposed to be 0 based.
     * It's a mad mad mad mad world.
     */
    {
        add_history("2");
        HIST_ENTRY *old_entry = replace_history_entry(1, "X", NULL);
        _py_free_history_entry_lock_held(old_entry);
        HIST_ENTRY *item = history_get(libedit_history_start);
        if (item && item->line && strcmp(item->line, "X")) {
            libedit_append_replace_history_offset = 0;
        } else {
            libedit_append_replace_history_offset = 1;
        }
    }
    clear_history();

I see this in the Readline docs, which seems suspiciously relevant:

Function: void rl_clear_history (void)

Clear the history list by deleting all of the entries, in the same manner as the History library’s clear_history() function. This differs from clear_history because it frees private data Readline saves in the history list.

(Though note that this is libedit, not actual GNU Readline.)

[geofft]

This is a bug in libedit that was fixed upstream (NetBSD) in 2022: NetBSD/src@86e4c4a

We might be able to work around it, but the ideal thing would be to report this to Apple and/or to ship our own private libedit (we do on Linux, not sure off hand why we don't on Mac).

[geofft]

I filed FB19762214 with Apple, asking them to update libedit (https://github.com/apple-oss-distributions/libedit does not have the fix), and provided the following reproducer:

#include <readline/readline.h>
#include <stdlib.h>

static void
_py_free_history_entry_lock_held(HIST_ENTRY *entry)
{
    if (entry->line)
        free((void *)entry->line);
    if (entry->data)
        free(entry->data);
    free(entry);
}

int main(void) {
    add_history("1");
    add_history("2");
    HIST_ENTRY *old_entry = replace_history_entry(1, "X", NULL);
    _py_free_history_entry_lock_held(old_entry);
    clear_history();
}

Compile with -lreadline and run leaks --atExit -- on the binary. (@janbridley, thanks for the clear and minimal reproducer in Python!)

Let me see how annoying it is to work around this ourselves, since I assume that will take a while to get into a macOS update. I'm guessing that Homebrew's Python links their own libedit, and they have a newer version of libedit than Apple's. We also have one more local patch to libedit from #652 so maybe shipping our own is actually the right call.

[janbridley]

Thanks for prompt follow-up everyone, uv has been excellent!