asyncio.StreamReader raises BufferError with active memoryview

[MadLongTom]

Bug report

Bug description:

Summary

When StreamReader has an active memoryview export of its internal buffer, methods that mutate the buffer in place can raise:

BufferError: Existing exports of data: object cannot be re-sized

This appears in real workloads with async database drivers, but the root cause is in StreamReader buffer mutation behavior.

Steps to reproduce

import asyncio

async def main():
    r = asyncio.StreamReader()
    r.feed_data(b"AABBCCDD")

    # Simulate a consumer that keeps a memoryview alive
    mv = memoryview(r._buffer)

    # Triggers BufferError in current implementation
    await r.readexactly(4)

    mv.release()

asyncio.run(main())

Expected behavior

readexactly(4) should return b"AABB" without raising BufferError.

Actual behavior

BufferError: Existing exports of data: object cannot be re-sized

Why this happens

Some StreamReader paths mutate bytearray in place, for example slice deletion and clear operations. In-place resize is not allowed while a memoryview export exists.

Affected methods

• read
• readline
• readuntil
• readexactly

Environment

• Python: 3.13.x, 3.14.x (also reproducible in local tests with newer builds)
• OS: macOS (also observed in Linux server workloads)
• Third-party context where this is frequently hit: async DB drivers using memoryview over packet buffers

Possible fix direction

Replace in-place buffer mutations with non-mutating replacement assignments, for example:

• replace slice delete with reassignment to remaining bytes
• replace clear with assigning a new empty bytearray

If useful, I can open a PR with tests covering the regression paths.

CPython versions tested on:

3.13

Operating systems tested on:

macOS

Linked PRs

• gh-146379: Avoid StreamReader BufferError with active memoryview exports #146380

[kumaraditya303]

When StreamReader has an active memoryview export of its internal buffer, methods that mutate the buffer in place can raise:

The internal buffer is private, it can only have a nonzero export count when the buffer is accessed directly, or is there any other way it happens?

The buffer is private because we don't want users to access it.

[MadLongTom]

When StreamReader has an active memoryview export of its internal buffer, methods that mutate the buffer in place can raise:

The internal buffer is private, it can only have a nonzero export count when the buffer is accessed directly, or is there any other way it happens?

The buffer is private because we don't want users to access it.

You're right that _buffer is private. However, the BufferError can be triggered purely through CPython's own code path in readexactly (Python 3.14.2):

# streams.py line ~774-776 (3.14.2)
data = bytes(memoryview(self._buffer)[:n])
del self._buffer[:n]

In a single-expression evaluation, CPython's refcounting releases the temporary memoryview before the del line runs. But under certain async conditions — specifically when a previous coroutine frame or traceback holds a reference to a local that transitively pins a memoryview on the buffer — the export count remains nonzero when del self._buffer[:n] executes.

In our production workload we use asyncmy (a Cython-based async MySQL driver) with SQLAlchemy async. asyncmy calls reader.readexactly(n) in a tight loop to parse MySQL wire protocol packets (read_packet → _read_bytes → readexactly). The driver never accesses _buffer directly; it only calls the public readexactly API. The BufferError is triggered internally by readexactly's own memoryview(self._buffer) temporary when coroutine frame cleanup is delayed.

The main branch already migrated most callsites to take_bytes(), but readline's LimitOverrunError handler still uses del self._buffer[:n] and .clear(). And take_bytes() itself also calls _canresize() internally, so it has the same BufferError risk when an export exists.

[vadmium]

Wouldn’t the solution be to add explicit memoryview release call(s) or use a with statement; something like

with memoryview(self._buffer) as view, view[:n] as slice:
    data = bytes(slice)
del self._buffer[:n]

If the main branch has already migrated to take_bytes, is memoryview still used anywhere?

[MadLongTom]

Wouldn’t the solution be to add explicit memoryview release call(s) or use a with statement; something like

with memoryview(self._buffer) as view, view[:n] as slice:
data = bytes(slice)
del self._buffer[:n]
If the main branch has already migrated to take_bytes, is memoryview still used anywhere?

On main, memoryview is no longer used in StreamReader — all the bytes(memoryview(self._buffer)[:n]) patterns were replaced by take_bytes(). But the issue remains:

1. take_bytes() also calls _canresize() internally, so it raises the same BufferError when an export exists:

   buf = bytearray(b"AABBCCDD")
   mv = memoryview(buf)
   buf.take_bytes(4)  # → BufferError: Existing exports of data: object cannot be re-sized

2. readline's LimitOverrunError handler (line 565–567 on main) still uses del self._buffer[:n] and self._buffer.clear(), which also fail with active exports.

3. The with memoryview(...) approach doesn't apply here — the issue isn't that StreamReader itself creates a memoryview it forgets to release. The issue is that external code (e.g. an async DB driver like asyncmy) calls readexactly() from within a Cython C extension frame that holds a buffer export on self._buffer across the await boundary. StreamReader has no control over when those external exports are released.

So the fix needs to handle the case where take_bytes() (or del/.clear()) fails due to an externally-held export, which is why my pr added _consume_buffer() catches BufferError and falls back to non-mutating replacement.

[kumaraditya303]

So the fix needs to handle the case where take_bytes() (or del/.clear()) fails due to an externally-held export, which is why my pr added _consume_buffer() catches BufferError and falls back to non-mutating replacement.

It is still unclear to me how the buffer can have a non-zero export count, I don't see any place where a reference to memoryview is kept in a local, it is temporary in all cases so it should get cleaned up immediately once the bytes object is created.

[kumaraditya303]

If you can come up with a small cython reproducer for this, that would be great.

also cc @da-woods for any insight on this as this involves cython.