Context

In PR #10183's discussion, it was noted that the current internal sync.Pool implementation in brontide always allocates a max size buffer (65 KB) to minimize additional allocations. While this approach successfully reduces allocations, it may be wasteful when handling smaller messages that don't require the full buffer capacity.

Proposed Enhancement

We could implement a tiered buffer pool system that allocates appropriately sized buffers based on the actual message size needed. This would maintain the performance benefits of buffer pooling while reducing memory overhead for smaller messages.

Here's an example of what this might look like:

type TieredBuffer struct {
    pool256B  sync.Pool
    pool1KB   sync.Pool
    pool4KB   sync.Pool
    pool66KB  sync.Pool
}

func (t *TieredBuffer) Take(size int) *bytes.Buffer {
    if size <= 256 {
        return t.pool256B.Get().(*bytes.Buffer)
    }
    if size <= 1024 {
        return t.pool1KB.Get().(*bytes.Buffer)
    }
    // ... and so on
}

This approach would give us better memory allocation efficiency by right-sizing buffers to their actual usage patterns, while still maintaining the allocation reduction benefits that PR #10183 provides.

Related to PR #10183: brontide+peer: use internal sync/pool to reduce allocations