messages::serialize: take/put buffers from/into a SerializeBufferPool

[Centril]

Description of Changes

Fixes #1858.

In ws_client_actor_inner, after having ws.feed(msg), keep underlying allocations around, and use a default size of 4KiB.

API and ABI breaking changes

None

Expected complexity level and risk

The changes are somewhat straight forward, and it's not a deep change, but it's in a fairly important place.

Testing

• Run a perf test and inpect flamegraphs and friends.

[gefjon]

Code looks reasonable. I'd prefer to have the pool size metric(s) in before we merge, so we can know that the pool isn't growing without bound. For these metrics, I'd recommend:

• Number of buffers in the pool.
• Total size in bytes of all buffers in the pool.
• Number of newly-allocated buffers.

I think these would give us sufficient information to decide if we need to impose a cap, and what that cap should be if so.

[coolreader18]

Reviewed this yesterday but didn't hit the button - LGTM, this is very straightforward yet effective! Wondering if there's a potential concern with use of a SegQueue, if we happen to allocate a bunch of memory at one point and then it hangs around forever, but choosing the right capacity for an ArrayQueue seems hard.

[coolreader18]

Is there a benefit between having a global queue and just reusing the same buffer per client? Given it seems that each client would only use 1 or 2 buffers, and then the buffer capacities are also more specialized to the specific client (and also the module) rather than growing towards the global maximum.

[coolreader18]

Something like this

[Centril]

Is there a benefit between having a global queue and just reusing the same buffer per client? Given it seems that each client would only use 1 or 2 buffers, and then the buffer capacities are also more specialized to the specific client (and also the module) rather than growing towards the global maximum.

A few benefits to global pools:

• Less memory usage overall for the buffers (not a huge difference... at 1024 CCU we're talking around 8 MiB).
• Better cache locality as the actual OS thread is not jumping between different buffers.
• In case of clients leaving and joining frequently, the local buffers don't help while the global pool does.

That said, the benefit of local buffers is the avoided synchronization overhead.
I think we can try local buffers first. That way, we also don't have to care about metrics the same way.