Skip to content

feat: Add finishInto() for zero-copy finish into existing buffers#2196

Open
jiang1997 wants to merge 2 commits intoprotobufjs:masterfrom
jiang1997:feat/finishTo
Open

feat: Add finishInto() for zero-copy finish into existing buffers#2196
jiang1997 wants to merge 2 commits intoprotobufjs:masterfrom
jiang1997:feat/finishTo

Conversation

@jiang1997
Copy link
Copy Markdown
Contributor

@jiang1997 jiang1997 commented Apr 28, 2026

Introduce finishInto(), which allows writing directly into an existing buffer (e.g., WebAssembly memory), thereby avoiding extra copying.

@dcodeIO
Copy link
Copy Markdown
Member

dcodeIO commented Apr 28, 2026

Do you plan to use finishTo somewhere within the library to speed it up? If not, then this should likely not live in protobufjs source itself, but be provided externally.

@jiang1997
Copy link
Copy Markdown
Contributor Author

Do you plan to use finishTo somewhere within the library to speed it up? If not, then this should likely not live in protobufjs source itself, but be provided externally.

To be honest, my initial motivation was simply to solve an issue in my own project (serializing directly into Wasm memory to avoid extra copies). I haven't really explored using it to speed up the library internally yet.
I understand that users could technically implement this externally by iterating over writer.head. However, I was concerned that this might not be the best approach, as it would require external code to break encapsulation and rely on internal implementation details.

@dcodeIO
Copy link
Copy Markdown
Member

dcodeIO commented Apr 29, 2026

One approach to integrate it into protobuf.js itself might be to use finishTo to implement finish. Then it's not just dead code. Ideally, though, such a change should not require a bunch of additional runtime checks that only exist because of that split.

@jiang1997 jiang1997 force-pushed the feat/finishTo branch 3 times, most recently from 205e22e to 2f41276 Compare April 29, 2026 08:24
@jiang1997
Copy link
Copy Markdown
Contributor Author

One approach to integrate it into protobuf.js itself might be to use finishTo to implement finish. Then it's not just dead code. Ideally, though, such a change should not require a bunch of additional runtime checks that only exist because of that split.

Yeah, totally agree. I've already made the changes — finish() now delegates to finishTo(), all runtime checks have been removed, and the documentation has been updated to note that callers are responsible for ensuring sufficient buffer space.

Comment thread src/writer.js Outdated
* @returns {Uint8Array} The provided buffer
*/
Writer.prototype.finishTo = function finishTo(buf, offset) {
offset |= 0;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This turns offsets >= 2 GiB negative. Perhaps:

Suggested change
offset |= 0;
offset >>>= 0;

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, learned something new! Updated to offset >>>= 0.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The offset >>>= 0 operation reduces the value to a 32-bit number. Node lets TypedArray indices go beyond 2³², so a large offset might end up near the start of the buffer.
I’d suggest making offset required, not optional, to avoid any unexpected behavior.

What are your thoughts on this, @dcodeIO ?

Comment thread src/writer.js Outdated
Comment on lines +456 to +458
* @param {Uint8Array} buf Target buffer
* @param {number} [offset=0] Offset to start writing at
* @returns {Uint8Array} The provided buffer
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this returns the provided buffer, I think we can make the JSDoc generic here:

Suggested change
* @param {Uint8Array} buf Target buffer
* @param {number} [offset=0] Offset to start writing at
* @returns {Uint8Array} The provided buffer
* @param {T} buf Target buffer
* @param {number} [offset=0] Offset to start writing at
* @returns {T} The provided buffer
* @template T extends Uint8Array

The definitions generator should then generate finishTo<T extends Uint8Array>(buf: T, offset?: number): T, preserving Buffer and other Uint8Array subclasses.

@dcodeIO
Copy link
Copy Markdown
Member

dcodeIO commented May 5, 2026

Thinking about naming a bit here, I guess a good name could be finishInto, then mirroring APIs like TextEncoder#encodeInto. Wdyt?

@dcodeIO dcodeIO changed the title feat: add finishTo() for zero-copy serialization into existing buffers feat: Add finishInto() for zero-copy finish into existing buffers May 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants