Read binary strings/blobs in bulk chunks with a memcpy fast path#5233
Conversation
The binary reader read CBOR/MessagePack/BSON/UBJSON strings and byte
arrays one byte at a time via get()/push_back(), and the iterator input
adapter's get_elements() fallback was itself a per-byte loop, so even
contiguous inputs never benefited from a block copy.
Two changes:
1. iterator_input_adapter::get_elements() gains a contiguous fast path
that copies the whole requested range with std::memcpy. Contiguity is
detected via std::is_pointer (all standards) and, in C++20,
std::contiguous_iterator (so std::vector/std::string iterators also
qualify). Non-contiguous iterators keep the element-by-element loop.
2. get_string()/get_binary() now share get_bytes(), which reads into the
result in bounded chunks through get_elements() instead of byte by
byte. Capping the chunk size preserves the deliberate "do not
reserve(len) for an untrusted length" DoS protection while turning the
inner loop into block copies. The min(chunk_size, len) computation is
width-safe so narrow length types (e.g. MessagePack ext-8's uint8_t)
cannot truncate chunk_size to zero.
Microbenchmark (2 MiB string + 2 MiB blob + 2000x1 KiB strings, Apple
clang, -O2):
C++20 from_cbor(vector) 20.1 ms -> 1.0 ms (~20x)
from_cbor(pointer) 18.9 ms -> 1.0 ms (~19x)
C++17 from_cbor(pointer) 18.7 ms -> 1.0 ms (~19x, memcpy)
from_cbor(vector) 18.7 ms -> 4.0 ms (~4.6x, tight loop)
Behavior is unchanged: truncated input still throws parse_error.110 at
the same byte offset, and all binary-format unit tests pass.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This comment was marked as outdated.
This comment was marked as outdated.
- get_bytes(): replace `len -= static_cast<NumberType>(wanted)` with an explicit `len = static_cast<NumberType>(len - ...)`. The compound assignment promoted the operands to int and narrowed back, which GCC rejects under -Werror=arith-conversion when NumberType is narrower than int (e.g. UBJSON's int16 length). This also fixes the CodeQL build. - get_bytes(): add JSON_ASSERT(read == wanted) documenting that get_elements() never returns more than requested (per review). - iterator_input_adapter: de-duplicate the iterator_is_contiguous definition with the #if inside the initializer (per review). - Add a comment explaining why the memcpy source is &*current (needed for non-pointer contiguous iterators), and drop the redundant parentheses. - Run astyle on the new unit test so the amalgamation/format check passes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Flawfinder flags the bare identifier `read` as the POSIX read() syscall (CWE-120/CWE-20), producing several false-positive code-scanning alerts. Renaming the local variable removes them; the byte-copy itself is unchanged and remains bounds-checked (get_elements caps the copy to the bytes available). This is a pure rename with identical semantics. Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
(Claude Code) Addressed the Flawfinder alerts:
Also fixed the GCC |
Make the "buffers are large enough" reasoning explicit: before the copy, assert that `copied` fits both the caller-provided destination (`wanted` bytes) and the remaining input range (`available` bytes). These hold by construction (`copied = min(wanted, available)`) but were previously only implied; the asserts document the invariant and would catch a future regression that breaks it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
get_elements() only receives a raw pointer and cannot know the size of its destination, so the destination-size guarantee is asserted at the caller that owns the buffer: after resizing `result`, assert it has room for `wanted` bytes at `old_size`. Together with the `copied <= wanted` assert at the memcpy, this makes the "destination is large enough" invariant explicit on both sides of the call. Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
std::vector::resize(n) (and string_t's resize) is required to make size() exactly n; only capacity() is permitted to overshoot. The destination-size assert in get_bytes should reflect that exact guarantee rather than a weaker `>=`. Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-6a6c1c Resolves a conflict in include/nlohmann/detail/input/input_adapters.hpp between this branch's memcpy fast path for get_elements() and develop's lazy-diagnostics feature (#5234), which added a `begin` iterator member, `get_consumed_count()`, and `copy_consumed_range()` to the same class. The two features are orthogonal (lexer.hpp uses the new diagnostics helpers; binary_reader.hpp uses get_elements()) and both are preserved. single_include/nlohmann/json.hpp is regenerated from the merged sources. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
(Claude Code) Merged latest |
Summary
The binary reader (
binary_reader::get_string/get_binary) read CBOR/MessagePack/BSON/UBJSON strings and byte arrays one byte at a time viaget()+push_back(). Worse, the iterator input adapter'sget_elements()fallback (input_adapters.hpp) was itself a per-byte loop, so even contiguous inputs (const char*,std::vector/std::stringiterators) never benefited from a block copy —get_number's multi-byte reads didn't either.Changes
iterator_input_adapter::get_elements()contiguous fast path. When the underlying iterator refers to a contiguous range, the whole requested range is copied with a singlestd::memcpy. Contiguity is detected viastd::is_pointeron all standards and, under C++20, additionally viastd::contiguous_iterator(sostd::vector/std::stringiterators qualify too). Non-contiguous iterators keep the element-by-element loop.get_string()/get_binary()read in bounded chunks. Both now delegate to a sharedget_bytes()that pulls data throughget_elements()in capped chunks instead of byte by byte. Capping the chunk size preserves the deliberate "don'treserve(len)for an untrusted length" DoS protection (peak allocation stays bounded regardless of the announced length) while turning the inner loop into block copies. Themin(chunk_size, len)computation is width-safe, so a narrow length type (e.g. MessagePack ext-8'suint8_t) cannot truncatechunk_sizeto zero.Benchmark
Payload: a 2 MiB text string + 2 MiB binary blob + 2000×1 KiB strings, re-encoded to CBOR/MessagePack and decoded 200× (Apple clang,
-O2):from_cbor(vector)from_cbor(pointer)from_cbor(pointer)from_cbor(vector)from_msgpackshows the same pattern.Correctness
parse_error.110at the same byte offset (verified for lengths straddling the chunk boundary).test-cbor,test-msgpack,test-ubjson,test-bson,test-bjdata,test-binary_formats— several million assertions, 0 failures) under C++11.CBOR large strings and binaries (chunked reader)) covering string/binary roundtrips and truncation at lengths of 0, 1, 4095, 4096, 4097, 8192, and 100000 bytes, for both vector and pointer inputs.single_includeregenerated;make check-amalgamationpasses.🤖 Generated with Claude Code