fix(datadog): cap coalesced DogStatsD datagram size, fall back to per-metric sends#13665
Open
shreemaan-abhishek wants to merge 1 commit into
Open
Conversation
Coalescing an entry's metrics into one datagram repeats the tag suffix on every line, so with long constant_tags / names / paths the packet can exceed the DogStatsD agent's default receive buffer (dogstatsd_buffer_size = 8192) and be silently truncated. Send the coalesced datagram only when it fits within 8192 bytes; otherwise fall back to one datagram per metric (the original size profile). Adds a test with oversized tags asserting all metrics still arrive. Signed-off-by: Abhishek Choudhary <shreemaan.abhishek@gmail.com>
dosubot
Bot
added
the
size:M
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #13653 (already merged), addressing the review note there about the coalesced datagram having no size guard.
#13653 coalesces an entry's metrics into a single newline-delimited DogStatsD datagram. Because each line repeats the full tag suffix, the coalesced packet is ~6x a single metric line. With long tag sources (
constant_tagshas nomaxItems, route/service/consumer names can be long,include_pathadds the route path), the packet can exceed the DogStatsD agent's default receive buffer (dogstatsd_buffer_size, 8192 bytes) and be silently truncated —sock:send()only returns an error above the OS max UDP size (~64KB on Linux), so nothing catches it. This is a regression versus the previous one-datagram-per-metric behaviour, where each datagram was ~6x smaller.This PR sends the coalesced datagram only when it fits within 8192 bytes; otherwise it falls back to one datagram per metric (the original size profile). A single socket is still reused across the batch, so the syscall win from #13653 is preserved for the common case.
Tests
Adds a test with oversized
constant_tags(coalesced payload > 8192) asserting every metric still arrives via the fallback path.Checklist