[IGNORE] PR 5962#5977
Draft
vpellan wants to merge 8 commits into
Draft
Conversation
vpellan
commented
Jul 1, 2026
vpellan
commented
Contributor
Instrument Racecar consumers and producers for Data Streams Monitoring. Consume checkpoints are set from the existing ActiveSupport notification events: single-message consume extracts the upstream pathway context from the message headers carried in the event payload, while batch consume sets a topic-level checkpoint (the batch payload carries no per-message headers). Both also track the consumed offset for consumer lag. Produce checkpoints are injected by prepending instrumentation onto Racecar::Consumer#produce and the standalone Racecar::Producer, mirroring the ruby-kafka integration.
Move consume-side DSM from the ActiveSupport event handlers to a prepend on Racecar::Runner#process and #process_batch, where the individual rdkafka messages and their headers are available. This lets batch consumption set a checkpoint per message rather than a single topic-level checkpoint, so each message keeps its own upstream pathway context. Pipelines that merge multiple input sources into one processing step (N:M topology) are now represented as distinct fan-in edges in Data Streams Monitoring.
Match stringified header keys instead of converting the lookup key to a symbol, so the lookup does not depend on symbol garbage collection.
Racecar 1.x consumes through Kafka::Consumer#each_message/#each_batch (ruby-kafka) rather than Runner#process, so the Runner prepend records nothing there. Gate the Runner prepend on the rdkafka-based runner and fall back to the kafka integration's consumer instrumentation on older, ruby-kafka-based versions.
Contributor
|
👋 Hey @DataDog/ruby-guild, please fill "Change log entry" section in the pull request description. If changes need to be present in CHANGELOG.md you can state it this way **Change log entry**
Yes. A brief summary to be placed into the CHANGELOG.md(possible answers Yes/Yep/Yeah) Or you can opt out like that **Change log entry**
None.(possible answers No/Nope/None) Visited at: 2026-07-01 14:08:14 UTC |
Contributor
Typing analysisNote: Ignored files are excluded from the next sections. Untyped methodsThis PR introduces 3 partially typed methods. It increases the percentage of typed methods from 65.59% to 65.68% (+0.09%). Partially typed methods (+3-0)❌ Introduced:If you believe a method or an attribute is rightfully untyped or partially typed, you can add |
|
Replace untyped signatures for the racecar instrumentation with concrete types, and split the DSM specs into consumer_spec.rb and producer_spec.rb to mirror the lib layout.
The method returns its headers argument unchanged when DSM is disabled, so the declared return type must be nilable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.