Skip to content

feat: source-transformer-map-sink spans#3390

Closed
adarsh0728 wants to merge 6 commits intomainfrom
tracing-spans
Closed

feat: source-transformer-map-sink spans#3390
adarsh0728 wants to merge 6 commits intomainfrom
tracing-spans

Conversation

@adarsh0728
Copy link
Copy Markdown
Member

What this PR does / why we need it

  • add a per-message numaflow.platform.process root span
  • add numaflow.monovertex.source.dispatch to measure source-stage work
  • add numaflow.monovertex.source.transform for source-transformer UDF time
  • keep numaflow.monovertex.map and numaflow.monovertex.sink.write under the same root span
  • handle source-transformer fan-out correctly so one input that emits N outputs is traced

Fan-out Handling

This PR explicitly handles the source-transformer 1->N case:

  • platform.process is still created once per source input.
  • source.transform is still created once per source input.
  • source.dispatch stays open until the final emitted child message for that source input is dispatched/bypassed downstream.
  • The zero-output/filter case is handled as well by closing the dispatch span immediately after transform.

Note

  • numaflow.platform.process created in source once a message exists, i.e. after Self::read() returns and before tracker insertion.
  • PR does not create a per-message source.read span.
  • Per-message spans can only be created after the batch read returns
  • source.dispatch operation name can be changed if required.

Related issues

Fixes #3379

Testing

In Progress

Special notes for reviewers

Will change the base to main once tracing-base branch PR is merged

Signed-off-by: adarsh0728 <gooneriitk@gmail.com>
Signed-off-by: adarsh0728 <gooneriitk@gmail.com>
Signed-off-by: adarsh0728 <gooneriitk@gmail.com>
Signed-off-by: adarsh0728 <gooneriitk@gmail.com>
Signed-off-by: adarsh0728 <gooneriitk@gmail.com>

// Drop span guard now so spans are closed before returning. They will also close on any
// early-return from the error branch below.
drop(_span_guard);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you do not require this because i have added a drop guard on the other one.

Base automatically changed from tracing-base to main April 22, 2026 15:55
@adarsh0728
Copy link
Copy Markdown
Member Author

Will create a new PR.

@adarsh0728 adarsh0728 closed this Apr 24, 2026
@adarsh0728 adarsh0728 deleted the tracing-spans branch April 25, 2026 04:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OTel tracing core spans — vertex, source, transformer, map, sink, fallback, onSuccess

2 participants