Skip to content

feat: add OTLP export into tracing setup with context propagation helpers#3385

Merged
vigith merged 12 commits intomainfrom
tracing-base
Apr 22, 2026
Merged

feat: add OTLP export into tracing setup with context propagation helpers#3385
vigith merged 12 commits intomainfrom
tracing-base

Conversation

@adarsh0728
Copy link
Copy Markdown
Member

@adarsh0728 adarsh0728 commented Apr 20, 2026

What this PR does / why we need it

  • This PR introduces dependencies, context propagation helpers, and OTLP exporter wiring. No behavioral change unless OTEL_EXPORTER_OTLP_ENDPOINT env var is set.
  • OTEL_TRACES_SAMPLER && OTEL_TRACES_SAMPLER_ARG can be configured for sampling

Related issues

Fixes #3378

Testing

Deploy with OTEL_EXPORTER_OTLP_ENDPOINT pointed at Jaeger — verified there are no crashes, exporter connects

[setup_tracing] Configuring OTLP exporter: endpoint=http://simplest-jaeger-collector.default:4317, service_name=numaflow-test
[setup_tracing] Sampler: parentbased_traceidratio, ratio=0.01
[setup_tracing] OTLP tracing ENABLED
{"timestamp":"2026-04-20T14:02:25.475288Z","level":"INFO","message":"Forwarder is starting...","target":"numaflow_core::monovertex"}
{"timestamp":"2026-04-20T14:02:25.475536Z","level":"INFO","message":"Started streaming source with batch size","self.read_batch_size":"500","target":"numaflow_core::source"}
{"timestamp":"2026-04-20T14:02:25.475982Z","level":"INFO","message":"Starting sink writer","self.batch_size":"500","self.chunk_timeout":"1s","target":"numaflow_core::sinker::sink"}
{"timestamp":"2026-04-20T14:02:26.471891Z","level":"INFO","message":"Processed messages per second","processed":"81500","target":"numaflow_core::tracker"}
{"timestamp":"2026-04-20T14:02:27.471820Z","level":"INFO","message":"Processed messages per second","processed":"82500","target":"numaflow_core::tracker"}
  • No drop / queue full / BatchSpan warnings — the span-only filter is working

Spec used

apiVersion: numaflow.numaproj.io/v1alpha1
kind: MonoVertex
metadata:
  name: simple-mono-vertex
spec:
  source:
    udsource:
      container:
        image: quay.io/numaio/numaflow-rs/simple-source:stable
    transformer:
      container:
        image: quay.io/numaio/numaflow-rs/source-transformer-now:stable
  udf:
    container:
      image: quay.io/numaio/numaflow-go/map-cat:stable
      imagePullPolicy: IfNotPresent
  sink:
    udsink:
      container:
        image: quay.io/numaio/numaflow-rs/sink-log:stable
  containerTemplate:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: "http://simplest-jaeger-collector.default:4317"
      - name: OTEL_SERVICE_NAME
        value: "numaflow-test"
      - name: OTEL_TRACES_SAMPLER
        value: "parentbased_traceidratio"
      - name: OTEL_TRACES_SAMPLER_ARG
        value: "0.01"

Special notes for reviewers

  • Tokio runtime created before tracing init (required for tonic gRPC exporter)

Signed-off-by: adarsh0728 <gooneriitk@gmail.com>
Signed-off-by: adarsh0728 <gooneriitk@gmail.com>
@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 20, 2026

Codecov Report

❌ Patch coverage is 82.18924% with 96 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.23%. Comparing base (06440fc) to head (db894d3).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
rust/numaflow/src/setup_tracing.rs 72.24% 63 Missing ⚠️
rust/numaflow/src/main.rs 0.00% 26 Missing ⚠️
rust/numaflow-core/src/shared.rs 0.00% 4 Missing ⚠️
rust/numaflow-core/src/shared/otel.rs 98.93% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3385      +/-   ##
==========================================
- Coverage   82.55%   81.23%   -1.33%     
==========================================
  Files         306      307       +1     
  Lines       74445    78064    +3619     
==========================================
+ Hits        61460    63415    +1955     
- Misses      12427    14094    +1667     
+ Partials      558      555       -3     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@adarsh0728 adarsh0728 self-assigned this Apr 20, 2026
Signed-off-by: adarsh0728 <gooneriitk@gmail.com>
Signed-off-by: adarsh0728 <gooneriitk@gmail.com>
@adarsh0728
Copy link
Copy Markdown
Member Author

adarsh0728 commented Apr 21, 2026

Opentelemetry's general recommendation is to have sampling if we generate more than 1000 traces per second.
Currently, head sampling is implemented based on the env variables which user can configure OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG depending on their throughput.

azure sdk example

@adarsh0728 adarsh0728 marked this pull request as ready for review April 21, 2026 04:15
@adarsh0728 adarsh0728 requested a review from BulkBeing April 21, 2026 05:30
adarsh0728 and others added 5 commits April 21, 2026 11:05
Signed-off-by: Vigith Maurice <vigith@gmail.com>
Signed-off-by: Vigith Maurice <vigith@gmail.com>
Signed-off-by: Vigith Maurice <vigith@gmail.com>
…lp_layer does it anyways

Signed-off-by: Vigith Maurice <vigith@gmail.com>
Copy link
Copy Markdown
Member

@vigith vigith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have pushed some changes, please review, and feel free to revert it. I made "a change" per commit.

Could you please improve the test coverage?

Signed-off-by: adarsh0728 <gooneriitk@gmail.com>
Signed-off-by: adarsh0728 <gooneriitk@gmail.com>
Signed-off-by: adarsh0728 <gooneriitk@gmail.com>
@vigith vigith merged commit 4dc4734 into main Apr 22, 2026
45 of 46 checks passed
@vigith vigith deleted the tracing-base branch April 22, 2026 15:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OTel tracing foundation — dependencies, propagation helpers, OTLP exporter

2 participants