Using opentelemetry-http for distributed tracing #3207

de-sh · 2025-10-22T09:20:15Z

de-sh
Oct 22, 2025

Hi, I have been trying to make use of tracing + opentelemetry crates to instrument my code for distributed tracing purposes. I intend to have an http request pass through my service, there will be calls to subsequent services that I am addding traceparent headers to, according to the W3C trace-context spec:

/// Initializes the OpenTelemetry tracer and sets it as the global tracer.
/// Setup exporter to push traces to the collector over GRPC.
pub fn init_tracer() -> SdkTracerProvider {
    global::set_text_map_propagator(TraceContextPropagator::new());
    // Initialize OTLP exporter using GRPC binary protocol
    let otlp_exporter = SpanExporter::builder().with_tonic().build().unwrap();

    // Create a tracer provider with the exporter
    let tracer_provider = SdkTracerProvider::builder()
        .with_batch_exporter(otlp_exporter)
        .build();

    // Set it as the global provider
    global::set_tracer_provider(tracer_provider.clone());

    tracer_provider
}

/// Initializes the OpenTelemetry logger and sets up the tracing subscriber.
pub fn init_logger() -> SdkLoggerProvider {
    let log_exporter = LogExporter::builder().with_tonic().build().unwrap();
    let logger_provider = SdkLoggerProvider::builder()
        .with_simple_exporter(log_exporter)
        .build();
    let otel_layer = OpenTelemetryTracingBridge::new(&logger_provider);
    tracing_subscriber::registry()
        .with(otel_layer)
        .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| "info".into()))
        .with(fmt::layer())
        .init();

    logger_provider
}

#[instrument]
pub fn request() {
...
        // Propagate OpenTelemetry context to the downstream service.
        let mut headers = HeaderMap::new();
        let tracer = global::tracer("my_tracer");
        let span = tracer
            .span_builder("execute")
            .with_kind(SpanKind::Client)
            .start(&tracer);
        let context = Context::current_with_span(span);
        global::get_text_map_propagator(|propagator| {
            propagator.inject_context(&context, &mut HeaderInjector(&mut headers))
        });

        let response = client
            .post(&url)
            .headers(headers)
            .json(&body)
            .send()
            .await?;
...
}

pub async fn add_response_header(request: Request<Body>, next: Next) -> Response<Body> {
    let tracer = global::tracer("my_tracer");
    let span = tracer.start("request");
    let ctx = Context::current_with_span(span);

    let mut response = next.run(request).with_context(ctx.clone()).await;

    let span = ctx.span();
    let span_context = span.span_context();
    if span_context.is_valid() {
        let trace_id = span_context.trace_id().to_string();
        response.headers_mut().insert(
            HeaderName::from_static("x-trace-id"),
            HeaderValue::from_str(&trace_id).unwrap(),
        );
    }

    response
}

I am seeing different traceid when comparing traceparent sent to the downstream service and the x-trace-id response header, what am I doing wrong?

muturgan · 2026-03-10T21:17:00Z

muturgan
Mar 10, 2026

still have a same problem

0 replies

muturgan · 2026-03-10T22:41:15Z

muturgan
Mar 10, 2026

Omg at last it works!

use axum::{body::Body, http::Request, middleware::Next, response::Response};
use opentelemetry::{
    Context, SpanId, TraceId,
    trace::{SpanContext, TraceContextExt, TraceFlags, TraceState},
};
use tracing::{Level, span, warn};
use tracing_opentelemetry::OpenTelemetrySpanExt;

async fn tracing_middleware(req: Request<Body>, next: Next<Body>) -> Response {
    if let Some((trace_id, span_id)) = req
        .headers()
        .get("traceparent")
        .and_then(|h| h.to_str().ok())
        .and_then(parse_traceparent)
    {
        let span_context = SpanContext::new(
            trace_id,
            span_id,
            TraceFlags::default(),
            true, // is_remote
            TraceState::default(),
        );

        let otel_context = Context::new().with_remote_span_context(span_context);

        let tracing_span = span!(Level::INFO, "Incoming http");
        if let Err(e) = tracing_span.set_parent(otel_context) {
            warn!("set_parent err: {e}");
        }

        let _tracing_guard = tracing_span.enter();

        handle_request(req, next).await
    } else {
        handle_request(req, next).await
    }
}

async fn handle_request(req: Request<Body>, next: Next<Body>) -> Response {
    todo!()
}

fn parse_traceparent(header: &str) -> Option<(TraceId, SpanId)> {
    todo!()
}

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using opentelemetry-http for distributed tracing #3207

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Using opentelemetry-http for distributed tracing #3207

Uh oh!

Uh oh!

de-sh Oct 22, 2025

Replies: 2 comments

Uh oh!

muturgan Mar 10, 2026

Uh oh!

Uh oh!

muturgan Mar 10, 2026

de-sh
Oct 22, 2025

muturgan
Mar 10, 2026

muturgan
Mar 10, 2026