OTEP: Thread Context: Sharing Thread-Level Information with external readers

[scottgerring]

Changes

External readers like the OpenTelemetry eBPF Profiler operate outside the instrumented process and cannot collect information about active OpenTelemetry traces running within the process they are sampling. We (@ivoanjo and @scottgerring) propose a mechanism for OpenTelemetry SDKs to publish thread-level attributes — including trace ID, span ID, and configurable custom attributes — through a standard format based on the ELF Thread Local Storage (TLSDESC) dialect.

Because this mechanism relies on having a native component and knowing when a runtime switches contexts, we consider it optional for SDKs to support, as some runtimes (or even runtime versions) may not be able to feasibly/efficiently (or undesirable, maintenance-wise) to implement it.

When a request context is attached or detached from a thread, the SDK publishes select information to a thread-local variable that external readers such as the eBPF profiler can discover and read. This enables correlation of profiling samples with request context, even when the active span was not sampled by the SDK.

This builds on and extends OTEP 4719: Process Context, using its process context mechanism to store the static, process-scoped reference data (schema version and attribute key map) that the thread-local records reference.

Why open as draft: OTEP 4719 is a dependency for this OTEP, and thus we'll need to wait for that OTEP to land to ensure we have a solid underpinning to build on.

This OTEP is based and heavily inspired on the custom-labels work by [Polar Signals](https://www.polarsignals.com/) and the universal profiling integration by Elastic — big thanks to them for the prior art and inspiration.and everyone that collaborated with us on the draft google doc this is based on.

• Related issues #
• Related OTEP(s): Profiling Vision #212, Profiles Data Model #239, Process Context #4719
• Links to the prototypes (when adding or changing features)
• CHANGELOG.md file updated for non-trivial changes
• Spec compliance matrix updated if necessary

[scottgerring]

Now that #4719 has been accepted, this one is ready for review!

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: ivoanjo / name: Ivo Anjo (31dbc10, 5c9f504, f529e91)
• ✅ login: scottgerring / name: Scott Gerring (01dcdb4, 047db5a, 0504443, 0e4bc01, 0e7c7c8, 22e7d28, 4e16475, 6400099, 6b8f73a, 6fad842, 8ba154e, 92164ae, b2aa8e3, bc4958d, e5d5798, 5c9f504, 8e89fe8, f529e91, ffc4c4e)
• ✅ login: szegedi / name: Attila Szegedi (ffc4c4e)

[jsuereth]

Overall - this answers my one major question.

I am worried about picking the right amount of information to pass back/forth between contexts. Given how we do logging today - I suspect we want the entire w3c traceparent dataset here, with a notion of w3c baggage. I think you've captured most of this, but I'm curious about the omission of some pieces and the impact of that.

For context -

Logs does include trace flag - https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/logs/v1/logs.proto#L181

Profiling does not include trace flag - https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/profiles/v1development/profiles.proto#L362

So if this is a general protocol that would, e.g. allow eBPF collection of stdout logs and attaching trace information for OTLP export - We'd need flags. If this is only for profiling, today we do not need it.

Given this is an OTEP - this doesn't block my approval. However - I think you should figure this out before formalizing this protocol - as I suspect it'll be sticky. I'd prefer to see the flags added now as I think it may be harder to add later (I'd still like to keep a reserved set of bits for things in case we missed anything else)

[reyang]

For folks who haven't got a chance to review this PR yet, we plan to merge it by the end of Jul. 17th unless there are blocking/outstanding comments.

[ivoanjo]

Update: I've pushed a small commit to refresh the "prototypes" section with more implementations that we're involved with/aware of.

[szegedi]

Few tiny changes (a typo and a better URL.)

Unrelated to these, does it make sense to express somewhere that when attributes are appended to an existing record in-place (because it has a sufficiently large allocation slack to hold them), then flipping the valid flag to 0 and back to 1 is not necessary as long as attrs_size is updated with a signal fence that ensures its new, incremented value is only written after the attributes themselves were written? Or is that too much of an implementation detail for a specification? This spec has gone into rather detailed prescriptions for fencing (which I think is good!) so that's why I wonder if this has its place in it.

My "Request changes" status on this review pertains only to the two tiny changes, not to the question above.

[ivoanjo]

Unrelated to these, does it make sense to express somewhere that when attributes are appended to an existing record in-place (because it has a sufficiently large allocation slack to hold them), then flipping the valid flag to 0 and back to 1 is not necessary as long as attrs_size is updated with a signal fence that ensures its new, incremented value is only written after the attributes themselves were written? Or is that too much of an implementation detail for a specification? This spec has gone into rather detailed prescriptions for fencing (which I think is good!) so that's why I wonder if this has its place in it.

I like it. Added in 65937b3, let me know your thoughts!