Skip to content

Commit a994110

Browse files
lukeina2zCopilot
andcommitted
[EXPORTER ETW] Fix Span::End() use-after-free when caller drops Tracer
Fixes #3849. etw::TracerProvider::GetTracer() was newing up a fresh Tracer per call and returning the only shared_ptr to it. The catch is that etw::Span stores its parent as a raw reference (Tracer &owner_), so the moment a caller drops or reassigns that shared_ptr, every Span that came from that Tracer is left with a dangling owner_. Span::End() then crashes inside owner_.EndSpan(). The snippet from the issue makes it obvious: auto provider = std::make_unique<etw::TracerProvider>(); auto tracer = provider->GetTracer("Foo"); auto spanFoo = tracer->StartSpan("Span-Foo"); tracer = provider->GetTracer("Bar"); // Tracer "Foo" goes away auto spanBar = tracer->StartSpan("Span-Bar"); spanFoo->End(); // crash spanBar->End(); Under MSVC Debug + cdb this lands as an AV in Span::End dereferencing 0xFEEEFEEE -- the debug heap's freed-memory fill -- which is about as unambiguous as UAF gets. The fix is to let TracerProvider keep the Tracers it hands out, keyed by name and guarded by a mutex, and have GetTracer() return the cached entry on repeat calls. That makes the Tracer outlive any Span derived from it, which is the lifetime contract Span's raw owner_ already assumes. It is also the same pattern sdk::trace::TracerProvider has always used, which is why the issue reporter noted the SDK provider does not crash under the same usage. There was an earlier attempt in PR #4070 that switched Span to take a shared_ptr<Tracer> via shared_from_this(). That was turned down because StartSpan() is the ETW exporter's hot path and nobody wants an extra atomic refcount bump on every span just to keep the parent alive. Doing the lifetime work once in GetTracer() keeps owner_ as a plain reference and costs nothing per span. While putting this together I tripped over a second bug that the cache exposes -- worth describing here because the fix lives in the same file. etw_tracer.h and etw_provider.h between them have three independent function-local statics that get tangled at process shutdown: 1. etw::TracerProvider itself (now owning the tracer cache). 2. Tracer::etwProvider() returns a `static ETWProvider instance`. 3. ETWProvider::providers() returns a `static std::map<...> providers` that lives inside that singleton's method, not inside the singleton object, and is lazily created on the first open()/is_registered() call. Before this change, all three were initialized in that order on the very first GetTracer() call: the TracerProvider was already constructed, then the Tracer ctor called etwProvider().open(...), which constructed #2 and #3. Reverse-order destruction therefore took out #3 first, while the TracerProvider (and its cached Tracers, which hold references into #3) was still alive. ~Tracer then ran etwProvider().close(provHandle) against a dangling reference and we got an AV at exit. With the old "fresh Tracer per call, caller drops it immediately" pattern this was hidden because the Tracer (and its provHandle reference) was almost always gone long before shutdown. The cache extends Tracer lifetime to match TracerProvider lifetime, which is exactly what tripped this over locally and on the Windows CI legs. The fix is small: both TracerProvider constructors now do (void)Tracer::etwProvider().is_registered(std::string{}); which forces both ETWProvider singletons to be constructed before the TracerProvider's body finishes. Reverse destruction then guarantees they outlive the cached Tracers. is_registered() is a public, side- effect-free read on ETWProvider that internally touches the providers map, which is exactly the static-init we need. It requires Tracer to befriend TracerProvider so the constructor can call the private etwProvider() accessor. A few things worth calling out: - No API or ABI change. Only private members are added to TracerProvider, and <map>/<memory>/<mutex> are already included by etw_tracer.h. The friend declaration is purely internal. - The cache deliberately does not evict. Span holds its parent by raw reference, so dropping a Tracer that still has live Spans would put the original bug right back. In practice the ETW exporter is built around "one Tracer per provider name, reused for the lifetime of the process", and Windows itself caps registered ETW providers per process well below anything you would worry about for memory. - Keying on name only is consistent with the existing etw::Tracer constructor, which already declares args and schema_url UNREFERENCED_PARAMETER. - etw::LoggerProvider was checked for the same shape and does not currently cache Loggers, so the destruction-order fix is not needed there today. If a similar cache is ever added to LoggerProvider it will want the same one-line force-init in its constructor. - Updated ETWTracer.GlobalSingletonTracer: it previously asserted that two GetTracer() calls with the same provider name produce different trace ids, which only held because of the bug being fixed here. GetTracer(name) is now idempotent per name, so the assertion is flipped to EXPECT_EQ and the stale sample-event comment updated to match. All other ETW tracer tests continue to pass. Repro before the change: AV in Span::End with the freed-fill operand, plus exit-time AV in ~Tracer on ctest runs that exercise a single test in isolation. Same repros after: both run cleanly to completion. Verified locally with the full ctest suite on a Windows abiv2 maintainer-mode build (1104/1104 passing) and on an abiv1 Debug build (43/43 ETW tests passing). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent 72953ea commit a994110

2 files changed

Lines changed: 70 additions & 7 deletions

File tree

exporters/etw/include/opentelemetry/exporters/etw/etw_tracer.h

Lines changed: 64 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,10 @@ class Tracer : public opentelemetry::trace::Tracer,
169169
bool IsClosed() const noexcept { return isClosed_.load(); }
170170

171171
private:
172+
// Allow TracerProvider to force-construct the ETWProvider singleton ahead
173+
// of itself; see TracerProvider's constructor for the rationale.
174+
friend class TracerProvider;
175+
172176
/**
173177
* @brief Parent provider of this Tracer
174178
*/
@@ -1106,6 +1110,17 @@ class TracerProvider : public opentelemetry::trace::TracerProvider
11061110
id_generator_{std::move(id_generator)},
11071111
tail_sampler_{std::move(tail_sampler)}
11081112
{
1113+
// Force-initialize the process-wide ETWProvider singleton -- and its
1114+
// internal providers() map (lazily created on first ETWProvider method
1115+
// call) -- before this TracerProvider finishes constructing. Static
1116+
// destruction is reverse-of-construction order, so initializing both
1117+
// here guarantees that they outlive this provider's tracer cache:
1118+
// cached Tracer destructors call etwProvider().close(provHandle), and
1119+
// provHandle is a reference into that providers() map. Without this,
1120+
// either singleton could be destroyed first, leaving cached Tracer
1121+
// destructors with dangling state at process shutdown (#3849).
1122+
(void)Tracer::etwProvider().is_registered(std::string{});
1123+
11091124
// By default we ensure that all events carry their with TraceId and SpanId
11101125
GetOption(options, "enableTraceId", config_.enableTraceId, true);
11111126
GetOption(options, "enableSpanId", config_.enableSpanId, true);
@@ -1143,6 +1158,17 @@ class TracerProvider : public opentelemetry::trace::TracerProvider
11431158
tail_sampler_{
11441159
std::unique_ptr<opentelemetry::exporter::etw::TailSampler>(new AlwaysOnTailSampler())}
11451160
{
1161+
// Force-initialize the process-wide ETWProvider singleton -- and its
1162+
// internal providers() map (lazily created on first ETWProvider method
1163+
// call) -- before this TracerProvider finishes constructing. Static
1164+
// destruction is reverse-of-construction order, so initializing both
1165+
// here guarantees that they outlive this provider's tracer cache:
1166+
// cached Tracer destructors call etwProvider().close(provHandle), and
1167+
// provHandle is a reference into that providers() map. Without this,
1168+
// either singleton could be destroyed first, leaving cached Tracer
1169+
// destructors with dangling state at process shutdown (#3849).
1170+
(void)Tracer::etwProvider().is_registered(std::string{});
1171+
11461172
config_.enableTraceId = true;
11471173
config_.enableSpanId = true;
11481174
config_.enableActivityId = false;
@@ -1152,6 +1178,31 @@ class TracerProvider : public opentelemetry::trace::TracerProvider
11521178
config_.encoding = ETWProvider::EventFormat::ETW_MANIFEST;
11531179
}
11541180

1181+
/**
1182+
* @brief Cache of Tracers handed out by GetTracer(), keyed by name.
1183+
*
1184+
* The ETW Span class stores its parent Tracer as a raw reference
1185+
* (Tracer &owner_); if the only strong refcount on the Tracer is the
1186+
* nostd::shared_ptr returned by GetTracer(), the Tracer is destroyed
1187+
* as soon as the caller drops it, leaving every Span derived from it
1188+
* with a dangling owner_ that crashes on End(). Cache here so the
1189+
* provider keeps the Tracer alive as long as it is alive itself,
1190+
* matching the lifetime contract used by sdk::trace::TracerProvider.
1191+
*
1192+
* Growth: this cache is unbounded by design and mirrors
1193+
* sdk::trace::TracerProvider's tracers_ vector. Eviction is unsafe
1194+
* because the ETW Span stores its parent Tracer by raw reference, so
1195+
* dropping a Tracer that still has outstanding Spans would
1196+
* re-introduce the use-after-free this cache exists to prevent.
1197+
*
1198+
* In practice the cache stays small because the ETW exporter is
1199+
* designed around "one Tracer per ETW provider name, reused for the
1200+
* process/component lifetime" -- and Windows itself caps the number
1201+
* of registered ETW providers per process well below any RAM concern.
1202+
*/
1203+
std::map<std::string, std::shared_ptr<opentelemetry::trace::Tracer>> tracers_;
1204+
std::mutex tracers_lock_;
1205+
11551206
/**
11561207
* @brief Obtain ETW Tracer.
11571208
* @param name ProviderId (instrumentation name) - Name or GUID
@@ -1177,9 +1228,19 @@ class TracerProvider : public opentelemetry::trace::TracerProvider
11771228
UNREFERENCED_PARAMETER(args);
11781229
UNREFERENCED_PARAMETER(schema_url);
11791230
ETWProvider::EventFormat evtFmt = config_.encoding;
1180-
std::shared_ptr<opentelemetry::trace::Tracer> tracer{new (std::nothrow)
1181-
Tracer(*this, name, evtFmt)};
1182-
return nostd::shared_ptr<opentelemetry::trace::Tracer>{tracer};
1231+
// Cache Tracers by name. The exporter Span class holds a raw
1232+
// Tracer& as owner_, so the Tracer must live at least as long as
1233+
// the longest-lived Span derived from it. Without caching, each
1234+
// GetTracer() call mints a fresh heap Tracer that is destroyed the
1235+
// moment the caller drops the returned shared_ptr, orphaning any
1236+
// outstanding Spans (use-after-free in Span::End()).
1237+
std::lock_guard<std::mutex> lock(tracers_lock_);
1238+
auto &cached = tracers_[std::string{name.data(), name.size()}];
1239+
if (!cached)
1240+
{
1241+
cached.reset(new (std::nothrow) Tracer(*this, name, evtFmt));
1242+
}
1243+
return nostd::shared_ptr<opentelemetry::trace::Tracer>{cached};
11831244
}
11841245
};
11851246

exporters/etw/test/etw_tracer_test.cc

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -392,12 +392,14 @@ TEST(ETWTracer, GlobalSingletonTracer)
392392
}
393393
*/
394394

395-
// Obtain a different tracer withs its own trace-id.
395+
// GetTracer(name) is now idempotent per name: a second call with the same
396+
// provider name returns the same cached Tracer (see #3849), so the trace
397+
// id matches the global tracer's.
396398
auto localTracer = GetGlobalTracerProvider().GetTracer(kGlobalProviderName);
397399
auto s2 = localTracer->StartSpan("Span2");
398400
auto traceId2 = s2->GetContext().trace_id();
399401
s2->End();
400-
/* === Span 2 - "TraceId": "334bf9a1eed98d40a873a606295a9368"
402+
/* === Span 2 - "TraceId": "182a64258fb1864ca4e1a542eecbd9bf" (same trace as Span 1)
401403
{
402404
"Timestamp": "2021-05-10T11:45:27.0289654-07:00",
403405
"ProviderName": "OpenTelemetry-ETW-TLD",
@@ -418,7 +420,7 @@ TEST(ETWTracer, GlobalSingletonTracer)
418420
"StatusCode": 0,
419421
"StatusMessage": "",
420422
"Success": "True",
421-
"TraceId": "334bf9a1eed98d40a873a606295a9368",
423+
"TraceId": "182a64258fb1864ca4e1a542eecbd9bf",
422424
"_name": "Span"
423425
}
424426
}
@@ -455,7 +457,7 @@ TEST(ETWTracer, GlobalSingletonTracer)
455457
}
456458
}
457459
*/
458-
EXPECT_NE(traceId1, traceId2);
460+
EXPECT_EQ(traceId1, traceId2);
459461
EXPECT_EQ(traceId1, traceId3);
460462

461463
# if OPENTELEMETRY_ABI_VERSION_NO == 1

0 commit comments

Comments
 (0)