Support lazy metric point allocations by saguiitay · Pull Request #7346 · open-telemetry/opentelemetry-dotnet

saguiitay · 2026-05-25T16:34:38Z

Fixes #6620
Design discussion issue #

Changes

This PR adds opt-in lazy metric point allocation for high-cardinality metric streams. Today, AggregatorStore eagerly allocates storage for CardinalityLimit + 2 metric points per stream, which can consume a lot of memory even when little or no telemetry is emitted.

Default behavior is unchanged. Existing users continue to get eager allocation unless they explicitly opt in.

Public API

Adds experimental opt-in controls:

 builder.SetDefaultMetricPointLazyAllocation(true);

 new MetricStreamConfiguration
 {
     CardinalityLimit = 100_000,
     EnableMetricPointLazyAllocation = true,
 }

View configuration takes precedence over the provider default. The API is marked experimental with OTEL1006.

Implementation

Lazy mode uses segmented metric point storage, allocating metric points in fixed-size chunks as time series are observed instead of allocating the full cardinality limit up front. Eager allocation remains the default path.

Performance impact

For users who do not opt in, there should be no intentional CPU or memory behavior change.
For opt-in users with high CardinalityLimit and low actual cardinality, memory usage drops significantly.
Measured dry BenchmarkDotNet result for 100 metrics, each with CardinalityLimit = 100_000, emitting one time series:

Scenario	Mean	Allocated
Default eager	246.3 ms	725.39 MB
Lazy allocation	133.0 ms	2.75 MB

That is roughly ~99.6% less allocated memory and ~46% lower cold-start/setup time in this benchmark. The CPU improvement comes mostly from avoiding large eager initialization.

Test coverage

Added focused unit, integration, stress, and benchmark coverage for lazy allocation, while also verifying that default eager behavior remains unchanged.

Merge requirement checklist

CONTRIBUTING guidelines followed (license requirements, nullable enabled, static analysis, etc.)
Unit tests added/updated
Appropriate CHANGELOG.md files updated for non-trivial changes
Changes in public API reviewed (if applicable)

linux-foundation-easycla · 2026-05-25T16:34:45Z

The committers listed above are authorized under a signed CLA.

✅ login: saguiitay / name: Itay Sagui (3fade33)

martincostello · 2026-05-30T14:14:26Z

The CLA needs to be signed.

saguiitay · 2026-05-30T14:59:24Z

The CLA needs to be signed.

DONE

github-actions · 2026-06-09T04:37:55Z

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or pushing will instruct the bot to automatically remove the label. This bot runs once per day.

cijothomas · 2026-06-09T19:42:20Z

I haven't review this PR in detail but one comment to think though:

I am unsure if we need to let end users worry about this option - it is adding more cognitive overhead to endusers when setting up OTel.
it may be better to change the default itself to follow this. This will cause increased jitter due to re-allocations as things get warmed up, and then settles. But once in steady state, no re-allocation needed, as we don't ever reclaim the underlying array from the Dictionary.

Would be good if this can be discussed in SIG too.

codecov · 2026-06-09T19:46:07Z

Codecov Report

❌ Patch coverage is 94.59459% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.30%. Comparing base (d84d2af) to head (6ad0626).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/OpenTelemetry/Metrics/AggregatorStore.cs	94.28%	6 Missing ⚠️
src/OpenTelemetry/Metrics/MeterProviderSdk.cs	60.00%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7346   +/-   ##
=======================================
  Coverage   90.29%   90.30%           
=======================================
  Files         287      288    +1     
  Lines       15617    15704   +87     
=======================================
+ Hits        14102    14182   +80     
- Misses       1515     1522    +7

Flag	Coverage Δ
unittests-Project-Experimental	`90.29% <94.59%> (-0.04%)`	⬇️
unittests-Project-Stable	`90.20% <94.59%> (-0.07%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/OpenTelemetry/Metrics/Metric.cs	`100.00% <100.00%> (ø)`
...emetry/Metrics/MetricPoint/MetricPointsAccessor.cs	`100.00% <100.00%> (ø)`
...Metrics/MetricPoint/SegmentedMetricPointStorage.cs	`100.00% <100.00%> (ø)`
...rc/OpenTelemetry/Metrics/Reader/MetricReaderExt.cs	`92.24% <100.00%> (+0.18%)`	⬆️
src/OpenTelemetry/Metrics/MeterProviderSdk.cs	`93.41% <60.00%> (-0.73%)`	⬇️
src/OpenTelemetry/Metrics/AggregatorStore.cs	`90.65% <94.28%> (+0.99%)`	⬆️

... and 2 files with indirect coverage changes

Per PR review feedback, drop the verbose `MetricPoint` segment from the new lazy-allocation API surface. Also renames the matching `SetDefaultMetricPointLazyAllocation` extension/builder method, `UsesMetricPointLazyAllocation` test helper, the internal `enableMetricPointLazyAllocation` field/parameter names, and the `LazyMetricPointAllocationExperimentalApi` diagnostics constant for consistency. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Documents the new lazy metric point allocation APIs (`MetricStreamConfiguration.EnableLazyAllocation` and `MeterProviderBuilderExtensions.SetDefaultLazyAllocation`) under docs/diagnostics/experimental-apis/ and registers OTEL1006 in the Active section of the README. Feedback link points at open-telemetry#6620. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Group `MetricPointAllocationBenchmarks` benchmarks by scenario (NoMeasurements, OneTimeSeries, TenTimeSeries, ManyMetricsHighCardinality) and mark the eager variant in each pair as `[Benchmark(Baseline = true)]` so the BenchmarkDotNet output reports the lazy variant with a ratio relative to its matching eager baseline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Per PR review feedback, do not extend the default `MetricsBenchmarks` matrix with an `EnableLazyAllocation` `[Params(false, true)]` (and its accompanying `AddView`) — doubling every existing combination is not worth the runtime cost when the dedicated `MetricPointAllocationBenchmarks` already covers the lazy-vs-eager comparison. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Per PR review feedback, switch the `ObservableLongCounterInstrument` test fixture callback from `() => 0L` (the default value of `long`, which reads like a placeholder and would be indistinguishable from a missing observation if the SDK ever pumped the callback) to `() => 42L` so the value is visibly distinct from the manually-driven `aggregatorStore.Update(1, …)` in `LazyAsynchronousCumulativeSnapshotSkipsStalePoints`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Per PR review feedback, the timing adjustments to `CompositeMetricReader_PreservesOriginalTimeoutBudget` (`MultipleReadersTests`) and `CompositeActivityProcessor_PreservesOriginalTimeoutBudget` (`CompositeActivityProcessorTests`) are unrelated to the lazy metric point allocation work and should not be part of this PR. Reverts both files to their pre-PR state; the original tests still pass on net10.0 (11/11 and 10/10 respectively). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Adds two tests targeting code paths flagged by codecov as uncovered in AggregatorStore after the lazy allocation changes: * LazyAllocationWorksWithMultiTagDeltaTemporality - exercises the multi-tag (length > 1) allocation path in LookupAggregatorStoreForDeltaWithReclaim with lazy + delta. * LazyAllocationOverflowsDoubleCounter - exercises the overflow path in UpdateDoubleMetricPoint (GetMetricPoint(1).Update) and the TraceBased exemplar branch on a Counter<double>. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

saguiitay · 2026-06-11T14:47:46Z

@martincostello I resolved all comments

cijothomas

Please use the SIG meeting to discuss this more.
I don't think we should expand the configuration to add this feature - its forcing end users to make yet another decision.
If the original pre-allocation is not preferred, then it is better to change that itself and make it the default behavior.

cijothomas · 2026-06-11T16:05:53Z

@saguiitay Would you be able to join the community call to discuss more on this? (I won't be able to join, unfortunately, but I'll share some thoughts in the issue itself)

saguiitay · 2026-06-11T17:38:06Z

Sure, as long as it's in reasonable hours (I'm located in Israel time - GMT+3)

martincostello · 2026-06-11T17:44:39Z

That's probably not workable for you then, as I think that would make the SIG at 2200 for you (it's usually at 1900 for me and I'm in the UK).

saguiitay · 2026-06-14T11:02:57Z

That's probably not workable for you then, as I think that would make the SIG at 2200 for you (it's usually at 1900 for me and I'm in the UK).

I can work with that...

saguiitay · 2026-06-14T11:04:15Z

Please use the SIG meeting to discuss this more. I don't think we should expand the configuration to add this feature - its forcing end users to make yet another decision. If the original pre-allocation is not preferred, then it is better to change that itself and make it the default behavior.

If it's up to me, we keep the option for a while, gather feedback from users, and only enable it by default if it's proven in production. I'd hate changing the default behavior, and getting a backlash from users that we broke them.

martincostello · 2026-06-16T10:59:03Z

I've added it to today's SIG agenda.

cijothomas · 2026-06-24T03:55:24Z

I've added it to today's SIG agenda.

@martincostello was this discussed? Can you update if any decision was made.

if it's up to me, we keep the option for a while, gather feedback from users, and only enable it by default if it's proven in production. I'd hate changing the default behavior, and getting a backlash from users that we broke them.

@saguiitay Thanks, this idea makes sense. (And OTEL_DOTNET_EXPERIMENTAL_METRICS_ENABLE_LAZY_ALLOCATION approach would mean only advanced users will need to make this decision, and after evaluation, we can in future make this the default).

Copilot

Pull request overview

This PR introduces an opt-in “lazy” metric point allocation mode in the Metrics SDK to reduce memory usage for high-cardinality streams by allocating metric point storage in fixed-size chunks on demand, while keeping eager allocation as the default.

Changes:

Added segmented metric point storage and wired AggregatorStore to use it when lazy allocation is enabled.
Plumbed a provider-level lazy-allocation flag from MeterProviderSdk → MetricReader → Metric/AggregatorStore, plus added an experimental environment-variable switch.
Added unit tests and benchmarks to validate chunk allocation behavior and quantify memory/time improvements; updated docs and changelog.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
test/OpenTelemetry.Tests/Metrics/SegmentedMetricPointStorageTests.cs	New unit tests validating segmented storage allocation behavior and concurrency safety.
test/OpenTelemetry.Tests/Metrics/MetricViewTests.cs	Minor formatting-only change.
test/OpenTelemetry.Tests/Metrics/AggregatorTests.cs	Added tests covering lazy allocation behavior across cumulative/delta, overflow, reclaim/reuse, and tag handling.
test/Benchmarks/Metrics/MetricPointAllocationBenchmarks.cs	New benchmarks comparing eager vs lazy allocation in common scenarios.
src/OpenTelemetry/Metrics/Reader/MetricReaderExt.cs	Passes provider-level lazy allocation setting into `Metric` creation.
src/OpenTelemetry/Metrics/MetricPoint/SegmentedMetricPointStorage.cs	New internal segmented/chunked storage implementation for metric points.
src/OpenTelemetry/Metrics/MetricPoint/MetricPointsAccessor.cs	Updated public enumerator plumbing to support segmented storage.
src/OpenTelemetry/Metrics/Metric.cs	Threads lazy-allocation flag through to `AggregatorStore`.
src/OpenTelemetry/Metrics/MeterProviderSdk.cs	Adds experimental env-var config key and propagates the setting to readers.
src/OpenTelemetry/Metrics/Builder/MeterProviderBuilderExtensions.cs	Preprocessor change related to CodeAnalysis import guarding.
src/OpenTelemetry/Metrics/AggregatorStore.cs	Core implementation changes: optional segmented storage, lazy index reservation, and batch resizing logic.
src/OpenTelemetry/CHANGELOG.md	Documents the new experimental environment variable.
docs/metrics/customizing-the-sdk/README.md	Documents how to enable lazy metric point allocation via environment variable.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

 public readonly struct MetricPointsAccessor
 {
-    private readonly MetricPoint[] metricsPoints;
+    private readonly MetricPoint[]? metricsPoints;
+    private readonly SegmentedMetricPointStorage? segmentedMetricPoints;
    private readonly int[] metricPointsToProcess;


                    Metric metric = new(
                        metricStreamIdentity,
                        this.GetAggregationTemporality(metricStreamIdentity.InstrumentType),
                        metricStreamConfig?.CardinalityLimit ?? this.cardinalityLimit,
                        exemplarFilter,
-                        metricStreamConfig?.ExemplarReservoirFactory);
+                        metricStreamConfig?.ExemplarReservoirFactory,
+                        this.enableLazyAllocation);


martincostello · 2026-06-24T06:58:05Z

@martincostello was this discussed? Can you update if any decision was made.

Yes, Itay joined the SIG call and myself and Alan discussed it. TL;DR was Itay was going to speak to Raj about it internally and then look to make an experimental opt-in to allow for it to be tested in the wild. I forget exactly what else we discussed, but it'll be in the recording.

cijothomas · 2026-06-25T09:56:25Z

@martincostello was this discussed? Can you update if any decision was made.

Yes, Itay joined the SIG call and myself and Alan discussed it. TL;DR was Itay was going to speak to Raj about it internally and then look to make an experimental opt-in to allow for it to be tested in the wild. I forget exactly what else we discussed, but it'll be in the recording.

Got it. I agree with this approach. The experimental opt-in is via ENV variables right?

@saguiitay Can you make the opt-in via ENV variable, and fix CI. I'll help review.

martincostello · 2026-06-25T09:59:24Z

The experimental opt-in is via ENV variables right?

Yep.

Support lazy metric point allocations

3fade33

saguiitay requested a review from a team as a code owner May 25, 2026 16:34

github-actions Bot added pkg:OpenTelemetry Issues related to OpenTelemetry NuGet package perf Performance related labels May 25, 2026

saguiitay mentioned this pull request May 25, 2026

Memory optimization suggestion #6620

Open

martincostello reviewed Jun 1, 2026

View reviewed changes

github-actions Bot added the Stale Issues and pull requests which have been flagged for closing due to inactivity label Jun 9, 2026

github-actions Bot removed the Stale Issues and pull requests which have been flagged for closing due to inactivity label Jun 10, 2026

saguiitay and others added 7 commits June 11, 2026 15:32

github-actions Bot added the documentation Documentation related label Jun 11, 2026

Merge branch 'main' into user/itsagui/lazy-allocation

8d65c8a

cijothomas requested changes Jun 11, 2026

View reviewed changes

saguiitay and others added 2 commits June 17, 2026 08:00

Moved to use an EnvironmentVariable

21d3cd0

Merge branch 'main' into user/itsagui/lazy-allocation

4118ace

cijothomas requested a review from Copilot June 24, 2026 03:56

Copilot started reviewing on behalf of cijothomas June 24, 2026 03:56 View session

Copilot AI reviewed Jun 24, 2026

View reviewed changes

Addressed comments

d7627c6

saguiitay requested a review from cijothomas June 29, 2026 11:31

Merge branch 'main' into user/itsagui/lazy-allocation

ea6b61b

martincostello reviewed Jun 29, 2026

View reviewed changes

Comment thread src/OpenTelemetry/CHANGELOG.md Outdated

saguiitay and others added 2 commits July 7, 2026 13:02

Merge branch 'main' into user/itsagui/lazy-allocation

7ffa870

Update changelog

6ad0626

Uh oh!

Conversation

saguiitay commented May 25, 2026

Changes

Public API

Implementation

Performance impact

Test coverage

Merge requirement checklist

Uh oh!

linux-foundation-easycla Bot commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martincostello commented May 30, 2026

Uh oh!

saguiitay commented May 30, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jun 9, 2026

Uh oh!

cijothomas commented Jun 9, 2026

Uh oh!

codecov Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

saguiitay commented Jun 11, 2026

Uh oh!

cijothomas left a comment

Choose a reason for hiding this comment

Uh oh!

cijothomas commented Jun 11, 2026

Uh oh!

saguiitay commented Jun 11, 2026

Uh oh!

martincostello commented Jun 11, 2026

Uh oh!

saguiitay commented Jun 14, 2026

Uh oh!

saguiitay commented Jun 14, 2026

Uh oh!

martincostello commented Jun 16, 2026

Uh oh!

cijothomas commented Jun 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

martincostello commented Jun 24, 2026

Uh oh!

cijothomas commented Jun 25, 2026

Uh oh!

martincostello commented Jun 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

linux-foundation-easycla Bot commented May 25, 2026 •

edited

Loading

codecov Bot commented Jun 9, 2026 •

edited

Loading