Skip to content

feat(openfeature): server-side EVP flagevaluation emission#4886

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 39 commits into
mainfrom
workspace/flag-evaluations-cross-sdk
Jun 22, 2026
Merged

feat(openfeature): server-side EVP flagevaluation emission#4886
gh-worker-dd-mergequeue-cf854d[bot] merged 39 commits into
mainfrom
workspace/flag-evaluations-cross-sdk

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Motivation

Go server-side flag evaluations need an EVP flagevaluation emission path alongside the existing OTel feature_flag.evaluations path.

The backend contract is owned by the dd-source flageval-worker: batched EVP event items accept only schema-visible fields, and OpenFeature reason is not an EVP event field. targeting_rule.key is real targeting-rule metadata, so the Go path omits targeting_rule until that metadata is available.

Changes

Area Implementation
Hook path OpenFeature Finally hook records EVP evaluations; the existing OTel hook path remains separate.
Hot path Evaluation work is bounded to scalar extraction, flattened/pruned context snapshotting, and a non-blocking queue offer.
Aggregation Background writer aggregates into full and degraded tiers, then counts drops when bounded queues or maps overflow.
Bucket dimensions Full tier keys by flag, variant, allocation, runtime default, error message, targeting key, and pruned context. Degraded tier keeps flag, variant, allocation, runtime default, and error message.
Payload contract SDK-owned tests assert the emitted EVP field set without copying the dd-source worker schema or adding a JSON Schema dependency.
EVP transport Exposures and flag evaluations share one EVP client and HTTP transport; endpoints and payload shapes remain separate.
Benchmarks Baseline-backed OpenFeature evaluation benchmarks from #4902 remain in microbenchmarks-3; on this branch the default client/provider benchmark exercises the EVP-enabled path. The focused BenchmarkFlagEvaluation* helpers remain runnable locally for overhead analysis, but are not listed in the benchmark gate until matching baseline names exist on main.
flowchart LR
    eval["OpenFeature evaluation"] --> provider["DatadogProvider<br/>UFC result"]
    provider --> hooks["Finally hooks<br/>OTel + EVP"]
    hooks --> writer["EVP writer<br/>bounded queue + aggregation"]
    writer --> intake["Agent EVP proxy<br/>dd-source flageval-worker"]
Loading

Decisions

  • The queue stores pruned evaluation snapshots, not raw OpenFeature evaluation context.
  • Full-tier context identity is a comparable canonical context string, not a hash digest.
  • Degraded aggregation drops targeting key and context for cardinality control while retaining schema-visible EVP dimensions.
  • Queue overflow and degraded overflow are observable counted drops; evaluation is never blocked on EVP telemetry.
  • Shutdown coordinates enqueue/drain/flush so the writer has one final flush path.
  • runtime_default_used comes from typed conversion/default behavior.
  • The worker schema stays in dd-source; the Go SDK only tests the fields it emits.

Validation Evidence

Local validation on June 20, 2026:

  • SDK commit: f2a6df08644a15aecaead0f68ca1e646dac276b3
  • system-tests commit: 95edcc8de242e0c70ddce87b6fab8a76e585f1e3
  • Weblog: golang / net-http
  • Runtime context: Agent 7.80.1; library golang@2.10.0-dev; Linux aarch64

Commands:

SYSTEM_TEST_BUILD_TIMEOUT=1200 ./build.sh --library golang --weblog-variant net-http
TEST_LIBRARY=golang WEBLOG_VARIANT=net-http ./run.sh +l golang FEATURE_FLAGGING_AND_EXPERIMENTATION -k "test_flag_eval_evp"

Result: 8 passed, 2627 deselected in 67.22s.

Repo-local validation on June 20, 2026:

  • go test -race ./openfeature - pass
  • go test ./openfeature -run '^(TestEndToEnd|TestFlagEvaluation|TestEvaluateStampsEvalTimeMetadata)' -count=1 -v - pass
  • go test ./openfeature -run '^$' -bench='^BenchmarkFlagEvaluation' -benchtime=100ms -count=1 - pass
  • git diff --check - pass

Dogfood validation:

  • SDK=go APP_URL=http://localhost:8081 WAIT_SECONDS=40 scripts/validate-server-flagevaluations.sh with DD_TRACE_GO_PATH=/Users/leo.romanovsky/gsd-workspaces/flag-evaluations-cross-sdk/dd-trace-go - pass (EVP flagevaluations OK: count=1, OTel non-regression OK: otel_count=1, service=ffe-dogfooding-go)
  • Direct backend query confirmed flagevaluation rows for Go targeting keys and evaluation_count aggregation. June 20 real-backend run id 1781931066 used prefix server-evp-e2e-go-1781931066-user-.

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Jun 12, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 87.70%
Overall Coverage: 62.86% (+0.24%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: f2a6df0 | Docs | Datadog PR Page | Give us feedback!

@codecov

codecov Bot commented Jun 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.30807% with 56 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.08%. Comparing base (86ab9ad) to head (f2a6df0).

Files with missing lines Patch % Lines
openfeature/flagevaluation.go 88.30% 31 Missing and 9 partials ⚠️
openfeature/evp.go 63.88% 7 Missing and 6 partials ⚠️
openfeature/flagevaluation_hook.go 75.00% 1 Missing and 1 partial ⚠️
openfeature/exposure.go 75.00% 1 Missing ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
openfeature/evaluator.go 77.82% <100.00%> (-2.91%) ⬇️
openfeature/exposure_hook.go 90.62% <ø> (-3.13%) ⬇️
openfeature/provider.go 74.10% <100.00%> (-0.27%) ⬇️
openfeature/exposure.go 88.34% <75.00%> (-1.66%) ⬇️
openfeature/flagevaluation_hook.go 75.00% <75.00%> (ø)
openfeature/evp.go 63.88% <63.88%> (ø)
openfeature/flagevaluation.go 88.30% <88.30%> (ø)

... and 290 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

… for EVP flagevaluation

- flagevaluation.go: signature-only stubs for aggregator, writer, pruneContext,
  extractEvalDetails, types (evaluationAggregationKey, evaluationDegradedKey,
  evaluationUltraDegradedKey, evaluationEntry, flagEvaluationEvent, flagEvaluationPayload,
  flagEvalDDContext, flagEvaluationWriter, evalDetails), and constants (flagEvaluationEndpoint,
  maxContextFields, maxFieldLength, defaultEvalGlobalCap, defaultEvalPerFlagCap,
  defaultEvalDegradedCap, defaultFlagEvalFlushInterval); all methods panic("not implemented")
- flagevaluation_hook.go: signature-only stubs for flagEvaluationHook.Finally and
  isRuntimeDefault; panics with "not implemented"
- flagevaluation_test.go: failing RED tests: TestPruneContext, TestFlagEvaluationPayloadSchema,
  TestAggregatorCollisionSafety, TestAggregatorConcurrentMinMax, TestAggregatorCapOverflow
- flagevaluation_hook_test.go: failing RED tests: TestIsRuntimeDefault,
  TestFlagEvaluationHookFinally, TestFlagEvaluationHookContextCancelled
- flageval_metrics.go untouched (PRES-01 invariant)
- Tests fail on panic("not implemented"), not compile errors (clean RED state)
…chmark scenarios

- BenchmarkFlagEvaluationNoop: raw evaluator baseline (zero hooks)
- BenchmarkFlagEvaluationOTelOnly: OTel hook only (existing Path A reference)
- BenchmarkFlagEvaluationOTelPlusEVP: OTel + EVP hook scaffold (plan 02 wires the hook)
- Each scenario has typical (100 flags, 50 users, 10-field context) and stress
  (10 flags, 1000 users, 250-field context) sub-benchmarks via b.Run
- makeBenchmarkConfig() and makeBenchmarkContext() helpers added
- All three compile and run with -benchtime=1x (CONT-08 scaffold)
- EVP hook created as nil-writer stub in OTelPlusEVP; body not called until plan 02 wires it
…reuse

- struct-keyed aggregation maps (collision-safe, CONT-05)
- three-tier cascade: full → degraded → ultra-degraded (CONT-04/CONT-10)
- first/last via min/max inside lock (CONT-06)
- pruneContext 256-field/256-char limits before buffering (CONT-03)
- phantom attempt counting: perFlagFull incremented even on globalCap drops
  so per-flag overflow path stays alive after full-tier cap is exhausted
- reused exposure.go transport: UDS/HTTP branch, evpSubdomainHeader/Value (D-04)
- dedicated 10s flush ticker, flush-on-shutdown, panic recovery (D-05)
- FlagEvaluationFlushInterval added to ProviderConfig
…helpers

- flagEvaluationHook.Finally: ctx.Done check, nil-guard, calls writer.record
- isRuntimeDefault: variant=="" primary signal; DEFAULT/DISABLED reason secondary (CONT-07)
- extractEvalDetails: shared helper for EVP hook only; flageval_metrics.go untouched (D-06/PRES-01)
…k 3)

- Add DD_FLAGGING_EVALUATION_COUNTS_ENABLED killswitch to provider (default true)
- Register flagEvaluationWriter + flagEvaluationHook in newDatadogProvider when enabled
- Start/flush/stop writer in InitWithContext / ShutdownWithContext
- Append EVP hook in Hooks() alongside existing OTel flagEvalHook (PRES-01 preserved)
- Add FlagEvaluationFlushInterval to ProviderConfig
- Register DD_FLAGGING_EVALUATION_COUNTS_ENABLED in supported_configurations.gen.go
- Add TestFlagEvaluationKillswitch: 3 subtests verifying killswitch behaviour
- Update TestNewDatadogProvider to expect 3 hooks (exposure + OTel + EVP)
…iter)

- BenchmarkFlagEvaluationOTelPlusEVP now uses ProviderConfig.FlagEvaluationFlushInterval=24h
  to construct a real EVP writer+hook without HTTP round-trips in the hot path (Pitfall 4)
- Removed the plan-01 stub comment and nil-writer construction
- Added time import for 24h duration constant
@leoromanovsky leoromanovsky force-pushed the workspace/flag-evaluations-cross-sdk branch from f53799f to 4b75012 Compare June 12, 2026 03:40
@pr-commenter

pr-commenter Bot commented Jun 12, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-06-20 04:11:48

Comparing candidate commit f2a6df0 in PR branch workspace/flag-evaluations-cross-sdk with baseline commit 86ab9ad in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 297 metrics, 2 unstable metrics, 1 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

  • scenario:BenchmarkOTLPTraceWriterFlush

…le field (CR-01, WR-01, WR-02)

- CR-01: hashContext now hashes over sorted keys so identical contexts produce a
  stable FNV-1a hash; Go map iteration is randomized and was fragmenting buckets
  and corrupting evaluation_count.
- WR-01: remove the never-populated targetingRuleKey field, its aggregation-key
  member, the flagEvalTargetingRule type, and the dead targeting_rule emission
  branch (no metadata source exists yet).
- WR-02: correct the globalCap comment to state it bounds the full tier only
  (degraded/ultra-degraded are bounded separately).
… (WR-03)

record() is a non-blocking in-memory add with no network call, so gating it on
ctx.Done() silently dropped legitimate evaluation counts when the request context
was already cancelled (undercut Gate-7). Remove the ctx.Done() guard and update
TestFlagEvaluationHookContextCancelled to assert the evaluation IS still counted.
The benchmark used b.N inside a b.Loop() body, which is a single fixed total (not a
per-iteration index), so every evaluation used an identical targeting key and the
50/1000-user cardinality profiles collapsed to one bucket. Use a per-iteration
counter with strconv.Itoa(i%numUsers) so the Gate-6 aggregation cost is exercised.
…d regenerate

The supported_configurations.json entry had placeholder FIX_ME values for type and
default, failing validate_supported_configurations_v2_local_file. Set type=boolean,
default=true (killswitch defaults on) and regenerate supported_configurations.gen.go
via scripts/configinverter so the JSON and generated map are canonical and in sync.
… EVP flagevaluation contract

- #1 (blocker): route globalCap overflow to ultra-degraded instead of dropping; count total preserved at every tier
- #3: add TestSaturationCountPreservation (total counts == add() calls past both globalCap and perFlagCap)
- #2: correct collision-safe wording to low-collision/not collision-proof; document count-preserving collision behavior
- #4: rotate flag keys + distinct context field names in benchmark so real cardinality is exercised
…; benchmark + comment fixes

The OpenFeature Go SDK runs hooks synchronously on the caller's eval goroutine
(client.go invokes finallyHooks via defer, no goroutine), so aggregating inside
Finally added +112%/+180% latency over OTel-only. Move the work off the hot path.

- async recording: Finally now does only cheap scalar extraction + the SDK's shallow
  Attributes() copy + a non-blocking bounded enqueue; a single background worker owns
  flatten/prune/hash/aggregate/flush. Queue-full drops with an observable counter
  (best-effort); stop() drains + final-flushes. Hot-path cost is a flat 5 allocs
  (~0.65us small / ~6us large context).
- hashContext: replace per-field fmt.Fprintf with direct byte writes (250-field hash
  drops from ~250 allocs to ~1).
- benchmark: drive evaluations through openfeature.Client so the hooks actually run
  (direct provider calls bypassed Hooks(), so prior overhead numbers measured nothing);
  isolate OTel-only vs OTel+EVP; rotate flag + user; add a hot-path-only micro-bench.
- tests: drain the async queue in the hook tests; add TestFlagEvaluationBackpressureDrops.
- comments: remove internal planning identifiers (this reference SDK is human-facing).
Comment thread openfeature/flagevaluation_hook.go Outdated
…egraded bound, post-stop record, and error message

- Deterministic context pruning: sort keys BEFORE the 256-field cap and
  oversized-string skip so identical logical contexts prune to the same
  subset and hash equal (no bucket fragmentation).
- Canonical context hash encoding: type-tagged + length-delimited key/value
  so int 1 != string "1" and '=' / '\n' in keys/values cannot alias a
  multi-field context.
- Bounded ultra-degraded tier: explicit ultraDegCap (<=0 treated as default)
  with a single count-preserving sentinel overflow bucket (__other__) so
  dynamic/malicious flag keys cannot grow the map without bound.
- Post-stop record() no-op: stopped is now atomic.Bool used as the single
  stop() idempotency gate; record() checks it lock-free and counts post-stop
  events as dropped instead of enqueuing into the never-drained channel.
- Error payload prefers OpenFeature ErrorMessage, falling back to ErrorCode
  only when ErrorMessage is empty.
- Doc comments on evaluationAggregationKey, hashContext, and
  addToUltraDegraded updated to match the fixed behavior.

OTel feature_flag.evaluations path (flageval_metrics.go) is unchanged.
… review)

- isRuntimeDefault now keys solely on an empty Variant; our evaluator sets a
  variant only on a matched allocation (TARGETING_MATCH/SPLIT/STATIC), so every
  DEFAULT/DISABLED/ERROR path already leaves it empty.
- Removes the dead secondary reason-clause (DEFAULT/DISABLED) flagged as
  belt-and-suspenders; a present variant now correctly means a real assignment,
  including for foreign providers.
- Aligns with PR #4886 reviewer concern: detect runtime default from absence of
  variant, not reason alone.
Comment thread openfeature/flagevaluation.go Outdated
Comment thread openfeature/flagevaluation.go Outdated
Comment thread openfeature/flagevaluation.go Outdated
Comment thread openfeature/provider_bench_test.go
… remove ultra-degraded; resize caps for 2,500-flag scale
…ier buckets

Address oleksii PR #4886 review #2: replace the lossy FNV-1a contextHash
discriminator in evaluationAggregationKey with an exact, comparable
canonical-context string (contextKey). The existing type-tagged,
length-delimited canonical encoding is now emitted AS THE MAP KEY instead of
hashed, so Go's map compares it byte-for-byte.

- Distinct contexts now ALWAYS land in distinct full-tier buckets: there is no
  hash, so no hash collision, so a count can never be misattributed to the
  wrong context (the prior collision-misattribution case is gone).
- Delete hash/fnv import, hashContext, and the collision-analysis doc block.
- Rename hashContext -> canonicalContextKey (returns the encoding as a string).
- Tests assert distinct contexts (int 1 vs string "1"; '='/newline-bearing
  values; multi-field) land in DISTINCT buckets with zero misattribution, and
  identical contexts still merge into one bucket.

Memory: the contextKey string is stored once per full-tier bucket, bounded by
globalCap and the 256-field/256-char prune. EVPRecord hot path unchanged (the
key is built in the async worker, not record()). OTel path untouched.
…rsal

Address oleksii PR #4886 review #4: the worker path previously ran
flattenContext (flatten.go) then pruneContext as two separate steps, each
allocating its own map. Merge them into a single flattenAndPruneContext used by
aggregate(): flatten nested objects, then apply the deterministic
sort-before-cut 256-field / 256-char prune in one pass.

Output is byte-for-byte identical to the prior two-step pipeline (same surviving
keys, same 256/256 limits, same deterministic cut). When the flattened context
already fits the limits — the common case — the flattened map is returned
directly, eliding the second pruned-output map the old pipeline always
allocated.

flattenContext is left unchanged because exposure_hook.go still calls it; only
the EVP aggregation path is merged. pruneContext is retained as the reference
behavior for tests. Added TestFlattenAndPruneContextEquivalence asserting the
merged pass equals flattenContext+pruneContext across nested, oversized, and
>256-field inputs, with preserved determinism.
…ook-fire time (review #3)

Capture the evaluation timestamp once at DatadogProvider.evaluate entry, thread it into
evaluateFlag (reusing the time.Now() the evaluator already computes for allocation windows),
and stamp it into FlagMetadata on every return path. The EVP flagevaluation hook reads it for
first/last_evaluation bounds instead of calling time.Now() at hook-fire time (slightly later
and slightly incorrect). Falls back to hook-fire time when absent (non-Datadog provider).
flageval_metrics.go untouched.
Comment thread openfeature/flagevaluation.go

@typotter typotter left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm no expert on metrics (but Vickie is). Overall, lgtm pending all of Oleksii's concerns being addressed

@petzel petzel left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking great so far.

Since the EvP intake only validates the API key (and not the structure of the payload), it would be good to verify this behavior end to end before we merge. Meaning, we should be able to show that we can query this data in the flagevaluation track using pup.

Comment thread openfeature/flagevaluation.go Outdated
Comment thread openfeature/flagevaluation.go Outdated
Comment thread openfeature/flagevaluation.go Outdated
Comment thread openfeature/flagevaluation.go Outdated
@leoromanovsky

leoromanovsky commented Jun 19, 2026

Copy link
Copy Markdown
Contributor Author

looking great so far.

Since the EvP intake only validates the API key (and not the structure of the payload), it would be good to verify this behavior end to end before we merge. Meaning, we should be able to show that we can query this data in the flagevaluation track using pup.

@petzel Certainly a good idea - here's the evidence.

FYI removed from public repo.

gh-worker-dd-mergequeue-cf854d Bot pushed a commit that referenced this pull request Jun 19, 2026
## Motivation

OpenFeature flag evaluation already has Go benchmarks, but they are not wired into the microbenchmark CI framework. The server-side flag-evaluation work needs a stable main-branch benchmark group for the client-facing evaluation path before adding EVP flagevaluation emission.

The top-line performance question is whether returning a variant through the public OpenFeature client regresses, and whether concurrent evaluations remain viable under load.

## Changes

- Add `BenchmarkOpenFeatureClientEvaluation` for the public OpenFeature client path with the provider default hooks.
- Add `BenchmarkOpenFeatureClientConcurrentEvaluations` for concurrent public-client evaluations.
- Add `microbenchmarks-3` with the new client-path benchmarks and the existing evaluator-focused OpenFeature benchmarks.

## Decisions

- The new top-line benchmarks use the public OpenFeature client instead of direct provider methods, because hook overhead is visible only on the client path.
- Existing direct-provider benchmarks are included as evaluator-focused diagnostics.
- `microbenchmarks-3` is intentionally not added to `pr-performance-gates` in this PR. The two new client benchmark names do not exist on `main` yet, so the gate cannot produce same-name baseline artifacts for them until after this PR lands. A follow-up feature PR can opt group 3 into the gate once these names are present on `main`.
- This PR intentionally does not add EVP-specific benchmark names; those belong in the flagevaluation PR after this baseline exists on `main`.

## Next Steps

After this PR lands on `main`, the server-side EVP flagevaluation PR ([#4886](#4886)) should:

- Reuse the `microbenchmarks-3` group introduced here. The EVP PR should not create a separate benchmark group for flagevaluations.
- Opt the existing `microbenchmarks-3` job into `pr-performance-gates` once `main` has baseline artifacts for these benchmark names.
- Keep the main-branch client-facing benchmark names as the primary regression signal, so the gate compares normal public-client evaluation against the same path with EVP emission wired in.
- Keep the EVP-specific benchmarks in that PR as diagnostics for no-op evaluation, OTel-only evaluation, OTel plus EVP aggregation, direct EVP record ingestion, and parallel EVP aggregation. Those explain where overhead comes from, while the client-facing benchmark remains the top-line signal.

## Validation

```
go test ./openfeature
go test ./openfeature -run='^$' -bench='^(BenchmarkOpenFeatureClientEvaluation|BenchmarkOpenFeatureClientConcurrentEvaluations|BenchmarkBooleanEvaluation|BenchmarkStringEvaluation|BenchmarkIntEvaluation|BenchmarkFloatEvaluation|BenchmarkEvaluation|BenchmarkEvaluationWithVaryingContextSize|BenchmarkEvaluationWithVaryingFlagCounts|BenchmarkConcurrentEvaluations)$' -benchmem -benchtime=100ms
```

## E2E: Staging Datadog APM proof

- Ran the dogfooding staging APM proof against a locally built SDK and the staging Agent using `DD_SITE=datad0g.com`.
- Validation id: `ffe-span-dotnet-20260619121831-b294dd1`; service: `ffe-dogfooding-dotnet`; env: `staging`; scenario: `span_enrichment.aggregate`; SDK sha: `b294dd1`.
- Staging Datadog MCP query used `base_url=https://dd.datad0g.com` and found `8` matching spans for `service:ffe-dogfooding-dotnet env:staging @validation_id:ffe-span-dotnet-20260619121831-b294dd1`.
- Trace Explorer: https://dd.datad0g.com/apm/traces?end=1781904227862&historicalData=true&paused=true&query=service%3Affe-dogfooding-dotnet+env%3Astaging+%40validation_id%3Affe-span-dotnet-20260619121831-b294dd1&start=1781817827862
- Indexed span-enrichment evidence:
  - `trace_id=6a35358d0000000091168b7fb4b815a2`, `span_id=15600877720192993068`, `resource=POST /scenario`, `ffe_flags_enc=vQblBKMP`
  - `trace_id=6a3534fd000000002e93ea8b4f41ec5d`, `span_id=12009548985036661845`, `resource=POST /scenario`, `ffe_flags_enc=vQblBKMP`


Co-authored-by: leo.romanovsky <leo.romanovsky@datadoghq.com>
@gh-worker-ownership-write-b05516 gh-worker-ownership-write-b05516 Bot removed the request for review from a team June 20, 2026 03:50
@leoromanovsky

Copy link
Copy Markdown
Contributor Author

I dug into the OpenFeature benchmark failures and now have a clearer read on them.

Async EVP delivery remains the right architecture, but pre-enqueue work is still on the hot path and must remain small. I reduced avoidable timestamp/context/event work, kept non-blocking best-effort delivery with bounded drops under pressure, and preserved evaluation timestamps for correctness even though that means accepting a small allocation cost.

@bm1549 bm1549 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Config add LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants