feat(openfeature): server-side EVP flagevaluation emission#4886
Conversation
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: f2a6df0 | Docs | Datadog PR Page | Give us feedback! |
Codecov Report❌ Patch coverage is Additional details and impacted files
🚀 New features to boost your workflow:
|
… for EVP flagevaluation
- flagevaluation.go: signature-only stubs for aggregator, writer, pruneContext,
extractEvalDetails, types (evaluationAggregationKey, evaluationDegradedKey,
evaluationUltraDegradedKey, evaluationEntry, flagEvaluationEvent, flagEvaluationPayload,
flagEvalDDContext, flagEvaluationWriter, evalDetails), and constants (flagEvaluationEndpoint,
maxContextFields, maxFieldLength, defaultEvalGlobalCap, defaultEvalPerFlagCap,
defaultEvalDegradedCap, defaultFlagEvalFlushInterval); all methods panic("not implemented")
- flagevaluation_hook.go: signature-only stubs for flagEvaluationHook.Finally and
isRuntimeDefault; panics with "not implemented"
- flagevaluation_test.go: failing RED tests: TestPruneContext, TestFlagEvaluationPayloadSchema,
TestAggregatorCollisionSafety, TestAggregatorConcurrentMinMax, TestAggregatorCapOverflow
- flagevaluation_hook_test.go: failing RED tests: TestIsRuntimeDefault,
TestFlagEvaluationHookFinally, TestFlagEvaluationHookContextCancelled
- flageval_metrics.go untouched (PRES-01 invariant)
- Tests fail on panic("not implemented"), not compile errors (clean RED state)
…chmark scenarios - BenchmarkFlagEvaluationNoop: raw evaluator baseline (zero hooks) - BenchmarkFlagEvaluationOTelOnly: OTel hook only (existing Path A reference) - BenchmarkFlagEvaluationOTelPlusEVP: OTel + EVP hook scaffold (plan 02 wires the hook) - Each scenario has typical (100 flags, 50 users, 10-field context) and stress (10 flags, 1000 users, 250-field context) sub-benchmarks via b.Run - makeBenchmarkConfig() and makeBenchmarkContext() helpers added - All three compile and run with -benchtime=1x (CONT-08 scaffold) - EVP hook created as nil-writer stub in OTelPlusEVP; body not called until plan 02 wires it
…reuse - struct-keyed aggregation maps (collision-safe, CONT-05) - three-tier cascade: full → degraded → ultra-degraded (CONT-04/CONT-10) - first/last via min/max inside lock (CONT-06) - pruneContext 256-field/256-char limits before buffering (CONT-03) - phantom attempt counting: perFlagFull incremented even on globalCap drops so per-flag overflow path stays alive after full-tier cap is exhausted - reused exposure.go transport: UDS/HTTP branch, evpSubdomainHeader/Value (D-04) - dedicated 10s flush ticker, flush-on-shutdown, panic recovery (D-05) - FlagEvaluationFlushInterval added to ProviderConfig
…helpers - flagEvaluationHook.Finally: ctx.Done check, nil-guard, calls writer.record - isRuntimeDefault: variant=="" primary signal; DEFAULT/DISABLED reason secondary (CONT-07) - extractEvalDetails: shared helper for EVP hook only; flageval_metrics.go untouched (D-06/PRES-01)
…k 3) - Add DD_FLAGGING_EVALUATION_COUNTS_ENABLED killswitch to provider (default true) - Register flagEvaluationWriter + flagEvaluationHook in newDatadogProvider when enabled - Start/flush/stop writer in InitWithContext / ShutdownWithContext - Append EVP hook in Hooks() alongside existing OTel flagEvalHook (PRES-01 preserved) - Add FlagEvaluationFlushInterval to ProviderConfig - Register DD_FLAGGING_EVALUATION_COUNTS_ENABLED in supported_configurations.gen.go - Add TestFlagEvaluationKillswitch: 3 subtests verifying killswitch behaviour - Update TestNewDatadogProvider to expect 3 hooks (exposure + OTel + EVP)
…iter) - BenchmarkFlagEvaluationOTelPlusEVP now uses ProviderConfig.FlagEvaluationFlushInterval=24h to construct a real EVP writer+hook without HTTP round-trips in the hot path (Pitfall 4) - Removed the plan-01 stub comment and nil-writer construction - Added time import for 24h duration constant
f53799f to
4b75012
Compare
BenchmarksBenchmark execution time: 2026-06-20 04:11:48 Comparing candidate commit f2a6df0 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 297 metrics, 2 unstable metrics, 1 flaky benchmarks without significant changes.
|
…le field (CR-01, WR-01, WR-02) - CR-01: hashContext now hashes over sorted keys so identical contexts produce a stable FNV-1a hash; Go map iteration is randomized and was fragmenting buckets and corrupting evaluation_count. - WR-01: remove the never-populated targetingRuleKey field, its aggregation-key member, the flagEvalTargetingRule type, and the dead targeting_rule emission branch (no metadata source exists yet). - WR-02: correct the globalCap comment to state it bounds the full tier only (degraded/ultra-degraded are bounded separately).
… (WR-03) record() is a non-blocking in-memory add with no network call, so gating it on ctx.Done() silently dropped legitimate evaluation counts when the request context was already cancelled (undercut Gate-7). Remove the ctx.Done() guard and update TestFlagEvaluationHookContextCancelled to assert the evaluation IS still counted.
The benchmark used b.N inside a b.Loop() body, which is a single fixed total (not a per-iteration index), so every evaluation used an identical targeting key and the 50/1000-user cardinality profiles collapsed to one bucket. Use a per-iteration counter with strconv.Itoa(i%numUsers) so the Gate-6 aggregation cost is exercised.
…d regenerate The supported_configurations.json entry had placeholder FIX_ME values for type and default, failing validate_supported_configurations_v2_local_file. Set type=boolean, default=true (killswitch defaults on) and regenerate supported_configurations.gen.go via scripts/configinverter so the JSON and generated map are canonical and in sync.
… EVP flagevaluation contract - #1 (blocker): route globalCap overflow to ultra-degraded instead of dropping; count total preserved at every tier - #3: add TestSaturationCountPreservation (total counts == add() calls past both globalCap and perFlagCap) - #2: correct collision-safe wording to low-collision/not collision-proof; document count-preserving collision behavior - #4: rotate flag keys + distinct context field names in benchmark so real cardinality is exercised
…; benchmark + comment fixes The OpenFeature Go SDK runs hooks synchronously on the caller's eval goroutine (client.go invokes finallyHooks via defer, no goroutine), so aggregating inside Finally added +112%/+180% latency over OTel-only. Move the work off the hot path. - async recording: Finally now does only cheap scalar extraction + the SDK's shallow Attributes() copy + a non-blocking bounded enqueue; a single background worker owns flatten/prune/hash/aggregate/flush. Queue-full drops with an observable counter (best-effort); stop() drains + final-flushes. Hot-path cost is a flat 5 allocs (~0.65us small / ~6us large context). - hashContext: replace per-field fmt.Fprintf with direct byte writes (250-field hash drops from ~250 allocs to ~1). - benchmark: drive evaluations through openfeature.Client so the hooks actually run (direct provider calls bypassed Hooks(), so prior overhead numbers measured nothing); isolate OTel-only vs OTel+EVP; rotate flag + user; add a hot-path-only micro-bench. - tests: drain the async queue in the hook tests; add TestFlagEvaluationBackpressureDrops. - comments: remove internal planning identifiers (this reference SDK is human-facing).
…egraded bound, post-stop record, and error message - Deterministic context pruning: sort keys BEFORE the 256-field cap and oversized-string skip so identical logical contexts prune to the same subset and hash equal (no bucket fragmentation). - Canonical context hash encoding: type-tagged + length-delimited key/value so int 1 != string "1" and '=' / '\n' in keys/values cannot alias a multi-field context. - Bounded ultra-degraded tier: explicit ultraDegCap (<=0 treated as default) with a single count-preserving sentinel overflow bucket (__other__) so dynamic/malicious flag keys cannot grow the map without bound. - Post-stop record() no-op: stopped is now atomic.Bool used as the single stop() idempotency gate; record() checks it lock-free and counts post-stop events as dropped instead of enqueuing into the never-drained channel. - Error payload prefers OpenFeature ErrorMessage, falling back to ErrorCode only when ErrorMessage is empty. - Doc comments on evaluationAggregationKey, hashContext, and addToUltraDegraded updated to match the fixed behavior. OTel feature_flag.evaluations path (flageval_metrics.go) is unchanged.
… review) - isRuntimeDefault now keys solely on an empty Variant; our evaluator sets a variant only on a matched allocation (TARGETING_MATCH/SPLIT/STATIC), so every DEFAULT/DISABLED/ERROR path already leaves it empty. - Removes the dead secondary reason-clause (DEFAULT/DISABLED) flagged as belt-and-suspenders; a present variant now correctly means a real assignment, including for foreign providers. - Aligns with PR #4886 reviewer concern: detect runtime default from absence of variant, not reason alone.
… remove ultra-degraded; resize caps for 2,500-flag scale
…regator scale harness
…ier buckets Address oleksii PR #4886 review #2: replace the lossy FNV-1a contextHash discriminator in evaluationAggregationKey with an exact, comparable canonical-context string (contextKey). The existing type-tagged, length-delimited canonical encoding is now emitted AS THE MAP KEY instead of hashed, so Go's map compares it byte-for-byte. - Distinct contexts now ALWAYS land in distinct full-tier buckets: there is no hash, so no hash collision, so a count can never be misattributed to the wrong context (the prior collision-misattribution case is gone). - Delete hash/fnv import, hashContext, and the collision-analysis doc block. - Rename hashContext -> canonicalContextKey (returns the encoding as a string). - Tests assert distinct contexts (int 1 vs string "1"; '='/newline-bearing values; multi-field) land in DISTINCT buckets with zero misattribution, and identical contexts still merge into one bucket. Memory: the contextKey string is stored once per full-tier bucket, bounded by globalCap and the 256-field/256-char prune. EVPRecord hot path unchanged (the key is built in the async worker, not record()). OTel path untouched.
…rsal Address oleksii PR #4886 review #4: the worker path previously ran flattenContext (flatten.go) then pruneContext as two separate steps, each allocating its own map. Merge them into a single flattenAndPruneContext used by aggregate(): flatten nested objects, then apply the deterministic sort-before-cut 256-field / 256-char prune in one pass. Output is byte-for-byte identical to the prior two-step pipeline (same surviving keys, same 256/256 limits, same deterministic cut). When the flattened context already fits the limits — the common case — the flattened map is returned directly, eliding the second pruned-output map the old pipeline always allocated. flattenContext is left unchanged because exposure_hook.go still calls it; only the EVP aggregation path is merged. pruneContext is retained as the reference behavior for tests. Added TestFlattenAndPruneContextEquivalence asserting the merged pass equals flattenContext+pruneContext across nested, oversized, and >256-field inputs, with preserved determinism.
…ook-fire time (review #3) Capture the evaluation timestamp once at DatadogProvider.evaluate entry, thread it into evaluateFlag (reusing the time.Now() the evaluator already computes for allocation windows), and stamp it into FlagMetadata on every return path. The EVP flagevaluation hook reads it for first/last_evaluation bounds instead of calling time.Now() at hook-fire time (slightly later and slightly incorrect). Falls back to hook-fire time when absent (non-Datadog provider). flageval_metrics.go untouched.
typotter
left a comment
There was a problem hiding this comment.
I'm no expert on metrics (but Vickie is). Overall, lgtm pending all of Oleksii's concerns being addressed
There was a problem hiding this comment.
looking great so far.
Since the EvP intake only validates the API key (and not the structure of the payload), it would be good to verify this behavior end to end before we merge. Meaning, we should be able to show that we can query this data in the flagevaluation track using pup.
@petzel Certainly a good idea - here's the evidence. FYI removed from public repo. |
## Motivation OpenFeature flag evaluation already has Go benchmarks, but they are not wired into the microbenchmark CI framework. The server-side flag-evaluation work needs a stable main-branch benchmark group for the client-facing evaluation path before adding EVP flagevaluation emission. The top-line performance question is whether returning a variant through the public OpenFeature client regresses, and whether concurrent evaluations remain viable under load. ## Changes - Add `BenchmarkOpenFeatureClientEvaluation` for the public OpenFeature client path with the provider default hooks. - Add `BenchmarkOpenFeatureClientConcurrentEvaluations` for concurrent public-client evaluations. - Add `microbenchmarks-3` with the new client-path benchmarks and the existing evaluator-focused OpenFeature benchmarks. ## Decisions - The new top-line benchmarks use the public OpenFeature client instead of direct provider methods, because hook overhead is visible only on the client path. - Existing direct-provider benchmarks are included as evaluator-focused diagnostics. - `microbenchmarks-3` is intentionally not added to `pr-performance-gates` in this PR. The two new client benchmark names do not exist on `main` yet, so the gate cannot produce same-name baseline artifacts for them until after this PR lands. A follow-up feature PR can opt group 3 into the gate once these names are present on `main`. - This PR intentionally does not add EVP-specific benchmark names; those belong in the flagevaluation PR after this baseline exists on `main`. ## Next Steps After this PR lands on `main`, the server-side EVP flagevaluation PR ([#4886](#4886)) should: - Reuse the `microbenchmarks-3` group introduced here. The EVP PR should not create a separate benchmark group for flagevaluations. - Opt the existing `microbenchmarks-3` job into `pr-performance-gates` once `main` has baseline artifacts for these benchmark names. - Keep the main-branch client-facing benchmark names as the primary regression signal, so the gate compares normal public-client evaluation against the same path with EVP emission wired in. - Keep the EVP-specific benchmarks in that PR as diagnostics for no-op evaluation, OTel-only evaluation, OTel plus EVP aggregation, direct EVP record ingestion, and parallel EVP aggregation. Those explain where overhead comes from, while the client-facing benchmark remains the top-line signal. ## Validation ``` go test ./openfeature go test ./openfeature -run='^$' -bench='^(BenchmarkOpenFeatureClientEvaluation|BenchmarkOpenFeatureClientConcurrentEvaluations|BenchmarkBooleanEvaluation|BenchmarkStringEvaluation|BenchmarkIntEvaluation|BenchmarkFloatEvaluation|BenchmarkEvaluation|BenchmarkEvaluationWithVaryingContextSize|BenchmarkEvaluationWithVaryingFlagCounts|BenchmarkConcurrentEvaluations)$' -benchmem -benchtime=100ms ``` ## E2E: Staging Datadog APM proof - Ran the dogfooding staging APM proof against a locally built SDK and the staging Agent using `DD_SITE=datad0g.com`. - Validation id: `ffe-span-dotnet-20260619121831-b294dd1`; service: `ffe-dogfooding-dotnet`; env: `staging`; scenario: `span_enrichment.aggregate`; SDK sha: `b294dd1`. - Staging Datadog MCP query used `base_url=https://dd.datad0g.com` and found `8` matching spans for `service:ffe-dogfooding-dotnet env:staging @validation_id:ffe-span-dotnet-20260619121831-b294dd1`. - Trace Explorer: https://dd.datad0g.com/apm/traces?end=1781904227862&historicalData=true&paused=true&query=service%3Affe-dogfooding-dotnet+env%3Astaging+%40validation_id%3Affe-span-dotnet-20260619121831-b294dd1&start=1781817827862 - Indexed span-enrichment evidence: - `trace_id=6a35358d0000000091168b7fb4b815a2`, `span_id=15600877720192993068`, `resource=POST /scenario`, `ffe_flags_enc=vQblBKMP` - `trace_id=6a3534fd000000002e93ea8b4f41ec5d`, `span_id=12009548985036661845`, `resource=POST /scenario`, `ffe_flags_enc=vQblBKMP` Co-authored-by: leo.romanovsky <leo.romanovsky@datadoghq.com>
…tions-cross-sdk # Conflicts: # .gitlab/benchmarks/micro/gitlab-ci.yml # openfeature/provider_bench_test.go
|
I dug into the OpenFeature benchmark failures and now have a clearer read on them. Async EVP delivery remains the right architecture, but pre-enqueue work is still on the hot path and must remain small. I reduced avoidable timestamp/context/event work, kept non-blocking best-effort delivery with bounded drops under pressure, and preserved evaluation timestamps for correctness even though that means accepting a small allocation cost. |
Motivation
Go server-side flag evaluations need an EVP
flagevaluationemission path alongside the existing OTelfeature_flag.evaluationspath.The backend contract is owned by the
dd-sourceflageval-worker: batched EVP event items accept only schema-visible fields, and OpenFeaturereasonis not an EVP event field.targeting_rule.keyis real targeting-rule metadata, so the Go path omitstargeting_ruleuntil that metadata is available.Changes
Finallyhook records EVP evaluations; the existing OTel hook path remains separate.dd-sourceworker schema or adding a JSON Schema dependency.microbenchmarks-3; on this branch the default client/provider benchmark exercises the EVP-enabled path. The focusedBenchmarkFlagEvaluation*helpers remain runnable locally for overhead analysis, but are not listed in the benchmark gate until matching baseline names exist onmain.flowchart LR eval["OpenFeature evaluation"] --> provider["DatadogProvider<br/>UFC result"] provider --> hooks["Finally hooks<br/>OTel + EVP"] hooks --> writer["EVP writer<br/>bounded queue + aggregation"] writer --> intake["Agent EVP proxy<br/>dd-source flageval-worker"]Decisions
runtime_default_usedcomes from typed conversion/default behavior.dd-source; the Go SDK only tests the fields it emits.Validation Evidence
Local validation on June 20, 2026:
f2a6df08644a15aecaead0f68ca1e646dac276b395edcc8de242e0c70ddce87b6fab8a76e585f1e3golang/net-http7.80.1; librarygolang@2.10.0-dev; Linuxaarch64Commands:
SYSTEM_TEST_BUILD_TIMEOUT=1200 ./build.sh --library golang --weblog-variant net-http TEST_LIBRARY=golang WEBLOG_VARIANT=net-http ./run.sh +l golang FEATURE_FLAGGING_AND_EXPERIMENTATION -k "test_flag_eval_evp"Result:
8 passed, 2627 deselected in 67.22s.Repo-local validation on June 20, 2026:
go test -race ./openfeature- passgo test ./openfeature -run '^(TestEndToEnd|TestFlagEvaluation|TestEvaluateStampsEvalTimeMetadata)' -count=1 -v- passgo test ./openfeature -run '^$' -bench='^BenchmarkFlagEvaluation' -benchtime=100ms -count=1- passgit diff --check- passDogfood validation:
SDK=go APP_URL=http://localhost:8081 WAIT_SECONDS=40 scripts/validate-server-flagevaluations.shwithDD_TRACE_GO_PATH=/Users/leo.romanovsky/gsd-workspaces/flag-evaluations-cross-sdk/dd-trace-go- pass (EVP flagevaluations OK: count=1,OTel non-regression OK: otel_count=1,service=ffe-dogfooding-go)flagevaluationrows for Go targeting keys andevaluation_countaggregation. June 20 real-backend run id1781931066used prefixserver-evp-e2e-go-1781931066-user-.