Move Feature Flagging evaluation into the provider runtime - #12097
Draft
leoromanovsky wants to merge 4 commits into
Draft
Move Feature Flagging evaluation into the provider runtime#12097leoromanovsky wants to merge 4 commits into
leoromanovsky wants to merge 4 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
Contributor
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Before this change, Java Feature Flags required the application to start with the Datadog Java agent:
The provider delegated UFC parsing and evaluation to classes owned by the Java agent. Without
-javaagent, the provider could not load configuration or evaluate flags.Despite its name,
feature-flagging-libwas never a reusable Feature Flagging library. Since its introduction, it owned agent-side Remote Configuration and telemetry delivery. It depended on agent communication, bootstrap, configuration, and tracer classes. It was not published indd-openfeatureand could not run as a provider-only dependency.Reusing that module would retain the agent coupling. This change creates a dependency-light provider core. It renames the historical module to describe its agent-runtime role.
Managed serverless platforms make the startup requirement costly:
JAVA_TOOL_OPTIONS=-javaagent:/opt/java/lib/dd-java-agent.jar.JAVA_TOOL_OPTIONS. First-generation functions require two agents.languageWorkers__java__argumentsorJAVA_OPTS, depending on the hosting plan.These steps couple flag evaluation to tracing packages, platform-specific runtime configuration, and agent compatibility. The default CDN path must work with
dd-openfeaturealone.The project must also preserve Remote Configuration. Existing customers need the agent-backed source. Released providers must continue to work with a new agent.
Changes
The default OpenFeature startup is now provider-only:
The application supplies
DD_API_KEY,DD_ENV, andDD_SITE. It does not set a configuration-source selector or attach-javaagent. The provider starts and owns the internal CDN poller.ffe-dogfooding #102 uses this startup path. The script runs
java -jar app.jarunlessremote_configis explicit.HTTP cancellation now handles both direct and
ExecutionException-wrappedCancellationExceptionresults. Both paths returnInterruptedIOException.The published coordinate remains
com.datadoghq:dd-openfeature. The provider JAR embedsfeature-flagging-coreandfeature-flagging-http.flowchart TD APP["Application and OpenFeature SDK"] --> API["feature-flagging-api<br/>published as dd-openfeature"] subgraph PROVIDER["Provider-owned modules embedded in dd-openfeature"] API --> CORE["feature-flagging-core<br/>UFC model, parser, evaluator,<br/>snapshot, last-known-good state"] API --> HTTP["feature-flagging-http<br/>Java 11 HttpClient, ETag,<br/>retry, scheduling, lifecycle"] HTTP --> CORE end API -. "reflective JDK-only calls" .-> BRIDGE["feature-flagging-bootstrap<br/>raw compatibility bridge"] subgraph AGENT["Java agent modules"] ENTRY["feature-flagging-agent<br/>agent lifecycle"] --> RUNTIME["feature-flagging-agent-runtime<br/>Remote Configuration,<br/>EVP transport, trace adapter"] RUNTIME --> TELEMETRY["feature-flagging-telemetry<br/>exposure deduplication,<br/>span-tag state and encoding"] end RUNTIME <--> BRIDGEBoth delivery sources feed the same provider-owned evaluator:
flowchart TD CDN["Default: Datadog CDN"] --> HTTP["HTTP source"] RC["Explicit: Remote Configuration"] --> AGENT["Java agent"] AGENT --> BRIDGE["Raw byte[] bridge"] HTTP --> CORE["Core snapshot and evaluator"] BRIDGE --> CORE CORE --> RESULT["OpenFeature result"]feature-flagging-libis nowfeature-flagging-agent-runtime.feature-flagging-telemetrymodule owns exposure deduplication and span-tag state and encoding.feature-flagging-agent-runtime.feature-flagging-telemetryis not embedded indd-openfeatureyet. The provider does not emit telemetry through it in this change.Decisions
feature-flagging-core. Future CDN, Remote Configuration, offline, tracer API, and Lambda entry points can reuse one implementation.feature-flagging-http. Offline and Remote Configuration sources do not inherit HTTP dependencies or polling behavior.core. UFC evaluation remains independent from telemetry state and transport.feature-flagging-telemetry. The module depends only on the JDK.feature-flagging-libtofeature-flagging-agent-runtime. The old module was never a reusable library.dd-openfeature.This organization is the last required configuration and evaluator ownership refactor. It is not the final telemetry architecture.
The bootstrap bridge abstracts provider-to-agent communication across classloaders. It does not abstract local relay or direct intake transport. It also cannot create or enrich a trace when no tracer exists.
Next steps
The provider must own the agentless telemetry runtime because it is the only component present in provider-only deployments. The agent runtime remains an optional adapter.
Before agentless telemetry delivery, the follow-up should:
feature-flagging-telemetryindd-openfeature.flowchart TB EVAL["Provider evaluation"] --> EVENT["Immutable evaluation event"] EVENT --> PIPE["Provider-owned telemetry runtime"] PIPE --> EVP["Exposure and aggregate EVP"] PIPE --> METRIC["Evaluation OTLP metric"] PIPE --> TRACE["Trace enrichment"] EVP --> EVP_ROUTE{"Agent accepts EVP?"} EVP_ROUTE -- Yes --> EVP_AGENT["Raw bridge to agent writer"] EVP_ROUTE -- No --> EVP_DIRECT["Local relay or direct EVP"] METRIC --> METRIC_ROUTE{"Local OTLP configured?"} METRIC_ROUTE -- Yes --> OTLP_LOCAL["OTLP HTTP on 4318"] METRIC_ROUTE -- No --> OTLP_DIRECT["Direct OTLP metrics intake"] TRACE --> TRACE_ROUTE{"Trace producer?"} TRACE_ROUTE -- "Datadog Java agent" --> TRACE_AGENT["Raw bridge to local-root span"] TRACE_ROUTE -- "OpenTelemetry SDK" --> TRACE_OTEL["Future OpenTelemetry span adapter"] TRACE_ROUTE -- None --> TRACE_NONE["No trace enrichment"]The required follow-up depends on the deployment:
serverless-initserverless-initor the Java agent4318only when explicitly enabledDirect OTLP trace intake makes an agentless trace path possible, but
dd-openfeaturemust not become a tracer. A future OpenTelemetry adapter can enrich an active OpenTelemetry span. The application must still install a trace SDK and exporter.PR risk
The customer-facing goal is narrow. CDN evaluation can run without the Java agent, while Remote Configuration remains available. The implementation is substantial because the previous design made the agent the configuration and lifecycle owner.
Net-new or changed behavior:
dd-openfeaturecan fetch, parse, store, and evaluate CDN configuration without-javaagent.Moved or ported behavior:
DDEvaluatorintofeature-flagging-core. The expected OpenFeature results remain unchanged, but the implementation and model changed.HttpClientimplementation.304, retry, timeout, overlap prevention, scheduling, and shutdown behavior were ported rather than added for the first time.feature-flagging-librename and telemetry extraction are primarily structural. They do not add telemetry delivery.The main residual risks are UFC semantic drift, provider lifecycle leaks, source-selection compatibility, HTTP behavior differences, and cross-classloader compatibility. The dogfooding proof, system tests, child-JVM tests, last-known-good tests, and compatibility matrix reduce these risks. They do not make the change risk-free.
This assessment comes from an LLM review of the static diff and recorded validation. It is not a formal safety proof or a substitute for human review.
Validation
ffe-dogfooding #102 validated Java commit
8bfa6f072a97ac8a7d4977c42e04fe8d63652e25.1.65.0-SNAPSHOTran beside published Node.jsdd-trace5.118.0.-javaagentand reportedjavaAgentAttached=false.PROVIDER_READYin 1,094 ms. Node.js reachedPROVIDER_READYin 706 ms.truefor the same configured flag.providerShutdown=true.system-tests #7300 used
java@1.65.0-SNAPSHOT+8bfa6f072a.FFL-2446,FFL-1729, andFFL-2184.javaAgentAttached=false.on-value.The compatibility matrix passed:
1.64.2agent + CDN.1.64.2provider + new agent + Remote Configuration through the legacy bridge.1.64.2agent + Remote Configuration returned the expected unsupported-bridge error.