test(ffe): cover agentless CDN configuration-source contract by leoromanovsky · Pull Request #7277 · DataDog/system-tests

leoromanovsky · 2026-07-07T22:17:22Z

Motivation

Server SDK agentless flag delivery needs one shared system-test contract before the SDK implementations land. These tests are intentionally red behind missing_feature; they define the cross-SDK behavior we expect for configuration-source selection, CDN polling, cold failures, warm preservation, and no-Agent reachability.

Tracking: FFL-2731, under epic FFL-2662.

Internal RFC: Supporting OpenFeature Agentless mode in Server SDKs

Changes

Configuration-source tests

Adds parametric FFE tests for remote_config, explicit cdn, default cdn, customer endpoint override, invalid source, cold failure/recovery, warm preservation, timeout, and poller no-overlap.
Uses existing DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED=true to enable the current provider; the renamed contract under test is DD_FLAGGING_CONFIGURATION_SOURCE*.

Mock FFE CDN

Adds /mock/ufc/config with test-owned response timelines.
Reuses existing tests/parametric/test_ffe/flags-v1.json; malformed UFC stays inline.
Removes invented configuration-source request signals. Tests prove CDN by actual mock requests, path, auth, ETag, status codes, and OpenFeature evaluation.
Uses standard 500 for 5xx retryable failures. No 509.

Docker reachability

Adds host.docker.internal:host-gateway for the test agent and for parametric SDK containers when their env points at host.docker.internal.
Adds a live smoke test that fetches the mock from inside the test-agent container on the same Docker network.

Decisions

Mock scope
Keep the fixture FFE/UFC-specific for this PR. A generic HTTP fixture can come later if another suite needs it.

Configuration model
Customer-hosted HTTP endpoints are covered as cdn with a different base URL, not a fourth SDK mode.

Provider enablement
This suite does not rename provider enablement. It uses the current system-tests/SDK provider flag and focuses the new contract on configuration-source env vars.

SDK Gates

SDK / manifest	Implementation ticket	System-test ticket	Manifest reason
Java	`FFL-2693`	`FFL-2731`	Java agentless CDN implementation plus shared configuration-source contract
Go	`FFL-2695`	`FFL-2731`	Go agentless CDN implementation plus shared configuration-source contract
Node.js	`FFL-2697`	`FFL-2731`	Node.js agentless CDN implementation plus shared configuration-source contract
Python	`FFL-2699`	`FFL-2731`	Python agentless CDN implementation plus shared configuration-source contract
Ruby	`FFL-2701`	`FFL-2731`	Ruby agentless CDN implementation plus shared configuration-source contract
.NET	`FFL-2703`	`FFL-2731`	.NET agentless CDN implementation plus shared configuration-source contract
PHP	`FFL-2705`	`FFL-2731`	PHP agentless CDN implementation plus shared configuration-source contract
C++	`irrelevant`	`FFL-2731`	Not in scope for v1
Rust	`irrelevant`	`FFL-2731`	Not in scope for v1

Coverage

Source Selection

Scenario	Test	What it proves
Explicit RC baseline	`test_remote_config_positive_ignores_cdn_env`	RC applies `flags-v1.json`; CDN base URL can be present but mock CDN must receive zero requests.
Remote config absent	`test_remote_config_without_rc_does_not_fallback_to_cdn`	`remote_config` without RC stays default/not-ready and does not fall back to CDN.
Explicit CDN	`test_explicit_cdn_positive`	`DD_FLAGGING_CONFIGURATION_SOURCE=cdn` fetches the mock and evaluates the same flag value as RC.
Default CDN	`test_default_cdn_positive`	Omitted source selects CDN and does not require Agent RC.
Customer endpoint	`test_customer_http_endpoint_default_cdn_positive`	Exact endpoint base URL works as CDN delivery, proving customer-hosted HTTP is just a base URL override.
Invalid source	`test_invalid_configuration_source_fails_closed`	Unknown source does not silently fall back to CDN.

remote_config positive: Agent RC flags-v1.json -> value "green"; mock requests_total == 0
remote_config absent: no RC payload + CDN env present -> default/not-ready; mock requests_total == 0
explicit/default/customer CDN: valid -> HTTP 200 + flags-v1.json + ETag "ufc-v1" -> value "green"
invalid source: DD_FLAGGING_CONFIGURATION_SOURCE=invalid + CDN env present -> no mock request

Cold Failure And Recovery

Scenario	Test	What it proves
Missing auth cold	`test_missing_auth_cold`	Missing API key cannot make a cold provider ready.
Malformed cold	`test_malformed_cold`	HTTP 200 with malformed UFC cannot make a cold provider ready.
Request timeout cold	`test_request_timeout_cold`	A response slower than `DD_FLAGGING_CONFIGURATION_SOURCE_REQUEST_TIMEOUT_SECONDS` is treated as cold failure.
Bad-to-good retry	`test_bad_to_good_cold_recovery`	Initial 5xx does not permanently poison cold startup; later valid UFC recovers.
Bad-to-unchanged cold	`test_bad_to_unchanged_cold_preserves_not_ready`	5xx followed by 304 before any accepted UFC does not create fake last-known-good state.

missing auth: no DD_API_KEY -> HTTP 401 -> default/not-ready
malformed cold: malformed -> HTTP 200 + {"flags": [ -> default/not-ready
timeout cold: timeout -> sleeps 1.5s; SDK timeout=1s -> default/not-ready
bad-to-good: server_error -> valid == HTTP 500 -> HTTP 200 flags-v1.json -> value "green"
bad-to-unchanged: server_error -> not_modified == HTTP 500 -> HTTP 304 -> default/not-ready

Warm State Preservation

Scenario	Test	What it proves
Missing auth warm	`test_missing_auth_warm`	Auth failure after valid UFC preserves last-known-good evaluator state.
Good-to-bad	`test_good_to_bad_warm_preservation`	5xx after valid UFC preserves last-known-good evaluator state.
Good-to-unchanged	`test_good_to_unchanged_etag_sequence`	304 sends `If-None-Match` and keeps evaluator state unchanged.
Malformed warm	`test_malformed_warm`	Malformed UFC after valid UFC cannot corrupt last-known-good state.

missing auth warm: valid -> unauthorized == HTTP 200 flags-v1.json -> HTTP 401 -> value "green"
good-to-bad: valid -> server_error == HTTP 200 flags-v1.json -> HTTP 500 -> value "green"
good-to-unchanged: valid -> not_modified == HTTP 200 ETag "ufc-v1" -> HTTP 304 with If-None-Match "ufc-v1" -> value "green"
malformed warm: valid -> malformed == HTTP 200 flags-v1.json -> HTTP 200 malformed bytes -> value "green"

Poller And Fixture

Scenario	Test	What it proves
Delayed no-overlap	`test_delayed_no_overlap`	Poll interval shorter than response latency does not create concurrent CDN requests.
Metadata-only status	`test_mock_ffe_cdn_status_is_metadata_only`	Mock status exposes counters/request metadata only, not UFC bytes.
Docker reachability	`test_mock_ffe_cdn_reachable_from_test_agent_network`	The default mock URL is reachable from a Docker container on the test-agent network.

delayed no-overlap: delayed_valid -> sleeps 0.5s -> HTTP 200; poll interval=0.2s; max_in_flight == 1
reachability: test-agent container -> GET http://host.docker.internal:<port>/mock/ufc/config with DD-API-KEY -> HTTP 200; requests_total == 1

Validation

$ ./run.sh -S PARAMETRIC -L python --skip-parametric-build tests/parametric/test_ffe/test_mock_ffe_cdn_reachability.py -q
Scenario: PARAMETRIC
Library: python@4.10.7
1 passed in 5.14s

$ ./run.sh -S PARAMETRIC -L python --skip-parametric-build --collect-only -q tests/parametric/test_ffe/test_configuration_sources.py tests/parametric/test_ffe/test_mock_ffe_cdn_reachability.py
19 tests collected

System Tests (python, prod) / parametric / parametric (1)
  https://github.com/DataDog/system-tests/actions/runs/28916585866/job/85785079227

  report.json records:

  {
    "nodeid": "tests/parametric/test_ffe/
    test_mock_ffe_cdn_reachability.py::Test_Feature_Flag_Mock_FFE_CDN_Reachability::test_mock
    _ffe_cdn_reachable_from_test_agent_network",
    "outcome": "passed",
    "setup": "passed",
    "call": "passed",
    "teardown": "passed"
  }

github-actions · 2026-07-07T22:17:57Z

CODEOWNERS have been resolved as:

tests/parametric/test_ffe/test_configuration_sources.py                 @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
tests/parametric/test_ffe/test_mock_ffe_cdn_reachability.py             @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core
utils/docker_fixtures/_mock_ffe_cdn.py                                  @DataDog/system-tests-core
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/nodejs.yml                                                    @DataDog/dd-trace-js
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
manifests/rust.yml                                                      @DataDog/apm-rust
utils/docker_fixtures/_core.py                                          @DataDog/system-tests-core
utils/docker_fixtures/_test_agent.py                                    @DataDog/system-tests-core
utils/docker_fixtures/_test_clients/_test_client_parametric.py          @DataDog/system-tests-core

datadog-datadog-prod-us1 · 2026-07-07T22:25:56Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 1 passed on retry

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 81bbcdd | Docs | Datadog PR Page | Give us feedback!}

- Add parametric FFE source-mode test class - Cover RC, CDN, default CDN, and negative fixture cases

- Add mock CDN control and metadata-only status endpoints - Add valid, malformed, and unchanged UFC fixtures

- Add source-mode class rows to all target SDK manifests - Gate rows pending agentless CDN source-mode implementations

- Register mock CDN fixture via pytest_plugins - Keep mock handler typing compatible with instance server state

- Add endpoint-form CDN base URL coverage - Add 509 and 304 transition cases for source-mode L3 - Assert CDN source metadata across mock-backed cases

- add deterministic retry/etag response sequences for CDN source-mode tests - expose metadata-only status code evidence and endpoint URL handling

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 81bbcdd41a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-07-08T04:47:39Z

+
+        status = _wait_for_status(
+            mock_ffe_cdn,
+            lambda current: current["status_codes"][-2:] == expected_status_codes,


Match CDN status transitions anywhere in history

When the SDK keeps polling after the expected transition, this predicate can miss a successful 500 -> 200 sequence and time out: for example, if another poll lands before _wait_for_status samples /status, the history becomes [500, 200, 200] and status_codes[-2:] is no longer [500, 200]. The same last-two-codes pattern is used for the other recovery/warm sequence tests, so these tests can be flaky once SDK polling is fast enough; checking for the expected adjacent pair anywhere in status_codes would make the assertion stable.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-08T04:47:39Z

+            self.in_flight += 1
+            self.max_in_flight = max(self.max_in_flight, self.in_flight)
+            self.last_path = parsed.path
+            self.last_if_none_match = headers.get("If-None-Match")


Normalize If-None-Match before storing it

If an SDK sends the ETag validator with different casing, e.g. if-none-match, this case-sensitive lookup records None even though HTTP header names are case-insensitive. test_good_to_unchanged_etag_sequence then times out on last_if_none_match == '"ufc-v1"' despite the SDK making a valid 304 request, so the mock can falsely fail conforming implementations.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-08T04:47:39Z

+        assert apply_state["apply_state"] == RemoteConfigApplyState.ACKNOWLEDGED.value
+        assert apply_state["product"] == RC_PRODUCT
+
+        assert test_library.ffe_start(UFC_VALID_DATA), "failed to start FFE provider in remote_config mode"


Do not inject UFC into the remote-config source test

Passing UFC_VALID_DATA directly to /ffe/start lets SDK test servers that consume the configuration request body, such as PHP's parametric endpoint, load the flag payload locally instead of proving that DD_FLAGGING_CONFIGURATION_SOURCE=remote_config read it from Agent RC. In that context the test can pass with zero CDN requests even if the remote-config source is broken, so this source-selection check should start the provider without the direct configuration payload and rely on the RC set above.

Useful? React with 👍 / 👎.

leoromanovsky added 11 commits July 7, 2026 18:27

test(03-03): add failing source mode scaffold

ee5aa77

- Add parametric FFE source-mode test class - Cover RC, CDN, default CDN, and negative fixture cases

feat(03-03): add mocked CDN source-mode fixture

492c9c1

- Add mock CDN control and metadata-only status endpoints - Add valid, malformed, and unchanged UFC fixtures

test(03-03): gate source-mode system tests

fdc6516

- Add source-mode class rows to all target SDK manifests - Gate rows pending agentless CDN source-mode implementations

refactor(03-03): clean up source-mode scaffold imports

82e64ee

- Register mock CDN fixture via pytest_plugins - Keep mock handler typing compatible with instance server state

fix(03-03): tighten source-mode scaffold assertions

bbdccdd

fix(03-03): stabilize source-mode mock CDN checks

4448cfb

test(05-03): add failing source-mode sequence contract

8371fd2

- Add endpoint-form CDN base URL coverage - Add 509 and 304 transition cases for source-mode L3 - Assert CDN source metadata across mock-backed cases

feat(05-03): implement source-mode mock CDN evidence support

ca65538

- add deterministic retry/etag response sequences for CDN source-mode tests - expose metadata-only status code evidence and endpoint URL handling

test(06-01): harden source-mode red contract coverage

a69fc66

test(ffe): cover customer CDN endpoint source mode

92271fe

test(ffe): link source-mode manifests to FFL ticket

2874813

leoromanovsky force-pushed the gsd/03-03-source-modes-scaffold branch from 73df0cf to 2874813 Compare July 7, 2026 22:30

leoromanovsky added 5 commits July 7, 2026 18:47

test(ffe): reuse UFC fixture for source-mode CDN mock

d91fa37

test(ffe): configure CDN response sequences from tests

f6e9404

test(ffe): rename source modes to configuration sources

994bb07

test(ffe): annotate configuration-source manifests with SDK tickets

6ab9c61

test(ffe): use configuration source env vars

a9b81dd

leoromanovsky changed the title ~~test(ffe): cover agentless CDN source-mode contract~~ test(ffe): cover agentless CDN configuration-source contract Jul 7, 2026

leoromanovsky added 5 commits July 7, 2026 19:39

test(ffe): group configuration source scenarios

7e1a2e1

test(ffe): ignore configuration source suite for cpp and rust

4669f70

test(ffe): simplify mock fixture metadata

7ab6e88

test(ffe): keep mock status metadata only

9c93287

test(ffe): harden configuration-source contract

81bbcdd

leoromanovsky marked this pull request as ready for review July 8, 2026 04:38

leoromanovsky requested review from a team as code owners July 8, 2026 04:39

leoromanovsky requested review from Kyle-Verhoog, PerfectSlayer, cbeauchesne, claponcet, daniel-romano-DD, ncybul, pavlokhrebto, sameerank and xlamorlette-datadog and removed request for a team July 8, 2026 04:39

chatgpt-codex-connector Bot reviewed Jul 8, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(ffe): cover agentless CDN configuration-source contract#7277

test(ffe): cover agentless CDN configuration-source contract#7277
leoromanovsky wants to merge 21 commits into
mainfrom
gsd/03-03-source-modes-scaffold

leoromanovsky commented Jul 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 8, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 8, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

leoromanovsky commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Decisions

SDK Gates

Coverage

Source Selection

Cold Failure And Recovery

Warm State Preservation

Poller And Fixture

Validation

Uh oh!

github-actions Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 8, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

leoromanovsky commented Jul 7, 2026 •

edited

Loading

github-actions Bot commented Jul 7, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Jul 7, 2026 •

edited

Loading