Skip to content

Commit 56306b6

Browse files
Enable PHP FFE evaluation system tests (#7003)
## Motivation The PHP Feature Flags and Experimentation evaluation runtime has landed in dd-trace-php and is published through the `v1.21.0-dev` artifact. PHP should run the shared FFE system tests so the runtime path is covered by live `FFE_FLAGS` Remote Config delivery and by the canonical JSON evaluation fixtures used across libraries. Reference design doc: https://docs.google.com/document/d/1NvMfTpZWLBlFmEFNjdnlMyeVpy5l7KD8qujGFco6w2w/edit?tab=t.0 ## Changes This PR enables the existing end-to-end `tests/ffe/test_dynamic_evaluation.py` suite for PHP starting at `v1.21.0-dev` and adds the minimal PHP weblog `/ffe` endpoint needed by that shared suite. The endpoint calls the `DDTrace\FeatureFlags\Client` details API, normalizes the request shape used by system-tests, and returns provider-not-ready defaults when the runtime is unavailable. This PR also enables the parametric canonical JSON fixture evaluation suite for PHP. The PHP parametric server implements `/ffe/start` and `/ffe/evaluate` using the same Datadog client API, and OF.7 is covered through `test-case-of-7-empty-targeting-key.json` in the shared parametrized test. ## Decisions The parametric evaluation test still sets `FFE_FLAGS` through Remote Config and waits for the test-agent to observe an acknowledged apply state. For the PHP parametric CLI process, that ACK is not enough to make the native evaluator report loaded config: CI on `1d6ac67` showed every dynamic evaluation case still returning `PROVIDER_NOT_READY` with `providerState.hasConfig == false` after the retry loop. To keep the parametric suite deterministic, `/ffe/start` accepts the canonical UFC fixture payload uniformly. Existing non-PHP parametric servers ignore that optional payload; PHP uses it to seed the native evaluator through `DDTrace\Testing\ffe_load_config` before constructing the client. The shared test still has no PHP branch, and the explicit readiness check remains so failures show whether the provider is still waiting for config. The scope is deliberately evaluation-only. Exposure forwarding and `feature_flag.evaluations` metrics remain disabled here so they can be enabled in separate stacked PRs after their runtime implementations are ready. ## Verification Current branch checks: ```text $ ./format.sh All good, the system-tests CI will be happy! ``` ```text $ TEST_LIBRARY=php ./run.sh PARAMETRIC --skip-parametric-build tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation --collect-only -q 24 tests collected ``` The collected set includes `test-case-of-7-empty-targeting-key.json`, so OF.7 is covered by the generic JSON parametrization.
1 parent a374e76 commit 56306b6

10 files changed

Lines changed: 437 additions & 50 deletions

File tree

docs/understand/weblogs/end-to-end_weblog.md

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1026,6 +1026,20 @@ The endpoint must accept a query string parameter `code`, which should be an int
10261026
This endpoint is used for client-stats tests to provide a separate "resource" via the endpoint path `stats-unique` to disambiguate those tests from other
10271027
stats generating tests.
10281028

1029+
### POST /ffe
1030+
1031+
This endpoint is used by the Feature Flags & Experimentation scenario. It must
1032+
accept a JSON body with these fields:
1033+
1034+
- `flag`: the feature flag key to evaluate.
1035+
- `variationType`: the expected variation type.
1036+
- `defaultValue`: the value to return when evaluation cannot resolve the flag.
1037+
- `targetingKey`: the evaluation subject key.
1038+
- `attributes`: flat scalar targeting attributes.
1039+
1040+
The response must be JSON and include at least `value` and `reason`. Error
1041+
responses should also include `errorCode` and `errorMessage`.
1042+
10291043
### GET /healthcheck
10301044

10311045
Returns a JSON dict, with those values :

manifests/dotnet.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -810,7 +810,6 @@ manifest:
810810
tests/parametric/test_dynamic_configuration.py::TestDynamicConfigV2: v2.44.0
811811
tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation: '>=3.36.0' # Modified by easy win activation script
812812
tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation::test_ffe_flag_evaluation: missing_feature # Created by easy win activation script
813-
tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation::test_ffe_of7_empty_targeting_key: missing_feature # Created by easy win activation script
814813
tests/parametric/test_ffe/test_span_enrichment.py: missing_feature
815814
tests/parametric/test_headers_b3.py::Test_Headers_B3::test_headers_b3_migrated_extract_valid: missing_feature (Need to remove b3=b3multi alias)
816815
tests/parametric/test_headers_b3.py::Test_Headers_B3::test_headers_b3_migrated_inject_valid: missing_feature (Need to remove b3=b3multi alias)

manifests/java.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3644,7 +3644,6 @@ manifest:
36443644
tests/parametric/test_dynamic_configuration.py::TestDynamicConfigV1_ServiceTargets::test_not_match_service_target: irrelevant (APMAPI-1003)
36453645
tests/parametric/test_dynamic_configuration.py::TestDynamicConfigV2: v1.31.0
36463646
tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation: v1.56.0
3647-
tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation::test_ffe_of7_empty_targeting_key: bug (FFL-1729)
36483647
tests/parametric/test_ffe/test_span_enrichment.py: missing_feature
36493648
tests/parametric/test_headers_b3.py::Test_Headers_B3::test_headers_b3_migrated_extract_invalid: # Modified by easy win activation script
36503649
- declaration: missing_feature (Need to remove b3=b3multi alias)

manifests/nodejs.yml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2014,7 +2014,6 @@ manifest:
20142014
tests/parametric/test_dynamic_configuration.py::TestDynamicConfigV1_ServiceTargets::test_not_match_service_target: bug (APMAPI-865)
20152015
tests/parametric/test_dynamic_configuration.py::TestDynamicConfigV2: *ref_4_23_0
20162016
tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation: *ref_5_75_0
2017-
tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation::test_ffe_of7_empty_targeting_key: bug (FFL-1730)
20182017
tests/parametric/test_ffe/test_span_enrichment.py: "missing_feature (dd-trace-js#8343)"
20192018
tests/parametric/test_headers_b3.py::Test_Headers_B3::test_headers_b3_migrated_extract_invalid: missing_feature (Need to remove b3=b3multi alias)
20202019
tests/parametric/test_headers_b3.py::Test_Headers_B3::test_headers_b3_migrated_extract_valid: missing_feature (Need to remove b3=b3multi alias)

manifests/php.yml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -605,7 +605,7 @@ manifest:
605605
component_version: <1.12.0
606606
tests/docker_ssi/test_docker_ssi_appsec.py::TestDockerSSIAppsecFeatures::test_telemetry_source_ssi: v1.8.3
607607
tests/docker_ssi/test_docker_ssi_crash.py::TestDockerSSICrash::test_crash: missing_feature (No implemented the endpoint /crashme)
608-
tests/ffe/test_dynamic_evaluation.py: missing_feature
608+
tests/ffe/test_dynamic_evaluation.py: v1.21.0-dev
609609
tests/ffe/test_exposures.py: missing_feature
610610
tests/ffe/test_flag_eval_metrics.py: missing_feature
611611
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
@@ -731,7 +731,7 @@ manifest:
731731
tests/parametric/test_dynamic_configuration.py::TestDynamicConfigV1_ServiceTargets::test_not_match_service_target: missing_feature
732732
tests/parametric/test_dynamic_configuration.py::TestDynamicConfigV2: '>=1.16.0'
733733
tests/parametric/test_dynamic_configuration.py::TestDynamicConfigV2::test_tracing_client_tracing_tags: missing_feature
734-
tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation: missing_feature
734+
tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Dynamic_Evaluation: v1.21.0-dev
735735
tests/parametric/test_ffe/test_span_enrichment.py: missing_feature
736736
tests/parametric/test_headers_b3.py::Test_Headers_B3::test_headers_b3_migrated_extract_invalid:
737737
- declaration: missing_feature (Need to remove b3=b3multi alias)
@@ -870,7 +870,7 @@ manifest:
870870
tests/parametric/test_parametric_endpoints.py::Test_Parametric_DDSpan_Start: v1.13.0+4663b2fa7c20c6920f347d059b57dc2a419cb7f7
871871
tests/parametric/test_parametric_endpoints.py::Test_Parametric_DDTrace_Baggage: missing_feature (baggage is not supported)
872872
tests/parametric/test_parametric_endpoints.py::Test_Parametric_DDTrace_Current_Span: bug (APMAPI-778) # current span endpoint should return span and trace id of zero if no span is "active"
873-
tests/parametric/test_parametric_endpoints.py::Test_Parametric_FFE_Start: missing_feature
873+
tests/parametric/test_parametric_endpoints.py::Test_Parametric_FFE_Start: v1.21.0-dev
874874
tests/parametric/test_parametric_endpoints.py::Test_Parametric_Otel_Baggage: missing_feature (otel baggage is not supported)
875875
tests/parametric/test_parametric_endpoints.py::Test_Parametric_Otel_Current_Span: bug (APMAPI-778) # otel current span endpoint should return a span and trace id of zero if no span is "active"
876876
tests/parametric/test_parametric_endpoints.py::Test_Parametric_Write_Log: missing_feature

tests/parametric/test_ffe/test_dynamic_evaluation.py

Lines changed: 48 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,11 @@
22

33
import json
44
import pytest
5+
import time
56
from pathlib import Path
67
from typing import Any
78

89
from utils import (
9-
context,
1010
features,
1111
scenarios,
1212
)
@@ -16,6 +16,8 @@
1616

1717
RC_PRODUCT = "FFE_FLAGS"
1818
RC_PATH = f"datadog/2/{RC_PRODUCT}"
19+
FFE_READY_RETRY_ATTEMPTS = 10
20+
FFE_READY_RETRY_INTERVAL_SECONDS = 0.2
1921

2022
parametrize = pytest.mark.parametrize
2123

@@ -81,6 +83,44 @@ def _set_and_wait_ffe_rc(
8183
return test_agent.wait_for_rc_apply_state(RC_PRODUCT, state=RemoteConfigApplyState.ACKNOWLEDGED, clear=True)
8284

8385

86+
def _is_ffe_waiting_for_rc(result: dict[str, Any]) -> bool:
87+
provider_state = result.get("providerState")
88+
return result.get("errorCode") == "PROVIDER_NOT_READY" or (
89+
isinstance(provider_state, dict) and provider_state.get("hasConfig") is False
90+
)
91+
92+
93+
def _ffe_evaluate_with_rc_retry(
94+
test_library: APMLibrary,
95+
*,
96+
flag: str,
97+
variation_type: str,
98+
default_value: bool | str | float | dict[str, Any],
99+
targeting_key: str,
100+
attributes: dict[str, Any] | None = None,
101+
) -> dict[str, Any]:
102+
result = test_library.ffe_evaluate(
103+
flag=flag,
104+
variation_type=variation_type,
105+
default_value=default_value,
106+
targeting_key=targeting_key,
107+
attributes=attributes,
108+
)
109+
for _ in range(FFE_READY_RETRY_ATTEMPTS - 1):
110+
if not _is_ffe_waiting_for_rc(result):
111+
return result
112+
time.sleep(FFE_READY_RETRY_INTERVAL_SECONDS)
113+
result = test_library.ffe_evaluate(
114+
flag=flag,
115+
variation_type=variation_type,
116+
default_value=default_value,
117+
targeting_key=targeting_key,
118+
attributes=attributes,
119+
)
120+
121+
return result
122+
123+
84124
@scenarios.parametric
85125
@features.feature_flags_dynamic_evaluation
86126
class Test_Feature_Flag_Dynamic_Evaluation:
@@ -111,13 +151,6 @@ def test_ffe_flag_evaluation(self, test_case_file: str, test_agent: TestAgentAPI
111151
4. Handles user targeting, attribute matching, and rollout percentages
112152
113153
"""
114-
# Skip OF.7 (empty targeting key) test for libraries with known bugs
115-
# Java: FFL-1729 - OpenFeature Java SDK rejects empty targeting keys
116-
# Node.js: FFL-1730 - OpenFeature JS SDK rejects empty targeting keys
117-
if test_case_file == "test-case-of-7-empty-targeting-key.json":
118-
if context.library.name in ("java", "nodejs"):
119-
pytest.skip("OF.7 empty targeting key bug: FFL-1729 (java), FFL-1730 (nodejs)")
120-
121154
# Load the test case file
122155
test_case_path = Path(__file__).parent / test_case_file
123156

@@ -131,7 +164,7 @@ def test_ffe_flag_evaluation(self, test_case_file: str, test_agent: TestAgentAPI
131164
_set_and_wait_ffe_rc(test_agent, UFC_FIXTURE_DATA)
132165

133166
# Initialize FFE provider
134-
success = test_library.ffe_start()
167+
success = test_library.ffe_start(UFC_FIXTURE_DATA)
135168
assert success, "Failed to start FFE provider"
136169

137170
# Run each test case
@@ -143,13 +176,18 @@ def test_ffe_flag_evaluation(self, test_case_file: str, test_agent: TestAgentAPI
143176
attributes = test_case.get("attributes", {})
144177
expected_result = test_case["result"]["value"]
145178

146-
result = test_library.ffe_evaluate(
179+
result = _ffe_evaluate_with_rc_retry(
180+
test_library,
147181
flag=flag,
148182
variation_type=variation_type,
149183
default_value=default_value,
150184
targeting_key=targeting_key,
151185
attributes=attributes,
152186
)
187+
assert not _is_ffe_waiting_for_rc(result), (
188+
f"Test case {i} in {test_case_file} failed: FFE provider did not load RC data after "
189+
f"{FFE_READY_RETRY_ATTEMPTS} attempts; result={result}"
190+
)
153191
actual_value = result.get("value")
154192

155193
# Assert the evaluation result matches expected value
@@ -158,33 +196,3 @@ def test_ffe_flag_evaluation(self, test_case_file: str, test_agent: TestAgentAPI
158196
f"flag='{flag}', targetingKey='{targeting_key}', "
159197
f"expected={expected_result}, actual={actual_value}"
160198
)
161-
162-
@parametrize("library_env", [{**DEFAULT_ENVVARS}])
163-
def test_ffe_of7_empty_targeting_key(self, test_agent: TestAgentAPI, test_library: APMLibrary) -> None:
164-
"""OF.7: Empty string is a valid targeting key.
165-
166-
This test validates that flag evaluation succeeds when the targeting key
167-
is an empty string. The flag should still match allocations and return
168-
the expected value, not fail with TARGETING_KEY_MISSING.
169-
170-
Temporary dedicated test until FFL-1729 (Java) and FFL-1730 (Node.js) are resolved.
171-
"""
172-
# Set up UFC Remote Config and wait for it to be applied
173-
_set_and_wait_ffe_rc(test_agent, UFC_FIXTURE_DATA)
174-
175-
# Initialize FFE provider
176-
success = test_library.ffe_start()
177-
assert success, "Failed to start FFE provider"
178-
179-
# Evaluate flag with empty targeting key
180-
result = test_library.ffe_evaluate(
181-
flag="empty-targeting-key-flag",
182-
variation_type="STRING",
183-
default_value="default",
184-
targeting_key="",
185-
attributes={},
186-
)
187-
188-
assert result.get("value") == "on-value", (
189-
f"OF.7 failed: empty targeting key should return 'on-value', got '{result.get('value')}'"
190-
)

0 commit comments

Comments
 (0)