Skip to content

Commit fa3b13d

Browse files
[ffe] Activate test_flag_eval_metrics.py across released tracer versions (#6972)
Co-authored-by: dd-oleksii <oleksii.shmalko@datadoghq.com> Co-authored-by: sameeran.kunche <sameeran.kunche@datadoghq.com>
1 parent 03abb49 commit fa3b13d

6 files changed

Lines changed: 20 additions & 25 deletions

File tree

manifests/dotnet.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -702,7 +702,7 @@ manifest:
702702
tests/ffe/test_dynamic_evaluation.py::Test_FFE_Flag_Parse_Error_Isolation: bug (FFL-2184)
703703
tests/ffe/test_dynamic_evaluation.py::Test_FFE_Unknown_Operator_Tolerance: bug (FFL-2184)
704704
tests/ffe/test_exposures.py: v3.36.0
705-
tests/ffe/test_flag_eval_metrics.py: missing_feature (FFL-2257 - Requires Datadog.FeatureFlags.OpenFeature 2.3.0 NuGet package to be published)
705+
tests/ffe/test_flag_eval_metrics.py: v3.44.0
706706
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
707707
tests/integrations/crossed_integrations/test_kafka.py::Test_Kafka: v2.0.0-prerelease
708708
tests/integrations/crossed_integrations/test_kinesis.py::Test_Kinesis_PROPAGATION_VIA_MESSAGE_ATTRIBUTES: missing_feature

manifests/golang.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -877,7 +877,7 @@ manifest:
877877
tests/ffe/test_dynamic_evaluation.py::Test_FFE_RC_Unavailable: v2.4.0
878878
tests/ffe/test_dynamic_evaluation.py::Test_FFE_Unknown_Operator_Tolerance::test_unknown_operator_errors: bug (FFL-2182)
879879
tests/ffe/test_exposures.py: v2.6.0-dev # Easy win for chi, echo, gin, net-http, net-http-orchestrion, uds-echo and version 2.5.0
880-
tests/ffe/test_flag_eval_metrics.py: v2.9.0-dev
880+
tests/ffe/test_flag_eval_metrics.py: v2.8.0
881881
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex: irrelevant (Go validates regex at config load time)
882882
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Nested_Attributes_Ignored: irrelevant (FFL-1980)
883883
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)

manifests/java.yml

Lines changed: 4 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -3161,26 +3161,11 @@ manifest:
31613161
"*": irrelevant
31623162
spring-boot: v1.56.0
31633163
tests/ffe/test_exposures.py::Test_FFE_EXP_5_Missing_Targeting_Key: bug (FFL-1729)
3164-
tests/ffe/test_flag_eval_metrics.py: missing_feature (FFL-1972)
3165-
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Config_Exists_Flag_Missing::test_ffe_eval_config_exists_flag_missing: bug (FFL-1972)
3166-
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Lowercase_Consistency::test_ffe_lowercase_error_type: bug (FFL-1972)
3167-
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Lowercase_Consistency::test_ffe_lowercase_reason: bug (FFL-1972)
3168-
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Basic::test_ffe_eval_metric_basic: bug (FFL-1972)
3164+
tests/ffe/test_flag_eval_metrics.py:
3165+
- weblog_declaration:
3166+
"*": irrelevant
3167+
spring-boot: v1.62.0
31693168
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Count::test_ffe_eval_metric_count: bug (FFL-1972)
3170-
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Different_Flags::test_ffe_eval_metric_different_flags: bug (FFL-1972)
3171-
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Numeric_To_Integer::test_ffe_eval_metric_numeric_to_integer: bug (FFL-1972)
3172-
? tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex::test_ffe_eval_metric_parse_error_invalid_regex
3173-
: bug (FFL-1972)
3174-
? tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Variant_Type_Mismatch::test_ffe_eval_metric_parse_error_variant_type_mismatch
3175-
: bug (FFL-1972)
3176-
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Type_Mismatch::test_ffe_eval_metric_type_mismatch: bug (FFL-1972)
3177-
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Nested_Attributes_Ignored::test_ffe_eval_nested_attributes_ignored: bug (FFL-1972)
3178-
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_No_Config_Loaded::test_ffe_eval_no_config_loaded: bug (FFL-1972)
3179-
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Default::test_ffe_eval_reason_default: bug (FFL-1972)
3180-
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Disabled::test_ffe_eval_reason_disabled: bug (FFL-1972)
3181-
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Split::test_ffe_eval_reason_split: bug (FFL-1972)
3182-
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Targeting::test_ffe_eval_reason_targeting: bug (FFL-1972)
3183-
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional::test_ffe_eval_targeting_key_optional: bug (FFL-1972)
31843169
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
31853170
tests/integration_frameworks/llm/openai/test_openai_apm.py: v1.61.0
31863171
tests/integration_frameworks/llm/openai/test_openai_llmobs.py: v1.61.0

manifests/nodejs.yml

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1679,8 +1679,18 @@ manifest:
16791679
"*": incomplete_test_app
16801680
express4: *ref_5_77_0
16811681
tests/ffe/test_exposures.py::Test_FFE_EXP_5_Missing_Targeting_Key: bug (FFL-1730)
1682-
tests/ffe/test_flag_eval_metrics.py: missing_feature
1683-
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional: irrelevant (JS SDK requires targeting key)
1682+
tests/ffe/test_flag_eval_metrics.py:
1683+
- weblog_declaration:
1684+
"*": incomplete_test_app
1685+
express4: *ref_5_99_0
1686+
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Basic::test_ffe_eval_metric_basic: bug (FFL-2313)
1687+
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Numeric_To_Integer::test_ffe_eval_metric_numeric_to_integer: bug (FFL-2313)
1688+
? tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Invalid_Regex::test_ffe_eval_metric_parse_error_invalid_regex
1689+
: bug (FFL-2313)
1690+
? tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Metric_Parse_Error_Variant_Type_Mismatch::test_ffe_eval_metric_parse_error_variant_type_mismatch
1691+
: bug (FFL-2313)
1692+
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Reason_Split::test_ffe_eval_reason_split: bug (FFL-2313)
1693+
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Targeting_Key_Optional: bug (FFL-1730)
16841694
tests/integration_frameworks/llm/anthropic/test_anthropic_apm.py::TestAnthropicApmMessages: *ref_5_71_0
16851695
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages: *ref_5_71_0
16861696
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create: missing_feature # ephemeral cache TTL metrics not yet released

manifests/python.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1316,7 +1316,7 @@ manifest:
13161316
tests/ffe/test_dynamic_evaluation.py::Test_FFE_RC_Down_From_Start: v4.0.0
13171317
tests/ffe/test_dynamic_evaluation.py::Test_FFE_RC_Unavailable: flaky (FFL-1622)
13181318
tests/ffe/test_exposures.py: v4.2.0-dev
1319-
tests/ffe/test_flag_eval_metrics.py: v4.8.0-dev
1319+
tests/ffe/test_flag_eval_metrics.py: v4.7.0
13201320
tests/ffe/test_flag_eval_metrics.py::Test_FFE_Eval_Nested_Attributes_Ignored: irrelevant (FFL-1980)
13211321
tests/integration_frameworks/llm/anthropic/test_anthropic_apm.py::TestAnthropicApmMessages: v3.16.0
13221322
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages: v3.16.0

manifests/ruby.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1857,7 +1857,7 @@ manifest:
18571857
tests/ffe/test_flag_eval_metrics.py:
18581858
- weblog_declaration:
18591859
"*": irrelevant
1860-
rails72: v2.32.0-dev
1860+
rails72: v2.32.0
18611861
tests/integration_frameworks/llm/anthropic/test_anthropic_llmobs.py::TestAnthropicLlmObsMessages::test_create_error: bug (MLOB-1234)
18621862
tests/integrations/crossed_integrations/test_kafka.py::Test_Kafka:
18631863
- weblog_declaration:

0 commit comments

Comments
 (0)