Skip to content

Enable EVP flagevaluation system tests for Ruby#7184

Draft
leoromanovsky wants to merge 7 commits into
mainfrom
leo.romanovsky/ffe-evp-flagevaluation-enable-ruby-clean
Draft

Enable EVP flagevaluation system tests for Ruby#7184
leoromanovsky wants to merge 7 commits into
mainfrom
leo.romanovsky/ffe-evp-flagevaluation-enable-ruby-clean

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

Motivation

Ruby needs the shared server-side EVP flagevaluation system-test contract enabled so APM reviewers can validate that Ruby evaluations produce the same backend-visible signal as the other SDKs. This PR provides the public system-tests companion for the Ruby SDK fanout.

Changes

  • Enables tests/ffe/test_flag_eval_evp.py for the validated Ruby rails72 weblog in manifests/ruby.yml at v2.36.0-dev; other Ruby weblogs remain irrelevant.
  • Splits and parallelizes the degradation test requests for Ruby so the test proves SDK aggregation without relying on one oversized app request.
  • Keeps the EVP path constant aligned with the current merged system-tests contract.

Decisions

  • Enable only rails72 for EVP flagevaluation; broader non-metric OpenFeature evaluation coverage for older Ruby weblogs lives in the separate Ruby-oldest PR.
  • Keep degradation request shaping in the system test because app request body/time limits are separate from the SDK EVP payload-splitting behavior under review.
  • Existing OTel metric coverage stays in tests/ffe/test_flag_eval_metrics.py; this PR is only for EVP flagevaluation.

Validation Evidence

System Tests

@github-actions

github-actions Bot commented Jun 20, 2026

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/ffe/test_flag_eval_evp.py                                         @DataDog/feature-flagging-and-experimentation-sdk @DataDog/system-tests-core

@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Jun 20, 2026

Copy link
Copy Markdown

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 4 Pipeline jobs failed

Testing the test | System Tests (python, dev) / End-to-end #1 / uwsgi-poc 1   View in Datadog   GitHub Actions

🧪 2 Tests failed

tests.appsec.iast.sink.test_unvalidated_redirect.TestUnvalidatedHeader.test_secure[uwsgi-poc] from system_tests_suite   View in Datadog
ValueError: Unexpected vulnerability reported: UNVALIDATED_REDIRECT

self = <tests.appsec.iast.sink.test_unvalidated_redirect.TestUnvalidatedHeader object at 0x7fc54186b9b0>

    def test_secure(self):
>       return super().test_secure()

tests/appsec/iast/sink/test_unvalidated_redirect.py:62: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/appsec/iast/utils.py:186: in test_secure
...
tests.appsec.iast.sink.test_unvalidated_redirect.TestUnvalidatedRedirect.test_secure[uwsgi-poc] from system_tests_suite   View in Datadog
ValueError: Unexpected vulnerability reported: UNVALIDATED_REDIRECT

self = <tests.appsec.iast.sink.test_unvalidated_redirect.TestUnvalidatedRedirect object at 0x7fc54186bd40>

    def test_secure(self):
>       super().test_secure()

tests/appsec/iast/sink/test_unvalidated_redirect.py:47: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/appsec/iast/utils.py:186: in test_secure
...

Testing the test | System Tests (ruby, dev) / End-to-end #1 / rails72 1   View in Datadog   GitHub Actions

🧪 1 Test failed

pytest.internal[rails72] from system_tests_suite   View in Datadog
internal error

Traceback (most recent call last):
  File "/home/runner/work/system-tests/system-tests/venv/lib/python3.12/site-packages/_pytest/main.py", line 268, in wrap_session
    session.exitstatus = doit(config, session) or 0
                         ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/system-tests/system-tests/venv/lib/python3.12/site-packages/_pytest/main.py", line 321, in _main
    config.hook.pytest_collection(session=session)
  File "/home/runner/work/system-tests/system-tests/venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
...

Testing the test | System Tests (ruby, prod) / End-to-end #1 / rails72 1   View in Datadog   GitHub Actions

🧪 1 Test failed

pytest.internal[rails72] from system_tests_suite   View in Datadog
internal error

Traceback (most recent call last):
  File "/home/runner/work/system-tests/system-tests/venv/lib/python3.12/site-packages/_pytest/main.py", line 268, in wrap_session
    session.exitstatus = doit(config, session) or 0
                         ^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/system-tests/system-tests/venv/lib/python3.12/site-packages/_pytest/main.py", line 321, in _main
    config.hook.pytest_collection(session=session)
  File "/home/runner/work/system-tests/system-tests/venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
...

View all 4 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5235629 | Docs | Datadog PR Page | Give us feedback!

@datadog-system-tests-org

datadog-system-tests-org Bot commented Jun 20, 2026

Copy link
Copy Markdown

Pipelines

⚠️ Warnings

🚦 2 Pipeline jobs failed

Testing the test | System Tests (ruby, dev) / End-to-end #1 / rails72 1   View in Datadog   GitHub Actions

Testing the test | all-jobs-are-green   View in Datadog   GitHub Actions

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7b30ebb | Docs | Give us feedback!

Base automatically changed from leo.romanovsky/ffe-evp-flagevaluation-contract-fix to main June 22, 2026 19:40
…p-flagevaluation-enable-ruby-clean

# Conflicts:
#	tests/ffe/test_flag_eval_evp.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant