ci: expand test optimization dogfooding (#17567)

## Description

Removes all remaining `--no-ddtrace` flags from riotfile.py, expanding dogfooding of the new Test Optimization plugin (`ddtrace/testing`) to more test suites.

Most of these `--no-ddtrace` flags were originally added when the old `_plugin_v2.py`-backed plugin was in use for outer sessions. Now that the new plugin (`ddtrace/testing`) is the default and is used for outer sessions, they are no longer needed.

**Inner sessions** (spawned by tests such as pytest-bdd, pytest-benchmark, pytest-flaky, and selenium) are not affected:
- `inline_run`-based inner sessions (pytest-bdd, pytest-benchmark, pytest-flaky) use `PytestTestCaseBase.inline_run()`, which suspends/resumes the outer CIVisibility instance and explicitly controls the `--ddtrace` flag per inner session.
- Subprocess-based inner sessions (selenium) are isolated processes and unaffected by the outer session's plugin state.

For now, all inner sessions use the new plugin (since `DD_PYTEST_USE_NEW_PLUGIN=false` in `_ci_override_env` has no effect on already-imported modules in in-process sessions). This is intentional — the new plugin is the focus of current efforts.

---

Also addresses comment by codex:
> In the GitLab riot workflow (.gitlab/tests.yml, line 43), every suite is invoked with --ddtrace, so removing --no-ddtrace here enables CI Visibility in the outer pytest process for this suite. That breaks assumptions in tests/integration/test_integration_civisibility.py (for example, test_civisibility_intake_with_missing_apikey expects CIVisibility to start disabled and asserts _instance is None after enable()), because a pre-existing singleton makes CIVisibility.enable() return early and bypasses the test’s patched initialization path.

By removing those env vars in conftest.py with a fixture, and also pause/resume any CIVisibility instance (not needed for ddtrace/testing plugin really, but it is needed if we switched to v2).

## Testing

Covered by the dogfooded test suites themselves running under the new plugin.

## Risks

Low. The new plugin is already the default. Removing `--no-ddtrace` only affects the outer pytest session instrumentation; inner sessions already control their own plugin activation.

## Additional Notes

Suites like `pytest_bdd` and `pytest_benchmark` have separate bdd/benchmark implementations in both plugins. Future work could add explicit coverage for the new plugin's code paths for those integrations.

Co-authored-by: federico.mon <federico.mon@datadoghq.com>

Author: gnufede

riotfile.py

@@ -132,7 +132,7 @@ def select_pys(min_version: str = MIN_PYTHON_VERSION, max_version: str = MAX_PYT
         Venv(
             name="meta-testing",
             pys=["3.10"],
-            command="pytest {cmdargs} --no-ddtrace tests/meta",
+            command="pytest {cmdargs} tests/meta",
             env={
                 "DD_CIVISIBILITY_FLAKY_RETRY_ENABLED": "0",
             },
@@ -475,7 +475,7 @@ def select_pys(min_version: str = MIN_PYTHON_VERSION, max_version: str = MAX_PYT
             name="integration",
             # Enabling coverage for integration tests breaks certain tests in CI
             # Also, running two separate pytest sessions, the ``civisibility`` one with --no-ddtrace
-            command="pytest -vv --no-ddtrace --no-cov --ignore-glob='*civisibility*' {cmdargs} tests/integration/",
+            command="pytest -vv --no-cov --ignore-glob='*civisibility*' {cmdargs} tests/integration/",
             pkgs={"msgpack": [latest], "coverage": latest, "pytest-randomly": latest},
             pys=select_pys(),
             venvs=[
@@ -497,7 +497,7 @@ def select_pys(min_version: str = MIN_PYTHON_VERSION, max_version: str = MAX_PYT
             name="integration-civisibility",
             # Enabling coverage for integration tests breaks certain tests in CI
             # Also, running two separate pytest sessions, the ``civisibility`` one with --no-ddtrace
-            command="pytest --no-cov --no-ddtrace {cmdargs} tests/integration/test_integration_civisibility.py",
+            command="pytest --no-cov {cmdargs} tests/integration/test_integration_civisibility.py",
             pkgs={"msgpack": [latest], "coverage": latest, "pytest-randomly": latest},
             pys=select_pys(),
             venvs=[
@@ -2068,7 +2068,7 @@ def select_pys(min_version: str = MIN_PYTHON_VERSION, max_version: str = MAX_PYT
         ),
         Venv(
             name="unittest",
-            command="pytest --no-ddtrace {cmdargs} tests/contrib/unittest/",
+            command="pytest {cmdargs} tests/contrib/unittest/",
             pkgs={
                 "msgpack": latest,
                 "pytest-randomly": latest,
@@ -2084,7 +2084,7 @@ def select_pys(min_version: str = MIN_PYTHON_VERSION, max_version: str = MAX_PYT
         ),
         Venv(
             name="asynctest",
-            command="pytest --no-ddtrace {cmdargs} tests/contrib/asynctest/",
+            command="pytest {cmdargs} tests/contrib/asynctest/",
             pkgs={
                 "pytest-randomly": latest,
             },
@@ -2105,7 +2105,7 @@ def select_pys(min_version: str = MIN_PYTHON_VERSION, max_version: str = MAX_PYT
         ),
         Venv(
             name="pytest_bdd",
-            command="pytest --no-ddtrace {cmdargs} tests/contrib/pytest_bdd/",
+            command="pytest {cmdargs} tests/contrib/pytest_bdd/",
             pkgs={
                 "msgpack": latest,
                 "more_itertools": "<8.11.0",
@@ -2140,7 +2140,7 @@ def select_pys(min_version: str = MIN_PYTHON_VERSION, max_version: str = MAX_PYT
         Venv(
             name="pytest_benchmark",
             pys=select_pys(),
-            command="pytest {cmdargs} --no-ddtrace --no-cov tests/contrib/pytest_benchmark/",
+            command="pytest {cmdargs} --no-cov tests/contrib/pytest_benchmark/",
             pkgs={
                 "msgpack": latest,
                 "pytest-randomly": latest,
@@ -2161,7 +2161,7 @@ def select_pys(min_version: str = MIN_PYTHON_VERSION, max_version: str = MAX_PYT
         Venv(
             name="pytest:flaky",
             pys=select_pys(),
-            command="pytest {cmdargs} --no-ddtrace --no-cov -p no:flaky tests/contrib/pytest_flaky/",
+            command="pytest {cmdargs} --no-cov -p no:flaky tests/contrib/pytest_flaky/",
             pkgs={
                 "flaky": latest,
                 "pytest-randomly": latest,
@@ -3358,7 +3358,7 @@ def select_pys(min_version: str = MIN_PYTHON_VERSION, max_version: str = MAX_PYT
         ),
         Venv(
             name="aws_lambda",
-            command="pytest --no-ddtrace {cmdargs} tests/contrib/aws_lambda",
+            command="pytest {cmdargs} tests/contrib/aws_lambda",
             pys=select_pys(min_version="3.9", max_version="3.13"),
             pkgs={
                 "boto3": latest,
@@ -3874,7 +3874,7 @@ def select_pys(min_version: str = MIN_PYTHON_VERSION, max_version: str = MAX_PYT
                 "selenium": "~=4.0",
                 "webdriver-manager": latest,
             },
-            command="pytest --no-cov {cmdargs} -c /dev/null --no-ddtrace tests/contrib/selenium",
+            command="pytest --no-cov {cmdargs} -c /dev/null tests/contrib/selenium",
             env={
                 "DD_AGENT_PORT": "9126",
             },

tests/integration/test_integration_civisibility.py

@@ -1,6 +1,6 @@
 import os
+from unittest import mock
 
-import mock
 import pytest
 
 from ddtrace.internal.ci_visibility import CIVisibility
@@ -18,6 +18,31 @@
 AGENT_VERSION = os.environ.get("AGENT_VERSION")
 
 
+@pytest.fixture(autouse=True)
+def _isolate_from_outer_session(monkeypatch):
+    """Strip DD_* env vars and suspend CIVisibility so each test starts with a clean slate.
+
+    When the outer pytest session runs with --ddtrace in CI, DD_API_KEY and other DD_* vars
+    are present in the process environment. Tests asserting on specific env conditions (e.g.
+    missing API key) must not see those vars. Each test sets its own DD_* vars explicitly via
+    override_env(), so stripping them all here is safe.
+
+    The CIVisibility suspension is belt-and-suspenders for the old plugin path where a
+    pre-existing singleton makes enable() return early.
+    """
+    for key in list(os.environ):
+        if key.startswith(("DD_", "_CI_DD_")):
+            monkeypatch.delenv(key, raising=False)
+
+    suspended = CIVisibility._suspend()
+    try:
+        yield
+    finally:
+        if CIVisibility.enabled:
+            CIVisibility.disable()
+        CIVisibility._resume(suspended)
+
+
 @pytest.fixture(autouse=True, scope="module")
 def _dummy_check_enabled_features():
     """By default, assume that _check_enabled_features() returns an ITR-disabled response.
@@ -77,8 +102,7 @@ def test_civisibility_intake_with_apikey():
 @pytest.mark.subprocess()
 def test_civisibility_intake_payloads():
     import gzip
-
-    import mock
+    from unittest import mock
 
     from ddtrace.internal.ci_visibility.constants import COVERAGE_TAG_NAME
     from ddtrace.internal.ci_visibility.recorder import CIVisibilityWriter