Remove black, use ruff for generated config_models (DataDog#23588)

* Drop black as a direct dependency, use ruff format for generated config models

- Remove explicit `black==23.12.1` from `datadog_checks_dev`
- Replace `apply_black` calls in the model consumer with `ruff format -`,
  using the repo's centralized `[tool.ruff]` configuration
- Drop the now-unused `code_formatter` plumbing through `ModelConsumer` /
  `build_model_file` / `validate models`
- Drop the [tool.black] block from `ddev/pyproject.toml` and the matching
  python-version-bump logic in `update_py_config.py` (with test fixture)
- Update README badge from black to ruff

The root `[tool.black]` section is kept (with an explanatory comment)
because `datamodel-code-generator` reads it transitively through its own
internal formatter, and removing it changes line-length to 88 which
breaks our list[...] -> tuple[..., ...] line-by-line transform.

* Add changelog entries

* Make ruff a hard dep of datadog_checks_dev[cli], invoke via python -m ruff

CI installs ddev with `pip install -e ./datadog_checks_dev[cli]` and never
adds ruff to PATH. The previous helper called `shutil.which('ruff')` and
silently returned the input unchanged when ruff was missing — leaving long
lines and missing wraps in 401 generated config-model files, surfacing as
"not in sync" in the validate workflow.

- Declare `ruff>=0.11` in `datadog_checks_dev[cli]` so the package is
  always installed alongside the model generator.
- Switch the helper to `sys.executable -m ruff` so the in-venv package is
  used regardless of PATH.
- Raise loudly on missing ruff or non-zero ruff exit instead of silently
  degrading, so any future regression fails the workflow with a clear error.

* Pin ruff to 0.11.10 to match ddev's hatch lint env

* Remove [tool.black] from root pyproject.toml and update _fix_types

Make `_fix_types` operate on the joined document (as UTF-8 bytes) instead
of line by line, so the bracket-tracking pass works regardless of how
datamodel-code-generator's internal formatter wrapped `list[...]`. Place
the `, ...` sentinel right after the last non-whitespace byte before the
closing `]`, so output stays on the previous content line even when the
parser pre-wrapped the closing bracket onto its own line.

With those changes the generator no longer relies on `[tool.black]`
existing in the repo, so the section and its accompanying comment are
removed from `pyproject.toml`. The black-related comment near the
config_models lint exclusion and the black badge in `README.md` go too.

Four config_models files (kafka_actions, win32_event_log, yarn x2)
regenerate with different — but semantically identical — wrapping. They
were the only ones whose pre-wrapped form was sensitive to the change in
upstream line-length default; future regens are stable.

* Add changelogs for regenerated config_models in kafka_actions, win32_event_log, yarn

* Address PR review: docs, error-path hint, focused _fix_types tests

- Replace remaining "code style - black" references in developer docs
  (`docs/developer/index.md` badge, `docs/developer/guidelines/style.md`
  style section, link reference) with ruff equivalents.
- Update the stale `ddev test postgres -l` example output in
  `docs/developer/testing.md` to drop `black==22.12.0` and reflect the
  current lint env contents (`ruff==0.11.10`, `pydantic==2.11.5`).
- Move the "ruff is not installed" install hint in `format_with_ruff`
  from the `FileNotFoundError` branch to the `CalledProcessError` branch
  and gate it on `"No module named 'ruff'"` in stderr — the previous
  layout was effectively dead code because `sys.executable` always
  resolves, so missing-ruff surfaces as a non-zero exit, not a missing
  binary.
- Add `tests/tooling/configuration/consumers/model/test_fix_types.py`
  with focused coverage for the `_fix_types` post-processing pass: the
  multiline-wrapped `list[Literal[...]]` regression case the PR was
  written to fix, dict and nested-list translations, unicode in
  descriptions, and verbatim pass-through when no `list[`/`dict[`
  is present.

* Address PR review (round 2): code_formatter robustness + direct tests

code_formatter.py:
- Guard `_resolve_ruff_config` with `if root_str:` so an unset
  `get_root()` (returns '') doesn't fall into the `Path('').is_dir()`
  branch (which is True — it resolves to CWD), and unit tests actually
  walk back to the repo pyproject.toml as the docstring claims.
- Replace the loose `'[tool.ruff' in text` substring with a line-anchored
  scan that only matches actual TOML table headers (`[tool.ruff]` or
  `[tool.ruff.…]`), so a comment or string value can't false-positive.
- Surface argv (via `shlex.join`), stderr, and stdout in the error
  message for non-missing-package failures, so a future ruff config
  change emitting actionable output is debuggable from the message alone.

Tests:
- New `test_code_formatter.py` (17 tests): direct coverage for
  `format_with_ruff` (line wrapping, quote-style preservation, short
  passthrough, missing-ruff hint, full-context error on other failures)
  and `_resolve_ruff_config` (root path success, fallback walk on empty
  root, fallback walk when root has no `[tool.ruff]`, returns None when
  nothing is found, parametrized header recognition for `_has_ruff_section`).
- `test_update_py_config.py`: add explicit content assertions on the
  rewritten `ddev/pyproject.toml` — no `[tool.black]` block survives,
  `[tool.ruff].target-version` is updated to the new pinned version, the
  old token is gone. Captures the actual contract instead of relying on
  the success-counter being 9.

* Relax ddev's datadog-checks-dev pin to span the dcd 39 release gap

This PR bumps `datadog-checks-dev` to 39 (`black` is dropped from `[cli]`
extras, so per semver it's a major). The current `~=38.0` constraint in
ddev's `pyproject.toml` would block any GitHub Action that installs both
packages from the local repo — `pip install -e ./ddev` would fail to
resolve the local dcd 39 against ddev's pin.

Relax the pin to `>=38.0,<40` for the duration of the gap between this
PR landing and the next ddev release. The release PR for ddev MUST
tighten this back to `~=39.0`.

* Apply ruff format to new helper and tests

* Restore relaxed datadog-checks-dev pin for transition period

Author: AAraKKe

README.md

@@ -4,7 +4,7 @@
 | --- | --- |
 | CI/CD | [![CI - Test][1]][2] |
 | Docs | [![Docs - Release][19]][20] |
-| Meta | [![Hatch project][26]][27] [![Linting - Ruff][24]][25] [![Code style - black][21]][22] [![Typing - Mypy][28]][29] [![License - BSD-3-Clause][30]][31] |
+| Meta | [![Hatch project][26]][27] [![Linting - Ruff][24]][25] [![Typing - Mypy][28]][29] [![License - BSD-3-Clause][30]][31] |
 
 This repository contains open source integrations that Datadog officially develops and supports.
 To add a new integration, please see the [Integrations Extras][5] repository and the
@@ -43,10 +43,8 @@ For more information on integrations, please reference our [documentation][11] a
 [16]: https://github.com/DataDog/integrations-core/blob/ea2dfbf1e8859333af4c8db50553eb72a3b466f9/requirements-agent-release.txt
 [19]: https://github.com/DataDog/integrations-core/workflows/docs/badge.svg
 [20]: https://github.com/DataDog/integrations-core/actions?workflow=docs
-[21]: https://img.shields.io/badge/code%20style-black-000000.svg
-[22]: https://github.com/ambv/black
-[24]: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v0.json
-[25]: https://github.com/charliermarsh/ruff
+[24]: https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json
+[25]: https://github.com/astral-sh/ruff
 [26]: https://img.shields.io/badge/%F0%9F%A5%9A-Hatch-4051b5.svg
 [27]: https://github.com/pypa/hatch
 [28]: https://img.shields.io/badge/typing-Mypy-blue.svg

datadog_checks_dev/changelog.d/23588.changed

@@ -0,0 +1 @@
+Stop declaring `black` as a direct dependency. The `apply_black` calls used to format auto-generated config-model files now go through `ruff format`, using the repo's centralized `[tool.ruff]` configuration.

datadog_checks_dev/datadog_checks/dev/tooling/commands/validate/models.py

@@ -95,8 +95,6 @@ def models(ctx, check, sync, verbose):
     license_header_lines = get_license_header().splitlines(True) + ['\n', '\n']
     documentation_header_lines = get_config_models_documentation().splitlines(True) + ['\n']
 
-    code_formatter = ModelConsumer.create_code_formatter()
-
     if is_core_check:
         checks = checks.difference(INTEGRATIONS_WITHOUT_MODELS)
 
@@ -135,7 +133,7 @@ def models(ctx, check, sync, verbose):
             if not sync and not dir_exists(models_location) and not is_core_check:
                 continue
 
-        model_consumer = ModelConsumer(spec.data, code_formatter)
+        model_consumer = ModelConsumer(spec.data)
 
         # So formatters see config files
         with chdir(root):

datadog_checks_dev/datadog_checks/dev/tooling/configuration/consumers/model/code_formatter.py

@@ -0,0 +1,89 @@
+# (C) Datadog, Inc. 2026-present
+# All rights reserved
+# Licensed under a 3-clause BSD style license (see LICENSE)
+import shlex
+import subprocess
+import sys
+from pathlib import Path
+
+from datadog_checks.dev.tooling.constants import get_root
+
+
+def format_with_ruff(source: str) -> str:
+    """Format Python source via ``ruff format -`` (stdin/stdout).
+
+    Replaces the line-wrapping role previously played by black on auto-generated
+    config_models files. Uses the repo's centralized ruff configuration so the
+    output matches the rest of the codebase. Invokes ruff through the active
+    interpreter (``python -m ruff``) so the package installed alongside
+    ``datadog_checks_dev[cli]`` is always picked up, regardless of PATH.
+    """
+    args = [sys.executable, '-m', 'ruff', 'format', '--quiet', '--stdin-filename=model.py']
+    config_path = _resolve_ruff_config()
+    if config_path is not None:
+        args.extend(['--config', str(config_path)])
+    else:
+        args.extend(['--isolated', '--config', "format.quote-style='preserve'", '--line-length=120'])
+    args.append('-')
+
+    try:
+        result = subprocess.run(
+            args,
+            input=source,
+            capture_output=True,
+            text=True,
+            check=True,
+        )
+    except subprocess.CalledProcessError as e:
+        # `python -m ruff` exits non-zero when the ruff package is missing,
+        # surfacing as ModuleNotFoundError on stderr. Promote that to a
+        # clearer install hint; otherwise propagate the underlying error
+        # with enough context to reproduce the failure manually.
+        stderr = e.stderr or ''
+        if "No module named 'ruff'" in stderr:
+            raise RuntimeError(
+                "Cannot format auto-generated config models: the `ruff` package is not installed in the active "
+                "interpreter. Reinstall `datadog_checks_dev[cli]` (or run `pip install ruff`) and retry."
+            ) from e
+        details = [f'{shlex.join(args)} failed', f'stderr: {stderr.strip()}']
+        if e.stdout:
+            details.append(f'stdout: {e.stdout.strip()}')
+        raise RuntimeError(
+            '`ruff format` failed while formatting auto-generated config models. ' + '; '.join(details)
+        ) from e
+    return result.stdout
+
+
+def _resolve_ruff_config() -> Path | None:
+    """Locate the repo pyproject.toml that holds the central ruff configuration.
+
+    Prefer the path reported by ``get_root`` (set by ddev commands). Fall back
+    to walking up from this module so unit tests, which never call ``set_root``,
+    still pick up the same configuration as model regeneration.
+    """
+    root_str = get_root()
+    if root_str:
+        root = Path(root_str)
+        if root.is_dir():
+            candidate = root / 'pyproject.toml'
+            if _has_ruff_section(candidate):
+                return candidate
+
+    for parent in Path(__file__).resolve().parents:
+        candidate = parent / 'pyproject.toml'
+        if _has_ruff_section(candidate):
+            return candidate
+    return None
+
+
+def _has_ruff_section(pyproject: Path) -> bool:
+    if not pyproject.is_file():
+        return False
+    try:
+        text = pyproject.read_text()
+    except OSError:
+        return False
+    return any(
+        stripped == '[tool.ruff]' or stripped.startswith('[tool.ruff.')
+        for stripped in (line.strip() for line in text.splitlines())
+    )

datadog_checks_dev/datadog_checks/dev/tooling/configuration/consumers/model/model_consumer.py

@@ -2,20 +2,19 @@
 # All rights reserved
 # Licensed under a 3-clause BSD style license (see LICENSE)
 import warnings
-from pathlib import Path
 from typing import Dict, List, Tuple
 
 import yaml
 from datamodel_code_generator import DataModelType
-from datamodel_code_generator.format import CodeFormatter, PythonVersion
+from datamodel_code_generator.format import PythonVersion
 from datamodel_code_generator.model import get_data_model_types
 from datamodel_code_generator.parser import LiteralType
 from datamodel_code_generator.parser.openapi import OpenAPIParser
 
+from datadog_checks.dev.tooling.configuration.consumers.model.code_formatter import format_with_ruff
 from datadog_checks.dev.tooling.configuration.consumers.model.model_file import build_model_file
 from datadog_checks.dev.tooling.configuration.consumers.model.model_info import ModelInfo
 from datadog_checks.dev.tooling.configuration.consumers.openapi_document import build_openapi_document
-from datadog_checks.dev.tooling.constants import get_root
 
 PYTHON_VERSION = PythonVersion.PY_39
 
@@ -32,9 +31,8 @@
 
 
 class ModelConsumer:
-    def __init__(self, spec: dict, code_formatter: CodeFormatter = None):
+    def __init__(self, spec: dict):
         self.spec = spec
-        self.code_formatter = code_formatter or self.create_code_formatter()
 
     def render(self) -> Dict[str, Dict[str, str]]:
         """
@@ -137,7 +135,6 @@ def _process_section(self, section) -> (List[Tuple[str, str]], dict, ModelInfo):
             model_id,
             section_name,
             model_info,
-            self.code_formatter,
         )
         # instance.py or shared.py
         model_files[model_file_name] = (model_file_contents, errors)
@@ -207,11 +204,6 @@ def _merge_instances(self, section: dict, errors: List[str]) -> dict:
 
         return new_section
 
-    @staticmethod
-    def create_code_formatter():
-        path = Path(get_root())
-        return CodeFormatter(PYTHON_VERSION, settings_path=path if path.is_dir() else None)
-
     def _build_deprecation_file(self, deprecation_data):
         file_needs_formatting = False
         deprecations_file_lines = []
@@ -226,7 +218,7 @@ def _build_deprecation_file(self, deprecation_data):
         deprecations_file_lines.append('')
         deprecations_file_contents = '\n'.join(deprecations_file_lines)
         if file_needs_formatting:
-            deprecations_file_contents = self.code_formatter.apply_black(deprecations_file_contents)
+            deprecations_file_contents = format_with_ruff(deprecations_file_contents)
         return deprecations_file_contents
 
     @staticmethod
@@ -255,5 +247,5 @@ def _build_defaults_file(self, model_info: ModelInfo):
         model_info.defaults_file_lines.append('')
         defaults_file_contents = '\n'.join(model_info.defaults_file_lines)
         if model_info.defaults_file_needs_value_normalization:
-            defaults_file_contents = self.code_formatter.apply_black(defaults_file_contents)
+            defaults_file_contents = format_with_ruff(defaults_file_contents)
         return defaults_file_contents

datadog_checks_dev/datadog_checks/dev/tooling/configuration/consumers/model/model_file.py

@@ -1,8 +1,7 @@
 # (C) Datadog, Inc. 2021-present
 # All rights reserved
 # Licensed under a 3-clause BSD style license (see LICENSE)
-from datamodel_code_generator.format import CodeFormatter
-
+from datadog_checks.dev.tooling.configuration.consumers.model.code_formatter import format_with_ruff
 from datadog_checks.dev.tooling.configuration.consumers.model.model_info import ModelInfo
 
 
@@ -11,14 +10,12 @@ def build_model_file(
     model_id: str,
     section_name: str,
     model_info: ModelInfo,
-    code_formatter: CodeFormatter,
 ):
     """
     :param parsed_document: OpenApi parsed document
     :param model_id: instance or shared
     :param section_name: init or instances
     :param model_info: Information to build the model file
-    :param code_formatter:
     """
     # Whether or not there are options with default values
     options_with_defaults = len(model_info.defaults_file_lines) > 0
@@ -54,7 +51,7 @@ def build_model_file(
     model_file_lines.append('')
     model_file_contents = '\n'.join(model_file_lines)
     if any(len(line) > 120 for line in model_file_lines):
-        model_file_contents = code_formatter.apply_black(model_file_contents)
+        model_file_contents = format_with_ruff(model_file_contents)
     return model_file_contents
 
 
@@ -109,27 +106,44 @@ def _add_imports(model_file_lines, need_defaults, need_deprecations):
 
 
 def _fix_types(model_file_lines):
-    for i, line in enumerate(model_file_lines):
-        line = model_file_lines[i] = line.replace('dict[', 'MappingProxyType[')
-        if 'list[' not in line:
-            continue
-
-        buffer = bytearray()
-        containers = []
-
-        for char in line:
-            if char == '[':
-                if buffer[-4:] == b'list':
-                    containers.append(True)
-                    buffer[-4:] = b'tuple'
-                else:
-                    containers.append(False)
-            elif char == ']' and containers.pop():
-                buffer.extend(b', ...')
-
-            buffer.append(ord(char))
-
-        model_file_lines[i] = buffer.decode('utf-8')
+    # Operate on the joined document (as UTF-8 bytes) so the bracket-tracking
+    # pass below works even when the upstream parser pre-wraps `list[...]`
+    # across multiple lines. Iterating bytes keeps the algorithm safe for
+    # non-ASCII content (descriptions, examples) since `[`, `]`, and `list`
+    # are all single-byte ASCII while UTF-8 continuation bytes never collide
+    # with them.
+    content = '\n'.join(model_file_lines).replace('dict[', 'MappingProxyType[')
+    if 'list[' not in content:
+        model_file_lines[:] = content.split('\n')
+        return
+
+    encoded = content.encode('utf-8')
+    buffer = bytearray()
+    containers = []
+    open_bracket = ord(b'[')
+    close_bracket = ord(b']')
+    whitespace = (ord(b' '), ord(b'\t'), ord(b'\n'))
+
+    for byte in encoded:
+        if byte == open_bracket:
+            if buffer[-4:] == b'list':
+                containers.append(True)
+                buffer[-4:] = b'tuple'
+            else:
+                containers.append(False)
+        elif byte == close_bracket and containers and containers.pop():
+            # Insert `, ...` after the last non-whitespace byte already in the
+            # buffer so the sentinel sits on the same line as the previous
+            # content (`tuple[X], ...` style) even when the parser wrapped the
+            # closing `]` onto its own line.
+            insert_at = len(buffer)
+            while insert_at > 0 and buffer[insert_at - 1] in whitespace:
+                insert_at -= 1
+            buffer[insert_at:insert_at] = b', ...'
+
+        buffer.append(byte)
+
+    model_file_lines[:] = buffer.decode('utf-8').split('\n')
 
 
 def _add_secure_fields_constant(model_file_lines, require_trusted_providers):

datadog_checks_dev/pyproject.toml

@@ -52,7 +52,6 @@ cli = [
     "aiomultiprocess",
     "atomicwrites",
     "beautifulsoup4==4.12.3",
-    "black==23.12.1",  # TODO Remove once https://github.com/koxudaxi/datamodel-code-generator/issues/1821 is fixed
     "build>=0.7.0",
     "click~=8.1.6",
     "codespell",
@@ -69,6 +68,13 @@ cli = [
     "platformdirs>=2.0.0a3",
     "pydantic>=2.0.2",
     "pysmi==1.6.2",
+    # ruff is invoked as a subprocess by the model generator to format
+    # auto-generated config_models files (see datadog_checks/dev/tooling/
+    # configuration/consumers/model/code_formatter.py). Keep this pin in sync
+    # with the version used by ddev's hatch lint env (currently `ruff==0.11.10`
+    # in ddev/src/ddev/plugin/external/hatch/environment_collector.py) until
+    # the commands here are migrated into ddev itself.
+    "ruff==0.11.10",
     "securesystemslib[crypto]==0.28.0",
     "semver>=2.13.0",
     "tabulate>=0.8.9",