[pull] master from DataDog:master#624
Merged
Merged
Conversation
#24126) * Remove __discovery_provides__ attribute and dead-code tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add discovery config spec validation and model generation to ddev. Adds a discovery section to the spec schema (port_hints, strategy field), extends the example and model consumers to render/generate discovery config, adds spec.py validation for discovery spec correctness, and covers the new paths with tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * add annotations import * Add krakend discovery config support with e2e discovery test helpers. Adds krakend config_models/discovery.py and auto_conf.yaml for port-based config autodiscovery. Adds get_e2e_discovery_metadata() to ddev docker helpers and a dd_agent_check_discovery pytest fixture so integrations can run discovery-path E2E tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * regen * add test to force probe of all ports/configs to check no side effects on server To ensure that this no risk with probing the ports of the container for configuration discovery, add a test which instanciates the integration which each of the generated configurations in turn and checks that the container logs don't have any obvious crashes. https://datadoghq.atlassian.net/wiki/spaces/DSCVR/pages/6671862234/Configuration+Discovery+for+Agent+Integrations#Probing-causes-problems While the logic for detecting "bad" effects is best-effort, the test also the logs from this scenario to be seen easier during development. For example, for krakend: ``` $ ddev env test --base --new-env krakend py3.13-2.10 -- -k test_e2e_discovery_candidates_do_not_destabilize_container -rP --log-level=DEBUG ... _________ test_e2e_discovery_candidates_do_not_destabilize_container __________ ------------------------------ Captured log call ------------------------------- DEBUG root:docker.py:90 Probing candidate #1: {'init_config': {}, 'instances': [{'openmetrics_endpoint': 'http://172.17.129.3:9090/metrics'}]} DEBUG root:docker.py:90 Probing candidate #2: {'init_config': {}, 'instances': [{'openmetrics_endpoint': 'http://172.17.129.3:8080/metrics'}]} DEBUG root:docker.py:94 Error probing candidate #2: {'init_config': {}, 'instances': [{'openmetrics_endpoint': 'http://172.17.129.3:8080/metrics'}]} DEBUG root:docker.py:103 New log line: [GIN] 2026/06/17 - 15:31:46 | 404 | 2.803µs | 172.17.129.1 | GET "/metrics" DEBUG root:docker.py:103 New log line: [GIN] 2026/06/17 - 15:31:47 | 404 | 2.753µs | 172.17.129.1 | GET "/metrics" DEBUG root:docker.py:90 Probing candidate #3: {'init_config': {}, 'instances': [{'openmetrics_endpoint': 'http://172.17.129.3:8090/metrics'}]} DEBUG root:docker.py:94 Error probing candidate #3: {'init_config': {}, 'instances': [{'openmetrics_endpoint': 'http://172.17.129.3:8090/metrics'}]} DEBUG root:docker.py:103 New log line: [GIN] 2026/06/17 - 15:31:49 | 200 | 51.868µs | ::1 | GET "/__health" ``` * Spec-driven discovery redesign: tooling + krakend PoC Move from_ports from a runtime function in datadog_checks_base to a codegen strategy in datadog_checks_dev, and introduce a registry-driven model consumer that generates candidates() using Pydantic models. Key changes: - New discovery registry package with @strategy decorator and REGISTRY - core_strategies.py registers from_ports as a codegen strategy - _build_discovery_file is now registry-driven (no hardcoded from_ports) - Generated candidates() use InstanceConfig/SharedConfig model_dump (D1) - Remove ad_identifiers from spec discovery stanza (D2: hand-maintained) - Remove auto_conf.yaml generation from example consumer (D2) - Add discovery_strategies.py and discovery_overrides.py to CUSTOM_FILES - Regenerate krakend discovery.py with model-backed candidate output - Add hand-maintained krakend/auto_conf.yaml with ad_identifiers Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Use model_validate with context for discovery candidate generation Direct InstanceConfig/SharedConfig construction fails because the field validators access info.context['configured_fields'], which is None without explicit context. Switch to model_validate with the required context dict so that all defaults from defaults.py are applied correctly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * T3/T4: extract expand_template_items, restructure discovery validation Extract expand_template_items from the template expansion loop in options_validator so it can be shared with discovery strategy lists. Add handle_discovery which runs after options_validator so discovery validation sees resolved options. Discovery now supports enabled:false kill-switch, local: strategy names, and candidate field cross-check against resolved instance options. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Refactor options_validator to call expand_template_items Remove the inline template expansion loop from options_validator and replace it with a call to expand_template_items, which was extracted in T3 precisely so both callers share one implementation. The only behavioral difference from the original was a per-item overrides dict; aligning expand_template_items to use shared overrides (accumulated across all items) preserves the existing semantics. hide_template was redundant: template.update(option) already carries any hidden:true from intermediate templates into the expanded item, so setdefault('hidden', False) produces the same result in all cases. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Make spec-driven discovery tooling fully registry-driven Align the discovery tooling with the implementation plan: a strategy now owns both its contract (provides/inputs) and its codegen (emit_context) in the dev-side registry, and the generator collects each strategy's runtime imports and emits the model-backed candidate body. No strategy name or input is special-cased in the generator or validator. - registry: add Input and Strategy(provides, inputs, runtime_imports, emit_context); add guarded SERVICE_FIELDS/PORT_FIELDS constants - core_strategies: from_ports emits only the candidate loop and ctx - model_consumer: registry-driven generator that builds candidates through the config models, supports local: strategies, and emits the discovery_strategies.py/discovery_overrides.py first-render stubs - spec: derive candidate placeholders from provides + the field constants, validate strategy inputs generically, honour enabled:false as a kill switch, resolve discovery-level templates - base: add a guard test asserting the dev field constants match the Service/Port pydantic models so the hand-copied fact cannot drift - fix the datadog_checks_dev discovery changelog entry PR number Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * krakend: regenerate discovery models and drop stray auto_conf.yaml - add the regenerated discovery_overrides.py custom-file stub - remove the misplaced package-root auto_conf.yaml; the Agent-facing opt-in file lives in data/auto_conf.yaml - point the discovery changelog entry at the correct PR number Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix discovery strategy validation * Wire discovery_overrides.py seam into generated discovery.py Generated discovery.py now delegates to the discovery_overrides.py custom file via the same pattern validators.py already uses: import the always- co-generated module and resolve candidates() through getattr with a fallback to the generated generator. Behaviorally identical while the stub is empty; live once an integration fills in candidates(service, default), with no further tooling PR. Regenerated krakend's discovery.py accordingly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * krakend: regenerate discovery stub docs from current constants The discovery_strategies.py / discovery_overrides.py stubs are write-once CUSTOM_FILES, so they kept the wording from an earlier render and never re-synced with the updated documentation constants. Delete and regenerate them so the in-tree stubs match the current model_consumer constants. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * krakend: match changelog text to phase 2 spec Use the wording prescribed by the phase 2 plan (T9): "Add configuration discovery support." Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Fix template-expansion regressions in discovery config tooling Two issues surfaced by review of the discovery config tooling refactor: - expand_template_items dropped wrapper attributes (e.g. `hidden: true`) when a template resolved to a list, so the common `- template: instances/pdh_legacy` + `hidden: true` pattern used by aspdotnet, dotnetclr, hyperv, exchange_server, and active_directory stopped hiding the expanded options. Propagate the wrapper's leftover attributes to each expanded item via setdefault, restoring the pre-refactor behaviour. - The discovery model consumer emitted candidate templates inside a single-quoted literal, producing invalid discovery.py when a template contained a single quote or backslash. Use repr() to emit a properly escaped literal. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Add test for hidden propagation on list-template expansion Locks in the fix: a `hidden: true` wrapper on a list template applies to every expanded item, while an item's own explicit `hidden` value wins. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Replace chr(10) workaround with a plain newline join chr(10) was the pre-3.12 workaround for using a backslash inside an f-string expression. Build the joined message first, matching the existing style in expand_template_items. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Restore master template-expansion behavior in spec tooling The discovery-config refactor changed two aspects of option-template expansion in `expand_template_items`, both of which broke config/models validation for existing integrations: - Overrides were scoped per-item, so an override targeting a nested template (e.g. `extra_metrics.value.example` in the Windows perf-counter specs) was attempted prematurely on the parent template, reported as an error, and discarded before the nested template was spliced in and expanded. Restore the shared-override semantics: `apply_overrides` pops each override on success, so unresolved ones are retried against later items, and only truly-unused overrides are reported. - `hidden` propagation was scoped to a template's own expanded items, dropping the historical cross-sibling leak where a template wrapper's `hidden: true` carries onto following plain options until the next template. haproxy relies on this to hide its legacy (non-OpenMetrics) option block. Reintroduce it behind a `propagate_hidden` flag (on for options, off for discovery strategies, which have no `hidden` concept). Rendered example/model output is now byte-identical to master across all 263 integration specs. Adds a regression test for the nested-template override case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * remove test_discovery_stanza_does_not_emit_auto_conf * Stop reserving auto_conf.yaml name in discovery handling Discovery no longer generates auto_conf.yaml (it is hand-maintained per integration), so reserving the example name produced a false-positive collision for the legitimate "discovery stanza + hand-maintained auto_conf.yaml" combination. Drop the reservation, its plumbing through handle_discovery, and the test that pinned the false positive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Drop redundant hidden setdefault in options_validator `expand_template_items(propagate_hidden=True)` now sets `hidden` on every resolved option during expansion, so the `option.setdefault('hidden', False)` in options_validator was a dead no-op that misleadingly read as the source of the default. Remove it; rendered example/model output is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Add discovery/openmetrics_from_ports and auto_conf/discovery spec templates Adds two reusable spec tooling templates: - discovery/openmetrics_from_ports: a from_ports strategy candidate that generates an openmetrics_endpoint URL from the discovered host and port. Integrations override port_hints to set the expected port. - auto_conf/discovery: the common options block for fleet-discovery auto_conf.yaml files (discovery: {}, init_config, instances: []). Each integration sets ad_identifiers independently and includes this template for the rest. Converts the krakend spec to use both templates and regenerates the krakend auto_conf.yaml and config_models/discovery.py from the spec. Also fixes ModelConsumer._process_section to return early for leaf options (no sub-options key) so that fleet-discovery auto_conf.yaml files, which use leaf instances: [] rather than a full instances section, do not crash validate models. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * modify auto_config * Narrow model consumer leaf-instances guard to instances section only The previous fix skipped any section without an `options` key before even checking whether it was init_config or instances. Tighten it to only apply inside the `instances` branch, where auto_conf.yaml files using the fleet-discovery pattern emit a leaf `instances: []` with no model schema. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Restrict leaf-instances guard to auto_conf.yaml files only Pass the spec file name into _process_section and apply the no-options early-return only when processing auto_conf.yaml, where a leaf instances: [] is valid for fleet discovery. Any other file with a leaf instances section still hits the normal code path and fails visibly. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Use by_alias=True when dumping discovery candidate configs Without this, options with hyphenated names (e.g. slowlog-max-len) are emitted as Python field names (slowlog_max_len) in the generated candidate dict. The Agent passes this dict verbatim as the raw instance config, so checks that look up the original hyphenated key would not find it. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Skip SharedConfig import/usage when spec has no init_config section When a spec declares discovery but omits init_config, no shared.py is generated. The discovery file unconditionally imported SharedConfig, causing ModuleNotFoundError at import time and silently preventing any candidates from being produced. Pass has_shared=False to _build_discovery_file in that case; the generated file emits `shared = {}` instead and omits the import. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * regen --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
…nch (#24197) * Fix license header validation to compare against the correct base branch When a file is present in a release branch but deleted from master, the validator was treating it as a new file and enforcing the current-year copyright template, causing false failures in PRs targeting that release branch. The fix detects the appropriate base ref (GITHUB_BASE_REF in CI, the current branch name when it is master or a release branch, or the closest ancestor via git merge-base otherwise) so the comparison is always made against the actual parent branch. * Add changelog entry * Add real-git integration tests for base ref detection * Resolve license-header base ref once per run instead of per file validate_license_headers resolved the base ref for every scanned file, and the CLI invokes it once per check, so a local full run triggered a git for-each-ref + per-branch merge-base/show storm O(files x checks) times. A shared build_get_previous() resolver now resolves the base ref lazily at most once and is reused across every check. * Add type annotations to the license-header methods touched here * Modernize validate_license_headers hints and share the restore_root fixture - Use `X | None`/`list[...]` on the edited validate_license_headers signature, matching the new helpers; trim unused typing imports. - Move the restore_root fixture into tests/tooling/conftest.py so test_git and test_license_headers share it. - Clarify build_get_previous's docstring: the returned instance caches the base ref; callers should share one to amortize across checks. * Use restore_root in the mocked base-ref tests and cover all-merge-base-fail - Have the three mocked _find_closest_base_ref tests take the restore_root fixture so set_root('/foo/') no longer leaks into the session; drop the unused monkeypatch param. - Add a test for the branch where for-each-ref succeeds but every merge-base fails, which must fall back to origin/master. * Simplify base-ref test mock lookup and drop a redundant comment - Pre-build a sha-to-timestamp dict in the _find_closest_base_ref mock helper for an O(1) lookup. - Remove the inline comment in the license-headers command; the factory and its docstring already convey the intent.
* Sunset tokumx integration * Remove all other references to tokumx in the codebase * Add changelog entry
* Bump datadog-checks-dev pin to >=39.0,<41 in ddev * Add changelog entry for #24110
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )