[pull] master from DataDog:master#567
Merged
Merged
Conversation
…23850) * Wait for the gamesim_primary index to be online before running tests test_index_stats_metrics asserts metrics tagged with index_name:gamesim_primary, which the Couchbase indexer only reports once the bundled GSI index for gamesim-sample finishes building. Add a WaitFor against /api/v1/stats on port 9102 (Couchbase 7+) so the environment fixture blocks until the indexer is actually reporting that keyspace. Also fix the inverted exit condition in load_sample_bucket(): the loop was breaking the moment the task appeared in /pools/default/tasks instead of when it disappeared, so it returned while the sample load was still running. * Confirm gamesim_primary build completes before yielding environment Three follow-ups from review: - Check `initial_build_progress == 100` on the keyspace; the indexer also publishes stats while an index is still building. - Use `raise_for_status()` so 401/404/5xx surface immediately instead of collapsing into a 60s WaitFor timeout. - Drop the >=7 conditional around the new WaitFor; the test matrix only covers 7.x and the test that needs the index is itself gated on 7+. - Bail out of load_sample_bucket if the install response carries no matching task entry, instead of spinning on `None == None`. * Require every gamesim_primary keyspace to report build progress 100 * Condense gamesim_primary_index_ready docstring to a one-liner * Clarify gamesim_primary_index_ready and log keyspaces on the negative path - Refactor to a list comprehension that separates 'no keyspace yet' from 'still building' so each retry path is self-documenting. - Print the keyspaces seen on the negative path so a future WaitFor timeout doesn't leave a debugging dead end. - Document why load_sample_bucket bails out when the install response carries no matching task — the next gamesim_primary_index_ready WaitFor is the real readiness gate.
* Honor source ref in release dispatch Keep release workflow tooling checked out separately from the source tree being released. This lets the workflow use current setup actions and release scripts while ddev tags and validates the requested source-repo-ref. Add source-repo-branch to manual release dispatches so stable versus pre-release validation follows the branch that contains source-repo-ref instead of the branch used to launch the workflow. * align source-repo-branch description with release-trigger.yml
* [LOGSP-29] Add bypass annotations for linter checks A linter rule is being added or changed and some integrations are expected to remain non-compliant. Add permanent bypass annotations so CI stays green and does not fail for new PRs. * fix(logs-linter): bypass date-remapper-parse-failure-checks for known false positives These integrations either use log samples that intentionally lack timestamp source fields, or rely on date formats the test harness cannot parse. The date-remapper is functioning correctly in production for all affected integrations. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> * Trigger CI --------- Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* WIP * Change orchestrator message * Add changelog and fix tests * Adjust Datadog code coverage gates * Sync Datadog code coverage services * Add integrations that don't pass threshold * Validate stale Datadog coverage services Detect services present in the Datadog Code Coverage config but no longer expected by ddev validate ci, and remove them when running with --sync. * Lower cilium threshold * Validate code coverage gates * Validate duplicate coverage services * Remove tests from coverage config * Exclude tests from coverage services * Lower coverage gates for migration exceptions
* Store ClickHouse advanced-queries metric definitions as JSON The four advanced_queries Python modules shipped ~250 KB of redundant dict literals (per-entry 'name' was always '<prefix>.<key>', and every entry repeated its type). Move that data to compact per-system-table JSON files under datadog_checks/clickhouse/data/ and build the QueryManager-shaped dicts at runtime. The check registers a check_initializations callable so the JSON files are parsed once on the first check run. Module attributes like advanced_queries.SystemMetrics remain available through __getattr__ backed by the same cache, so tests that read those attributes directly keep working. The metric generator emits the new JSON format directly; the three system_*.tpl templates and the four old Python modules are removed. * Add changelog entry * Simplify advanced_queries loader and tighten the generator - Replace the initializer(check) factory with a plain warm_cache callable; the closure never depended on the check instance. - Restore __all__ on the package so the public surface is explicit. - Rename _cache to cache per the AGENTS.md module-name rule. - Mirror Python's default AttributeError format in __getattr__. - Tighten return-type annotations on load, _build_items, and __getattr__. - Raise loudly in generate_queries when a metric type appears with mixed scaled/unscaled entries instead of silently producing a wrong container. - Reclass the changelog from .changed to .fixed: the refactor preserves runtime behaviour byte-for-byte, so a patch bump is the right semver. * Tighten the advanced_queries loader - Guard warm_cache with an explicit "key not in cache" check so the JSON files are read at most once per process, even when __getattr__ populated the cache first. - Wrap load() exceptions as RuntimeError so a missing or malformed data file produces an actionable message in the check-init failure. - Discriminate the two JSON shapes on positive presence of "columns" rather than absence of "items". - Reclass the changelog as .changed: this is a significant internal refactor, not a bug fix, so .fixed was misleading. * Drop system_errors from the generator and widen load() error wrap - Wrap KeyError in load()'s error path so a malformed JSON file (one that parses but is missing expected keys) raises the same context-rich RuntimeError as a missing or invalid file. - Remove SYSTEM_ERRORS_SPEC and generate_system_errors() from the generator. The system_errors data is static; the committed data/system_errors.json is the single source of truth, and the other three queries continue to be generator-driven from ClickHouse source. * Tighten the advanced_queries loader and rename the generator's Template - Comment the load() branch that handles system_errors so the asymmetry is named at the call site instead of requiring readers to audit the JSON files. - Widen load()'s except tuple with TypeError and AttributeError so a malformed JSON shape (e.g. items shipped as a list) still raises the wrapped RuntimeError with the file name rather than leaking the bare underlying exception. - Narrow _build_items's compact parameter type to dict[str, list[str] | dict[str, str]] to mirror the producer annotation in generate_metrics.py. - Rename the generator's Template dataclass to FileTemplate so it doesn't silently shadow string.Template if the stdlib import is ever reintroduced. * Rename module cache to _cache and drop the one-member Templates enum - Rename advanced_queries.cache to _cache so the underscore signals it as module-internal mutable state (PEP 8 module-private). The module's __all__ already advertises only the four System* names. - Rephrase the load() inline comment around the columns shortcut so it names the discriminating shape rather than the system_errors file, which would mislead if a second verbatim file were added later. - Replace the single-member Templates enum in the generator with a TESTS_METRICS_TEMPLATE constant; the enum was a vestige from when it held the three QUERY_* templates that now live in QUERY_SPECS. * Test the advanced_queries loader directly Adds tests/test_advanced_queries.py covering the new loader logic: - module-level __getattr__ resolution + caching - compact format: source/match column shape, sorted items, name derivation including the dotted-key edge case (jemalloc.epoch) - verbatim format: system_errors columns pass through with the boolean: true tag preserved - RuntimeError wrap on every malformed-JSON path the load() except tuple is meant to cover (missing file, invalid JSON, items as list, items as scalar, missing required keys) with the cause chain preserved - warm_cache populates every known name and is idempotent * Move changelog to fixed * Scope the advanced_queries loader to bulk match queries The advanced_queries package is now organised around one named pattern: the SQL-returns-(value, metric_name)-and-dispatches-via-lookup-table shape that SystemEvents, SystemMetrics, and SystemAsynchronousMetrics all share. The compact JSON files exist specifically to compress that pattern. - Rename load() to load_match_query(); _build_items() to _expand_match_items(); NAMES to MATCH_QUERIES; _cache to _match_query_cache. Names now say what the loader does. - Inline SystemErrors as a plain Python literal in __init__.py. Its shape doesn't fit the bulk-match pattern, so the JSON compression has nothing to compress; data/system_errors.json is removed and the verbatim-columns branch in load() goes away with it. - Add a top-level docstring that describes the JSON schema, names the generator that produces the files, and points operators at the hatch run metrics:generate command. - Add clickhouse/AGENTS.md (with a CLAUDE.md @AGENTS.md indirection) giving anyone opening this directory a short orientation note plus the "don't hand-edit the JSON files" warning that JSON has no comment syntax to carry. - Update tests/test_advanced_queries.py for the rename and add coverage that SystemErrors stays out of the match-query cache. Runtime dict shape and metric names are byte-identical to before; verified by diffing the four module-attribute dumps against the pre-refactor master.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )