[pull] master from DataDog:master#628
Merged
Merged
Conversation
…tegration (#24236) * Add account configuration JSON for Palo Alto Networks Cortex XSOAR integration * Fix account config schema error by removing unsupported pattern field The account config schema does not support a `pattern` field on text fields, which caused a schema parse error. Removing it also resolves the review concern that the regex rejected valid Cortex XSOAR tenant FQDNs. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
* kafka_consumer: bound and refine estimated_consumer_lag Cap left-extrapolation of the broker timestamp cache so a consumer offset older than the oldest cached sample cannot extrapolate more than 10 minutes past it, keeping estimated_consumer_lag bounded. Use max(consumer_offset, low_watermark) as the offset basis for lag-in-time when cluster monitoring is enabled: messages below the low watermark are out of retention and unreachable, so they should not inflate the time lag. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * kafka_consumer: add changelog entry for PR #24167 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * kafka_consumer: compact and prune the broker-timestamp cache Replace single-oldest eviction with batch compaction (Visvalingam-Whyatt) triggered when the cache reaches capacity: keep the oldest and newest samples and drop the points that least distort the offset/timestamp curve, so the cache spans a longer history at a coarsening resolution and high lag is interpolated rather than extrapolated. At the same trigger, prune samples below the earliest consumer offset (keeping one anchor) since no consumer will ever interpolate there. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * kafka_consumer: prune broker-timestamp cache by low watermark Use the partition low watermark as the prune floor when cluster monitoring is enabled (the physically meaningful "lowest readable offset"), falling back to the earliest committed consumer offset otherwise. The low watermark is now fetched before the cache update and reused for both pruning and the lag-in-time floor, so there is no extra broker call. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * kafka_consumer: fetch low watermark offsets once and share them Previously the log-start (low watermark) offsets were fetched twice per run when cluster monitoring and data streams were both enabled: once by the metadata collector for partition.size/topic.size/throughput, and again by the lag path for the lag-in-time and cache-pruning floor. Fetch them once in check(), gated on cluster monitoring, over all non-internal topic partitions, and share the result with both the data-streams lag path and the metadata collector. Removes the duplicate list_offsets(earliest) call and the divergent internal-topic handling. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * kafka_consumer: reuse _fetch_earliest_offsets instead of a parallel fetch Drop the PR-added Client.get_low_watermark_offsets and the _get_low_watermark_offsets wrapper, which duplicated the existing ClusterMetadataCollector._fetch_earliest_offsets. The check now calls _fetch_earliest_offsets once under cluster monitoring and shares the result with both the data-streams lag/pruning path and the topic-metadata collection, so the earliest offsets are still fetched only once per run. This reverts client.py to master and keeps the cluster_metadata.py change to a small signature tweak. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * kafka_consumer: use low_watermark_offsets directly in topic metadata Drop the redundant earliest_offsets alias and reference the passed-in low_watermark_offsets directly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * kafka_consumer: address review feedback on lag bounding - Clarify that the left-extrapolation cap bounds lag-in-time regardless of cluster monitoring or the low-watermark floor, and document why there is no symmetric right-side clamp (the newest cached sample is the just-collected highwater, which the consumer offset can never exceed). - Promote ClusterMetadataCollector.fetch_earliest_offsets to a public method since KafkaCheck now calls it across the class boundary. - Log a debug line when the cache-prune floor falls back from the low watermark to the earliest consumer offset. - Extract the Visvalingam-Whyatt significance closure into a module-level _interpolation_error helper. - Parameterize the _visvalingam_whyatt tests; add direct tests for _earliest_consumer_offsets, _prune_below_anchor, and the left-extrapolation cap through report_consumer_offsets_and_lag without a low watermark. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * kafka_consumer: trim comments to a single note on the extrapolation cap Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * kafka_consumer: move extrapolation-cap comment to the clamp line Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * kafka_consumer: reuse fetched topic partitions in topic metadata collection Pass the topic-partition map computed in check() through collect_all_metadata into _collect_topic_metadata instead of fetching it again, so the cluster monitoring path makes the same number of get_topic_partitions calls as before. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * kafka_consumer: satisfy ruff formatting for collect_all_metadata call Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * kafka_consumer: clear full timestamp cache on reset, test pruning end-to-end When a reset is detected (any cached offset above the new highwater), clear the entire cache instead of only dropping entries above the highwater. The VW compactor always preserves the minimum cached offset as an endpoint, so old-generation low-offset entries would never age out and would poison lag interpolation indefinitely after a partial reset. Also replaces the direct private-method test for consumer-floor pruning with a dd_run_check test that exercises the full check() path, and adds tests for the new clear-on-reset behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * kafka_consumer: satisfy ruff formatting for new unit tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * kafka_consumer: shorten reset-detection comment Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * kafka_consumer: trim reset test comment Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * kafka_consumer: test timestamp compaction via dd_run_check instead of private method Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * kafka_consumer: replace _prune_below_anchor direct tests with dd_run_check tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * kafka_consumer: satisfy ruff formatting for prune_below_anchor replacement tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * kafka_consumer: replace private method tests with public method tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * kafka_consumer: test that lag accuracy is preserved after VW compaction Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * kafka_consumer: parametrize VW compaction test with 4 cases Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Update integration for SCFW v3.x * Respond to review feedback * Fix broken log test * More review feedback * Fix breaking test
* Implement agent_hostname in DatabaseCheck * Add changelog
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )