Event streaming for faster entity collection

[steveny91]

What does this PR do?

Adds opt-in event streaming to the generic-resources collector: Applications are watched via /api/v1/stream/applications for near-real-time updates, with a periodic full scrape as a TTL-refresh and backfill safety net. Collection cadences are now per resource type (applications, clusters, repositories) and fully configurable, with a single 30-minute TTL.

A new collect_openmetrics toggle lets an instance run generic-resource collection without scraping metrics. Also hardens the stream listener dedicated HTTP session, sticky stream.up gauge, non-blocking cancel(), and resilient reconnect/backoff. Streaming defaults on, gated behind collect_genresources.

Generic resources — enable & connection

┌─────────────────────────┬─────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│         Option          │ Default │                                                  What it does                                                   │
├─────────────────────────┼─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ collect_genresources    │ false   │ Enable the collector that ships Applications/Clusters/Repositories as generic resources.                        │
├─────────────────────────┼─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ genresources_endpoint   │ —       │ Base URL of the Argo CD REST API (e.g. https://argocd.example.com). Required when collect_genresources is true. │
├─────────────────────────┼─────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ genresources_auth_token │ —       │ Raw bearer token for the REST API (secret). Unset → inherits the instance's HTTP auth.                          │
└─────────────────────────┴─────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Generic resources — cadence & TTL

┌───────────────────────────────────────────────────────┬─────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│                        Option                         │ Default │                                               What it does                                               │
├───────────────────────────────────────────────────────┼─────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ genresources_stream_applications_enabled              │ true    │ Stream Application changes in near-real-time; when false, poll apps instead. Only active when            │
│                                                       │         │ collect_genresources is true.                                                                            │
├───────────────────────────────────────────────────────┼─────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ genresources_application_poll_interval_seconds        │ 120     │ Dedup poll of Applications — streaming-off only (ignored when streaming).                                │
├───────────────────────────────────────────────────────┼─────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ genresources_application_full_scrape_interval_seconds │ 600     │ Full Application re-scrape (re-submits all → refreshes TTL + backfills), both modes.                     │
├───────────────────────────────────────────────────────┼─────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ genresources_cluster_scrape_interval_seconds          │ 300     │ Full Cluster scrape (Clusters are always polled, never streamed).                                        │
├───────────────────────────────────────────────────────┼─────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ genresources_repository_scrape_interval_seconds       │ 300     │ Full Repository scrape (always polled).                                                                  │
├───────────────────────────────────────────────────────┼─────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ genresources_ttl_seconds                              │ 1800    │ TTL per emitted resource; also bounds how long a deleted resource lingers. Must be ≥ the longest scrape  │
│                                                       │         │ interval.                                                                                                │
├───────────────────────────────────────────────────────┼─────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ genresources_stream_backoff_max_seconds               │ 30      │ Max stream reconnect backoff; starts at 1s, doubles, resets after a connection that delivered data.      │
└───────────────────────────────────────────────────────┴─────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────┘

[datadog-datadog-prod-us1-2]

   

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 2 Pipeline jobs failed

PR | test / check     

PR | test / test (linux, ubuntu-22.04, argocd, ArgoCD (py3.13-2.4.7), py3.13-2.4.7) / ArgoCD (py3.13-2.4.7)-py3.13-2.4.7     

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 96.34%
• Overall Coverage: 93.06% (+5.04%)

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 231e871 | Docs | Datadog PR Page | Give us feedback!}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3b39503199

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Don't shadow inherited auth headers for genresources requests

When genresources_auth_token is omitted, the documented fallback is to inherit the instance's HTTP auth (for example headers, extra_headers, or the structured auth_token writer), but this helper returns {} and the new callers pass it as the per-request headers value. RequestsWrapper gives per-request options precedence over its configured default headers, so these genresources REST calls (and the stream listener, which uses the same helper) go out without the inherited Authorization/header config and fail for users relying on that fallback; only pass headers when a dedicated genresources token is set, or merge with the wrapper defaults.

Useful? React with 👍 / 👎.

[dd-octo-sts]

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and code coverage settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run