Skip to content

chore: remove monitors test from pr workflow#695

Merged
ava-silver merged 3 commits intomainfrom
ava.silver/chore/remove-monitors-test-from-pr-workflow
Apr 14, 2026
Merged

chore: remove monitors test from pr workflow#695
ava-silver merged 3 commits intomainfrom
ava.silver/chore/remove-monitors-test-from-pr-workflow

Conversation

@ava-silver
Copy link
Copy Markdown
Contributor

@ava-silver ava-silver commented Apr 14, 2026

What does this PR do?

  1. Removes the "Test getting recommended monitors" step from the PR integration-tests workflow.
  2. Renames integration_tests/snapshot_tests/ (directory, script, workflow, and config references).

Motivation

Monitors test removal: This step hits api.datadoghq.com/api/v2/monitor/recommended and asserts that certain named recommended monitors exist. It tests nothing in this repo -- those monitors are configured server-side by Datadog's backend team, so no PR here can affect whether this test passes or fails.

The test was causing every Dependabot PR to fail on its first CI run because Dependabot PRs don't have access to regular repo secrets (DD_API_KEY/DD_APP_KEY), only the restricted Dependabot secret scope. The fix in #692 addressed a separate CloudFormation credentials issue but missed this one.

The test already has a dedicated home in monitor_api_integration_test.yml, which runs on a daily cron with a Slack alert on failure -- the right place for a canary that monitors external API availability.

Rename: The tests in integration_tests/ run sls package, generate a CloudFormation template, and diff it against checked-in snapshots. They never deploy anything and require no real AWS connectivity. Calling them "integration tests" implies live AWS interaction is expected -- which is exactly why the credential failure looked like a bug rather than an obvious non-issue. Renaming to snapshot_tests makes the nature of the tests self-evident.

Note: integrationTesting / testingMode config flags in src/env.ts, src/forwarder.ts, src/index.ts are a different concept (bypass forwarder ARN validation) and are left untouched.

Testing Guidelines

CI should pass on Dependabot PRs without any manual intervention.

Additional Notes

Fixes the remaining flakiness on Dependabot PRs not addressed by #692.

Types of changes

  • Bug fix
  • New feature
  • Breaking change
  • Misc (docs, refactoring, dependency upgrade, etc.)

Check all that apply

  • This PR's description is comprehensive
  • This PR contains breaking changes that are documented in the description
  • This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
  • This PR impacts documentation, and it has been updated (or a ticket has been logged)
  • This PR's changes are covered by the automated tests
  • This PR collects user input/sensitive content into Datadog

Copy link
Copy Markdown
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Apr 14, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.69%. Comparing base (feb68d8) to head (da26cb9).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #695   +/-   ##
=======================================
  Coverage   77.69%   77.69%           
=======================================
  Files          12       12           
  Lines        1112     1112           
  Branches      350      350           
=======================================
  Hits          864      864           
  Misses        118      118           
  Partials      130      130           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@ava-silver ava-silver marked this pull request as ready for review April 14, 2026 19:13
@ava-silver ava-silver requested a review from a team as a code owner April 14, 2026 19:13
@ava-silver ava-silver requested a review from avangelillo April 14, 2026 19:13
env:
DD_API_KEY: ${{ secrets.DD_API_KEY }}
DD_APP_KEY: ${{ secrets.DD_APP_KEY }}
run: npm run test:integration
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll be honest, I never really looked that closely into what these tests were doing but I assumed that it was deploying to AWS then pulling the deployed template down and asserting that it is both deployable and matches the snapshot. [In my opinion] just doing a snapshot test like this isn't really all that useful, and trying to add a deploy step here is worth doing if we have the credentials available.

It tests nothing in this repo -- those monitors are configured server-side by Datadog's backend team, so no PR here can affect whether this test passes or fails.
...
The test already has a dedicated home in monitor_api_integration_test.yml, which runs on a daily cron with a Slack alert on failure -- the right place for a canary that monitors external API availability.

I think I've added a monitor to this list before, but if we don't really own it I'd go even further and say that we shouldn't be the ones trying to test it.

Since the plugin does get the recommended monitors from an API, then creates / updates / deletes them based on the response, I think the including that in (actual) integration tests that run as a part of a PR is still the appropriate place to run these tests, if a PR breaks that we should not merge it in and we're missing that validation. Maybe not by specific names like how the test works now, but that something was created.

We also need a Datadog API / app key for the remote instrumentation tests, and they are stored in Secrets Manager in the sandbox account and retrieved during the integration tests, I would recommend doing something similar to that if there is a problem storing the keys in GitHub.

@ava-silver ava-silver merged commit 30e3e45 into main Apr 14, 2026
7 checks passed
@ava-silver ava-silver deleted the ava.silver/chore/remove-monitors-test-from-pr-workflow branch April 14, 2026 21:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants