Skip to content

feat(ci): run integration tests on Dependabot PRs#149

Open
zfarrell wants to merge 2 commits into
mainfrom
feat/dependabot-integration-tests
Open

feat(ci): run integration tests on Dependabot PRs#149
zfarrell wants to merge 2 commits into
mainfrom
feat/dependabot-integration-tests

Conversation

@zfarrell

@zfarrell zfarrell commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Dependabot's sandboxed pull_request runs get no secrets/vars, so the normal Integration Tests workflow skips every credentialed scenario — behavioral regressions in a bump slip through green. This adds a pull_request_target workflow that runs the real suite in the base-repo context where credentials exist.

Hardened: gated to dependabot[bot] (forks can't trigger it), contents: read only (no writable token exposed to the build), and no shared cargo cache.

Comment on lines +19 to +20
pull_request_target:
types: [opened, synchronize, reopened]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using unsafe GitHub Actions trigger may allow privilege escalation via CI/CD - critical severity
Using pull_request_target or workflow_run as a trigger is not recommended, as it may allow an attacker to elevate its privileges via the CI/CD pipeline by exfiltrating secrets (e.g. by reading out the caches of the GitHub Actions pipeline or listing loaded secrets in the environment). If the affected repository is open source, the attacker doesn't have to be an insider but could be any GitHub user.

Show fix

Remediation: Avoid using pull_request_target or workflow_run as a trigger.

Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AikidoSec ignore: Deliberate, mitigated pull_request_target use. The credentialed job is gated to dependabot[bot] (authenticated author, unspoofable; workflow always runs from main so the gate can't be removed via a PR), so external users can't reach it. The cited exfil vectors are closed: no cache is used, and permissions: contents: read means no writable token. Residual risk (insider pushing to a dependabot branch; the bumped crate running with a test-scoped key) is inherent to live dependency coverage and accepted.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Based on your feedback, we ignored this issue because of the following reason:

Deliberate, mitigated pull_request_target use. The credentialed job is gated to dependabot[bot] (authenticated author, unspoofable; workflow always runs from main so the gate can't be removed via a PR), so external users can't reach it. The cited exfil vectors are closed: no cache is used, and permissions: contents: read means no writable token. Residual risk (insider pushing to a dependabot branch; the bumped crate running with a test-scoped key) is inherent to live dependency coverage and accepted.

@sentry

sentry Bot commented Jun 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Comment on lines +42 to +46
env:
HOTDATA_SDK_TEST_API_URL: ${{ vars.HOTDATA_SDK_TEST_API_URL }}
HOTDATA_SDK_TEST_API_KEY: ${{ secrets.HOTDATA_SDK_TEST_API_KEY }}
HOTDATA_SDK_TEST_WORKSPACE_ID: ${{ vars.HOTDATA_SDK_TEST_WORKSPACE_ID }}
HOTDATA_SDK_TEST_CONNECTION_ID: ${{ vars.HOTDATA_SDK_TEST_CONNECTION_ID }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: This workflow will almost certainly run green while testing nothing.

The premise in the header comment — "pull_request_target executes in the base-repo context where the test credentials are available" — does not hold for Dependabot-authored events. GitHub treats any run where github.actor == 'dependabot[bot]' as untrusted regardless of trigger, including pull_request_target: the run gets a read-only GITHUB_TOKEN and no access to the Actions secrets/variables store. Dependabot-triggered runs can only read the separate Dependabot secrets/variables store (Settings → Secrets and variables → Dependabot).

So unless HOTDATA_SDK_TEST_API_KEY / HOTDATA_SDK_TEST_WORKSPACE_ID (and the URL/connection vars) are also configured as Dependabot secrets/variables, all four ${{ secrets.* }} / ${{ vars.* }} here resolve to empty strings. And the harness in tests/common/mod.rs treats empty as absent (non_empty()has_creds() false) and skips every scenario, exiting green:

"GitHub Actions sets env: keys even when the underlying secret/var is unset, producing empty strings rather than absent keys. We treat empty strings as absent."

Net result: the job passes, the PR looks covered, and zero credentialed scenarios actually ran — strictly worse than no workflow, because it manufactures false confidence.

Two things needed before merge:

  1. Confirm the four credentials are mirrored into the Dependabot secret/variable store (the Actions copies used by integration-tests.yml are not visible here).
  2. Fail loudly on misconfiguration. The silent-skip behavior is correct for fork PRs in integration-tests.yml, but it's wrong for a workflow whose sole purpose is credentialed runs. Add a guard so a missing/empty key fails the job instead of passing green, e.g.:
      - name: Require credentials
        env:
          HOTDATA_SDK_TEST_API_KEY: ${{ secrets.HOTDATA_SDK_TEST_API_KEY }}
          HOTDATA_SDK_TEST_WORKSPACE_ID: ${{ vars.HOTDATA_SDK_TEST_WORKSPACE_ID }}
        run: |
          test -n "$HOTDATA_SDK_TEST_API_KEY" || { echo "::error::HOTDATA_SDK_TEST_API_KEY is empty — is it set as a *Dependabot* secret?"; exit 1; }
          test -n "$HOTDATA_SDK_TEST_WORKSPACE_ID" || { echo "::error::HOTDATA_SDK_TEST_WORKSPACE_ID is empty — is it set as a *Dependabot* variable?"; exit 1; }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right, and thanks for the catch — the premise was wrong. Verified against GitHub's docs that the Dependabot sandbox keys on pull_request.user.login == 'dependabot[bot]' and applies to pull_request_target too, so it would have skipped green exactly as you describe. Reworked: dropped pull_request_target entirely (also clears the Aikido finding), switched to a plain pull_request job that reads the four creds from the Dependabot secrets store, and added the fail-loud guard so an empty credential errors instead of passing. The four values must be set under Settings → Secrets and variables → Dependabot for the job to run.

jobs:
integration:
if: github.event.pull_request.user.login == 'dependabot[bot]'
runs-on: ubuntu-latest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

super nit: this job checks out and executes untrusted dependency code (build scripts, proc-macros) with a live API key present, as the header comment acknowledges. Consider adding a timeout-minutes: to the job as cheap defense-in-depth so a runaway/abusive build script can't sit on the credentialed runner indefinitely. (not blocking)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added timeout-minutes: 20 to the job — good defense-in-depth, thanks.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review

Blocking Issues

  • The credentialed suite will silently no-op. pull_request_target runs triggered by Dependabot (github.actor == 'dependabot[bot]') are treated as untrusted by GitHub and do not receive Actions secrets/variables — only the separate Dependabot secrets/variables store is injected. Combined with the harness in tests/common/mod.rs, which treats empty credentials as "skip cleanly and exit green," this workflow will pass while running zero credentialed scenarios unless the secrets are mirrored into the Dependabot store. That produces false coverage, which is worse than no workflow. See the inline comment on the env: block.

Action Required

  1. Confirm HOTDATA_SDK_TEST_API_KEY, HOTDATA_SDK_TEST_WORKSPACE_ID, HOTDATA_SDK_TEST_API_URL, and HOTDATA_SDK_TEST_CONNECTION_ID are configured as Dependabot secrets/variables (not just Actions secrets).
  2. Add a guard step that fails the job when the required credentials are empty, so a misconfiguration surfaces loudly instead of as a green-but-untested run.

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cycle 2: all prior blocking threads resolved in code.

  • pull_request_target dropped — now plain pull_request (clears the Aikido finding).
  • Switched to the Dependabot secrets store + fail-loud guard. The guard checks HOTDATA_SDK_TEST_API_KEY and HOTDATA_SDK_TEST_WORKSPACE_ID, which are exactly the two creds TestEnv::has_creds() gates on, so the job can't pass green while silently skipping every scenario.
  • timeout-minutes: 20 added.

LGTM.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant