Skip to content

Add live test for partial failure exit code (core.setFailed) #15

@joshjohanning

Description

@joshjohanning

Context

PR #138 changed the action to use core.setFailed() instead of core.warning() when some repositories fail to update. This means the workflow step now exits with a non-zero status on partial failures.

What to test

  1. Prepare: Create a test scenario where at least one repo fails (e.g., a repo where the GitHub App doesn't have admin access, or a repo that's been deleted between prepare and test)
  2. Run action with a mix of accessible and inaccessible repos
  3. Assert:
    • The action step exits with a failure status
    • The failed-repositories output is > 0
    • Successfully processed repos still show their results
    • The failure message uses proper pluralization (1 repository vs N repositories)

Implementation notes

  • The workflow step running the action will need continue-on-error: true so the assertion step can still run
  • Capture the step's outcome to verify it's failure
  • This could be added to the existing test-failure suite

Files to modify

  • .github/workflows/live-integration.yml — add continue-on-error: true to the failure test action step, add outcome assertion
  • integration-test/configs/repos.failure.template.yml — add a repo that will fail (not just warn)
  • integration-test/scripts/prepare-live-test-repositories.js — add prepare for failure repo
  • integration-test/scripts/assert-live-test-failures.js — assert on exit code and failure count

Note

The current failure suite tests warning scenarios (invalid files, archived repos) which produce warnings but zero failures. This test needs an actual success: false repo to trigger core.setFailed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions