feat: add repeat and repeat-fail-on-threshold inputs

[tgvashworth]

• Adds repeat input that runs each test N times via promptfoo's --repeat flag
• Adds repeat-fail-on-threshold input that checks each individual test passes a minimum percentage of its repeated runs
• When thresholds are configured and pass, the action succeeds even if promptfoo exits non-zero (which it does whenever any test fails)
• Adds info logging showing repeat config and threshold results

Example usage

- uses: promptfoo/promptfoo-action@v1
  with:
    config: evals/skills.yaml
    repeat: 3
    repeat-fail-on-threshold: 66
    fail-on-threshold: 90

This runs each test 3 times and requires each test to pass at least 2/3 runs. The suite-level threshold requires 90% overall pass rate. Both are checked independently.

Testing

• 96 tests pass locally (npm test)
• Tested in CI on and internal repo with real LLM evals
• Verified --repeat 3 is passed to promptfoo and NxR test cases run
• Verified per-test grouping works correctly (groups by description across repeats)
• Verified threshold pass suppresses exec error
• dist/ rebuilt and included

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 469431f8-2d61-4da3-9331-ff3ce12530f8

📥 Commits

Reviewing files that changed from the base of the PR and between 88c88ba and 7427029.

⛔ Files ignored due to path filters (2)

• dist/index.js is excluded by !**/dist/**
• dist/index.js.map is excluded by !**/dist/**, !**/*.map

📒 Files selected for processing (5)

• README.md
• __tests__/main.test.ts
• action.yml
• src/main.ts
• src/utils/errors.ts

---

Walkthrough

This pull request introduces repeat functionality for handling flaky LLM evaluations. Two new action inputs are added: repeat (number of test repetitions, default 1) and repeat-fail-on-threshold (minimum per-test pass rate percentage). The implementation includes validation for both inputs, per-test aggregation logic to compute pass rates across repeated runs, and enhanced error handling to tolerate non-zero exit codes when thresholds pass. Documentation is updated with usage examples, and comprehensive test coverage is added for repeat scenarios, threshold validation, and combined behaviors. No code logic changes affect existing functionality when these inputs are not used.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The PR title accurately describes the main feature additions: adding repeat and repeat-fail-on-threshold inputs as shown across action.yml, src/main.ts, and test files.
Description check	✅ Passed	The description directly addresses the changeset, explaining the repeat and repeat-fail-on-threshold features with usage examples, test coverage, and implementation details.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[mldangelo-oai]

Thank you so much for putting this together, @tgvashworth. This was a really useful contribution: the original repeat support, threshold idea, docs, and tests gave us the right foundation for handling noisy LLM evals in the action. I really appreciate the thoughtful implementation and the concrete examples in the PR. I’m going to merge this first so your original work lands with proper credit, then I’ll rebase the follow-up redesign/hardening PR on top of it.