feat: add repeat and repeat-fail-on-threshold inputs

README.md

@@ -40,6 +40,8 @@ The action can be configured using the following inputs:
 | `no-progress-bar`    | Run promptfoo with `--no-progress-bar` flag to keep output minimal. Defaults to `false`                                                                   | No       |
 | `no-cache`           | Run promptfoo with `--no-cache` flag to avoid reading or writing results to the disk cache. Defaults to `false`                                             | No       |
 | `disable-comment`    | Disable posting comments to the PR. Defaults to `false`                                                                                                   | No       |
+| `repeat`             | Number of times to run each test. Useful for flaky LLM evals where a single run is noisy. Defaults to `1`                                                 | No       |
+| `repeat-fail-on-threshold` | Fail the action if any individual test passes less than this percentage (0-100) of its repeated runs. Only meaningful when `repeat > 1`. Example: `66` means each test must pass at least 2 out of 3 runs. | No       |
 | `force-run`          | Force evaluation to run even if no files changed. Defaults to `false`                                                                                      | No       |
 
 The following API key parameters are supported:
@@ -315,6 +317,25 @@ If you need to run evaluations regardless of file changes, use the `force-run` o
     force-run: true
 ```
 
+### Handling flaky LLM evals with repeat
+
+LLM eval outputs are non-deterministic. Use `repeat` to run each test multiple times and `repeat-fail-on-threshold` to require a minimum per-test pass rate:
+
+```yaml
+- name: Run skill evals
+  uses: promptfoo/promptfoo-action@v1
+  with:
+    github-token: ${{ secrets.GITHUB_TOKEN }}
+    config: evals/skills.yaml
+    use-config-prompts: 'true'
+    repeat: 3
+    repeat-fail-on-threshold: 66
+```
+
+This runs each test 3 times and requires each test to pass at least 2 out of 3 runs (66%). Tests that consistently fail will be flagged, while random grader variance is tolerated.
+
+You can combine this with `fail-on-threshold` for a suite-level check — both thresholds are evaluated independently.
+
 ## Caching for Better Performance
 
 promptfoo-action integrates with both GitHub Actions caching and promptfoo's internal caching to significantly reduce API costs and evaluation time.