Skip to content

Commit 3d5675f

Browse files
authored
Improve spam detection evals (cli#11419)
* ci: improve spam detection evals Signed-off-by: Babak K. Shandiz <babakks@github.com> * ci: make test case names consistent Signed-off-by: Babak K. Shandiz <babakks@github.com> * ci: remove ill-indented/redundant test case Signed-off-by: Babak K. Shandiz <babakks@github.com> --------- Signed-off-by: Babak K. Shandiz <babakks@github.com>
1 parent a2e23a2 commit 3d5675f

1 file changed

Lines changed: 100 additions & 151 deletions

File tree

.github/workflows/scripts/spam-detection/eval-prompts.yml

Lines changed: 100 additions & 151 deletions
Original file line numberDiff line numberDiff line change
@@ -10,126 +10,6 @@ evaluators:
1010
string:
1111
equals: "{{expected}}"
1212
testData:
13-
- name: not-spam, staff issue
14-
expected: PASS
15-
input: |
16-
<TITLE>
17-
Automatically update third party licenses during Dependabot PRs
18-
</TITLE>
19-
20-
<BODY>
21-
## Overview
22-
23-
With `cli/cli` lint process erring if 3rd party license information is not updated in https://github.com/cli/cli/pull/11047, Dependabot PRs will require maintainers to manually run `make licenses`.
24-
25-
Recently, @williammartin opened https://github.com/cli/cli/pull/11269 with the [`script/fix-dependabot-licenses.sh`](https://github.com/cli/cli/blob/26d70bfb7bcc0b41dbdd50bfc51f827f1a5ad4c4/script/fix-dependabot-licenses.sh) script for maintainers to run that will find all Dependabot PRs and attempt to fix them where the lint workflow failed. This script is a manual repair effort, however it is possible to [use a GitHub Actions workflow to run the `make license` script for Dependabot PRs](https://docs.github.com/en/code-security/dependabot/working-with-dependabot/automating-dependabot-with-github-actions):
26-
27-
> ```yaml
28-
> name: Dependabot fetch metadata
29-
> on: pull_request
30-
>
31-
> permissions:
32-
> pull-requests: write
33-
> issues: write
34-
>
35-
> jobs:
36-
> dependabot:
37-
> runs-on: ubuntu-latest
38-
> if: github.event.pull_request.user.login == 'dependabot[bot]' && github.repository == 'owner/my_repo'
39-
> steps:
40-
> - name: Dependabot metadata
41-
> id: metadata
42-
> uses: dependabot/fetch-metadata@d7267f607e9d3fb96fc2fbe83e0af444713e90b7
43-
> with:
44-
> github-token: "${{ secrets.GITHUB_TOKEN }}"
45-
> # The following properties are now available:
46-
> # - steps.metadata.outputs.dependency-names
47-
> # - steps.metadata.outputs.dependency-type
48-
> # - steps.metadata.outputs.update-type
49-
> ```
50-
51-
This issue is aimed at implementing GitHub Actions workflow changes that will automatically update `third-party` license source code and `third-party-*.md` reports, eliminating the need for maintainers to manually repair Dependabot PRs.
52-
53-
> [!NOTE]
54-
> To download the `script/fix-dependabot-licenses.sh` script, run the following command:
55-
> ```shell
56-
> curl -o fix-dependabot-licenses.sh https://raw.githubusercontent.com/cli/cli/26d70bfb7bcc0b41dbdd50bfc51f827f1a5ad4c4/script/fix-dependabot-licenses.sh
57-
> ```
58-
>
59-
> Or checkout the original PR:
60-
>
61-
> ```shell
62-
> gh pr checkout https://github.com/cli/cli/pull/11269
63-
> ```
64-
65-
## Expected outcomes
66-
67-
- When Dependabot PRs are opened, automation attempts to regenerate and commit updated license information via `make licenses`
68-
- When Dependabot PRs are updated, status checks pass without maintainer action outside of reviewing PR
69-
- name: not-spam, template-based
70-
spam: true
71-
title: |
72-
Incorrect check summary with v2.45 and v2.75
73-
body: |
74-
### Describe the bug
75-
76-
I got below confusing reports with v2.45 CLI on Ubuntu 22.04:
77-
78-
```sh
79-
$ gh pr status
80-
81-
Relevant pull requests in micropython/micropython
82-
...
83-
Created by you
84-
#17660 tests/extmod: Close UDP timely. [yf13:pull-udp-close]
85-
✓ Checks passing
86-
#17638 unix/make: Drop i686-linux-gnu path. [yf13:pull-drop-i686-linux-gnu]
87-
× 1/94 checks failing
88-
$ gh pr checks 17638
89-
All checks were successful
90-
0 cancelled, 0 failing, 48 successful, 0 skipped, and 0 pending checks
91-
```
92-
I downloaded latest v2.75 CLI but it is the same.
93-
94-
Meanwhile. from browser UI it seems checks passed.
95-
96-
### Affected version
97-
98-
Please run `gh version` and paste the output below.
99-
100-
```
101-
$ gh version
102-
gh version 2.75.0 (2025-07-09)
103-
https://github.com/cli/cli/releases/tag/v2.75.0
104-
```
105-
106-
### Steps to reproduce the behavior
107-
108-
see above descriptions.
109-
110-
### Expected vs actual behavior
111-
112-
A clear and concise description of what you expected to happen and what actually happened.
113-
114-
### Logs
115-
116-
Paste the activity from your command line. Redact if needed.
117-
118-
<!-- Note: Set `GH_DEBUG=true` for verbose logs or `GH_DEBUG=api` for verbose logs with HTTP traffic details. -->
119-
120-
</BODY>
121-
- name: not-spam, short/focused
122-
expected: PASS
123-
input: |
124-
<TITLE>
125-
Include `isImmutable` in `release list`
126-
</TITLE>
127-
128-
<BODY>
129-
Update the list of available JSON fields in the `release list` command to include `isImmutable` flag.
130-
131-
This boolean flag indicates whether a particular release has been marked as immutable.
132-
</BODY>
13313
- name: spam, two words
13414
expected: FAIL
13515
input: |
@@ -295,34 +175,6 @@ testData:
295175
296176
Add any other context like screenshots or mockups are helpful, if applicable.
297177
</BODY>
298-
- name: 'spam, legit but too general #10368 (https://github.com/cli/cli/issues/10368)'
299-
expected: FAIL
300-
input: |-
301-
<TITLE>
302-
Instructions in install_linux.md do not result in installation
303-
</TITLE>
304-
305-
<BODY>
306-
### Describe the bug
307-
308-
Bug: the instructions meant to install gh instead don't install gh.
309-
310-
### Affected version
311-
312-
Latest
313-
314-
### Steps to reproduce the behavior
315-
316-
Follow instructions in install_linux.md
317-
318-
### Expected vs actual behavior
319-
320-
Expect: gh is installed and can be used.
321-
322-
### Logs
323-
324-
A bunch of errors
325-
</BODY>
326178
- name: 'spam, #11304 (https://github.com/cli/cli/issues/11304)'
327179
expected: FAIL
328180
input: |-
@@ -820,6 +672,106 @@ testData:
820672
- name: 'spam, #9928 (https://github.com/cli/cli/issues/9928)'
821673
expected: FAIL
822674
input: "<TITLE>\nNote that an earlier version of the instructions used the location `/usr/share/keyrings` instead of `/etc/apt/keyrings` in the `sources.list.d` file, so I had to update that to make it work with the above update instructions, and remove the old keyring file from `/usr/share/keyrings`.\n</TITLE>\n\n<BODY>\n Note that an earlier version of the instructions used the location `/usr/share/keyrings` instead of `/etc/apt/keyrings` in the `sources.list.d` file, so I had to update that to make it work with the above update instructions, and remove the old keyring file from `/usr/share/keyrings`.\r\n\r\nAlternatively, one could of course download the updated key to `/usr/share/keyrings`, but we don't really want to pollute `/usr` with non-packaged files!\r\n\r\n_Originally posted by @rrthomas in https://github.com/cli/cli/issues/9569#issuecomment-2333981674_\r\n \n</BODY>"
675+
- name: 'spam, #10075 (https://github.com/cli/cli/issues/10075)'
676+
expected: FAIL
677+
input: "<TITLE>\nRHEL 9 installation update\n</TITLE>\n\n<BODY>\n### Describe the bug\r\n\r\nsteps to install on RHEL9 \r\n\r\n### Steps to reproduce the behavior\r\n\r\n\r\n### Expected vs actual behavior\r\n\r\n```\r\nsudo dnf install dnf-plugins-core.noarch\r\nsudo dnf config-manager --add-repo https://cli.github.com/packages/rpm/gh-cli.repo\r\nsudo dnf install gh --repo gh-cli\r\n```\n</BODY>"
678+
- name: not spam, staff issue
679+
expected: PASS
680+
input: |
681+
<TITLE>
682+
Automatically update third party licenses during Dependabot PRs
683+
</TITLE>
684+
685+
<BODY>
686+
## Overview
687+
688+
With `cli/cli` lint process erring if 3rd party license information is not updated in https://github.com/cli/cli/pull/11047, Dependabot PRs will require maintainers to manually run `make licenses`.
689+
690+
Recently, @williammartin opened https://github.com/cli/cli/pull/11269 with the [`script/fix-dependabot-licenses.sh`](https://github.com/cli/cli/blob/26d70bfb7bcc0b41dbdd50bfc51f827f1a5ad4c4/script/fix-dependabot-licenses.sh) script for maintainers to run that will find all Dependabot PRs and attempt to fix them where the lint workflow failed. This script is a manual repair effort, however it is possible to [use a GitHub Actions workflow to run the `make license` script for Dependabot PRs](https://docs.github.com/en/code-security/dependabot/working-with-dependabot/automating-dependabot-with-github-actions):
691+
692+
> ```yaml
693+
> name: Dependabot fetch metadata
694+
> on: pull_request
695+
>
696+
> permissions:
697+
> pull-requests: write
698+
> issues: write
699+
>
700+
> jobs:
701+
> dependabot:
702+
> runs-on: ubuntu-latest
703+
> if: github.event.pull_request.user.login == 'dependabot[bot]' && github.repository == 'owner/my_repo'
704+
> steps:
705+
> - name: Dependabot metadata
706+
> id: metadata
707+
> uses: dependabot/fetch-metadata@d7267f607e9d3fb96fc2fbe83e0af444713e90b7
708+
> with:
709+
> github-token: "${{ secrets.GITHUB_TOKEN }}"
710+
> # The following properties are now available:
711+
> # - steps.metadata.outputs.dependency-names
712+
> # - steps.metadata.outputs.dependency-type
713+
> # - steps.metadata.outputs.update-type
714+
> ```
715+
716+
This issue is aimed at implementing GitHub Actions workflow changes that will automatically update `third-party` license source code and `third-party-*.md` reports, eliminating the need for maintainers to manually repair Dependabot PRs.
717+
718+
> [!NOTE]
719+
> To download the `script/fix-dependabot-licenses.sh` script, run the following command:
720+
> ```shell
721+
> curl -o fix-dependabot-licenses.sh https://raw.githubusercontent.com/cli/cli/26d70bfb7bcc0b41dbdd50bfc51f827f1a5ad4c4/script/fix-dependabot-licenses.sh
722+
> ```
723+
>
724+
> Or checkout the original PR:
725+
>
726+
> ```shell
727+
> gh pr checkout https://github.com/cli/cli/pull/11269
728+
> ```
729+
730+
## Expected outcomes
731+
732+
- When Dependabot PRs are opened, automation attempts to regenerate and commit updated license information via `make licenses`
733+
- When Dependabot PRs are updated, status checks pass without maintainer action outside of reviewing PR
734+
</BODY>
735+
- name: not spam, short/focused
736+
expected: PASS
737+
input: |
738+
<TITLE>
739+
Include `isImmutable` in `release list`
740+
</TITLE>
741+
742+
<BODY>
743+
Update the list of available JSON fields in the `release list` command to include `isImmutable` flag.
744+
745+
This boolean flag indicates whether a particular release has been marked as immutable.
746+
</BODY>
747+
- name: 'not spam, legit but too general #10368 (https://github.com/cli/cli/issues/10368)'
748+
expected: PASS
749+
input: |-
750+
<TITLE>
751+
Instructions in install_linux.md do not result in installation
752+
</TITLE>
753+
754+
<BODY>
755+
### Describe the bug
756+
757+
Bug: the instructions meant to install gh instead don't install gh.
758+
759+
### Affected version
760+
761+
Latest
762+
763+
### Steps to reproduce the behavior
764+
765+
Follow instructions in install_linux.md
766+
767+
### Expected vs actual behavior
768+
769+
Expect: gh is installed and can be used.
770+
771+
### Logs
772+
773+
A bunch of errors
774+
</BODY>
823775
- name: 'not spam, #11277 (https://github.com/cli/cli/issues/11277)'
824776
expected: PASS
825777
input: |-
@@ -4482,9 +4434,6 @@ testData:
44824434
- name: 'not spam, #10076 (https://github.com/cli/cli/issues/10076)'
44834435
expected: PASS
44844436
input: "<TITLE>\n`gh run list` does not work with organization ruleset required workflows\n</TITLE>\n\n<BODY>\n### Describe the bug\r\n\r\nSimilar bug mentioned https://github.com/cli/cli/issues/3437, but gh run view or list all return a 404. The URL returned seems right based on REST api docs but not getting any response. When comparing the ID `gh run list` doesn't seem to be correct based on the ids from `gh workflow list`\r\n\r\n**gh cli version:** `gh version 2.63.2 (2024-12-05)`\r\n**ghe version:** `3.13.4`\r\n\r\n### Steps to reproduce the behavior\r\n\r\n1. Complete login to the enterprise server with Github CLI\r\n2. Go to a repository directory that uses that server as a remote\r\n3. Run gh run list returns 404\r\n\r\n### Expected vs actual behavior\r\n\r\nThe gh run list prints out list of workflow runs for the repo to choose from\r\n\r\n### Logs\r\n\r\n```bash\r\n[git remote -v]\r\n[git config --get-regexp ^remote\\..*\\.gh-resolved$]\r\n* Request at 2024-12-13 00:23:19.723417 -0600 CST m=+0.101249251\r\n* Request to https://{SERVER_URL}/api/graphql\r\n* Request took 281.385ms\r\n⣾* Request at 2024-12-13 00:23:20.040818 -0600 CST m=+0.418510918\r\n* Request to https://{SERVER_URL}/api/v3/repos/{ORG}/{REPO}/actions/runs?per_page=20&exclude_pull_requests=true\r\n⢿* Request took 421.362291ms\r\n⡿* Request at 2024-12-13 00:23:20.534045 -0600 CST m=+0.911535293\r\n* Request to https://{SERVER_URL}/api/v3/repos/{ORG}/{REPO}/actions/workflows?per_page=100&page=1\r\n⣟* Request took 105.218541ms\r\n* Request at 2024-12-13 00:23:20.700194 -0600 CST m=+1.077616418\r\n* Request to https://{SERVER_URL}/api/v3/repos/{ORG}/{REPO}/actions/workflows/63737\r\n⣯* Request took 121.476458ms\r\nfailed to get runs: HTTP 404: Not Found (https://{SERVER_URL}/api/v3/repos/{ORG}/{REPO}/actions/workflows/63737)\r\n```\r\n</BODY>"
4485-
- name: 'not spam, #10075 (https://github.com/cli/cli/issues/10075)'
4486-
expected: PASS
4487-
input: "<TITLE>\nRHEL 9 installation update\n</TITLE>\n\n<BODY>\n### Describe the bug\r\n\r\nsteps to install on RHEL9 \r\n\r\n### Steps to reproduce the behavior\r\n\r\n\r\n### Expected vs actual behavior\r\n\r\n```\r\nsudo dnf install dnf-plugins-core.noarch\r\nsudo dnf config-manager --add-repo https://cli.github.com/packages/rpm/gh-cli.repo\r\nsudo dnf install gh --repo gh-cli\r\n```\n</BODY>"
44884437
- name: 'not spam, #10073 (https://github.com/cli/cli/issues/10073)'
44894438
expected: PASS
44904439
input: "<TITLE>\n`gh gist delete` does not prompt for a gist to delete or prompt for confirmation before deletion\n</TITLE>\n\n<BODY>\n### Describe the bug\n\n- `gh gist delete` doesn't prompt for a gist to delete. This seems like it might be an oversight when compared to the behavior of other `gh gist` and `gh <some-command> delete` operations.\n- `gh gist delete` should prompt for a gist to delete and confirm the selection to delete.\n- `gh gist delete` also does not currently support `--yes` for non-interactive confirmation - perhaps it should? \n\n### Steps to reproduce the behavior\n\n```\ngh gist delete\n```\n### Expected vs actual behavior\n\n**Expected**\n\n```\n❯ gh gist delete\n? Select a gist to delete [Use arrows to move, type to filter]\n> test.md test gist about 4 days ago\n draft.md about 2 months ago\n? Are you sure you want to delete gist test.md (Y/n)\n```\n\n**Actual**\n```\ngh gist delete\n❯ gh gist delete\ncannot delete: gist argument required\n\nUsage: gh gist delete {<id> | <url>} [flags]\n```\n\n### Notes\n\nDiscovered in #10042 \n</BODY>"

0 commit comments

Comments
 (0)