Skip to content

[pull] trunk from cli:trunk #61

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pull merged 15 commits into from
Jul 22, 2025
Merged

[pull] trunk from cli:trunk #61

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
c7c6892
ci: add spam issue detection scripts
babakks Jul 16, 2025
43e1634
ci: add spam issue detection workflow
babakks Jul 16, 2025
e90336a
ci: remove unused env vars
babakks Jul 16, 2025
1447400
ci: use `issue.html_url` instead of `issue.url`
babakks Jul 16, 2025
7996a6c
ci: fix potentially confusing typo in system prompt
babakks Jul 16, 2025
9aae4a5
docs: fix typo in script docs
babakks Jul 16, 2025
1779103
ci: correct `sed` usage to remove Markdown front matter
babakks Jul 16, 2025
6a5fbdd
ci: add `models: read` permission
babakks Jul 16, 2025
8a235ec
ci: echo spam detection result
babakks Jul 16, 2025
8610d8b
First pass to optimize and improve
andyfeller Jul 21, 2025
03cc1d8
Remove unnecessary file for heredoc
andyfeller Jul 21, 2025
f7448c1
Update eval script comments
andyfeller Jul 21, 2025
0c105af
Use gh go templating for user prompt
andyfeller Jul 21, 2025
aa955e1
Update .github/workflows/scripts/spam-detection/generate-sys-prompt.sh
andyfeller Jul 21, 2025
b2348f8
Merge pull request #11316 from cli/babakks/automate-spam-issue-detection
BagToad Jul 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .github/workflows/detect-spam.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
name: Spam Issue Detection
on:
issues:
types: [opened]

permissions:
contents: none
issues: write
models: read

jobs:
issue-spam:
runs-on: ubuntu-latest
environment: cli-automation
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Run spam detection
env:
GH_TOKEN: ${{ secrets.AUTOMATION_TOKEN }}
ISSUE_URL: ${{ github.event.issue.html_url }}
run: |
./.github/workflows/scripts/spam-detection/process-issue.sh "$ISSUE_URL"
if [[ $? -ne 0 ]]; then
echo "error processing issue"
exit 1
fi
7 changes: 7 additions & 0 deletions .github/workflows/scripts/spam-detection/check-issue-prompts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
name: Detect spam
model: openai/gpt-4o-mini
messages:
- role: system
content: "" # Since it's not a fix value, it should be generated and replaced at runtime
- role: user
content: "" # This will be replaced at runtime
48 changes: 48 additions & 0 deletions .github/workflows/scripts/spam-detection/check-issue.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
#!/bin/bash

# Check if an issue is spam or not and output "PASS" (not spam) or "FAIL" (spam).
#
# Regardless of the spam detection result, the script always exits with a zero
# exit code, unless there's a runtime error.
#
# This script must be run from the root directory of the repository.

set -euo pipefail

# Determine absolute path to script directory based on where it is called from.
# This allows the script to be run from any directory.
SPAM_DIR="$(dirname "$(realpath "$0")")"

# Retrieve and prepare information about issue for detection
_issue_url="$1"
if [[ -z "$_issue_url" ]]; then
echo "error: issue URL is empty" >&2
exit 1
fi

_user_prompt_template='
<TITLE>
{{ .title }}
</TITLE>

<BODY>
{{ .body }}
</BODY>
'

_user_prompt="$(gh issue view --json title,body --template "$_user_prompt_template" "$_issue_url")"

# Generate dynamic prompts for inference
_system_prompt="$($SPAM_DIR/generate-sys-prompt.sh)"
_final_prompt="$(_system="$_system_prompt" _user="$_user_prompt" yq eval ".messages[0].content = strenv(_system) | .messages[1].content = strenv(_user)" "$SPAM_DIR/check-issue-prompts.yml")"

gh extension install github/gh-models 2>/dev/null

_result="$(gh models run --file <(echo "$_final_prompt") | cat)"

if [[ "$_result" != "PASS" && "$_result" != "FAIL" ]]; then
echo "error: expected PASS or FAIL but got an unexpected result: $_result" >&2
exit 1
fi

echo "$_result"
Loading
Loading