Skip to content

Add GitHub Models support#12

Merged
slifty merged 1 commit into
mainfrom
11-add-github-models-support
Feb 20, 2026
Merged

Add GitHub Models support#12
slifty merged 1 commit into
mainfrom
11-add-github-models-support

Conversation

@slifty

@slifty slifty commented Feb 20, 2026

Copy link
Copy Markdown
Member

This PR updates the workflow to default to github models (still allowing anthropic as a configurable option). The idea is that since this is a GitHub action this might be a more seamless "default" for some users, and also might better support FOSS projects that are hosted on GitHub.

It also add adds the workflow... to itself! Now we should be able to get QA advice.

Resolves #11

@slifty slifty force-pushed the 11-add-github-models-support branch 3 times, most recently from d137467 to 6b22445 Compare February 20, 2026 23:01
This updates the workflow to default to github models (still allowing
anthropic as a configurable option).  The idea is that since this is
a GitHub action this might be a more seamless "default" for some users,
and also might better support FOSS projects that are hosted on GitHub.

Issue #11 Support GitHub Models
@slifty slifty force-pushed the 11-add-github-models-support branch from 6b22445 to 787fc51 Compare February 20, 2026 23:03
@github-actions

Copy link
Copy Markdown

QA Instructions

Summary

This PR introduces support for GitHub Models as an AI provider within the QA Instructions GitHub Action, making it the default option. Anthropic support remains available as a configurable option. The goal is to enhance accessibility for GitHub-hosted projects and simplify usage by leveraging GitHub's built-in models API. The PR also adds the workflow to generate QA instructions to its own repository.

Test Environment Setup

  1. Ensure you have access to a GitHub repository where workflows can run.
  2. Verify the required permissions for the GitHub token:
    • Set permissions: models: read and permissions: pull-requests: write in the workflow YAML.
  3. Optionally, add ANTHROPIC_API_KEY as a secret for testing the Anthropic provider.

Test Scenarios

1. Verify GitHub Models Workflow (Default):

  1. Update a repository's .github/workflows directory with the provided qa.yml workflow.
  2. Open a pull request or update an existing one.
  3. Confirm that the QA Instructions action is triggered and completes successfully.
  4. Check the pull request for a comment from the action with generated QA instructions.
  5. Validate that the default provider is github-models and no additional API keys were required.

2. Verify Anthropic Provider Workflow:

  1. Modify the workflow YAML to include the provider: anthropic input and set up an ANTHROPIC_API_KEY secret.
  2. Trigger the workflow via a pull request.
  3. Confirm that the action uses the Anthropic provider (verify the value logged in the mock outputs for createProvider).
  4. Ensure the pull request contains generated QA instructions.

3. Validate Custom Model Input:

  1. Update the workflow YAML to include a model input, overriding the default model for either the GitHub or Anthropic provider.
  2. Verify that the specified model is used by checking the logs or API call parameters.

4. Test Edge Cases for Missing Inputs:

  • Remove or misconfigure the github-token input and validate that the action fails cleanly with appropriate error messages.
  • For Anthropic, omit the anthropic-api-key input and confirm the error is descriptive.

5. Regression Testing:

  • Verify that the new implementation does not break existing functionality, particularly:
    • Check that PR details (metadata, diff, etc.) are still processed as expected.
    • Confirm backward compatibility with workflows configured to use the Anthropic provider.

Regression Risks

  1. Potential misconfiguration of permissions (models: read) can result in unexpected failures.
  2. Changes to the provider logic (e.g., createProvider in provider-factory.ts) may impact behavior for Anthropic users.
  3. Default values for inputs (e.g., setting the model and provider) may inadvertently override explicit user configurations.

Things to Watch For

  1. Ensure that the context truncation logic works correctly for both GitHub models and Anthropics since the context limits differ between providers.
  2. Verify that error handling correctly conveys issues through GitHub Actions logs or failure annotations if provider-specific configurations are missing.
  3. Check that comments on PRs update as expected on re-runs of the workflow.

Generated by QA Instructions Action

@slifty slifty merged commit 6437d93 into main Feb 20, 2026
4 checks passed
@slifty slifty deleted the 11-add-github-models-support branch February 20, 2026 23:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support GitHub Models

1 participant