feat: install PR Behavioral Analysis workflow by dev-punia-altimate · Pull Request #69 · AltimateAI/altimate-code

dev-punia-altimate · 2026-03-06T11:19:40Z

Summary

Install QA Autopilot's PR Behavioral Analysis workflow on altimate-code. When PRs are opened or marked ready for review, this workflow will:

Gate: skip docs-only, draft, and bot PRs
Analyze: use Claude to find semantic bugs (null access, type mismatches, missing error handling, etc.)
Dispatch fixes: relay critical findings to QA Autopilot workers for auto-fix
Generate tests: auto-generate unit tests for changed code (bun:test for TS, pytest for Python)
Report: post findings as PR comments + Slack notifications

Jobs pipeline

gate → analyze → dispatch-fix     (semantic bug detection + relay)
     → generate-tests → run-unit-tests   (auto-generate + validate)
     → report-unfixable           (PR comment for human review)
     → report                     (comprehensive quality report)

Required Secrets

Please configure the required secrets in Settings → Secrets and variables → Actions (see internal documentation for details).

Test plan

Configure the required secrets
Open a test PR with source code changes
Verify the workflow triggers and the gate job runs
Verify Claude analysis produces findings JSON
Verify PR comment is posted with findings

Install QA Autopilot's PR behavioral analysis on this repo. When PRs are opened/updated, this workflow will: - Gate: skip docs-only, draft, and bot PRs - Analyze: find behavioral gaps, missing tests, contract violations - Dispatch fixes to qa-autopilot workers - Generate and run unit tests - Report findings as PR comments + Slack

sentry · 2026-03-06T11:22:54Z

+        id: process
+        run: |
+          # Extract JSON from Claude's response
+          RESPONSE='${{ steps.analyze.outputs.result }}'


Bug: The shell variable assignment RESPONSE='${{ steps.analyze.outputs.result }}' is vulnerable to injection and will fail if the API response contains a single quote, causing the step to be skipped.
_{Severity: HIGH}

Suggested Fix

Pass the API response to the script using an environment variable instead of direct shell interpolation. This allows the shell to handle special characters safely.

- name: Process findings id: process env: RESPONSE: ${{ steps.analyze.outputs.result }} run: | echo "$RESPONSE" | python3 -c "..."

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not valid. Location: .github/workflows/pr-behavioral-analysis.yml#L228 Potential issue: The GitHub Actions workflow at `.github/workflows/pr-behavioral-analysis.yml:228` directly interpolates an API response into a single-quoted shell variable assignment: `RESPONSE='${{ steps.analyze.outputs.result }}'`. The API response is a JSON string containing natural language, which is very likely to include single-quote characters (e.g., in words like "it's"). When this occurs, the single quote will prematurely terminate the string, causing a shell syntax error. This error is caught by a fallback handler that silently marks the analysis as skipped, preventing any findings from being reported and effectively disabling the workflow step.

_{Did we get this right? 👍 / 👎 to inform future reviews.}

Single-quoted shell interpolation breaks when Claude's JSON response contains apostrophes (e.g., "it's"), silently disabling the workflow. Pass via env var instead.

dev-punia-altimate · 2026-03-06T11:33:02Z

Re: the shell quoting bug flagged by the review bot — confirmed real, already fixed in commit 14d1475.

The bug: RESPONSE='${{ steps.analyze.outputs.result }}' breaks on any single quote in Claude's JSON response (e.g., "it's a null access"), silently triggering the fallback that marks analysis_ran=false and disabling findings.

The fix: Pass via env: block instead of shell interpolation:

env:
  CLAUDE_RESPONSE: ${{ steps.analyze.outputs.result }}
run: |
  python3 -c "
  text = os.environ.get('CLAUDE_RESPONSE', '')
  ..."

Also fixed a secondary bug: analyze job outputs referenced steps.analyze.outputs.* for findings, but they're set in the process step — updated to steps.process.outputs.*.

sentry · 2026-03-06T11:33:07Z

+      - name: Process findings
+        id: process
+        env:
+          CLAUDE_RESPONSE: ${{ steps.analyze.outputs.result }}


Bug: The workflow references steps.analyze.outputs.result, but the claude-code-action provides its output in steps.analyze.outputs.structured_output, causing a silent failure.
_{Severity: CRITICAL}

Suggested Fix

Change the environment variable assignment on line 227 to use the correct output name. Replace CLAUDE_RESPONSE: ${{ steps.analyze.outputs.result }} with CLAUDE_RESPONSE: ${{ steps.analyze.outputs.structured_output }}.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not valid. Location: .github/workflows/pr-behavioral-analysis.yml#L227 Potential issue: The workflow at line 227 attempts to use the output `steps.analyze.outputs.result` from the `claude-code-action`. However, this action provides its JSON data in the `structured_output` output. Because `result` does not exist, the `CLAUDE_RESPONSE` environment variable becomes an empty string. The subsequent Python script fails to parse this empty string, causing the analysis to silently fail and fall back to a default state. The job incorrectly reports success, meaning no semantic code analysis is ever actually performed.

sentry · 2026-03-06T11:33:07Z

+    if: >-
+      always() &&
+      needs.gate.result == 'success' &&
+      needs.analyze.outputs.skip != 'true'


Bug: The generate-tests job condition is missing a check for needs.analyze.result == 'success', causing it to run even when the analyze job fails.
_{Severity: HIGH}

Suggested Fix

Add needs.analyze.result == 'success' to the if condition for the generate-tests job. This ensures the job only runs if the upstream analyze job has completed successfully and produced valid outputs.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not valid. Location: .github/workflows/pr-behavioral-analysis.yml#L390-L393 Potential issue: The `if` condition for the `generate-tests` job does not check if the upstream `analyze` job succeeded via `needs.analyze.result == 'success'`. It only checks if an output `skip` is not `'true'`. If the `analyze` job fails, its outputs are empty, the condition incorrectly evaluates to true, and `generate-tests` runs. The job then fails when it tries to use the empty outputs from the failed `analyze` job, such as `feature_branch` and `pr_number`, wasting CI resources and causing cascading failures.

anandgupta42 · 2026-03-06T15:49:03Z

@dev-punia-altimate we plan to open-source this package. I am not sure, it will be a good idea to put so much information on how we are doing this out there.

dev-punia-altimate · 2026-03-09T06:45:56Z

@dev-punia-altimate we plan to open-source this package. I am not sure, it will be a good idea to put so much information on how we are doing this out there.

@anandgupta42 - sure i'll replan this

github-actions · 2026-03-17T01:16:57Z

This PR doesn't fully meet our contributing guidelines and PR template.

What needs to be fixed:

PR description is missing required template sections. Please use the PR template.

Please edit this PR description to address the above within 2 hours, or it will be automatically closed.

If you believe this was flagged incorrectly, please let a maintainer know.

sentry bot reviewed Mar 6, 2026

View reviewed changes

fix: use env var for Claude response to avoid shell quoting bug

14d1475

Single-quoted shell interpolation breaks when Claude's JSON response contains apostrophes (e.g., "it's"), silently disabling the workflow. Pass via env var instead.

sentry bot reviewed Mar 6, 2026

View reviewed changes

dev-punia-altimate requested review from anandgupta42 and suryaiyer95 March 6, 2026 12:41

anandgupta42 closed this Mar 13, 2026

dev-punia-altimate deleted the qa-autopilot/install-behavioral-analysis branch March 16, 2026 06:58

github-actions bot added the needs:compliance label Mar 17, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: install PR Behavioral Analysis workflow#69

feat: install PR Behavioral Analysis workflow#69
dev-punia-altimate wants to merge 2 commits intomainfrom
qa-autopilot/install-behavioral-analysis

dev-punia-altimate commented Mar 6, 2026 •

edited by anandgupta42

Loading

Uh oh!

sentry bot Mar 6, 2026

Uh oh!

dev-punia-altimate commented Mar 6, 2026

Uh oh!

sentry bot Mar 6, 2026

Uh oh!

sentry bot Mar 6, 2026

Uh oh!

anandgupta42 commented Mar 6, 2026

Uh oh!

dev-punia-altimate commented Mar 9, 2026

Uh oh!

github-actions bot commented Mar 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dev-punia-altimate commented Mar 6, 2026 • edited by anandgupta42 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Jobs pipeline

Required Secrets

Test plan

Uh oh!

sentry bot Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

dev-punia-altimate commented Mar 6, 2026

Uh oh!

sentry bot Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

sentry bot Mar 6, 2026

Choose a reason for hiding this comment

Uh oh!

anandgupta42 commented Mar 6, 2026

Uh oh!

dev-punia-altimate commented Mar 9, 2026

Uh oh!

github-actions bot commented Mar 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dev-punia-altimate commented Mar 6, 2026 •

edited by anandgupta42

Loading