Skip to content

feat(quality): add AI-assisted OSPS-QA-06.02 evaluation#346

Open
vinayada1 wants to merge 1 commit into
ossf:mainfrom
vinayada1:feat/osps-qa-06-02-ai-validation
Open

feat(quality): add AI-assisted OSPS-QA-06.02 evaluation#346
vinayada1 wants to merge 1 commit into
ossf:mainfrom
vinayada1:feat/osps-qa-06-02-ai-validation

Conversation

@vinayada1

@vinayada1 vinayada1 commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

Summary

Implements an AI-assisted evaluator for OSPS-QA-06.02 (documentation explains when and how tests are run), replacing the previous hard-coded NeedsReview stub.

  • Maps OSPS-QA-06.02 to the new quality.TestExecutionDocumentation evaluator.
  • Sends a conservative evidence boundary (README + CONTRIBUTING) to the configured AI provider and parses a strict, schema-validated verdict.
  • Persists AI evidence packets via the SDK writer (sdkai.WritePacket), gated behind the SDK's double opt-in (config.Write + ai_write_evidence: true), with SDK-side redaction.
  • Caches successful verdicts per run; provider/config/schema failures remain retryable.

Backward compatibility

  • AI not configured: behavior is identical to today. When Config is nil, the client is nil, or ai_provider/ai_api_key are missing, the evaluator returns the same gemara.NeedsReview result and message as the previous stub. All failure paths fall back to the same NeedsReview verdict.
  • SDK version: requires privateer-sdk v1.28.0 symbols (WritePacket, PacketAttempt, NewSanitizer, RedactPatterns, NewClientFromConfig). go.mod already pins v1.28.0, and Go's minimum version selection guarantees the floor at build time, so there is no silent old-SDK runtime risk.

@vinayada1 vinayada1 requested a review from a team as a code owner June 12, 2026 15:05
@vinayada1 vinayada1 force-pushed the feat/osps-qa-06-02-ai-validation branch 2 times, most recently from 833cf87 to ac70c26 Compare June 12, 2026 15:10
@vinayada1 vinayada1 force-pushed the feat/osps-qa-06-02-ai-validation branch 3 times, most recently from 27604ac to ae28216 Compare July 9, 2026 17:02
@kusari-inspector

kusari-inspector Bot commented Jul 9, 2026

Copy link
Copy Markdown

Kusari Inspector

Kusari Analysis Results:

Do not proceed without addressing issues

Caution

Flagged Issues Detected
These changes contain flagged issues that may introduce security risks.

The code security analysis found zero issues across 10 files — no secrets, no workflow vulnerabilities, and no insecure code patterns. However, the dependency analysis identified a confirmed HIGH severity vulnerability (CVE-2026-50163, CVSS 3.1 High - AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:L/A:N) introduced via the privateer-sdk upgrade from v1.28.0 to v1.31.1. This upgrade pulls in oras.land/oras-go/v2 v2.6.1 as a transitive dependency, which contains a hardlink tar-extraction path escape flaw. A crafted OCI artifact layer can cause os.Link() to resolve against the process working directory rather than the intended extraction directory, potentially exposing sensitive files such as .env files, .git/config, and CI workspace credentials. Critically, no upstream fix is currently available — v2.6.1 is the latest release with fix_status confirmed as NO_FIX. The clean code scan does not reduce the dependency risk, as these are independent analyses. We strongly recommend addressing this before merging. Recommended actions: (1) Revert privateer-sdk to v1.28.0 until a patched version of oras-go is released and incorporated into privateer-sdk. (2) If the privateer-sdk upgrade is required for other reasons, assess whether your application invokes oras-go content.File extraction against untrusted OCI sources — if it does not, the practical risk may be lower, but the exposure remains. (3) Avoid any privateer-sdk APIs that trigger oras-go content.File extraction with io.deis.oras.content.unpack enabled against untrusted registry sources as a temporary mitigation. (4) Monitor the oras-project/oras-go repository for a patched release and update privateer-sdk to incorporate the fix as soon as one is available.

Note

View full detailed analysis result for more information on the output and the checks that were run.

Required Dependency Mitigations

  • CRITICAL: oras.land/oras-go/v2 v2.6.1 is affected by CVE-2026-50163 (GHSA-fxhp-mv3v-67qp) - CVSS 3.1 HIGH (AV:N/AC:L/PR:N/UI:R/S:U/C:H/I:L/A:N). Dependency path: github.com/privateerproj/privateer-sdk v1.31.1 (direct dependency of root) → oras.land/oras-go/v2 v2.6.1 (vulnerable). The vulnerability is a hardlink tar-extraction path escape: when extracting OCI artifact layers with io.deis.oras.content.unpack annotation enabled, a crafted tarball containing a TypeLink entry with a relative Linkname causes os.Link() to resolve against the process CWD rather than the extract directory. This creates a hardlink inside the extract tree pointing to an arbitrary file in the invoker CWD, enabling read and write access to CWD files (e.g., .env, .git/config, CI workspace credentials). There is NO upstream fix available - v2.6.1 is currently the latest release and the vulnerabilityFixReport confirms fix_status is NO_FIX. Recommended actions: (1) Consider reverting privateer-sdk to v1.28.0 until a patched version of oras-go is released and incorporated into privateer-sdk. (2) If upgrading privateer-sdk is required for other reasons, assess whether your application invokes oras-go content.File extraction against untrusted OCI sources - if it does not, the practical risk may be lower. (3) Monitor the oras-project/oras-go repository for a patched release and update privateer-sdk to a version that incorporates the fix as soon as one is available. (4) As a temporary code-level mitigation, avoid using any privateer-sdk APIs that trigger oras-go content.File extraction with io.deis.oras.content.unpack enabled against untrusted registry sources.

@kusari-inspector rerun - Trigger a re-analysis of this PR
@kusari-inspector feedback [your message] - Send feedback to our AI and team
See Kusari's documentation for setup and configuration.
Commit: eab9e8a, performed at: 2026-07-09T17:07:12Z

Found this helpful? Give it a 👍 or 👎 reaction!

@kusari-inspector

Copy link
Copy Markdown

Kusari PR Analysis rerun based on - ae28216 performed at: 2026-07-09T17:04:52Z - link to updated analysis

Implement AI-assisted assessment for OSPS-QA-06.02 (test execution
documentation). When AI is configured, the step feeds README and
CONTRIBUTING content to the model and records a structured verdict
with evidence. Falls back gracefully to manual review when AI is
unavailable.

Updated to work with the TypedStep refactor (ossf#302) and SDK v1.28.0
bump (ossf#333).

Changes:
- Add TestExecutionDocumentation step with AI-assisted evaluation
- Add testExecutionDocumentationEvidence to gather README/CONTRIBUTING
- Add reusable_steps.AIFallback for generic AI degradation handling

Signed-off-by: Vinaya Damle <vinayada1@users.noreply.github.com>
@vinayada1 vinayada1 force-pushed the feat/osps-qa-06-02-ai-validation branch from ae28216 to eab9e8a Compare July 9, 2026 17:05
@kusari-inspector

Copy link
Copy Markdown

Kusari PR Analysis rerun based on - eab9e8a performed at: 2026-07-09T17:07:44Z - link to updated analysis

@jmeridth

Copy link
Copy Markdown
Member

@vinayada1 we merged the CVE fix and an sdk version with the fix also. Please rebase or merge main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants