GitHub Actions: PR-Optimized Test Evidence Comment Renderer

[mk3008]

Context

We already have a working Test Evidence pipeline with the following properties:

• JSON is the single source of truth

  • test-specification.preview.json
  • test-specification.pr.json (diff JSON)

• A local Diff MD renderer exists and is useful for developer-side inspection

• Diff logic is stable and based on structured JSON comparison (not MD diff)

The next step is to integrate this into GitHub Actions so that Test Evidence is automatically generated and posted to Pull Requests in a reviewer-friendly way.

---

Design Principles (Non-Negotiable)

• Diff calculation must remain JSON-based

• Rendering must be separated from diff logic

• Multiple MD outputs are allowed as projections, not as independent sources

• No duplicated diff logic

• GitHub PR comments should be:

  • Quiet (no noise)
  • Mechanical (no human judgment text)
  • Clearly indicate whether evidence exists or not

---

Goal

Introduce a GitHub-optimized renderer for Test Evidence and wire it into GitHub Actions.

This renderer is specifically for PR comments and may differ from the existing local Diff MD format.

---

Scope

1. Core remains unchanged

Do not modify:

• Diff JSON schema
• Diff calculation logic

The following function is considered the single source of truth:

buildDiffJson(basePreviewJson, headPreviewJson, options) -> diffJson

---

2. Introduce a new renderer for GitHub PR comments

Add a new renderer function, for example:

renderDiffForGitHubPR(diffJson, options)
  -> { mode: "full" | "short" | "skip", body?: string }

This renderer is GitHub-specific and separate from the local MD renderer.

---

3. Rendering behavior rules (PR comment)

Case A: Diff exists (normal case)

Conditions:

• Any of catalogs/tests has + / - / ~

Behavior:

• Post a full PR comment

• Content is derived from diff JSON

• Format may include:

  • Summary header
  • Optional <details> folding (to be refined later)

Example summary line (conceptual):

Test Evidence: +1 catalog, -1 catalog, ~2 catalogs; +3 tests, -2 tests, ~1 test.

---

Case B: No diff

Conditions:

• catalogs: +0 / -0 / ~0
• tests: +0 / -0 / ~0

Behavior:

• Do not post the full report
• Post a short status comment only

Example:

Test Evidence: no changes detected.

---

Case C: No supported catalogs / evidence not applicable

Conditions (examples):

• head totals: catalogs=0 tests=0
• Or evidence collection reports “no supported test catalogs”

Behavior:

• Do not post the full report
• Post a short explanatory comment

Example:

Test Evidence: no supported test catalogs found in this PR.

---

Case D: Generation failure

Conditions:

• Diff generation failed
• Base preview could not be generated

Behavior:

• Post a failure status comment only
• No partial or misleading output

Example:

Test Evidence: failed to generate.

---

4. Comment update policy

• The Action must update a single comment, not create new comments repeatedly
• Use a stable identifier (bot user + hidden marker) to find and update the comment
• Prevent comment spam

---

5. GitHub Actions integration

This Issue does not require full YAML authoring, but assumes:

• Triggered on Pull Requests

• Renderer output decides:

  • full comment
  • short comment
  • or skip (no comment at all)

The Action should:

1. Generate preview JSON (base/head)
2. Build diff JSON
3. Invoke GitHub PR renderer
4. Post/update comment according to renderer result

---

Non-Goals (Explicitly Out of Scope)

• Migrating existing tests to catalog format
• AST-based migration tooling
• Public package extraction
• HTML or web UI rendering
• Snapshot support

These may be future work.

---

Why this approach

• Keeps JSON as the only authoritative data
• Avoids double maintenance of diff logic
• Allows local MD and GitHub PR views to evolve independently
• Makes CI output predictable and machine-trustable
• Reduces PR noise while preserving confidence

---

Acceptance Criteria

• A new GitHub-specific renderer exists
• Renderer behavior matches the rules above
• No duplicate diff logic introduced
• PR comments are updated, not spammed
• Short “no diff / not applicable” comments are visible and unambiguous

---

This Issue should be sufficient for the next AI or developer to implement the feature end-to-end.

[mk3008]

Performance & Cost Considerations for GitHub Actions

One additional consideration for this extraction work is GitHub Actions cost and execution time.

Since GitHub Actions may incur billing once free minutes are exceeded, it is important that the Test Evidence pipeline is designed to run as fast as reasonably possible, especially on Pull Request events that can trigger multiple runs.

This impacts the design in the following ways:

Why this matters

• PR workflows can run many times (opened, synchronize, etc.)
• Slow or redundant work directly increases CI minutes
• High cost or long wait times reduce adoption, even if the feature is valuable

Design implications (to consider during extraction)

The extracted diff core should enable (or at least not prevent) the following optimizations at the Action / caller level:

1. Early exit when evidence is not applicable

   • If no supported catalogs/tests exist, skip heavy generation
   • Emit a short status comment instead

2. Early exit when no relevant files changed

   • Callers may detect irrelevant PRs via path-based diff
   • Diff/rendering should not require full preview generation in such cases

3. Base preview reuse / caching

   • In PR workflows, the resolved base (merge-base) often stays the same
   • Callers may cache base preview JSON by base SHA
   • The diff core should be pure and cache-friendly

4. Pure, deterministic execution

   • No hidden IO or environment dependencies
   • Allows aggressive caching and reuse by callers

These concerns reinforce the decision to keep the diff core pure, fast, and environment-agnostic, and to leave performance strategies (caching, skipping, triggers) to the integration layer (GitHub Actions, CLI, etc.).

This is not a request to implement performance optimizations inside the core immediately, but a reminder that cost-aware CI usage is a first-class design constraint for this system.