You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# ADR-26148: Deterministic Audit Metrics via run_summary.json Cache and workflow-logs/ Exclusion
2
+
3
+
**Date**: 2026-04-14
4
+
**Status**: Draft
5
+
**Deciders**: pelikhan, Copilot
6
+
7
+
---
8
+
9
+
## Part 1 — Narrative (Human-Friendly)
10
+
11
+
### Context
12
+
13
+
The `audit` command reported wildly inconsistent `token_usage` and `turns` on repeated invocations for the same workflow run (observed: 9 turns / 381k tokens on one call, 22 turns / 4.7M tokens on another). Two compounding bugs caused this: (1) `AuditWorkflowRun` unconditionally re-processed all local log files on every call, even when a fully-computed `run_summary.json` was already on disk; and (2) the log-file walk in `extractLogMetrics` did not exclude the `workflow-logs/` directory, which `downloadWorkflowRunLogs` populates with GitHub Actions step-output — files that capture the same agent stdout already present in the agent artifact logs, inflating token counts by approximately 12×.
14
+
15
+
### Decision
16
+
17
+
We will adopt a **cache-first strategy** for `AuditWorkflowRun`: before performing any API calls or log processing, check whether a valid `run_summary.json` exists on disk (validated by CLI version). If a cache hit is found, reconstruct `ProcessedRun` from the cached summary and return immediately via a shared `renderAuditReport` helper, bypassing all re-download and re-parse logic. We will additionally **exclude the `workflow-logs/` directory** from the `extractLogMetrics` log walk by returning `filepath.SkipDir` whenever the walk visits a directory named `workflow-logs`, preventing GitHub Actions runner captures from being counted as agent artifact data. Together, these two changes ensure that repeated `audit` calls for the same run produce identical metrics.
18
+
19
+
### Alternatives Considered
20
+
21
+
#### Alternative 1: Invalidate and Overwrite the Cache on Every Call
22
+
23
+
Rather than treating the cached `run_summary.json` as authoritative, re-process logs on every call and overwrite the cache. This would keep the cache "fresh" but would perpetuate the inconsistency problem: log re-processing can produce different values depending on which files are present at the time (e.g., if `workflow-logs/` has been populated between calls). This was rejected because consistency of audit metrics across repeated calls is the primary requirement.
24
+
25
+
#### Alternative 2: Exclude `workflow-logs/` Files by Name Pattern Instead of Directory Skip
26
+
27
+
Rather than skipping the entire `workflow-logs/` directory with `filepath.SkipDir`, selectively exclude individual files whose names match known GitHub Actions runner-log patterns (e.g., `*_Run log step.txt`). This would be fragile: GitHub Actions file naming conventions can change, and any unrecognized file would silently inflate metrics again. Skipping the entire directory by name is simpler, robust, and aligns with how `downloadWorkflowRunLogs` places its output.
28
+
29
+
#### Alternative 3: Store Canonical Metrics in a Separate Lock File
30
+
31
+
Record only the metrics (token usage, turns) in a dedicated lock file separate from `run_summary.json`, and read that lock file on subsequent calls. This adds file-system complexity without meaningful benefit over reusing the existing `run_summary.json` structure. The current `loadRunSummary` already performs CLI-version validation, providing a clean automatic invalidation mechanism.
32
+
33
+
### Consequences
34
+
35
+
#### Positive
36
+
- Repeated `audit` calls for the same run are now deterministic and produce identical output.
37
+
- The cache-hit path avoids all API calls and file re-parsing, making subsequent audits significantly faster.
38
+
- The `renderAuditReport` helper function eliminates the duplicated render + finalization logic that previously existed in both the fresh-download and (now) cache-hit code paths.
39
+
- Cache invalidation on CLI upgrade is automatic via the existing `CLIVersion` check in `loadRunSummary`.
40
+
41
+
#### Negative
42
+
- The first successful `audit` call becomes the canonical source of truth. If log files were incomplete on the first run (e.g., partial download), the cached metrics will be wrong until the cache is manually cleared or the CLI is upgraded.
43
+
- The `workflow-logs/` exclusion is a directory-name-based heuristic. If `downloadWorkflowRunLogs` ever changes the output directory name, the exclusion silently stops working.
44
+
- Adding a new top-level helper (`renderAuditReport`) increases the surface area of the package's internal API.
45
+
46
+
#### Neutral
47
+
- The `run_summary.json` format is unchanged; only the read/write ordering is adjusted (save-before-render in the fresh-download path).
48
+
- Existing tests for `loadRunSummary` and `saveRunSummary` remain valid; new regression tests were added for the cache-hit path and the `workflow-logs/` exclusion.
49
+
50
+
---
51
+
52
+
## Part 2 — Normative Specification (RFC 2119)
53
+
54
+
> The key words **MUST**, **MUST NOT**, **REQUIRED**, **SHALL**, **SHALL NOT**, **SHOULD**, **SHOULD NOT**, **RECOMMENDED**, **MAY**, and **OPTIONAL** in this section are to be interpreted as described in [RFC 2119](https://www.rfc-editor.org/rfc/rfc2119).
55
+
56
+
### Cache-First Audit Strategy
57
+
58
+
1. Implementations **MUST** check for a valid `run_summary.json` on disk before initiating any API calls or log-file processing in `AuditWorkflowRun`.
59
+
2. Implementations **MUST** treat a cache hit (valid `run_summary.json` with matching `CLIVersion`) as the authoritative source of metrics and return immediately without re-processing logs.
60
+
3. Implementations **MUST NOT** overwrite an existing `run_summary.json` when serving a cache hit; the cached file **MUST** remain unmodified.
61
+
4. Implementations **MUST** persist `run_summary.json` to disk before calling the render step in the fresh-download path, so that a render failure does not prevent future cache hits.
62
+
5. Implementations **SHOULD** log a message (at the appropriate verbosity level) indicating that a cached summary is being used, including the original `ProcessedAt` timestamp.
63
+
64
+
### Log Metric Extraction
65
+
66
+
1. Implementations **MUST** skip the `workflow-logs/` directory (and its contents) when walking the run output directory in `extractLogMetrics`.
67
+
2. Implementations **MUST** use `filepath.SkipDir` (or equivalent) to exclude the entire `workflow-logs/` subtree, not individual files within it.
68
+
3. Implementations **MUST NOT** include token-usage data found in `workflow-logs/` in the `LogMetrics.TokenUsage` or `LogMetrics.Turns` totals.
69
+
4. Implementations **MAY** log a debug message when skipping the `workflow-logs/` directory to aid in future diagnostics.
70
+
71
+
### Shared Render Path
72
+
73
+
1. Implementations **MUST** use a single shared function (currently `renderAuditReport`) to build and emit the audit report, invoked by both the cache-hit path and the fresh-download path.
74
+
2. The shared render function **MUST NOT** re-extract metrics from log files; it **MUST** use only the metrics passed to it by the caller.
75
+
76
+
### Conformance
77
+
78
+
An implementation is considered conformant with this ADR if it satisfies all **MUST** and **MUST NOT** requirements above. Failure to meet any **MUST** or **MUST NOT** requirement constitutes non-conformance.
79
+
80
+
---
81
+
82
+
*This is a DRAFT ADR generated by the [Design Decision Gate](https://github.com/github/gh-aw/actions/runs/24396807146) workflow. The PR author must review, complete, and finalize this document before the PR can merge.*
// Use cached run summary when available to ensure deterministic metrics across repeated calls.
193
+
// Re-processing the same log files can produce different results (e.g. when GitHub's API
194
+
// returns aggregated data that differs from the locally-stored firewall logs), so we always
195
+
// prefer the first fully-processed summary written to disk. The cache is automatically
196
+
// invalidated whenever the CLI version changes (see loadRunSummary).
197
+
ifsummary, ok:=loadRunSummary(runOutputDir, verbose); ok {
198
+
auditLog.Printf("Using cached run summary for run %d (processed at %s)", runID, summary.ProcessedAt.Format(time.RFC3339))
199
+
ifverbose {
200
+
fmt.Fprintln(os.Stderr, console.FormatInfoMessage(fmt.Sprintf("Using cached run summary for run %d (processed at %s)", runID, summary.ProcessedAt.Format(time.RFC3339))))
0 commit comments