Merge pull request #1882 from Hack23/copilot/fix-agentic-workflow-errors

Fix safeoutputs MCP session expiry causing silent PR failures across all news workflows

Author: pethers

.github/aw/SHARED_PROMPT_PATTERNS.md

@@ -71,12 +71,15 @@ TOTAL_READ_BYTES=0
 TOTAL_ONDISK_BYTES=0
 while read -r f; do
   if [ -f "$f" ]; then
-    FSIZE=$(wc -c < "$f" | tr -d ' ')
+    # AWF-safe: no $(...) command substitution — use tempfile + read redirection, then clean up.
+    wc -c < "$f" | tr -d ' ' > /tmp/fsize-$$.txt
+    read FSIZE < /tmp/fsize-$$.txt
+    rm -f /tmp/fsize-$$.txt
     TOTAL_ONDISK_BYTES=$((TOTAL_ONDISK_BYTES + FSIZE))
-    echo "--- BEGIN ANALYSIS FILE: $f (size: ${FSIZE} bytes) ---"
+    echo "--- BEGIN ANALYSIS FILE: $f (size: $FSIZE bytes) ---"
     if [ "$FSIZE" -gt "$MAX_FILE_BYTES" ]; then
       # Extremely rare — emit the full file but warn.
-      echo "⚠️  File exceeds ${MAX_FILE_BYTES}-byte soft cap; emitting in full regardless (§P0-5)."
+      echo "⚠️  File exceeds $MAX_FILE_BYTES-byte soft cap; emitting in full regardless (§P0-5)."
     fi
     cat "$f"
     TOTAL_READ_BYTES=$((TOTAL_READ_BYTES + FSIZE))
@@ -497,7 +500,7 @@ if [ "$IS_TIER_C" = "1" ]; then
     monthly-review*)      MULT_NUM=15; MULT_DEN=10 ;;  # 1.5×
     *)                    MULT_NUM=10; MULT_DEN=10 ;;  # fallback baseline
   esac
-  echo "📐 Period-scope multiplier for '$ANALYSIS_SUBFOLDER': ${MULT_NUM}/${MULT_DEN}"
+  echo "📐 Period-scope multiplier for '$ANALYSIS_SUBFOLDER': $MULT_NUM/$MULT_DEN"
 
   declare -A TIER_C_BASE_SIZES=(
     ["README.md"]=3000
@@ -513,9 +516,12 @@ if [ "$IS_TIER_C" = "1" ]; then
       echo "🔴 MISSING Tier-C: $REQUIRED_FILE — Tier-C workflow MUST CREATE"
       TIER_C_MISSING=$((TIER_C_MISSING + 1))
     else
-      FSIZE=$(wc -c < "$ANALYSIS_DIR/$REQUIRED_FILE")
+      # AWF-safe: no $(...) command substitution — use tempfile + read redirection, then clean up.
+      wc -c < "$ANALYSIS_DIR/$REQUIRED_FILE" | tr -d ' ' > /tmp/fsize-$$.txt
+      read FSIZE < /tmp/fsize-$$.txt
+      rm -f /tmp/fsize-$$.txt
       if [ "$FSIZE" -lt "$MIN_SIZE" ]; then
-        echo "🔴 UNDERSIZED Tier-C: $REQUIRED_FILE ($FSIZE bytes < $MIN_SIZE scaled minimum — base $BASE_SIZE × ${MULT_NUM}/${MULT_DEN}) — MUST ENRICH"
+        echo "🔴 UNDERSIZED Tier-C: $REQUIRED_FILE ($FSIZE bytes < $MIN_SIZE scaled minimum — base $BASE_SIZE × $MULT_NUM/$MULT_DEN) — MUST ENRICH"
         TIER_C_MISSING=$((TIER_C_MISSING + 1))
       else
         echo "✅ OK Tier-C: $REQUIRED_FILE ($FSIZE bytes ≥ $MIN_SIZE scaled minimum)"
@@ -2663,9 +2669,10 @@ Then call the health gate:
 > 🚨 **UNIVERSAL SAFE OUTPUT RULES — ALL WORKFLOWS MUST FOLLOW:**
 >
 > 1. **Call `safeoutputs___create_pull_request` as EARLY as possible** — the moment you have committed files. The safeoutputs MCP session has a finite lifetime. Successful runs call it by minute ~25. Failed runs that delayed past minute 40 got "session not found" and lost all work.
-> 2. **NEVER call `safeoutputs___noop` when artifacts exist.** Noop means "I did nothing." If you created files, you DID something and MUST create a PR. Partial work in a PR is infinitely better than lost work via noop.
-> 3. **At HARD DEADLINE**: If ANY files were created → `safeoutputs___create_pull_request`. ONLY noop if truly ZERO files were created.
-> 4. **Architecture reminder**: `safeoutputs___create_pull_request` records your intent. A separate `safe_outputs` job executes the PR creation AFTER the agent job ends. If the MCP session expires before you record the intent, the `safe_outputs` job is SKIPPED and all work is lost.
+> 2. **Heartbeat PR to keep the safeoutputs session alive (`max: 2+`)** — the Streamable-HTTP safeoutputs MCP session has a ~30–35 min idle lifetime (observed in PR #1835 and run #24672037751). Every `safeoutputs___create_pull_request` call **resets the session idle timer**. If the workflow sets `create-pull-request.max: 2` or higher, call the tool once at minute ~22–25 as a **heartbeat PR** capturing work-in-progress (partial analysis + first article draft), then call it again at minute 40–45 with the polished final output. Each call creates a separate PR on a separate branch; the final PR supersedes the heartbeat. This pattern was proven in `news-translate` (`max: 5`, zero session expiries across 55-minute runs) and `news-realtime-monitor` (`max: 3`). **Single-PR workflows (`max: 1`) that delay their only call past minute 35 WILL lose all work to session expiry.**
+> 3. **NEVER call `safeoutputs___noop` when artifacts exist.** Noop means "I did nothing." If you created files, you DID something and MUST create a PR. Partial work in a PR is infinitely better than lost work via noop.
+> 4. **At HARD DEADLINE**: If ANY files were created → `safeoutputs___create_pull_request`. ONLY noop if truly ZERO files were created.
+> 5. **Architecture reminder**: `safeoutputs___create_pull_request` records your intent. A separate `safe_outputs` job executes the PR creation AFTER the agent job ends. If the MCP session expires before you record the intent, the `safe_outputs` job is SKIPPED and all work is lost.
 
 ### Layer 3: MCP Gateway Diagnostics (run when tools fail)
 
@@ -2932,9 +2939,9 @@ echo "=== 🔍 Analysis Enrichment Verification Gate ==="
 for f in "$ANALYSIS_DIR"/*.md; do
   [ ! -f "$f" ] && continue
   # Skip the factual data-download-manifest.md — its script marker is expected and not a stub
-  FBASE=$(basename "$f")
-  case "$FBASE" in
-    data-download-manifest.md) echo "⏭️  SKIP (factual manifest): $f"; continue ;;
+  # AWF-safe: use `$f` path-pattern tests instead of $(basename "$f")
+  case "$f" in
+    */data-download-manifest.md) echo "⏭️  SKIP (factual manifest): $f"; continue ;;
   esac
   # Match the legacy `pre-article-analysis script` marker present in historical stub files.
   # The current download script writes only the factual `data-download-manifest.md` (skipped above),

.github/workflows/news-article-generator.md

@@ -128,6 +128,7 @@ safe-outputs:
     labels: [agentic-news, analysis-data]
     draft: false
     expires: 14d
+    max: 2
   add-comment: {}
   dispatch-workflow:
     workflows: [news-translate]
@@ -379,7 +380,9 @@ read START_TIME < /tmp/start_time.txt
 | Setup | 0–3 | Date check, `get_sync_status()` warm-up, check recent generation |
 | Download | 3–6 | Run data download scripts (MCP data fetch) |
 | **AI Analysis Pass 1** | **6–21** | **🚨 MANDATORY 15 min minimum**: Read ALL methodology guides, create per-file analysis for EVERY document with Mermaid diagrams, evidence tables, SWOT entries. |
-| **AI Analysis Pass 2** | **21–28** | **🚨 MANDATORY 7 min minimum**: Read ALL analysis back completely, improve every section, replace ALL script stubs with AI analysis. |
+| **AI Analysis Pass 2 (Part A)** | **21–22** | Begin reading ALL analysis artifacts back and identify improvement targets. |
+| **Heartbeat PR** | **22–25** | 🫀 `git add && git commit` analysis + any drafts so far, then `safeoutputs___create_pull_request` (title `🫀 Heartbeat - Article Generator - {date}`). This refreshes the safeoutputs MCP session (which expires after ~30–35 min idle) AND guarantees no work is lost if later phases fail. After the call succeeds, run `git checkout main` so subsequent commits don't stack onto the frozen patch. |
+| **AI Analysis Pass 2 (Part B)** | **25–28** | **Complete improvements (6 min improvement work total across Parts A+B)**: improve every section, replace ALL script stubs with AI analysis. |
 | Gates | 28–30 | Run ENFORCED Minimum Time Gate + Enrichment Verification Gate (SHARED_PROMPT_PATTERNS.md). Both MUST pass. |
 | Generate | 30–36 | Run `generate-news-enhanced.ts` in batches |
 | **Article Improvement** | **36–40** | 🚨 Read ALL articles back, replace AI_MUST_REPLACE markers, improve content. Run article quality component gate. |
@@ -389,6 +392,7 @@ read START_TIME < /tmp/start_time.txt
 > ⚠️ **Analysis phase is 22 minutes minimum (Pass 1: 15 min + Pass 2: 7 min)** — every analysis file must contain color-coded Mermaid diagrams, structured evidence tables with dok_id citations, and follow template structure exactly. ALL script-generated stubs MUST be replaced with AI-enriched analysis. Run the ENFORCED gates from SHARED_PROMPT_PATTERNS.md before article generation.
 
 **Hard cutoffs** — check elapsed time before each phase:
+- `>= 25 min` and no safeoutputs call yet → 🚨 call `safeoutputs___create_pull_request` as a heartbeat with whatever files exist. Do NOT delay — the safeoutputs session expires at ~30–35 min idle.
 - `>= 35 min` → Stop generating, commit what you have, create PR immediately
 - `>= 43 min` → STOP ALL WORK, call safe output immediately
 

.github/workflows/news-committee-reports.md

@@ -119,6 +119,7 @@ safe-outputs:
     labels: [agentic-news, analysis-data]
     draft: false
     expires: 14d
+    max: 2
   add-comment: {}
   dispatch-workflow:
     workflows: [news-translate]
@@ -286,7 +287,9 @@ read START_TIME < /tmp/start_time.txt
 - **Minutes 0–3**: Date check, MCP warm-up with `get_sync_status()`
 - **Minutes 3–6**: Run download-parliamentary-data pipeline (download data)
 - **Minutes 6–21**: 🚨 **AI Analysis Pass 1 (15 min minimum)**: Read ALL methodology guides, create per-file analysis for EVERY document with Mermaid diagrams, evidence tables, SWOT entries.
-- **Minutes 21–28**: 🚨 **AI Analysis Pass 2 (7 min minimum)**: Read ALL analysis back completely, improve every section, replace ALL script stubs with AI analysis. Run enrichment verification gate.
+- **Minutes 21–22**: 🚨 **AI Analysis Pass 2 (Part A, start)**: Begin reading ALL analysis artifacts back and identify improvement targets.
+- **Minutes 22–25**: 🫀 **Heartbeat PR** — `git add && git commit` analysis artifacts so far, then `safeoutputs___create_pull_request` (title `🫀 Heartbeat - Committee Reports - {date}`). Refreshes the safeoutputs MCP session (idle timeout ~30–35 min) AND preserves work if later phases fail. Run `git checkout main` after the call so subsequent commits don't stack onto the frozen patch.
+- **Minutes 25–28**: 🚨 **AI Analysis Pass 2 (Part B, complete — 6 min improvement work total across Parts A+B)**: Improve every section, replace ALL script stubs with AI analysis. Run enrichment verification gate.
 - **Minutes 28–30**: Run ENFORCED Minimum Time Gate + Enrichment Verification Gate (SHARED_PROMPT_PATTERNS.md). Both MUST pass.
 - **Minutes 30–36**: Generate articles for core languages (EN, SV) using `npx tsx scripts/generate-news-enhanced.ts`
 - **Minutes 36–40**: 🚨 **Article Improvement Pass**: Read ALL articles back, replace AI_MUST_REPLACE markers, improve content. Run article quality component gate.

.github/workflows/news-evening-analysis.md

@@ -127,6 +127,7 @@ safe-outputs:
     labels: [agentic-news, analysis-data]
     draft: false
     expires: 14d
+    max: 2
   add-comment: {}
   dispatch-workflow:
     workflows: [news-translate]
@@ -317,7 +318,9 @@ read START_TIME < /tmp/start_time.txt
 | Setup | 0–3 | Date check, `get_sync_status()`, determine day type | MCP responds |
 | Download | 3–6 | Run `populate-analysis-data.ts` + `download-parliamentary-data.ts` (script-driven data download) | Data files exist |
 | **AI Analysis Pass 1** | **6–21** | **🚨 MANDATORY 15 min minimum**: Read ALL methodology guides, create per-file analysis for EVERY document with Mermaid diagrams, evidence tables, SWOT entries. **Create ALL 9 required artifacts.** | 9 artifact files exist |
-| **AI Analysis Pass 2** | **21–28** | **🚨 MANDATORY 7 min minimum**: Read ALL 9 analysis artifacts back completely, improve every section, add missing Mermaid diagrams and evidence tables, replace ALL script stubs with AI analysis. | All 9 files ≥500 bytes |
+| **AI Analysis Pass 2 (Part A)** | **21–22** | Begin reading ALL 9 analysis artifacts back and identify improvement targets. | Files opened for review |
+| **Heartbeat PR** | **22–25** | 🫀 `git add && git commit` analysis + any drafts so far, then `safeoutputs___create_pull_request` (title `🫀 Heartbeat - Evening Analysis - {date}`). This refreshes the safeoutputs MCP session (which expires after ~30–35 min idle) AND guarantees no work is lost if later phases fail. After the call succeeds, run `git checkout main` so subsequent commits don't stack onto the frozen patch. | Heartbeat PR created |
+| **AI Analysis Pass 2 (Part B)** | **25–28** | **Complete improvements (6 min improvement work total across Parts A+B)**: improve every section, add missing Mermaid diagrams and evidence tables, replace ALL script stubs with AI analysis. | All 9 files ≥500 bytes |
 | Gates | 28–30 | Run ENFORCED Minimum Time Gate + Enrichment Verification Gate + **9-artifact completeness check** (SHARED_PROMPT_PATTERNS.md). ALL MUST pass. | 0 failures |
 | Generate | 30–36 | Run generation script OR manual synthesis (see Step 3) | HTML files created |
 | **Article Improvement** | **36–40** | 🚨 **Article Improvement Pass**: Read ALL articles back, replace AI_MUST_REPLACE markers, improve content. Run article quality component gate. | 0 AI_MUST_REPLACE markers |
@@ -328,7 +331,7 @@ read START_TIME < /tmp/start_time.txt
 >
 > 🔴 **ANTI-PATTERN (REJECTED)**: Creating only synthesis-summary.md + significance-scoring.md + stakeholder-perspectives.md and skipping the other 6 artifacts. This produces shallow articles missing SWOT tables, risk matrices, threat analysis, and classification data.
 
-**Hard cutoffs**: `>= 35 min` → commit & PR now; `>= 43 min` → STOP ALL WORK, call safe output immediately.
+**Hard cutoffs**: `>= 25 min` and no safeoutputs call yet → 🚨 call `safeoutputs___create_pull_request` as a heartbeat with whatever files exist (do NOT delay — the safeoutputs session expires at ~30–35 min idle); `>= 35 min` → commit & PR now; `>= 43 min` → STOP ALL WORK, call safe output immediately.
 
 ## Required Skills
 
@@ -611,9 +614,12 @@ for REQUIRED_FILE in README.md executive-brief.md scenario-analysis.md comparati
     echo "🔴 MISSING Tier-C: $REQUIRED_FILE — aggregation workflow MUST CREATE"
     MISSING=$((MISSING + 1))
   else
-    FSIZE=$(wc -c < "$ANALYSIS_DIR/$REQUIRED_FILE")
+    # AWF-safe: no $(...) command substitution — use tempfile + read redirection, then clean up.
+    wc -c < "$ANALYSIS_DIR/$REQUIRED_FILE" | tr -d ' ' > /tmp/fsize-$$.txt
+    read FSIZE < /tmp/fsize-$$.txt
+    rm -f /tmp/fsize-$$.txt
     if [ "$FSIZE" -lt "$MIN" ]; then
-      echo "🔴 UNDERSIZED Tier-C: $REQUIRED_FILE ($FSIZE < $MIN — base $BASE_MIN × ${PERIOD_SCOPE_MULT_NUM}/${PERIOD_SCOPE_MULT_DEN}) — MUST ENRICH"
+      echo "🔴 UNDERSIZED Tier-C: $REQUIRED_FILE ($FSIZE < $MIN — base $BASE_MIN × $PERIOD_SCOPE_MULT_NUM/$PERIOD_SCOPE_MULT_DEN) — MUST ENRICH"
       MISSING=$((MISSING + 1))
     else
       echo "✅ OK Tier-C: $REQUIRED_FILE ($FSIZE bytes)"

.github/workflows/news-interpellations.md

@@ -119,6 +119,7 @@ safe-outputs:
     labels: [agentic-news, analysis-data]
     draft: false
     expires: 14d
+    max: 2
   add-comment: {}
   dispatch-workflow:
     workflows: [news-translate]
@@ -295,7 +296,9 @@ read START_TIME < /tmp/start_time.txt
 - **Minutes 0–3**: Date check, MCP warm-up with `get_sync_status()`
 - **Minutes 3–6**: Run download-parliamentary-data pipeline (download data)
 - **Minutes 6–21**: 🚨 **AI Analysis Pass 1 (15 min minimum)**: Read ALL methodology guides, create per-file analysis for EVERY document with Mermaid diagrams, evidence tables, SWOT entries.
-- **Minutes 21–28**: 🚨 **AI Analysis Pass 2 (7 min minimum)**: Read ALL analysis back completely, improve every section, replace ALL script stubs with AI analysis. Run enrichment verification gate.
+- **Minutes 21–22**: 🚨 **AI Analysis Pass 2 (Part A, start)**: Begin reading ALL analysis artifacts back and identify improvement targets.
+- **Minutes 22–25**: 🫀 **Heartbeat PR** — `git add && git commit` analysis artifacts so far, then `safeoutputs___create_pull_request` (title `🫀 Heartbeat - Interpellations - {date}`). Refreshes the safeoutputs MCP session (idle timeout ~30–35 min) AND preserves work if later phases fail. Run `git checkout main` after the call so subsequent commits don't stack onto the frozen patch.
+- **Minutes 25–28**: 🚨 **AI Analysis Pass 2 (Part B, complete — 6 min improvement work total across Parts A+B)**: Improve every section, replace ALL script stubs with AI analysis. Run enrichment verification gate.
 - **Minutes 28–30**: Run ENFORCED Minimum Time Gate + Enrichment Verification Gate (SHARED_PROMPT_PATTERNS.md). Both MUST pass.
 - **Minutes 30–36**: Generate articles for core languages (EN, SV) using `npx tsx scripts/generate-news-enhanced.ts`
 - **Minutes 36–40**: 🚨 **Article Improvement Pass**: Read ALL articles back, replace AI_MUST_REPLACE markers, improve content. Run article quality component gate.

.github/workflows/news-month-ahead.md

@@ -120,6 +120,7 @@ safe-outputs:
     labels: [agentic-news, analysis-data]
     draft: false
     expires: 14d
+    max: 2
   add-comment: {}
   dispatch-workflow:
     workflows: [news-translate]
@@ -274,7 +275,9 @@ read START_TIME < /tmp/start_time.txt
 - **Minutes 0–3**: Date check, MCP warm-up with `get_sync_status()`
 - **Minutes 3–5**: Run download-parliamentary-data pipeline (download data)
 - **Minutes 5–15**: 🚨 **AI Analysis Pass 1 (10 min minimum)**: Read ALL methodology guides, create analysis for EVERY document with Mermaid diagrams, evidence tables, SWOT entries.
-- **Minutes 15–22**: 🚨 **AI Analysis Pass 2 + Enrichment Verification (7 min minimum)**: Read ALL analysis back, improve every section, replace ALL script stubs with AI analysis, and complete enrichment verification before the shared minimum-time gate.
+- **Minutes 15–19**: 🚨 **AI Analysis Pass 2 (Part A)**: Read ALL analysis back, improve major sections, replace script stubs.
+- **Minutes 19–21**: 🫀 **Heartbeat PR** — `git add && git commit` analysis artifacts so far, then `safeoutputs___create_pull_request` (title `🫀 Heartbeat - Month Ahead - {date}`). Refreshes the safeoutputs MCP session AND preserves work if later phases fail. Run `git checkout main` after the call so subsequent commits don't stack onto the frozen patch.
+- **Minutes 21–22**: 🚨 **AI Analysis Pass 2 (Part B) + Enrichment Verification**: Complete remaining improvements and run enrichment verification before the shared minimum-time gate.
 - **Minutes 22–23**: Run ENFORCED Minimum Time Gate (set `MINIMUM_ANALYSIS_MINUTES=14` for 30-min workflows) + final Enrichment Verification Gate (SHARED_PROMPT_PATTERNS.md). Both MUST pass.
 - **Minutes 23–28**: Generate articles for all 14 languages. Read articles back, replace AI_MUST_REPLACE markers. Run article quality gate.
 - **Minutes 28–29**: Validate and commit analysis + articles