Bug: structural test validator should scan generated output, not file on disk

[Serhan-Asad]

Problem

detect_structural_test_patterns() scans test files on disk with a start_line offset to skip pre-existing content. The approach keeps breaking on edge cases:

• Enforce metadata finalization in CI auto-heal / preflight drift-heal paths #990 — validator scanned entire file, flagged pre-existing patterns. Fix: add start_line + line-count snapshot.
• feat: reduce no-argument global sync dry-run noise for non-Tier-1 states (#1016) #1026 — snapshot scope was wrong, retry was destructive (deleted 12,308 lines). Fix: broader snapshot, content regression guard.
• Now — off-by-one: effective_start = 1 if start_line > len(lines) fires when start_line = len(lines) + 1 (the normal "nothing appended" case). File has 4144 lines, start_line=4145, clamp rescans everything, flags pre-existing inspect.getsource at lines 3346/3362, triggers retry, CLI killed (SIGKILL) after 17 min.

Proof

# test_orchestrator.py has 4144 lines, inspect.getsource at lines 3346/3362

detect_structural_test_patterns(file, start_line=4145)  # -> 2 violations (WRONG)
detect_structural_test_patterns(file, start_line=4144)  # -> 0 violations (correct)

The clamp at pdd/agentic_bug_orchestrator.py:980 treats start_line > len(lines) as "file was truncated, scan everything." But start_line = len(lines) + 1 is the normal case — it means "no new lines were appended." These two cases are indistinguishable from line-count alone.

Impact

• pdd_cloud issue Define and emit a machine-readable claim/evidence ledger for pdd change #1064: pdd bug attempt 1 ran Step 9, validator false-flagged pre-existing patterns, triggered retry, killed after 17 min (exit -9). Attempt 2 repeated the same failure.
• Pattern: this is the third time this scanning approach has broken. Every fix adds a new edge-case handler that the next edge case breaks.

Root Cause

File-on-disk scanning with line-count offsets can't distinguish "nothing new" from "file truncated." Every patch on this logic has been fragile because line counts are a lossy summary of the file state — they can't express which lines are new.

Recommended Fix — Diff-Based Scanning

Replace the line-count snapshot with a full content snapshot, and use difflib.SequenceMatcher to compute the exact set of new/changed line numbers in the current file. Only report violations whose line number is in that set.

Why this works:

• nothing added -> diff is empty -> no lines reportable -> zero false positives (fixes this bug)
• lines appended -> diff marks just the appended lines -> only those get flagged
• file truncated / rewritten -> diff marks all current lines as new -> scanned correctly
• new file (no snapshot) -> previous_content=None -> scan everything (unchanged behavior)
• insertions in the middle -> difflib identifies them correctly (current offset-based approach cannot)

API change:

# Before:
def detect_structural_test_patterns(file_path: str, start_line: Optional[int] = None) -> List[str]: ...

# After:
def detect_structural_test_patterns(file_path: str, previous_content: Optional[str] = None) -> List[str]: ...

Callers switch from snapshotting len(content.splitlines()) to snapshotting content itself.

Variable-tracking semantics are preserved: the function still scans the whole file for source-variable assignments (var = file.read_text()) so a cross-boundary pattern — old line defines the variable, new line uses it in a string-match assertion — still flags correctly. Only violation reporting is gated by the diff result.

Why not the "scan LLM output" alternative: Step 9 output lacks reliable line numbers relative to the target file. Violations need file line numbers so the retry prompt can tell the LLM where to fix.

Affected Files

• pdd/agentic_bug_orchestrator.py:931 — detect_structural_test_patterns() function (replace start_line param with previous_content, remove clamp at line 980, add _compute_changed_lines helper using difflib)
• pdd/agentic_bug_orchestrator.py:1528 — rename pre_step9_line_counts: Dict[str, int] to pre_step9_content: Dict[str, str]
• pdd/agentic_bug_orchestrator.py:1537 — snapshot full read_text() instead of len(splitlines())
• pdd/agentic_bug_orchestrator.py:1868-1871 — initial Step 9 violation scan: pass previous_content=pre_step9_content.get(abs_path.resolve_str())
• pdd/agentic_bug_orchestrator.py:2004 — retry violation scan: unchanged (no snapshot needed — .bak rewrite means everything is new)
• pdd/agentic_bug_orchestrator.py:2136-2137 — coverage-retry violation scan: same caller update
• tests/test_structural_test_guard.py — update TestStartLineTruncationSafety and related tests to use the new previous_content API; add regression test for the exact start_line=N+1 scenario (expressed as previous_content=same_content -> 0 violations)

Test Cases (must pass)

1. Off-by-one regression — file unchanged relative to snapshot -> detect_structural_test_patterns(path, previous_content=original) returns [] even when the unchanged file contains structural patterns.
2. Appended violations — snapshot has no patterns, file has patterns appended -> only the appended line numbers are reported.
3. Pre-existing violations preserved — snapshot already contains patterns -> those lines are not reported.
4. Truncation safe — file rewritten shorter than snapshot -> remaining lines scanned correctly.
5. Cross-boundary variable tracking — var = x.read_text() in snapshot, assert "y" in var in new lines -> the new assertion is flagged.
6. New file — previous_content=None -> every line scanned (unchanged behavior).

History

• Enforce metadata finalization in CI auto-heal / preflight drift-heal paths #990 — first snapshot fix (introduced start_line)
• feat: reduce no-argument global sync dry-run noise for non-Tier-1 states (#1016) #1026 — destructive retry fix (4 cascading bugs on the same approach)
• This issue — third failure of the same line-count approach

[prompt-driven-github]

🚀 Job Queued!

Job ID: iB5Usu2mUa04c1wRpmhz
Triggered by: @Serhan-Asad
Label: bug

View Live Progress
Model: pdd-sonnet (claude-sonnet-4-6)

[prompt-driven-github]

Step 1: Duplicate Check

Status: No duplicates found

Search Performed

• Searched for: detect_structural_test_patterns start_line, structural test validator start_line, off-by-one start_line validator, structural test patterns pre-existing, inspect.getsource false positive, step 9 retry SIGKILL, line count snapshot truncated
• Issues reviewed: 5 (issues PDD bug accumulates unnecessary test files #642, pdd test --manual ignores --manual flag and invokes agentic test generation instead #717, _find_state_comment() missing --paginate causes workflow state loss on issues with 30+ comments #481, and the two referenced issues Enforce metadata finalization in CI auto-heal / preflight drift-heal paths #990 and feat: reduce no-argument global sync dry-run noise for non-Tier-1 states (#1016) #1026 — which do not exist in this repo)

Findings

This is a new issue in this repository. The referenced issues #990 and #1026 mentioned in the bug history do not exist in promptdriven/pdd (likely from a different/private repository). No existing open or closed issues in this repo describe the same root cause: the off-by-one in detect_structural_test_patterns() where start_line = len(lines) + 1 triggers the truncation clamp and rescans the entire file, producing false-positive violations.

---

Proceeding to Step 2: Documentation Check

[prompt-driven-github]

[prompt-driven-github]

Step 2: Documentation Check

Status: Confirmed Bug

Documentation Reviewed

• pdd/agentic_bug_orchestrator.py — detect_structural_test_patterns() function (lines 931–1000) and its docstring
• pdd/agentic_bug_orchestrator.py — pre_step9_line_counts snapshot logic (lines 1528–1539) and call sites (lines 1868–1871, 2136–2137)
• README.md, CONTRIBUTING.md, docs/ — no external documentation describes the internal behavior of this function

Findings

The off-by-one is confirmed directly in the source code. The docstring at line 947–951 states:

If start_line exceeds the file length (e.g. the file was rewritten shorter than the snapshot), it is clamped to 1 so the entire file is scanned.

The clamp condition at line 980 is:

effective_start = 1 if start_line > len(lines) else start_line

The snapshot at line 1537–1539 stores len(splitlines()), and callers always pass pre_count + 1 as start_line (lines 1870, 2137).

When the LLM appends nothing to a 4144-line file:

• pre_count = 4144 → start_line = 4145
• Clamp check: 4145 > 4144 → True → effective_start = 1 (rescan entire file)

But start_line = len(lines) + 1 is the normal "nothing appended" case, not a truncation. The clamp fires one step too early. Truncation would be start_line > len(lines) + 1 (old file had more lines than the current file).

The code's own intent (per its docstring) is to rescan only when the file was truncated/rewritten shorter. The current condition contradicts that intent for the common case where the LLM wrote no new lines, causing false-positive violations that trigger a destructive retry loop.

This is a confirmed bug — the behavior contradicts the documented intent of the function.

---

Proceeding to Step 3: Triage

[prompt-driven-github]

Step 3: Triage

Status: Fast-Track

Pre-Diagnosed Issue Detected

The issue author has provided a detailed root cause analysis:

Identified Root Cause:
The clamp at pdd/agentic_bug_orchestrator.py:980 uses effective_start = 1 if start_line > len(lines), which fires when start_line = len(lines) + 1 — the normal "nothing was appended" case. Callers always pass start_line = previous_line_count + 1, so when no lines are added, the condition is True and the entire file is rescanned. This causes pre-existing structural test patterns to be flagged as violations, triggering unnecessary retries. The root cause is that line-count offsets cannot distinguish between "nothing appended" and "file truncated.". The recommended fix is to replace line-count snapshots with full content snapshots and use difflib.SequenceMatcher to identify only newly changed/added lines.

Evidence Quality:

• File paths with line numbers: Yes
  • pdd/agentic_bug_orchestrator.py:931 — detect_structural_test_patterns() function
  • pdd/agentic_bug_orchestrator.py:980 — off-by-one clamp
  • pdd/agentic_bug_orchestrator.py:1528 — pre_step9_line_counts dict
  • pdd/agentic_bug_orchestrator.py:1537 — line-count snapshot
  • pdd/agentic_bug_orchestrator.py:1868-1871 — initial Step 9 violation scan
  • pdd/agentic_bug_orchestrator.py:2004 — retry violation scan
  • pdd/agentic_bug_orchestrator.py:2136-2137 — coverage-retry violation scan
  • tests/test_structural_test_guard.py — test updates needed
• Causal explanation: Yes — line-count offsets are a lossy summary that cannot distinguish "nothing new" from "file truncated"
• Reproduction evidence: Yes — minimal Python proof showing start_line=4145 (N+1) produces 2 violations while start_line=4144 (N) produces 0; confirmed by Step 2 code inspection

Summary

The bug, root cause, affected files with line numbers, fix approach, and required test cases are all fully specified by the author. Step 2 independently confirmed the off-by-one at line 980. No diagnosis or reproduction steps are needed — we can proceed directly to implementing the fix.

---

Fast-tracking to Step 6: Root Cause Analysis (skipping Steps 4-5)