pdd checkup --pr step 5 (test execution) times out on large repos and blocks step 7 verify

[Serhan-Asad]

Problem

pdd checkup --pr <pr-url> --issue <issue-url> runs an 8-step agentic checkup. Step 5 (test execution, prompt pdd/prompts/agentic_checkup_step5_test_LLM.prompt) instructs the agent to "Run ALL tests — unit tests, integration tests, and end-to-end tests". On large repos like pdd-gltanaka itself (~8,600+ unit tests, plus integration/e2e suites), the agent process exceeds even the retry timeout budget and returns "Timeout expired", aborting the entire pipeline before steps 6.1–6.3 (fix) and step 7 (verify) can run.

The user-facing error is misleading: the orchestrator prints Aborting after Step 5: agent providers unavailable even though providers are healthy — the agent simply ran out of wall-clock time.

Reproduction (verbatim, on PR #1111)

pdd checkup --pr https://github.com/promptdriven/pdd/pull/1111 \
            --issue https://github.com/promptdriven/pdd/issues/1106

Observed timing in .pdd/worktrees/.../checkup-pr-1111/.pdd/agentic-logs/session_*.jsonl:

Step	Duration	Cost	Result
1 discover	63s	$1.86	ok
2 deps	68s	$0.49	ok
3 build	34s	$0.26	ok
4 interfaces	127s	$2.96	ok
5 test	600s	$0	Timeout expired
5 test (retry)	1200s	$0	Timeout expired
6.1–6.3, 7, 8	—	—	unreached

Total: $5.57 spent, no verdict.

Root Causes

1. Step 5 prompt scope is unbounded. agentic_checkup_step5_test_LLM.prompt says "Run ALL tests" with no awareness that in PR mode the changed files (and therefore the relevant tests) are a tiny subset of the suite. For a PR touching 6 files, blanket-running the whole suite is wasteful even when it succeeds.
2. Per-step timeout budget (600s default, 1200s retry) is fixed regardless of repo size. Configurable via --timeout-adder but there's no auto-scaling heuristic.
3. Misleading error message. The orchestrator surfaces agent providers unavailable when the actual failure is a hard timeout; the JSONL log shows "response": "Timeout expired". Users reasonably assume API/auth problems and waste time investigating the wrong thing.
4. No way to resume past a failing step. GitHub state-resume picks up where the last successful comment was posted — step 4 here — so retries are stuck at step 5 forever. There's no --start-step / --skip-step flag to jump to step 7 verify when step 5 is the bottleneck.

Suggested Fix (in order of impact, smallest first)

1. PR-mode test scoping. In PR mode, derive changed_paths = gh pr diff → expand to associated test files (tests/test_<basename>.py and any <basename> mention in tests/). Pass those as the only mandatory test set in the step 5 prompt; let the agent expand if it finds extra signal. This should cut step 5 from minutes to seconds on most PRs.
2. Surface the real failure. Change "agent providers unavailable" to "step N timeout (Xs exceeded budget Ys)" when the underlying response is "Timeout expired".
3. --start-step flag. Add a CLI flag (or env var PDD_CHECKUP_START_STEP) that overrides the GitHub-state-derived resume point, so users can recover by skipping a known-broken step (with appropriate [warning] step N skipped manually in the issue comment trail).
4. Step 5 timeout heuristic. Scale the per-step timeout by repo size (test count × avg duration) instead of a hardcoded constant.

Acceptance Criteria

• pdd checkup --pr on a PR touching ≤10 files completes step 5 in < 5 minutes on the pdd-gltanaka repo.
• When step 5 does time out, the orchestrator's user-facing message correctly says "step 5 timed out after Xs", not "agent providers unavailable".
• A user can rerun pdd checkup --pr --start-step 7 (or equivalent) to obtain a verdict when an upstream step is broken.
• Regression test covers the step-5 PR-mode narrowing.

Context

Discovered while verifying PR #1111 against issue #1106 (metadata finalization alignment). Steps 1–4 posted clean findings; the merge was already independently validated via codex review + targeted pytest (321 passed) + broader pytest (8,616 passed) + manual repro confirmation, so this issue blocks tooling UX rather than the underlying fix.

[Serhan-Asad]

5/21/2026

[Serhan-Asad]

Step 1/8: Project Discovery

Tech Stack

• Languages: Python 3.12+ (primary), TypeScript/React (pdd/frontend/ UI), Shell (completion scripts, CI)
• Frameworks: Click (CLI), Pydantic v2 (validation), FastAPI + uvicorn + websockets (pdd/server/), Textual (TUI), Vite + React (frontend), pytest (testing)
• Package Managers: pip / setuptools (pyproject.toml, build backend setuptools.build_meta + setuptools-scm); npm for pdd/frontend/
• Build Tools: Makefile (top-level dev workflow), pyproject.toml, pytest.ini, pyrightconfig.json, vite.config.ts (frontend), ci/cloud-batch/ (Docker + Cloud Build for sharded test execution)
• Key runtime deps: litellm[caching]>=1.80.0,<=1.82.6, click==8.1.7, pydantic==2.11.4, GitPython==3.1.44, firebase_admin==6.6.0, boto3==1.35.99, google-cloud-aiplatform, openai>=1.99.5, keyring==25.6.0, rich==14.0.0, tiktoken>=0.7.0, filelock>=3.12

Project Structure

• pdd/ — main CLI package (~158 modules). Subpackages: pdd/commands/ (Click subcommands incl. checkup.py), pdd/core/ (CLI group, duplicate guard, cloud config), pdd/server/ (FastAPI job queue), pdd/frontend/ (TS/React UI), pdd/prompts/ (LLM prompt templates), pdd/data/ (CSV catalogs).
• tests/ — 336 entries; pytest with markers real, integration, slow, e2e, private_prompt, uses_real_cli_detector; default per-test timeout 300s.
• prompts/ → symlink to pdd/prompts/; data/ → symlink to pdd/data/.
• architecture.json — single source of truth for module/prompt graph (used by pdd sync, agentic workflows).
• .pddrc — defines contexts: pdd_cli (main), commands, core, server, routes, pdd_frontend, regression, examples, etc.
• ci/cloud-batch/ — Docker + Cloud Build for sharded pytest runs.
• Makefile, .github/workflows/ (auto-heal, release, unit-tests), scripts/, utils/.

Checkup Scope

PR-mode checkup of PR #1119 (`checkup/pr-1119`, head commit `1c4fa64fb` fix(checkup): scope PR tests and surface timeouts) against issue #1114.

PR #1119 directly addresses issue #1114's four root causes. Code changes (excluding .pdd/meta/, context/, docs/, CHANGELOG.md, .github/) touch ~50 files, with the checkup-critical surface being:

• `pdd/agentic_checkup.py`, `pdd/agentic_checkup_orchestrator.py` — PR-mode test scoping (merge-base changed-files → test-prompt narrowing), `--start-step` / `PDD_CHECKUP_START_STEP` resume override, step-5 timeout heuristic.
• `pdd/commands/checkup.py` — CLI surface (new flag wiring).
• `pdd/agentic_common.py` — timeout error classification ("step N timeout" vs misleading "agent providers unavailable").
• `pdd/agentic_e2e_fix_orchestrator.py`, `pdd/checkup_review_loop.py`, `pdd/ci_validation.py` — adjacent orchestrators kept in sync with the new gating.

The checkup will verify these changes (a) honor PR #1119's acceptance criteria from issue #1114, (b) do not regress the broader agentic workflow surface (bug/change/e2e/sync orchestrators also touched), and (c) keep tests, examples, and architecture metadata consistent.

---

Proceeding to Step 2: Dependency Audit

[Serhan-Asad]

Step 1/8: Discovering project structure and tech stack

Step 1 completed; no <step_report> block returned by agent. Raw output retained in workflow state.

[Serhan-Asad]

Step 2/8: Dependency Audit

Scope: files changed in PR #1119 (pdd/agentic_checkup.py, pdd/agentic_checkup_orchestrator.py, pdd/commands/checkup.py, tests/commands/test_checkup.py, tests/test_agentic_checkup_orchestrator.py). The PR diff adds no new third-party imports — the only import-line change is typing.Union added to agentic_checkup.py:14.

Missing Dependencies

None. Third-party imports in PR-touched files (rich.console, click, click.testing, pytest) are all declared in pyproject.toml (rich==14.0.0, click==8.1.7, pytest==8.3.5).

Broken Local References

None. Every relative import resolves:

Import site	Target	Status
agentic_checkup.py → .agentic_change (_check_gh_cli, _escape_format_braces, _parse_issue_url, _parse_pr_url, _run_gh_command)	pdd/agentic_change.py:19/27/34/53/72	ok
agentic_checkup.py → .agentic_checkup_orchestrator.run_agentic_checkup_orchestrator	pdd/agentic_checkup_orchestrator.py:872	ok
agentic_checkup.py → .checkup_review_loop (ReviewLoopConfig, ReviewLoopContext, parse_reviewers, parse_severity_list, parse_state_list, run_checkup_review_loop)	pdd/checkup_review_loop.py:608/655/754/1427/1465/1481	ok
agentic_checkup.py → .agentic_sync (_find_project_root, _load_architecture_json)	pdd/agentic_sync.py:42 (re-export) / :379	ok
agentic_checkup_orchestrator.py → .agentic_common (DEFAULT_MAX_RETRIES, _sanitize_comment_body, clear_workflow_state, extract_step_report, load_workflow_state, normalize_step_comments_state, post_pr_comment, post_step_comment, post_step_comment_once, run_agentic_task, save_workflow_state, substitute_template_variables)	pdd/agentic_common.py (all defined or aliased, e.g. extract_step_report = _extract_step_report at :3507)	ok
agentic_checkup_orchestrator.py → .load_prompt_template.load_prompt_template, .preprocess.preprocess	pdd/load_prompt_template.py:10, pdd/preprocess.py:228	ok
commands/checkup.py → ..agentic_change._parse_pr_url, ..agentic_checkup.run_agentic_checkup, ..agentic_sync._is_github_issue_url, ..track_cost.track_cost, ..core.errors.handle_error	resolved at pdd/agentic_change.py:53, pdd/agentic_checkup.py:307, pdd/agentic_sync.py:63, pdd/track_cost.py:21, pdd/core/errors.py:92	ok

Unused Dependencies

None introduced or removed by the PR. Full manifest audit is out of scope for a PR-mode checkup; runtime deps in pyproject.toml are unchanged on this branch.

Version Pinning Notes

PR does not touch pyproject.toml. Existing pins remain (mix of strict == for most runtime deps and >= for litellm, psutil, fastapi, uvicorn, websockets, watchdog, tiktoken, filelock, lxml, openai, google-cloud-aiplatform). No new conflicts.

Issues Found

• [info] No dependency changes in this PR; all new code reuses already-declared packages (rich, click, stdlib re, shutil, subprocess, pathlib, typing, json).
• [info] typing.Union added at pdd/agentic_checkup.py:14 is stdlib — no manifest impact.

---

Proceeding to Step 3: Build Check

[Serhan-Asad]

Step 2/8: Auditing dependencies

Step 2 completed; no <step_report> block returned by agent. Raw output retained in workflow state.

[Serhan-Asad]

Step 3/8: Build Check (Iteration 1)

Build Commands Run

• python -m py_compile pdd/agentic_checkup.py pdd/agentic_checkup_orchestrator.py pdd/commands/checkup.py — clean (no output)
• python -c "import pdd.agentic_checkup, pdd.agentic_checkup_orchestrator, pdd.commands.checkup" — imports ok
• python -m compileall -q pdd/ — clean (no errors)

Scope note: PR #1119 touches Python only (pdd/agentic_checkup.py, pdd/agentic_checkup_orchestrator.py, pdd/commands/checkup.py, plus prompts and tests). No pdd/frontend/ or pdd/server/ source files modified, so tsc --noEmit skipped.

Build Errors

Severity	File	Error
—	—	none

Summary

0 errors found (0 critical, 0 medium, 0 low). Bytecode compilation and runtime import both succeed for every file changed in PR #1119 and for the broader pdd/ package.

---

Proceeding to Step 4: Interface Check

[Serhan-Asad]

Step 3/8: Running build/compile checks

Step 3 completed; no <step_report> block returned by agent. Raw output retained in workflow state.

[Serhan-Asad]

Step 4/8: Interface Check (Iteration 1)

Module Interface Issues

Module	Issue Type	Description
(none)	—	No missing exports, stub imports, or broken signatures detected in PR #1119

Cross-Module Compatibility

Traced the new start_step_override parameter end-to-end across all three layers:

• pdd/commands/checkup.py:368-384 — derives start_step_override: Optional[Union[int, float]] from the new --start-step Click option and forwards it to run_agentic_checkup(...).
• pdd/agentic_checkup.py:333,510 — accepts start_step_override: Optional[Union[int, float]] = None and forwards it to run_agentic_checkup_orchestrator(...). The Union import added on line 14 covers the annotation.
• pdd/agentic_checkup_orchestrator.py:893,1180-1188 — accepts the parameter and validates it against STEP_ORDER before overriding start_step. Union was already imported (line 21).

Other new symbols added in this PR are all internal to agentic_checkup_orchestrator.py and referenced only within the same module:

• _is_step_timeout_failure(output) — used at lines 1304-1308 and 1593-1594.
• _is_provider_failure(output) — promoted from a nested closure to module scope; same call sites as before plus line 1593.
• _format_step_abort_message(step_num, output) — used at line 1595 to replace the hard-coded "agent providers unavailable" abort string.
• _format_pr_changed_files_for_prompt(worktree, pr_metadata) — called at lines 1142 and 1503 with metadata_for_guard, which is initialized to {} at line 983 (unconditionally, before any PR branching), so the call site is always safe.

Set is imported at line 21 and used by the new helper at line 758. re (line 17) and subprocess (line 19) are already imported and reused by the new helpers — no new third-party dependencies.

The two new prompt placeholders are correctly wired:

• {pr_changed_files} — set in context at lines 1142 / 1503; consumed by prompts/agentic_checkup_step5_test_LLM.prompt:16-18,58 and prompts/agentic_checkup_step7_verify_LLM.prompt:24-26,78.
• {manual_start_step} — set at line 934; consumed by prompts/agentic_checkup_orchestrator_python.prompt.

Steps 5 and 7 prompt files reference the placeholders inside the existing context block, so substitute_template_variables (already used by _run_single_step) will resolve them with the values the orchestrator populates.

Frontend Navigation Reachability

Not applicable — PR #1119 changes are Python-only (pdd/agentic_checkup*.py, pdd/commands/checkup.py, and prompt templates). No frontend routes were added or modified.

Frontend→Backend API Consistency

Not applicable — no frontend or HTTP-client changes in this PR.

Summary

0 interface issues found. The --start-step CLI option, the new helper functions, and the PR-mode pr_changed_files context value all flow through consistent typed signatures, and every new symbol is reachable from its call sites with no missing imports.

---

Proceeding to Step 5: Test Execution

[Serhan-Asad]

Step 4/8: Checking cross-module interfaces

Step 4 completed; no <step_report> block returned by agent. Raw output retained in workflow state.