Skip to content

Add failing tests for #729: BUG_STEP_TIMEOUTS mismatch #731

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
prompt-driven-github wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Add failing tests for #729: BUG_STEP_TIMEOUTS mismatch #731

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 54 additions & 29 deletions pdd/prompts/agentic_bug_orchestrator_python.prompt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,41 +4,63 @@
Write the `pdd/agentic_bug_orchestrator.py` module.

% Role & Scope
Orchestrator for the 10-step agentic bug investigation workflow. Runs each step as a separate agentic task, accumulates context between steps, and tracks overall progress and cost.
Orchestrator for the 12-step agentic bug investigation workflow. Runs each step as a separate agentic task, accumulates context between steps, and tracks overall progress and cost.

% Requirements
1. Function: `run_agentic_bug_orchestrator(issue_url: str, issue_content: str, repo_owner: str, repo_name: str, issue_number: int, issue_author: str, issue_title: str, *, cwd: Path, verbose: bool = False, quiet: bool = False) -> Tuple[bool, str, float, str, List[str]]`
2. Return 5-tuple: (success, final_message, total_cost, model_used, changed_files)
3. Run 10 steps sequentially, each as a separate `run_agentic_task()` call
3. Run 12 steps sequentially, each as a separate `run_agentic_task()` call
4. Accumulate step outputs to pass as context to subsequent steps
5. Track total cost across all steps
6. For Step 7: Parse agent output for `FILES_CREATED: path1, path2` or `FILES_MODIFIED: path1, path2` lines to extract changed files (used for hard stop check and final summary)
7. For Step 9: Parse agent output for `E2E_FILES_CREATED: path1, path2` lines to extract E2E test files, extend changed_files list
8. Pass extracted files to Step 9 (E2E) and Step 10 (PR) via `files_to_stage` context variable for explicit git staging
6. For Step 9: Parse agent output for `FILES_CREATED: path1, path2` or `FILES_MODIFIED: path1, path2` lines to extract changed files (used for hard stop check and final summary)
7. For Step 11: Parse agent output for `E2E_FILES_CREATED: path1, path2` lines to extract E2E test files, extend changed_files list
8. Pass extracted files to Step 11 (E2E) and Step 12 (PR) via `files_to_stage` context variable for explicit git staging

% Step Execution
For each step (1-10):
For each step (1-12):
1. Load the step prompt template via `load_prompt_template(f"agentic_bug_step{n}_{name}_LLM")`
2. Format template with: issue_url, repo_owner, repo_name, issue_number, issue_content, issue_author, step1_output, step2_output, etc.
3. Call `run_agentic_task(formatted_prompt, cwd, ..., timeout=STEP_TIMEOUTS.get(step_num))`
- Import `STEP_TIMEOUTS` from `agentic_common` (Issue #261)
3. Call `run_agentic_task(formatted_prompt, cwd, ..., timeout=BUG_STEP_TIMEOUTS.get(step_num))`
- Define `BUG_STEP_TIMEOUTS: Dict[int, float]` as a module-level constant with integer keys 1-12
- Pass step-specific timeout to ensure complex steps get adequate time
4. Store the output for use by subsequent steps
5. Accumulate cost: `total_cost += step_cost`

% Per-Step Timeout Mapping
Define `BUG_STEP_TIMEOUTS` as a module-level dict with integer keys matching the 12-step `steps_config`:
```python
BUG_STEP_TIMEOUTS: Dict[int, float] = {
1: 240.0, # Duplicate Check
2: 400.0, # Docs Check
3: 400.0, # Triage
4: 500.0, # API Research
5: 600.0, # Reproduce (Complex)
6: 600.0, # Root Cause (Complex)
7: 600.0, # Prompt Classification (may auto-fix prompts)
8: 340.0, # Test Plan
9: 1000.0, # Generate Unit Test (Most Complex)
10: 600.0, # Verify Unit Test
11: 2000.0, # E2E Test (Complex - needs to discover env & run tests)
12: 240.0, # Create PR
}
```
Keys MUST be integers (not floats like 5.5) and MUST match the step numbers in `steps_config`.

% Step Sequence
| Step | Template Name | Purpose |
|------|---------------|---------|
| 1 | agentic_bug_step1_duplicate_LLM | Search for duplicate issues |
| 2 | agentic_bug_step2_docs_LLM | Check documentation for user error |
| 3 | agentic_bug_step3_triage_LLM | Assess if enough info to proceed |
| 4 | agentic_bug_step4_reproduce_LLM | Attempt to reproduce the bug |
| 5 | agentic_bug_step5_root_cause_LLM | Analyze root cause |
| 6 | agentic_bug_step6_test_plan_LLM | Design test strategy |
| 7 | agentic_bug_step7_generate_LLM | Generate failing unit test |
| 8 | agentic_bug_step8_verify_LLM | Verify test catches the bug |
| 9 | agentic_bug_step9_e2e_test_LLM | Generate and run E2E tests |
| 10 | agentic_bug_step10_pr_LLM | Create draft PR and link to issue |
| 4 | agentic_bug_step4_api_research_LLM | Research external API dependencies and constraints |
| 5 | agentic_bug_step5_reproduce_LLM | Attempt to reproduce the bug |
| 6 | agentic_bug_step6_root_cause_LLM | Analyze root cause |
| 7 | agentic_bug_step7_prompt_classification_LLM | Classify defect: code bug vs prompt defect |
| 8 | agentic_bug_step8_test_plan_LLM | Design test strategy |
| 9 | agentic_bug_step9_generate_LLM | Generate failing unit test |
| 10 | agentic_bug_step10_verify_LLM | Verify test catches the bug |
| 11 | agentic_bug_step11_e2e_test_LLM | Generate and run E2E tests |
| 12 | agentic_bug_step12_pr_LLM | Create draft PR and link to issue |

% Context Accumulation
- Step 1 receives: issue_content
Expand All @@ -47,15 +69,17 @@ For each step (1-10):
- Step 4 receives: issue_content, step1_output through step3_output
- ... and so on
- Step 8 receives: issue_content, step1_output through step7_output
- Step 9 receives: issue_content, step1_output through step8_output, worktree_path, files_to_stage
- Step 10 receives: issue_content, step1_output through step9_output, worktree_path, files_to_stage
- Step 9 receives: issue_content, step1_output through step8_output
- Step 10 receives: issue_content, step1_output through step9_output
- Step 11 receives: issue_content, step1_output through step10_output, worktree_path, files_to_stage
- Step 12 receives: issue_content, step1_output through step11_output, worktree_path, files_to_stage

% Files to Stage
After Step 7 parses FILES_CREATED/FILES_MODIFIED, pass the extracted file list to Steps 9 and 10:
After Step 9 parses FILES_CREATED/FILES_MODIFIED, pass the extracted file list to Steps 11 and 12:
- `context["files_to_stage"] = ", ".join(changed_files)`
- After Step 9 parses E2E_FILES_CREATED, extend changed_files and update files_to_stage
- This ensures Step 9 (E2E) and Step 10 (PR) know exactly which files to include
- Step 10 prompt uses `{files_to_stage}` to display the explicit list of files to stage
- After Step 11 parses E2E_FILES_CREATED, extend changed_files and update files_to_stage
- This ensures Step 11 (E2E) and Step 12 (PR) know exactly which files to include
- Step 12 prompt uses `{files_to_stage}` to display the explicit list of files to stage

% Early Exit Conditions (Hard Stops)

Expand All @@ -66,9 +90,10 @@ The orchestrator must parse step output to detect stop conditions:
| 1 | Issue is a duplicate | Output contains "Duplicate of #" |
| 2 | "Feature Request" or "User Error" | Output contains "Feature Request (Not a Bug)" or "User Error (Not a Bug)" |
| 3 | Needs more info from author | Output contains "Needs More Info" |
| 7 | No test file generated | No FILES_CREATED or FILES_MODIFIED line in output, or empty file list |
| 8 | Test doesn't fail correctly | Output contains "FAIL: Test does not work as expected" |
| 9 | E2E test doesn't catch bug | Output contains "E2E_FAIL: Test does not catch bug correctly" |
| 7 | Prompt defect needs human review | Output contains "PROMPT_REVIEW:" |
| 9 | No test file generated | No FILES_CREATED or FILES_MODIFIED line in output, or empty file list |
| 10 | Test doesn't fail correctly | Output contains "FAIL: Test does not work as expected" |
| 11 | E2E test doesn't catch bug | Output contains "E2E_FAIL: Test does not catch bug correctly" |

**Hard stop behavior:**
- Return `(False, "Stopped at step N: {reason}", total_cost, model_used, changed_files)`
Expand All @@ -82,7 +107,7 @@ The orchestrator must parse step output to detect stop conditions:

% Git Worktree Isolation

Before Step 7, create an isolated git worktree for test generation and PR creation. This prevents the workflow from disturbing the user's current branch.
Before Step 7, create an isolated git worktree for prompt classification, test generation, and PR creation. This prevents the workflow from disturbing the user's current branch.

1. Helper function: `_setup_worktree(cwd: Path, issue_number: int, quiet: bool) -> Tuple[Optional[Path], Optional[str]]`
- Create worktree at `.pdd/worktrees/fix-issue-{issue_number}/` relative to git root
Expand All @@ -96,8 +121,8 @@ Before Step 7, create an isolated git worktree for test generation and PR creati
2. In orchestrator loop, before Step 7:
- Call `_setup_worktree(cwd, issue_number, quiet)`
- If worktree creation fails, return early with error: `(False, "Failed to create worktree: {error}", ...)`
- Switch working directory to worktree path for Steps 7-10
- Add `worktree_path` to context dict (for Step 10 prompt formatting)
- Switch working directory to worktree path for Steps 7-12
- Add `worktree_path` to context dict (for Step 12 prompt formatting)
- Print: `"[blue]Working in worktree: {worktree_path}[/blue]"` (unless quiet)

3. Additional helper functions:
Expand All @@ -118,7 +143,7 @@ The orchestrator must provide real-time feedback to the user via rich console ou

2. **Step progress** (before each step):
```
[Step N/10] {step_description}...
[Step N/12] {step_description}...
```

3. **Agentic output** (during each step):
Expand Down Expand Up @@ -148,7 +173,7 @@ The orchestrator must provide real-time feedback to the user via rich console ou
Use `quiet` flag to suppress all output except errors. Use `verbose` flag to show full agentic task output.

% Dependencies
Import from `agentic_common`: `run_agentic_task`, `STEP_TIMEOUTS`
Import from `agentic_common`: `run_agentic_task`
<pdd.agentic_common><include>context/agentic_common_example.py</include></pdd.agentic_common>
<pdd.load_prompt_template><include>context/load_prompt_template_example.py</include></pdd.load_prompt_template>
<pdd.agentic_bug><include>context/agentic_bug_example.py</include></pdd.agentic_bug>
Expand Down
Loading
Loading