Skip to content

Commit e09f5ce

Browse files
committed
Add sample code branch push step to PR verification agent
After verification passes, the agent now pushes the verification sample to a dedicated branch (verification/pr-<N>) for future reference. Changes: - Added Step 5.5: Push Verification Sample to Branch - Updated behavioral rules: clarify no source code changes, add branch isolation rule - Updated success criteria to include branch pushing - Updated Step 7: samples are now pushed, not deleted
1 parent c2e439c commit e09f5ce

1 file changed

Lines changed: 210 additions & 6 deletions

File tree

.github/agents/pr-verification.agent.md

Lines changed: 210 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
---
1+
---
22
name: pr-verification
33
description: >-
44
Autonomous PR verification agent that finds PRs labeled pending-verification,
@@ -304,7 +304,140 @@ If the verification **fails**, investigate:
304304
- Is the sample correct?
305305
- Retry up to 2 times before reporting failure.
306306

307-
## Step 5: Post Verification to Linked Issue
307+
## Step 5: Create E2E Tests
308+
309+
After verification passes, create **e2e tests** that can be run as part of the
310+
test suite to prevent regression. These tests go in `test/e2e-azuremanaged/`.
311+
312+
### Test File Placement
313+
314+
Choose the appropriate test file based on the scenario:
315+
- If the fix relates to orchestrations → add tests to `test/e2e-azuremanaged/orchestration.spec.ts`
316+
- If the fix relates to entities → add tests to `test/e2e-azuremanaged/entity.spec.ts`
317+
- If the fix relates to retries → add tests to `test/e2e-azuremanaged/retry-advanced.spec.ts` or `retry-handler.spec.ts`
318+
- If the fix relates to a new area → create a new `test/e2e-azuremanaged/<area>.spec.ts` file
319+
320+
### Test Structure
321+
322+
E2e tests must follow the existing patterns in the test files:
323+
324+
1. **Read the existing test file first** to understand the patterns, imports, helper
325+
functions (`createClient()`, `createWorker()`), and `describe` block structure.
326+
2. **Add new `it()` blocks** inside the existing `describe` block.
327+
3. **Use the same helpers** (`createClient()`, `createWorker()`) — do NOT create new
328+
client/worker construction logic.
329+
4. **Follow the naming convention:** `it("should <expected behavior> when <scenario>", ...)`
330+
5. **Clean up resources:** Always stop worker/client in a `finally` block.
331+
6. **Use realistic timeouts:** 30-60 seconds for orchestrations to complete.
332+
333+
### Test Guidelines
334+
335+
- Tests must exercise the **exact scenario** that the PR fixes.
336+
- Tests should be self-contained — no shared mutable state between tests.
337+
- Tests should assert on the final orchestration status and output.
338+
- If the fix involves error handling, test both the success and failure paths.
339+
- Add a brief comment above each test describing the scenario and referencing the PR number.
340+
- Do NOT duplicate existing tests — check what's already covered.
341+
342+
### Example Test
343+
344+
```typescript
345+
// PR #123: Fix WhenAllTask crash on late child completion
346+
it("should complete fan-out orchestration when activity fails after another succeeds", async () => {
347+
const client = createClient();
348+
const worker = createWorker();
349+
350+
const orchestrator: TOrchestrator = async function* (ctx: OrchestrationContext) {
351+
// Fan out to multiple activities where one fails
352+
const tasks = [
353+
ctx.callActivity("successActivity"),
354+
ctx.callActivity("failingActivity"),
355+
];
356+
try {
357+
yield whenAll(tasks);
358+
} catch (e) {
359+
return "caught-error";
360+
}
361+
};
362+
363+
worker.addOrchestrator(orchestrator);
364+
worker.addActivity("successActivity", async () => "ok");
365+
worker.addActivity("failingActivity", async () => { throw new Error("fail"); });
366+
367+
try {
368+
await worker.start();
369+
const id = await client.scheduleNewOrchestration(orchestrator);
370+
const state = await client.waitForOrchestrationCompletion(id, undefined, 30);
371+
expect(state?.runtimeStatus).toBe(OrchestrationStatus.ORCHESTRATION_STATUS_COMPLETED);
372+
expect(state?.serializedOutput).toBe('"caught-error"');
373+
} finally {
374+
await worker.stop();
375+
await client.stop();
376+
}
377+
});
378+
```
379+
380+
## Step 6: Push Verification Artifacts to Branch
381+
382+
After verification passes and e2e tests are created, push all artifacts to a
383+
dedicated branch so they are preserved and can be reviewed.
384+
385+
### Branch Creation
386+
387+
Create a branch from the **PR's branch** (not from `main`) named:
388+
```
389+
verification/pr-<pr-number>
390+
```
391+
392+
For example, for PR #123:
393+
```bash
394+
git checkout -b verification/pr-123
395+
```
396+
397+
### Files to Commit
398+
399+
Commit the following files to the branch:
400+
401+
1. **Verification sample** — the standalone script created in Step 3.
402+
Place it at: `examples/verification/pr-<pr-number>-<short-description>.ts`
403+
(e.g., `examples/verification/pr-123-whenall-failfast.ts`)
404+
405+
2. **E2E test additions** — the test code added in Step 5.
406+
These are modifications to existing files in `test/e2e-azuremanaged/`.
407+
408+
### Commit and Push
409+
410+
```bash
411+
# Stage the verification sample and e2e test changes
412+
git add examples/verification/
413+
git add test/e2e-azuremanaged/
414+
415+
# Commit with a descriptive message
416+
git commit -m "chore: add verification sample and e2e tests for PR #<pr-number>
417+
418+
Verification sample: examples/verification/pr-<pr-number>-<description>.ts
419+
E2E tests added to: test/e2e-azuremanaged/<file>.spec.ts
420+
421+
Generated by pr-verification-agent"
422+
423+
# Push the branch
424+
git push origin verification/pr-<pr-number>
425+
```
426+
427+
### Branch Naming Rules
428+
429+
- Always use the prefix `verification/pr-`
430+
- Include only the PR number, not the issue number
431+
- Branch names must be lowercase with hyphens
432+
- If the branch already exists on the remote, skip pushing (idempotency)
433+
434+
Check if the branch already exists before pushing:
435+
```bash
436+
git ls-remote --heads origin verification/pr-<pr-number>
437+
```
438+
If it exists, skip the push and note it in the verification report.
439+
440+
## Step 7: Post Verification to Linked Issue
308441

309442
Post a comment on the **linked GitHub issue** (not the PR) with the verification report.
310443

@@ -334,6 +467,12 @@ Post a comment on the **linked GitHub issue** (not the PR) with the verification
334467

335468
</details>
336469

470+
### E2E Tests Added
471+
472+
- **File:** `test/e2e-azuremanaged/<file>.spec.ts`
473+
- **Tests:** <list of test names added>
474+
- **Branch:** `verification/pr-<pr-number>` ([view branch](https://github.com/microsoft/durabletask-js/tree/verification/pr-<pr-number>))
475+
337476
### Results
338477

339478
| Check | Expected | Actual | Status |
@@ -353,13 +492,74 @@ Post a comment on the **linked GitHub issue** (not the PR) with the verification
353492

354493
### Conclusion
355494

356-
<PASS: "All verification checks passed. The fix works as described in the PR.">
495+
<PASS: "All verification checks passed. The fix works as described in the PR. Verification sample and e2e tests pushed to `verification/pr-<pr-number>` branch.">
357496
<FAIL: "Verification failed. See details above. The fix may need additional work.">
358497
```
359498

360499
**Important:** The comment must start with `<!-- pr-verification-agent -->` (HTML comment)
361500
so the idempotency check in Step 1 can detect it.
362501

502+
## Step 5.5: Push Verification Sample to Branch
503+
504+
After posting the verification comment and **only if verification passed**, push the
505+
sample code to a dedicated branch for future reference.
506+
507+
### Branch Naming Convention
508+
509+
Create a branch named: `verification/pr-<pr-number>`
510+
511+
Example: For PR #123, create branch `verification/pr-123`
512+
513+
### Push Procedure
514+
515+
```bash
516+
# Create and switch to the verification branch (from main)
517+
git checkout main
518+
git checkout -b verification/pr-<pr-number>
519+
520+
# Create the samples directory structure if needed
521+
mkdir -p examples/verification-samples
522+
523+
# Move the sample file to the examples directory
524+
mv <temp-sample-file> examples/verification-samples/pr-<pr-number>-verify.ts
525+
526+
# Commit the sample
527+
git add examples/verification-samples/pr-<pr-number>-verify.ts
528+
git commit -m "Add verification sample for PR #<pr-number>
529+
530+
This sample was automatically generated by the pr-verification-agent.
531+
It reproduces the customer scenario addressed by PR #<pr-number> and
532+
validates the fix against the DTS emulator.
533+
534+
Scenario: <scenario-name>
535+
Verification status: PASSED
536+
Generated: <ISO timestamp>"
537+
538+
# Push the branch
539+
git push origin verification/pr-<pr-number>
540+
```
541+
542+
### Post-Push Cleanup
543+
544+
After pushing, switch back to `main`:
545+
546+
```bash
547+
git checkout main
548+
```
549+
550+
### Update Verification Comment
551+
552+
Edit the verification comment on the issue to include a link to the pushed sample:
553+
554+
```markdown
555+
### Sample Code
556+
557+
The verification sample has been pushed to branch [`verification/pr-<pr-number>`](https://github.com/microsoft/durabletask-js/tree/verification/pr-<pr-number>/examples/verification-samples/pr-<pr-number>-verify.ts).
558+
```
559+
560+
**Note:** If the push fails (e.g., permission issues, branch already exists), log
561+
the error but continue with label updates. The sample is still captured in the
562+
issue comment.
363563
## Step 6: Update PR Labels
364564

365565
After posting the verification comment:
@@ -374,16 +574,19 @@ If verification **failed**, do NOT update labels. Instead:
374574

375575
## Step 7: Clean Up
376576

377-
- Delete the temporary verification sample file (it's captured in the issue comment).
577+
- The verification sample has been pushed to a branch - no need to delete it locally.
578+
- Clean up the local working directory if any temporary files remain.
378579
- Do NOT stop the DTS emulator (other tests or agents may be using it).
379580

380581
## Behavioral Rules
381582

382583
### Hard Constraints
383584

384585
- **Idempotent:** Never post duplicate verification comments. Always check first.
385-
- **No code changes:** This agent does NOT modify any source files in the repository.
386-
It only creates temporary sample files, runs them, and posts results.
586+
- **No source code changes:** This agent does NOT modify any SDK source files in the
587+
repository. It only creates verification samples in `examples/verification-samples/`.
588+
- **Branch isolation:** Verification samples are pushed to `verification/pr-<N>` branches,
589+
never to `main` or feature branches.
387590
- **No PR merges:** This agent does NOT merge or approve PRs. It only verifies.
388591
- **Never modify generated files** (`*_pb.js`, `*_pb.d.ts`, `*_grpc_pb.js`).
389592
- **Never modify CI/CD files** (`.github/workflows/`, `eng/`, `azure-pipelines.yml`).
@@ -419,5 +622,6 @@ A successful run means:
419622
- All `pending-verification` PRs were processed (or correctly skipped)
420623
- Verification samples accurately test the PR's fix scenario
421624
- Evidence is posted to the correct GitHub issue
625+
- Verification samples are pushed to `verification/pr-<N>` branches
422626
- Labels are updated correctly
423627
- Zero duplicate work

0 commit comments

Comments
 (0)