Improve sync workflow conflict resolution. (#17)

## Description

When a cherry-pick fails during sync PR creation, the workflow now
creates a **conflict PR** instead of hard-failing with a
`PermanentSyncError`. This unblocks the sync pipeline by giving humans
(and eventually the conflict-resolution agent) a PR artifact to work
with, rather than requiring manual RUNBOOK intervention.

The conflict PR:
- Commits the raw conflict state (with markers or modify/delete
artifacts) so reviewers see exactly what git produced.
- Invokes the conflict-resolution agent to produce a separate resolution
commit on top (currently broken — fails gracefully; will be fixed in a
follow-up).
- Adds a `Repo-Sync-Conflict: cherry-pick` trailer, `repo-sync:conflict`
label, assigns a reviewer, and sends a Slack notification.
- Continues processing remaining commits (they stack on top of the
conflict branch).

The approve workflow checks for the `Repo-Sync-Conflict` trailer and
**never** approves a PR that has it — human approval is unconditionally
required. This is a structural guarantee independent of the bot's code
being correct.

Also threads `escalate_to` through the sync workflow so the conflict
handler can determine the right reviewer/fallback team.

## Testing

Existing tests pass (251/251). The new code paths
(`_handle_cherry_pick_conflict`, approve workflow gate) interact with
git and the GitHub API at runtime — manual end-to-end testing against a
real repo pair is needed.

Co-Authored-By: Oz <oz-agent@warp.dev>

Author: vorporeal

.github/workflows/sync.yml

@@ -132,6 +132,7 @@ jobs:
             --default-branch "${{ github.event.repository.default_branch }}" \
             --slack-webhook-url "${{ inputs.slack_webhook_url }}" \
             --private-to-public-fixup-script "${{ inputs.private_to_public_fixup_script }}" \
-            --public-to-private-fixup-script "${{ inputs.public_to_private_fixup_script }}"
+            --public-to-private-fixup-script "${{ inputs.public_to_private_fixup_script }}" \
+            --escalate-to "${{ inputs.escalate_to }}"
         env:
           WARP_API_KEY: ${{ secrets.warp_api_key }}

docs/RUNBOOK.md

@@ -2,11 +2,11 @@
 
 operational procedures for repo-sync failure scenarios.
 
-## cherry-pick failure during public-to-private sync
+## cherry-pick conflict during public-to-private sync
 
 ### symptoms
 
-the sync workflow fails with a cherry-pick conflict error.  the Slack notification reads something like: `repo-sync: cherry-pick failed for <sha> in <repo>`.  the watermark does not advance, so all subsequent commits are blocked.
+the sync workflow creates a conflict PR with the `repo-sync:conflict` label and `Repo-Sync-Conflict` trailer.  the Slack notification reads something like: `repo-sync: cherry-pick conflict for <sha> in <repo>. Conflict PR #<n> created.`  a reviewer is automatically assigned.
 
 ### cause
 
@@ -16,95 +16,120 @@ this is rare -- it can only happen when a public commit modifies lines adjacent
 
 ### remediation
 
-#### option 1: fix the root cause (preferred)
+#### option 1: resolve the conflict PR (preferred)
+
+the conflict PR contains the raw cherry-pick conflict (with conflict markers or modify/delete artifacts) and possibly an agent-proposed resolution commit on top.
+
+1. check out the conflict PR's branch locally
+2. resolve the conflicts (edit files to remove conflict markers, or handle modify/delete cases)
+3. push the resolution as a new commit
+4. approve the PR and merge it
+5. subsequent sync PRs in the stack will be restacked automatically
+
+the approve bot will **never** auto-approve a conflict PR (the `Repo-Sync-Conflict` trailer prevents this).  human approval is always required.
+
+#### option 2: fix the root cause
 
 adjust the private repo so the cherry-pick context matches:
 
-1. identify the conflicting file and lines from the workflow logs
+1. identify the conflicting file and lines from the conflict PR
 2. in the private repo, move the nearby `!repo-sync` marker regions so they don't overlap with the public commit's diff context.  options:
    - move the private-only code to a `private/` directory instead of using inline markers
    - rearrange the code so the marker region is further from the modified lines
 3. merge the fix into the private repo's default branch
-4. the next sync workflow run retries the cherry-pick against the updated `main`.  if the context now matches, the cherry-pick succeeds and the pipeline unblocks automatically
+4. close the conflict PR.  the next sync workflow run retries the cherry-pick against the updated `main`.  if the context now matches, the cherry-pick succeeds and creates a clean PR
 
-#### option 2: manually create the sync PR (escape hatch)
+#### option 3: manually create the sync PR (escape hatch)
 
-if fixing the root cause isn't practical or is too slow:
+if the conflict PR is unusable for some reason:
 
-1. determine the source commit SHA and the sync branch name from the workflow logs (format: `repo-sync/public-to-private/<short-sha>`)
-2. check out the private repo locally
-3. create the sync branch from the current stack top (or `main` if no stack):
+1. close the conflict PR
+2. determine the source commit SHA and the sync branch name from the workflow logs (format: `repo-sync/public-to-private/<short-sha>`)
+3. check out the private repo locally
+4. create the sync branch from the current stack top (or `main` if no stack):
    ```sh
    git checkout -b repo-sync/public-to-private/<short-sha> origin/main
    ```
-4. manually apply the public commit's changes, resolving any context mismatches:
+5. manually apply the public commit's changes, resolving any context mismatches:
    ```sh
    git cherry-pick <source-sha>
    # resolve conflicts, then:
    git cherry-pick --continue
    ```
-5. ensure the commit message includes the `Repo-Sync-Origin` trailer:
+6. ensure the commit message includes the `Repo-Sync-Origin` trailer:
    ```sh
    git commit --amend -m "$(git log -1 --format='%B')" -m "Repo-Sync-Origin: <source-repo>@<source-sha>"
    ```
-6. push the branch and create the PR with the correct `Repo-Sync-Origin` trailer in the description
-7. the sync workflow's idempotency guard will see the branch exists and skip this commit on the next run, unblocking the pipeline
+7. push the branch and create the PR with the correct `Repo-Sync-Origin` trailer in the description
+8. the sync workflow's idempotency guard will see the branch exists and skip this commit on the next run, unblocking the pipeline
 
 ### key properties
 
-- the watermark does not advance past the failing commit, so no commits are lost
+- the watermark does not advance past the conflicting commit, so no commits are lost
+- subsequent commits are stacked on top of the conflict PR and will be processed once it merges
 - the idempotency guard ensures manual intervention does not conflict with automation
-- once the failing commit is handled (by either path), the sync workflow processes all remaining unsynced commits automatically
+- once the conflict PR is resolved and merged, the sync workflow processes all remaining unsynced commits automatically
 
-## patch apply failure during private-to-public sync
+## cherry-pick conflict during private-to-public sync
 
 ### symptoms
 
-the sync workflow fails with a patch apply error.  the Slack notification reads something like: `repo-sync: patch apply failed for <sha> in <repo>. Un-synced public changes overlap.`  the watermark does not advance, so all subsequent commits are blocked.
+the sync workflow creates a conflict PR with the `repo-sync:conflict` label and `Repo-Sync-Conflict` trailer.  the Slack notification reads something like: `repo-sync: cherry-pick conflict for <sha> in <repo>. Conflict PR #<n> created.`  a reviewer is automatically assigned.
 
 ### cause
 
-a private commit modifies lines in a file that also has un-synced public changes nearby.  the sync workflow generates a patch (diff between consecutive clean snapshots of the private repo) and applies it to the public repo.  if the public repo's state in the affected area differs from the clean snapshot's context (because of un-synced public changes), `git apply` fails.
+a private commit modifies lines in a file that also has un-synced public changes nearby.  the sync workflow generates a clean delta (diff between consecutive stripped snapshots of the private repo) and cherry-picks it onto the public repo.  if the public repo's state in the affected area differs from the clean snapshot's context (because of un-synced public changes), the cherry-pick fails.
 
 this happens when both repos independently modify the same area of a file before the changes have been synced in both directions.
 
 ### remediation
 
-#### option 1: merge the pending public-to-private sync PRs first (preferred)
+#### option 1: resolve the conflict PR (preferred)
+
+the conflict PR contains the raw cherry-pick conflict (with conflict markers or modify/delete artifacts) and possibly an agent-proposed resolution commit on top.
 
-the most common cause is pending public-to-private sync PRs that haven't merged yet.  once they merge, the private repo includes the public changes, and the next sync run generates a patch with the correct context.
+1. check out the conflict PR's branch locally
+2. resolve the conflicts (edit files to remove conflict markers, or handle modify/delete cases)
+3. push the resolution as a new commit
+4. approve the PR and merge it
+5. subsequent sync PRs in the stack will be restacked automatically
 
-1. check for open public-to-private sync PRs in the private repo
-2. merge them (or wait for them to auto-merge)
-3. the next private-to-public sync run retries with updated context and should succeed
+the approve bot will **never** auto-approve a conflict PR (the `Repo-Sync-Conflict` trailer prevents this).  human approval is always required.
 
-#### option 2: manually create the sync PR (escape hatch)
+#### option 2: merge pending public-to-private sync PRs
 
-if merging pending sync PRs doesn't resolve the issue:
+the most common root cause is pending public-to-private sync PRs that haven't merged yet.  once they merge, the private repo includes the public changes, and a subsequent retry would generate a patch with the correct context.
 
-1. determine the source commit SHA and sync branch name from the workflow logs (format: `repo-sync/private-to-public/<short-sha>`)
-2. check out the public repo locally
-3. create the sync branch from the current stack top (or `main` if no stack):
+1. close the conflict PR
+2. check for open public-to-private sync PRs in the private repo
+3. merge them (or wait for them to auto-merge)
+4. the next private-to-public sync run retries with updated context and should succeed
+
+#### option 3: manually create the sync PR (escape hatch)
+
+if the conflict PR is unusable for some reason:
+
+1. close the conflict PR
+2. determine the source commit SHA and sync branch name from the workflow logs (format: `repo-sync/private-to-public/<short-sha>`)
+3. check out the public repo locally
+4. create the sync branch from the current stack top (or `main` if no stack):
    ```sh
    git checkout -b repo-sync/private-to-public/<short-sha> origin/main
    ```
-4. generate the patch locally and apply it with conflict resolution:
-   ```sh
-   # apply with --3way for merge-style conflict resolution, or apply manually
-   git apply --3way /path/to/patch.patch
-   ```
-5. commit with the `Repo-Sync-Origin` trailer:
+5. manually apply the changes with conflict resolution
+6. commit with the `Repo-Sync-Origin` trailer:
    ```sh
    git commit -m "repo-sync: sync from private" -m "Repo-Sync-Origin: <source-repo>@<source-sha>"
    ```
-6. push the branch and create the PR with the `Repo-Sync-Origin` trailer in the description
-7. the sync workflow's idempotency guard will see the branch exists and skip this commit on the next run
+7. push the branch and create the PR with the `Repo-Sync-Origin` trailer in the description
+8. the sync workflow's idempotency guard will see the branch exists and skip this commit on the next run
 
 ### key properties
 
-- the watermark does not advance past the failing commit, so no commits are lost
+- the watermark does not advance past the conflicting commit, so no commits are lost
+- subsequent commits are stacked on top of the conflict PR and will be processed once it merges
 - the idempotency guard ensures manual intervention does not conflict with automation
-- merging pending public-to-private sync PRs is usually sufficient to unblock the pipeline
+- merging pending public-to-private sync PRs is usually sufficient to prevent these conflicts
 
 ## missing Repo-Sync-Origin trailer on merge commit
 

docs/TECH-DESIGN.md

@@ -35,19 +35,31 @@ when a workflow run starts, it reads the watermark tag to determine the last-syn
    b. compute the diff between the two clean snapshots by committing both into a temporary git repo.  if the diff is empty, skip (all changes in this commit were internal-only)
    c. check if a PR with head branch `repo-sync/private-to-public/<short-sha>` was previously created (idempotency guard -- prevents duplicates if the workflow crashed and restarted mid-run).  if a MERGED PR exists, skip without updating the stack base.  if an OPEN PR exists, use its branch as the stack base and skip
    d. create a branch `repo-sync/private-to-public/<short-sha>` based on the top of the current stack (or `main` if no stack)
-   e. apply the delta to the public repo by cherry-picking the diff commit from the temporary repo.  cherry-pick uses three-way merge internally, so it handles context mismatches from un-synced public changes gracefully.  if the cherry-pick produces no changes (delta already present in the public repo), skip.  if it conflicts, fail loudly
+   e. apply the delta to the public repo by cherry-picking the diff commit from the temporary repo.  cherry-pick uses three-way merge internally, so it handles context mismatches from un-synced public changes gracefully.  if the cherry-pick produces no changes (delta already present in the public repo), skip.  if it conflicts, create a **conflict PR** (see below)
    f. amend the commit with a generic message (e.g., `"repo-sync: sync from private"`) and the `Repo-Sync-Origin` trailer.  **do not** use the source commit's message, as it could leak private information
    g. push the branch.  if the branch already exists on the remote (from a previous interrupted run where push succeeded but PR creation failed), verify the content matches and skip the push.  hard-fail if the content differs
    h. create a PR with the base set to the previous sync branch (or `main`)
 
 the sync workflow does **not** approve or enable auto-merge on any PRs.  approval is handled by the separate approve workflow when a PR reaches the bottom of the stack.
 
+**cherry-pick conflict PRs:**
+
+when a cherry-pick fails during sync PR creation (either direction), instead of failing the workflow, the sync creates a conflict PR:
+1. commit the raw conflict state as-is (with conflict markers or modify/delete artifacts) using `git add -A` + `git commit`.  this gives reviewers full visibility into the raw conflict
+2. invoke the conflict-resolution agent to produce a **separate resolution commit** on top.  if the agent fails (or is unavailable), the branch is left with just the raw conflict commit
+3. push the branch and create the PR with a `Repo-Sync-Conflict: cherry-pick` trailer, the `repo-sync:conflict` label, and a `Repo-Sync-Assigned` trailer
+4. request review from the person who merged the source PR (or the fallback team)
+5. send a Slack notification with the conflict PR number
+6. continue processing remaining unsynced commits (they stack on top of the conflict branch)
+
+the `Repo-Sync-Conflict` trailer is a structural guarantee: the approve workflow checks for it and **never** approves a PR with this trailer.  conflict PRs unconditionally require human approval, regardless of mergeability state.  this ensures no conflict PR is auto-merged without human review.
+
 **public-to-private:**
 1. identify unsynced commits on the public repo's default branch
 2. for each unsynced commit (in chronological order):
    a. check if a branch `repo-sync/public-to-private/<short-sha>` already exists or a PR with that head branch was previously created (idempotency guard)
    b. create a branch `repo-sync/public-to-private/<short-sha>` based on the top of the current stack (or `main` if no stack)
-   c. cherry-pick the commit, preserving author and message.  if the cherry-pick fails (rare -- caused by private-only code overlapping with the public commit's diff context), the workflow **fails loudly** and notifies oncall.  see [RUNBOOK.md](RUNBOOK.md) for remediation steps
+   c. cherry-pick the commit, preserving author and message.  if the cherry-pick fails (rare -- caused by private-only code overlapping with the public commit's diff context), create a **conflict PR** (see below)
    d. create a PR with the base set to the previous sync branch (or `main`)
 
 ### merge strategy
@@ -94,6 +106,7 @@ the workflow is serialized per PR via a concurrency group (`cancel-in-progress:
 
 **decision logic (in order):**
 1. **already handled?** if the PR has an existing approval OR a `Repo-Sync-Assigned` trailer → skip
+1b. **conflict PR?** if the PR has a `Repo-Sync-Conflict` trailer → skip.  these PRs unconditionally require human approval
 2. **commit count?** if the PR has ≠ 1 commit → skip.  the restack workflow handles restacking (either post-merge or via the `repo-sync:needs-restack` label, which can be added manually or by the escalation cron)
 3. **mergeable?** check via GitHub API (with retries for `UNKNOWN`):
    - `MERGEABLE` → approve + enable auto-merge (API-only, no git operations)

src/repo_sync/stack/trailers.py

@@ -15,6 +15,7 @@
 # Trailer prefixes.
 _ORIGIN_PREFIX = "Repo-Sync-Origin:"
 _ASSIGNED_PREFIX = "Repo-Sync-Assigned:"
+_CONFLICT_PREFIX = "Repo-Sync-Conflict:"
 
 
 @dataclass(frozen=True)
@@ -125,6 +126,22 @@ def format_origin_trailer(repo: str, sha: str) -> str:
     return f"{_ORIGIN_PREFIX} {repo}@{sha}"
 
 
+def parse_conflict(text: str) -> bool:
+    """Check if the text contains a Repo-Sync-Conflict trailer.
+
+    Returns True if the trailer is present.
+    """
+    for line in text.splitlines():
+        if line.strip().startswith(_CONFLICT_PREFIX):
+            return True
+    return False
+
+
+def format_conflict_trailer() -> str:
+    """Format a Repo-Sync-Conflict trailer line."""
+    return f"{_CONFLICT_PREFIX} cherry-pick"
+
+
 def format_assigned_trailer(username: str, timestamp: datetime) -> str:
     """Format a Repo-Sync-Assigned trailer line.
 

src/repo_sync/workflows/approve_logic.py

@@ -18,7 +18,7 @@
 
 from repo_sync.stack.gh_ops import GhOps
 from repo_sync.stack.git_ops import GitOps
-from repo_sync.stack.trailers import parse_origin
+from repo_sync.stack.trailers import parse_conflict, parse_origin
 from repo_sync.workflows.sync import determine_sync_reviewer
 
 logger = logging.getLogger(__name__)
@@ -251,6 +251,20 @@ def run_approve(
     if check_already_handled(gh, repo, pr_number):
         return
 
+    # Step 1b: Conflict trailer check.  PRs created by the sync workflow
+    # with a Repo-Sync-Conflict trailer unconditionally require human
+    # approval.  The approve bot must never approve these.
+    body = gh._run([
+        "pr", "view", str(pr_number), "--repo", gh.repo,
+        "--json", "body", "--jq", ".body",
+    ], check=False)
+    if body and parse_conflict(body):
+        logger.info(
+            "PR #%d has Repo-Sync-Conflict trailer. "
+            "Human approval required. Skipping.", pr_number,
+        )
+        return
+
     # Step 2: Commit count check.
     commit_count = check_commit_count(gh, pr_number)
     logger.info("PR #%d has %d commit(s).", pr_number, commit_count)

src/repo_sync/workflows/cli.py

@@ -253,6 +253,7 @@ def cmd_run_sync(args: argparse.Namespace) -> None:
             slack_webhook_url=args.slack_webhook_url,
             private_to_public_fixup_script=args.private_to_public_fixup_script,
             public_to_private_fixup_script=args.public_to_private_fixup_script,
+            escalate_to=args.escalate_to,
         )
     except PermanentSyncError as e:
         logging.error("%s", e)
@@ -426,6 +427,10 @@ def main() -> None:
         "--public-to-private-fixup-script", default="",
         help="Optional script to run after cherry-pick for public-to-private sync (not yet implemented).",
     )
+    p.add_argument(
+        "--escalate-to", default="@oncall-client-primary",
+        help="GitHub team or user to escalate to on timeout.",
+    )
     p.set_defaults(func=cmd_run_sync)
 
     # escalation-check.