Detect failed merges and hold write protection#9783
Conversation
…IRED The code for the new error was added in 995be34 but the generated artifacts and formatter tests were not; this brings them in sync. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New durable status set by failed-merge detection; cleared only by recovery. Auto-exposed to the GraphQL BranchStatus enum. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Grace-period setting, a lock-token worker-id parser beside the token format, a read-only merge-lock holder lookup, and a multi-status branch filter — the building blocks the detector composes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Flip a branch stuck in MERGING whose merge-lock holder is a dead worker to MERGE_FAILED (after a grace period) and keep the default branch write-blocked with the recovery error. Runs from a recurring merge-watcher cron and a best-effort check at API-server startup; the write gate reads the shared protection key and returns MERGE_RECOVERY_REQUIRED while failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…hape Knowledge-base note for the MERGE_FAILED status and detection mechanism, and internal-api contract updated to the scan()/module-function shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2ec412e to
2db7d95
Compare
There was a problem hiding this comment.
3 issues found and verified against the latest diff
Confidence score: 3/5
- In
schema/error-catalogue.json, the SDKBranchStatusenum missingMERGE_FAILEDcan leave clients unable to represent or handle a backend-reported merge state, which risks broken consumer logic whenMERGE_RECOVERY_REQUIREDis returned — addMERGE_FAILEDto the SDK enum before merging. - In
backend/infrahub/branch/status_checker.py, the cache-fallback path usesBranch.get_listwithout a status filter or explicit limit, so branches beyond the default page size may bypass merge gating and allow writes during active/failed merges — tighten the fallback query (status + pagination/limit handling) before merge. - In
backend/tests/functional/merge/test_failed_merge_protection.py, inconsistent assertions across the three error-response blocks make it easier for regressions in error contract fields to slip through unnoticed — align the assertions so each path verifies the same key fields (message,code,http_status, anddata) for reliable protection.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="backend/tests/functional/merge/test_failed_merge_protection.py">
<violation number="1" location="backend/tests/functional/merge/test_failed_merge_protection.py:85">
P3: The three error-response assertion blocks in this test are inconsistent in what they verify. The default-branch gate checks `message`, `code` (`MERGE_RECOVERY_REQUIRED`), `http_status` (`423`), and `data`; the source-branch gate skips `http_status`; and the restart/post-recovery gate only checks `code`. A partial regression in `http_status`, `data`, or the error message after recovery would go undetected. Align all three gates to verify the same set of fields so the error contract is uniformly validated, especially after the `scan_for_failed_merges` recovery path.</violation>
</file>
<file name="schema/error-catalogue.json">
<violation number="1" location="schema/error-catalogue.json:304">
P1: SDK `BranchStatus` enum is missing `MERGE_FAILED`. The backend already has this value (`backend/infrahub/core/branch/enums.py:8`), and the new `MERGE_RECOVERY_REQUIRED` error code communicates this state to consumers. But the SDK's `BranchStatus` enum at `opsmill/infrahub-sdk-python/infrahub_sdk/branch.py:5` is missing it, so any `branch.get()` or `branch.all()` call that encounters a branch with `status: "MERGE_FAILED"` will crash with a Pydantic validation error. Add `MERGE_FAILED = "MERGE_FAILED"` to the SDK's `BranchStatus` enum so SDK consumers can query and handle failed-merge branches.</violation>
</file>
<file name="backend/infrahub/branch/status_checker.py">
<violation number="1" location="backend/infrahub/branch/status_checker.py:110">
P2: Cache-fallback merge gating can allow writes during an active/failed merge when branch count exceeds the default page size. The fallback query uses `Branch.get_list` without status filter or explicit limit, so it only inspects up to 1000 branches before deciding no block is needed.</violation>
</file>
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
| } | ||
| }, | ||
| "MERGE_RECOVERY_REQUIRED": { | ||
| "description": "The write was rejected because a previous branch merge failed and left the default branch protected. Recovery is required: an administrator must run `infrahub recover`. Unlike MERGE_IN_PROGRESS this is not retryable.", |
There was a problem hiding this comment.
P1: SDK BranchStatus enum is missing MERGE_FAILED. The backend already has this value (backend/infrahub/core/branch/enums.py:8), and the new MERGE_RECOVERY_REQUIRED error code communicates this state to consumers. But the SDK's BranchStatus enum at opsmill/infrahub-sdk-python/infrahub_sdk/branch.py:5 is missing it, so any branch.get() or branch.all() call that encounters a branch with status: "MERGE_FAILED" will crash with a Pydantic validation error. Add MERGE_FAILED = "MERGE_FAILED" to the SDK's BranchStatus enum so SDK consumers can query and handle failed-merge branches.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At schema/error-catalogue.json, line 304:
<comment>SDK `BranchStatus` enum is missing `MERGE_FAILED`. The backend already has this value (`backend/infrahub/core/branch/enums.py:8`), and the new `MERGE_RECOVERY_REQUIRED` error code communicates this state to consumers. But the SDK's `BranchStatus` enum at `opsmill/infrahub-sdk-python/infrahub_sdk/branch.py:5` is missing it, so any `branch.get()` or `branch.all()` call that encounters a branch with `status: "MERGE_FAILED"` will crash with a Pydantic validation error. Add `MERGE_FAILED = "MERGE_FAILED"` to the SDK's `BranchStatus` enum so SDK consumers can query and handle failed-merge branches.</comment>
<file context>
@@ -238,6 +238,92 @@
+ }
+ },
+ "MERGE_RECOVERY_REQUIRED": {
+ "description": "The write was rejected because a previous branch merge failed and left the default branch protected. Recovery is required: an administrator must run `infrahub recover`. Unlike MERGE_IN_PROGRESS this is not retryable.",
+ "stability": "evolving",
+ "http_status": 423,
</file context>
There was a problem hiding this comment.
added in the SDK and the submodule commit has been bumped to include it
- regenerate gql.tada types (graphql-env/graphql-cache) so the BranchStatus enum includes MERGE_FAILED, matching the graphql-codegen output - add MERGE_RECOVERY_REQUIRED to the frontend error-code exhaustiveness switch - re-wrap Timestamp in the predicate test so ty resolves the core type rather than the SDK base class Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
2 issues found across 34 files
Confidence score: 4/5
- In
docs/docs/reference/error-catalogue.mdx, the recovery guidance points operators toinfrahub recover, which is not present here, so on-call responders could follow a non-existent command during an incident and lose time; update the catalogue to the actual supported recovery path before merging. - In
dev/specs/ifc-2437-merge-failure-recovery/contracts/internal-api.md, the predicate text still depends onmerge_locker.get_merge_lock_holder_worker_id(cache)even thoughmerge_lockeris no longer injected, which leaves the contract internally inconsistent and can cause incorrect implementations; revise the spec to document the new source of lock-holder worker ID before merge.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="schema/error-catalogue.json">
<violation number="1" location="schema/error-catalogue.json:304">
P1: SDK `BranchStatus` enum is missing `MERGE_FAILED`. The backend already has this value (`backend/infrahub/core/branch/enums.py:8`), and the new `MERGE_RECOVERY_REQUIRED` error code communicates this state to consumers. But the SDK's `BranchStatus` enum at `opsmill/infrahub-sdk-python/infrahub_sdk/branch.py:5` is missing it, so any `branch.get()` or `branch.all()` call that encounters a branch with `status: "MERGE_FAILED"` will crash with a Pydantic validation error. Add `MERGE_FAILED = "MERGE_FAILED"` to the SDK's `BranchStatus` enum so SDK consumers can query and handle failed-merge branches.</violation>
</file>
<file name="docs/docs/reference/error-catalogue.mdx">
<violation number="1" location="docs/docs/reference/error-catalogue.mdx:289">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**
The error catalogue claims that operators should run `infrahub recover`, but that CLI command does not exist in this PR. The PR description explicitly states that "No recovery path included in this PR; the recovery flow lands in a follow-up." Documentation that describes behavior not yet implemented is misleading and should be deferred until the recovery command actually exists, or at minimum should not prescribe a specific command that does not exist.</violation>
</file>
<file name="dev/specs/ifc-2437-merge-failure-recovery/contracts/internal-api.md">
<violation number="1" location="dev/specs/ifc-2437-merge-failure-recovery/contracts/internal-api.md:193">
P2: The constructor no longer injects `merge_locker`, but the predicate description still references `merge_locker.get_merge_lock_holder_worker_id(cache)` to read the lock-holder worker id. The spec needs to clarify how the lock-holder token is now parsed — either the reading/parsing logic is folded into `MergeWriteBlocker` (which currently owns the `merge:protected` key, not the merge-lock token) or is handled through a standalone helper. As written, an implementer following this spec has no way to get the lock-holder worker id.</violation>
</file>
Shadow auto-approve: would not auto-approve because issues were found.
Re-trigger cubic
|
|
||
| ### MERGE_RECOVERY_REQUIRED | ||
|
|
||
| The write was rejected because a previous branch merge failed and left the default branch protected. Recovery is required: an administrator must run `infrahub recover`. Unlike MERGE_IN_PROGRESS this is not retryable. |
There was a problem hiding this comment.
P2: Custom agent: Flag AI Slop and Fabricated Changes
The error catalogue claims that operators should run infrahub recover, but that CLI command does not exist in this PR. The PR description explicitly states that "No recovery path included in this PR; the recovery flow lands in a follow-up." Documentation that describes behavior not yet implemented is misleading and should be deferred until the recovery command actually exists, or at minimum should not prescribe a specific command that does not exist.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At docs/docs/reference/error-catalogue.mdx, line 289:
<comment>The error catalogue claims that operators should run `infrahub recover`, but that CLI command does not exist in this PR. The PR description explicitly states that "No recovery path included in this PR; the recovery flow lands in a follow-up." Documentation that describes behavior not yet implemented is misleading and should be deferred until the recovery command actually exists, or at minimum should not prescribe a specific command that does not exist.</comment>
<file context>
@@ -284,6 +284,41 @@ The write was rejected because a branch merge is in progress. The block is trans
+### MERGE_RECOVERY_REQUIRED
+
+The write was rejected because a previous branch merge failed and left the default branch protected. Recovery is required: an administrator must run `infrahub recover`. Unlike MERGE_IN_PROGRESS this is not retryable.
+
+- **Stability**: `evolving`
</file context>
There was a problem hiding this comment.
it will be added in this release
There was a problem hiding this comment.
The parent comment still applies for this PR: the recovery command is still listed as a follow-up in the PR body, so the docs line is premature here.
| merge_locker: MergeLocker, # read-side helper: current merge-lock holder worker_id (or None) | ||
| cache: InfrahubCache, # read the merge-lock holder token | ||
| component: InfrahubComponent, # list_workers() -> active-worker set for the liveness check | ||
| merge_write_blocker: MergeWriteBlocker, # owns the merge:protected key (set/get/delete) |
There was a problem hiding this comment.
P2: The constructor no longer injects merge_locker, but the predicate description still references merge_locker.get_merge_lock_holder_worker_id(cache) to read the lock-holder worker id. The spec needs to clarify how the lock-holder token is now parsed — either the reading/parsing logic is folded into MergeWriteBlocker (which currently owns the merge:protected key, not the merge-lock token) or is handled through a standalone helper. As written, an implementer following this spec has no way to get the lock-holder worker id.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At dev/specs/ifc-2437-merge-failure-recovery/contracts/internal-api.md, line 193:
<comment>The constructor no longer injects `merge_locker`, but the predicate description still references `merge_locker.get_merge_lock_holder_worker_id(cache)` to read the lock-holder worker id. The spec needs to clarify how the lock-holder token is now parsed — either the reading/parsing logic is folded into `MergeWriteBlocker` (which currently owns the `merge:protected` key, not the merge-lock token) or is handled through a standalone helper. As written, an implementer following this spec has no way to get the lock-holder worker id.</comment>
<file context>
@@ -179,42 +179,48 @@ deleted/reopened; the `updated_at >= merge_at` filter excludes already-restored
- merge_locker: MergeLocker, # read-side helper: current merge-lock holder worker_id (or None)
+ cache: InfrahubCache, # read the merge-lock holder token
+ component: InfrahubComponent, # list_workers() -> active-worker set for the liveness check
+ merge_write_blocker: MergeWriteBlocker, # owns the merge:protected key (set/get/delete)
+ # recovery (PR-5) additionally injects: diff_merger: DiffMerger (the range rollback query)
) -> None: ...
</file context>
There was a problem hiding this comment.
I'll clean the spec up for the next PR in the series
- regenerate graphql-cache.d.ts and update .betterer.results after a clean pnpm install so gql.tada ordering and tracked-error snapshots match CI (the added MERGE_FAILED value grew two existing tracked-error messages) - status_checker DB fallback: filter branches by status server-side rather than scanning the default page, so a merging/failed branch is never missed when the cache is unreachable and there are many branches - functional test: assert the full recovery-error contract uniformly across all three gates Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 4 files (changes from recent commits).
Shadow auto-approve: would not auto-approve. Auto-approval blocked by 3 unresolved issues from previous reviews.
Re-trigger cubic
Adds INFRAHUB_MERGE_FAILURE_GRACE_PERIOD_SECONDS to the generated env-var block so validate-dockercomposeenv passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Shadow auto-approve: would not auto-approve. Auto-approval blocked by 3 unresolved issues from previous reviews.
Re-trigger cubic
The new merge_failure_grace_period_seconds config field is surfaced in the OpenAPI schema and the configuration reference doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 2 files (changes from recent commits).
Shadow auto-approve: would not auto-approve. Auto-approval blocked by 3 unresolved issues from previous reviews.
Re-trigger cubic
Cascade from the openapi.json change so frontend-validate-openapi-types passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- set a ~1s merge-failure grace period in the test docker-compose stack so the merge-watcher scan flips a dead merge quickly instead of after ~3 minutes - wait until the merge is actually MERGING before SIGKILLing the worker, so the kill deterministically lands mid-merge, then poll for MERGE_FAILED well within the 300s pytest timeout (the old 360s poll could never finish) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
1 issue found across 2 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="schema/error-catalogue.json">
<violation number="1" location="schema/error-catalogue.json:304">
P1: SDK `BranchStatus` enum is missing `MERGE_FAILED`. The backend already has this value (`backend/infrahub/core/branch/enums.py:8`), and the new `MERGE_RECOVERY_REQUIRED` error code communicates this state to consumers. But the SDK's `BranchStatus` enum at `opsmill/infrahub-sdk-python/infrahub_sdk/branch.py:5` is missing it, so any `branch.get()` or `branch.all()` call that encounters a branch with `status: "MERGE_FAILED"` will crash with a Pydantic validation error. Add `MERGE_FAILED = "MERGE_FAILED"` to the SDK's `BranchStatus` enum so SDK consumers can query and handle failed-merge branches.</violation>
</file>
<file name="docs/docs/reference/error-catalogue.mdx">
<violation number="1" location="docs/docs/reference/error-catalogue.mdx:289">
P2: Custom agent: **Flag AI Slop and Fabricated Changes**
The error catalogue claims that operators should run `infrahub recover`, but that CLI command does not exist in this PR. The PR description explicitly states that "No recovery path included in this PR; the recovery flow lands in a follow-up." Documentation that describes behavior not yet implemented is misleading and should be deferred until the recovery command actually exists, or at minimum should not prescribe a specific command that does not exist.</violation>
</file>
<file name="dev/specs/ifc-2437-merge-failure-recovery/contracts/internal-api.md">
<violation number="1" location="dev/specs/ifc-2437-merge-failure-recovery/contracts/internal-api.md:193">
P2: The constructor no longer injects `merge_locker`, but the predicate description still references `merge_locker.get_merge_lock_holder_worker_id(cache)` to read the lock-holder worker id. The spec needs to clarify how the lock-holder token is now parsed — either the reading/parsing logic is folded into `MergeWriteBlocker` (which currently owns the `merge:protected` key, not the merge-lock token) or is handled through a standalone helper. As written, an implementer following this spec has no way to get the lock-holder worker id.</violation>
</file>
<file name="backend/tests/integration_docker/test_merge_kill_recovery.py">
<violation number="1" location="backend/tests/integration_docker/test_merge_kill_recovery.py:60">
P2: The merge mutation fires as a background task via `asyncio.create_task`, but its potential exception is never collected. If the GraphQL call raises an error for any reason unrelated to the SIGKILL (API timeout, schema validation, transient network issue), Python will emit a "Task exception was never retrieved" warning at garbage-collection time and the root cause will be silently lost, potentially masking CI flakiness or real failures. Consider short-circuiting the exception retrieval in the `finally` block so the test makes the failure explicit rather than letting it go unobserved.</violation>
</file>
Shadow auto-approve: would not auto-approve because issues were found.
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Points at the SDK commit adding MERGE_FAILED so SDK clients (and the integration test's branch.all()) can read a MERGE_FAILED branch without a pydantic validation error. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Shadow auto-approve: would not auto-approve. Auto-approval blocked by 4 unresolved issues from previous reviews.
Re-trigger cubic
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Shadow auto-approve: would not auto-approve. Auto-approval blocked by 3 unresolved issues from previous reviews.
Re-trigger cubic
There was a problem hiding this comment.
0 issues found across 9 files (changes from recent commits).
Shadow auto-approve: would not auto-approve. Auto-approval blocked by 2 unresolved issues from previous reviews.
Re-trigger cubic
There was a problem hiding this comment.
0 issues found across 1 file (changes from recent commits).
Shadow auto-approve: would not auto-approve. Auto-approval blocked by 2 unresolved issues from previous reviews.
Re-trigger cubic
| Deliberately a sibling of MergeInProgressError, not a subclass: the error-catalogue resolver | ||
| walks the MRO to map an exception to a code, so subclassing would make this resolve to the | ||
| transient MERGE_IN_PROGRESS code and be indistinguishable from a retryable in-progress merge. | ||
| This is the durable, non-retryable signal that an administrator must run `infrahub recover`. |
There was a problem hiding this comment.
I am not sure to understand this comment. I think we just remove the constructor from MergeRecoveryRequiredError and if we make it inherit from MergeInProgressError, the unit test test_merge_recovery_required_resolves_to_distinct_code passes.
If a code exists inside the catalogue for the subtype, it returns it. If not, it looks for parent types.
There was a problem hiding this comment.
yes I agree. this docstring has been updated. looks like you were reviewing older code
| if self.branch_filters.status: | ||
| param_name = "filter_status" | ||
| self._branch_filter_params[param_name] = self.branch_filters.status.value | ||
| conditions.append(f"n.status = ${param_name}") | ||
|
|
||
| if self.branch_filters.statuses: | ||
| param_name = "filter_statuses" | ||
| self._branch_filter_params[param_name] = [status.value for status in self.branch_filters.statuses] | ||
| conditions.append(f"n.status IN ${param_name}") | ||
|
|
There was a problem hiding this comment.
Is it worth to keep both of these filters? Does it make a difference in Cypher to use the IN for only asserting against one status?
There was a problem hiding this comment.
I kept it for backwards compatibility. I didn't think it was worth it to search for and update uses of status vs statuses
| def is_failed_merge( | ||
| self, | ||
| *, | ||
| status: BranchStatus, | ||
| lock_holder_worker_id: str | None, | ||
| active_worker_ids: set[str], | ||
| merge_started_at: Timestamp | None, | ||
| now: Timestamp, | ||
| grace_period_seconds: int, | ||
| ) -> bool: |
There was a problem hiding this comment.
As this is a component isolated from the rest, this is not really important, but having a method named is_failed_merge which does not consider the case when a branch comes in with a status of MERGE_FAILED looks confusing at first glance.
Based on the current usage inside the code, a branch can only come in with MERGING status and this is the first condition of the method, so I guess it's fine.
I get that this is to determine if a MERGING branch should transition to MERGE_FAILED status but I think there should be a dedicated comment explaining why MERGE_FAILED comes out as false or maybe the method's name is not totally accurate.
Maybe I'm also missing something.
There was a problem hiding this comment.
renamed to should_mark_as_failed_merge
| if not self.is_failed_merge( | ||
| status=branch.status, | ||
| lock_holder_worker_id=lock_holder_worker_id, | ||
| active_worker_ids=active_worker_ids, | ||
| merge_started_at=merge_started_at, | ||
| now=now, | ||
| grace_period_seconds=grace_period_seconds, | ||
| ): |
There was a problem hiding this comment.
If the upper comment makes sense around is_failed_merge there would be an impact over here
| ) | ||
|
|
||
| @pytest.fixture | ||
| async def merging_branch(self, db: InfrahubDatabase, default_branch: Branch) -> Branch: |
There was a problem hiding this comment.
It looks like this fixture should be named differently since this is a merging branch that specifically started 10 minutes+ ago. In the test_within_grace_is_not_flagged, there is also a merging branch created.
There was a problem hiding this comment.
updated to merging_branch_for_10m
| backend_branch.status = BranchStatus.OPEN | ||
| await backend_branch.save(db=db) | ||
|
|
||
| async def test_failed_merge_blocks_with_recovery_code( |
There was a problem hiding this comment.
Nice test, it clearly illustrates the different steps of the workflow
There was a problem hiding this comment.
Just thinking out loud here, but shouldn't we keep such long tests from running in every pipeline? Maybe we should have a test suite that we run once per release, or something like that?
Based on my tests, this test takes approximately 90–100s, which impacts the integration_docker test suite. That's also the longest test suite in the current CI since the component tests were split out.
This gives a lof of confidence, but also costs a lot in my opinion.
There was a problem hiding this comment.
yes this is a good point. I'll mark as skip for now and make a follow-up item to enable it for release testing only
https://opsmill.atlassian.net/browse/IFC-2878
There was a problem hiding this comment.
0 issues found across 4 files (changes from recent commits).
Shadow auto-approve: would not auto-approve. Auto-approval blocked by 2 unresolved issues from previous reviews.
Re-trigger cubic
Why
When a task worker is killed mid-merge (e.g. SIGKILL), the source branch is left stuck in
MERGINGwith a partially-merged graph and the global merge lock still held by the dead worker. Nothing detected this state, so the default-branch write-protection either lingered indefinitely or — worse — writes could resume against a half-merged default branch, and users got no signal that anything was wrong.Goal: deterministically detect a dead merge, record it durably as
MERGE_FAILED, and keep the default branch write-blocked with a distinct, non-retryable error that tells the operator recovery is required — while never mis-flagging a healthy in-progress merge.Non-goals (follow-ups):
infrahub recover/ range rollback / branch + proposed-change reset (US3, separate PR).Part of the merge-failure-recovery epic IFC-2559; this is the US2 (detection) slice of umbrella bug IFC-2437 and advances IFC-2565 (surface a crashed-merge error to the operator). It does not close IFC-2437 — recovery lands in a follow-up.
Closes IFC-2565
What changed
Behavioral changes (what you'll observe differently):
MERGE_FAILED, auto-exposed in the GraphQLBranchStatusenum. It is set only by detection and cleared only by recovery (not yet in this PR).MERGE_WATCHERworkflow (1-minute cron, single-flighted across workers) plus a best-effort check at API-server startup flip a branch that is stuck inMERGINGwhose merge-lock holder is no longer a live worker (after a grace period) toMERGE_FAILED, and hold the shared write-protection.MERGE_RECOVERY_REQUIREDGraphQL error code (HTTP 423) with a message naminginfrahub recoverand "contact an administrator" — separate from the transientMERGE_IN_PROGRESSused during a healthy merge. Payload:{branch_name, merging_branch}.INFRAHUB_MERGE_FAILURE_GRACE_PERIOD_SECONDS(default180).Implementation notes:
{timestamp}::{worker_id}) compared againstcomponent.list_workers()' active set — the same pattern asclean_up_deadlocks. The decision is a side-effect-free predicate (is_failed_merge) that is unit-tested without a DB; the grace period is injected at construction.MergeFailureIdentifier.scan()runs detection then reconciliation in one pass. Reconciliation re-aligns the volatilemerge:protectedcache key with the durableMERGING/MERGE_FAILEDbranch status, so protection self-heals after a restart or cache flush.MergeRecoveryRequiredErroris deliberately a sibling ofMergeInProgressError, not a subclass: the two states demand opposite reactions (retry vs. run recovery), and subclassing would let any handler written for the retryable error silently absorb the durable one.lock.pybeside the token format (aget_worker_id_from_lock_tokenhelper), and a multi-status filter (BranchListFilters.statuses→n.status IN …) lets reconcile query protected branches in one round-trip.python_sdksubmodule is bumped so SDK clients can deserialize aMERGE_FAILEDbranch without a validation error.What stayed the same: no database migration; the healthy-merge happy path is unchanged; the recovery flow is out of scope.
Suggested review order
945f7a18c+421f3828b— theMergeRecoveryRequiredErrorexception and regenerated error-catalogue bindings/tests forMERGE_RECOVERY_REQUIRED.cbff3a97c— add theMERGE_FAILEDstatus (+ GraphQL schema regen).426833218— detection primitives (grace config, lock-token parser, merge-lock read, multi-status filter).4ce36eba2— the detector, watcher cron, startup scan, and the write gate (the meat + tests).2db7d9548— docs + spec alignment.python_sdksubmodule bump, hardening of the SIGKILL integration test, and post-review refactors (rename toMergeFailureIdentifier/failure_identifier.py, grace-period constructor injection, docstring corrections).How to review
backend/infrahub/core/merge/failure_identifier.py(the predicate +scan), andbackend/infrahub/branch/status_checker.py(theMERGE_FAILEDgate raisingMergeRecoveryRequiredError).MergeRecoveryRequiredError.backend/tests/integration_docker/test_merge_kill_recovery.pySIGKILLs a task worker mid-merge against a real docker stack (with a ~1s grace period in the test compose file) and asserts the idle scan flips the branch toMERGE_FAILED. The write-gate contract test lives inbackend/tests/integration/merge/test_failed_merge_protection.py.schema/schema.graphql,schema/error-catalogue.json,schema/openapi.json,frontend/app/src/shared/api/**/generated,frontend/app/src/shared/api/rest/types.generated.ts,docs/docs/reference/error-catalogue.mdx,docs/docs/reference/configuration.mdx, regenerated docker-compose env vars.Impact & rollout
MERGE_FAILEDis a newBranchStatusenum value andMERGE_RECOVERY_REQUIREDa new catalogue code (markedevolving). Clients that exhaustively switch on either enum should handle the new value. No migration.infrahub recover) is not in this PR. Until the recovery PR lands, a branch that reachesMERGE_FAILEDstays write-blocked with no in-product way to clear it (manual DB intervention only). Land the recovery follow-up close behind this.GETper top-level mutation). Detection adds one parameterized branch query + one cache read per worker per minute.INFRAHUB_MERGE_FAILURE_GRACE_PERIOD_SECONDS(default 180).MERGE_WATCHERcron deployment is created automatically once the workflow definition is present.Checklist
MERGE_RECOVERY_REQUIRED; configuration reference reflects the new grace-period setting)dev/knowledge/backend/branch-status.md,dev/specs/ifc-2437-merge-failure-recovery/*)Summary by cubic
Detects and guards against failed merges to keep the default branch consistent. Adds
MERGE_FAILEDwith a distinctMERGE_RECOVERY_REQUIRED(HTTP 423) response, runs a 1‑min single‑flightMERGE_WATCHERplus an API‑server startup scan, and exposes a grace‑period setting (IFC-2437).New Features
MERGINGtoMERGE_FAILEDwhen the merge‑lock holder is dead and past grace; never flag a healthy merge. Hold shared write protection and reconcile themerge:protectedkey with durable status so it self‑heals after restarts or cache flushes. Runs viaMERGE_WATCHER(every minute, single‑flight) and on API‑server startup.MergeInProgressErrorforMERGINGandMergeRecoveryRequiredError(codeMERGE_RECOVERY_REQUIRED, HTTP 423) forMERGE_FAILEDwith payload{branch_name, merging_branch}; applies to the source and default branches. Cache‑down fallback enforces from DB and prioritizesMERGE_FAILED.BranchStatus.MERGE_FAILED(GraphQL/SDK); server‑side multi‑status filter (statuses) for safe DB fallback; lock‑token worker‑id parser and read‑only merge‑lock holder lookup. Error catalogue/formatter updated; frontend error bindings and GraphQL/OpenAPI/REST types regenerated; SDK submodule bumped. Tests include unit, component, functional, and a skipped-by-default SIGKILL integration.Migration
MERGE_FAILEDandMERGE_RECOVERY_REQUIRED.merge_failure_grace_period_seconds(default 180), exposed asINFRAHUB_MERGE_FAILURE_GRACE_PERIOD_SECONDS.MERGE_WATCHERworkflow.Written for commit f55ee89. Summary will update on new commits.