Fix broker stuck in SYNCHRONIZING on DB error during rollback

[kathap]

When a service broker update or create job fails, a database connection failure during state rollback could leave the broker permanently stuck in SYNCHRONIZING state with a FAILED job. This PR adds comprehensive error handling and a recovery hook to ensure brokers are marked as SYNCHRONIZATION_FAILED even when database outages occur during job failure.

Changes:

a) app/jobs/v3/services/update_broker_job.rb:

• Add recover_from_failure hook to set broker to SYNCHRONIZATION_FAILED when job transitions to FAILED
• Refactor rescue block to use rollback_broker_state and destroy_update_request helper methods with graceful error handling
• Add conditional WHERE clause (state: SYNCHRONIZING) to prevent overwriting newer broker states

b) app/jobs/v3/services/synchronize_broker_catalog_job.rb:

• Add recover_from_failure hook to set broker to SYNCHRONIZATION_FAILED when job transitions to FAILED
• Add conditional WHERE clause (state: SYNCHRONIZING) to prevent overwriting newer broker states

c) app/jobs/wrapping_job.rb:

• Add delegation for recover_from_failure hook to allow all wrappers to invoke recovery methods

d) app/jobs/pollable_job_wrapper.rb:

• Invoke recover_from_failure hook in failure method on the handler (delegated through WrappingJob)
• Log recovery failures without raising to prevent cascading errors

e) spec/unit/jobs/v3/services/update_broker_job_spec.rb:

• Add test for database disconnect during state rollback
• Add comprehensive tests for recover_from_failure method covering normal recovery, idempotency, state protection, and error handling

f) spec/unit/jobs/pollable_job_wrapper_spec.rb:

• Add integration test verifying recovery hook is called on job failure

g) spec/unit/jobs/wrapping_job_spec.rb:

• Add tests for recover_from_failure delegation through wrapper layers

A short explanation of the proposed change:

This PR adds multi-layered error handling for broker state management during job failures:

1. Immediate rollback: Best-effort state revert in the job's rescue block (extracted to rollback_broker_state helper for updates, inline for creates)
2. Failure hook: recover_from_failure method called when job exhausts retries and transitions to FAILED
3. Conditional updates: WHERE clause ensures only SYNCHRONIZING brokers are affected, protecting against overwriting newer states

An explanation of the use cases your change solves:

This change solves a critical production issue where service brokers become permanently stuck in SYNCHRONIZING state. This occurs when:

1. A service broker update or create job is executing and encounters an error (e.g., invalid catalog, validation failure)
2. The job's error handling attempts to set the broker state to SYNCHRONIZATION_FAILED
3. A database connection failure occurs during this state update operation
4. The job exhausts retries while the database remains unavailable

Without this fix: The broker remains stuck in SYNCHRONIZING state with a FAILED job, requiring manual intervention

With this fix: When the job transitions to FAILED, the recover_from_failure hook is invoked by PollableJobWrapper, which attempts to set the broker to SYNCHRONIZATION_FAILED. The conditional WHERE clause prevents overwriting any newer state that may have been set, ensuring safe recovery.

• I have reviewed the contributing guide

• I have viewed, signed, and submitted the Contributor License Agreement

• I have made this pull request to the main branch

• I have run all the unit tests using bundle exec rake

• I have run CF Acceptance Tests

[johha]

LGTM