Skip to content

Commit b002752

Browse files
committed
Fix broker stuck in SYNCHRONIZING on DB error during rollback
When a service broker update job fails and attempts to revert the broker state, a database connection failure could cause the job to crash without properly handling the original error. This left the broker stuck in SYNCHRONIZING state with a FAILED job. This change wraps the state rollback operation in error handling to catch database errors and allow the original exception to be raised and the job to be retried properly. Changes: - app/jobs/v3/services/update_broker_job.rb: Add error handling around ServiceBroker.where().update() call in rescue block to gracefully handle database disconnections during state rollback - spec/unit/jobs/v3/services/update_broker_job_spec.rb: Add test case for database disconnect during state rollback
1 parent b27da11 commit b002752

File tree

2 files changed

+35
-3
lines changed

2 files changed

+35
-3
lines changed

app/jobs/v3/services/update_broker_job.rb

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -66,13 +66,18 @@ def perform
6666

6767
@warnings
6868
rescue StandardError => e
69-
ServiceBroker.where(id: update_request.service_broker_id).update(state: previous_broker_state)
69+
begin
70+
ServiceBroker.where(id: update_request.service_broker_id).update(state: previous_broker_state) if update_request
71+
rescue StandardError
72+
# Swallow errors during state rollback (e.g., DB connection issues)
73+
# so the original exception can be raised and the job retried
74+
end
7075

7176
raise V3::ServiceBrokerUpdate::InvalidServiceBroker.new(e.message) if e.is_a?(Sequel::ValidationFailed)
7277

73-
raise e
78+
raise
7479
ensure
75-
update_request.destroy
80+
update_request.destroy if update_request
7681
end
7782

7883
private

spec/unit/jobs/v3/services/update_broker_job_spec.rb

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -446,6 +446,33 @@ module V3
446446
end
447447
end
448448

449+
context 'when database disconnects during state rollback' do
450+
let(:catalog_error) { StandardError.new('Catalog fetch failed') }
451+
452+
before do
453+
allow_any_instance_of(VCAP::CloudController::V3::ServiceBrokerCatalogUpdater).to receive(:refresh).and_raise(catalog_error)
454+
455+
mock_dataset = instance_double(Sequel::Postgres::Dataset)
456+
allow(mock_dataset).to receive(:update).and_raise(Sequel::DatabaseDisconnectError.new('connection lost'))
457+
458+
allow(ServiceBroker).to receive(:where).and_call_original
459+
allow(ServiceBroker).to receive(:where).with(id: broker.id).and_return(mock_dataset)
460+
end
461+
462+
it 'swallows the database error and re-raises the original catalog error' do
463+
expect { job.perform }.to raise_error(catalog_error)
464+
end
465+
466+
it 'does not raise a database connection error' do
467+
expect { job.perform }.not_to raise_error(Sequel::DatabaseDisconnectError)
468+
end
469+
470+
it 'still cleans up the update request' do
471+
expect { job.perform }.to raise_error(catalog_error)
472+
expect(ServiceBrokerUpdateRequest.where(id: update_broker_request.id).all).to be_empty
473+
end
474+
end
475+
449476
context 'when the broker ceases to exist during the job' do
450477
it 'raises a ServiceBrokerGone error' do
451478
broker.destroy

0 commit comments

Comments
 (0)