Skip to content

fix(transfer): Prevent JobPollingService thread crash on stale job claims#1491

Merged
yukthagaduputi merged 1 commit into
dtinit:masterfrom
yukthagaduputi:fix-job-polling-crash
May 19, 2026
Merged

fix(transfer): Prevent JobPollingService thread crash on stale job claims#1491
yukthagaduputi merged 1 commit into
dtinit:masterfrom
yukthagaduputi:fix-job-polling-crash

Conversation

@yukthagaduputi

Copy link
Copy Markdown
Collaborator

This PR fixes a critical process-terminating thread crash in the JobPollingService loop that occurs when a transfer worker attempts to claim a job that has already been claimed and has credentials stored by another worker instance.

Problem
In a distributed, highly concurrent deployment with multiple worker instances, a worker can poll a job ID from an eventually consistent database index that looks free (CREDS_AVAILABLE) but has actually already been claimed by a faster peer instance.

When our worker performs a strongly consistent read (store.findJob) to retrieve the job details, it detects the instanceId mismatch (indicating the job belongs to another worker) and marks the job's state as CANCELED in memory.

However, in the old implementation:

  1. JobPollingService.tryToClaimJob failed to verify whether the retrieved existingJob was null or canceled, proceeding blindly to build the updatedJob to claim it.
  2. Constructing this job object to transition to state CREDS_ENCRYPTION_KEY_GENERATED threw a validation exception (IllegalStateException) because credentials (encryptedAuthData) had already been set by the peer worker.
  3. Because the PortabilityJob builder execution occurred outside the main try-catch block in tryToClaimJob, this exception was uncaught.
  4. This uncaught thread failure propagated up, terminating the periodic JobPollingService thread and causing the entire container sandbox to crash.

Solution
Implemented two layers of safety in JobPollingService.java to resolve this:

  1. Proactive Prevention (Early Abort): Added null and CANCELED state checks immediately after retrieving the job from the JobStore. If the job has been deleted or marked canceled (which happens on instanceId mismatch), the worker aborts the claim attempt early and returns false safely.
  2. Reactive Safety (Validation Safety Net): Wrapped the PortabilityJob builder execution in a local try-catch block to safely handle any unexpected IllegalStateException validation errors during object construction, returning false (handled failure) instead of propagating and crashing the thread.

These changes ensure that failing to claim a job (due to losing a race) is treated as a handled, temporary failure, allowing the worker to complete the current polling iteration normally and try again in the next cycle instead of crashing.

@CLAassistant

CLAassistant commented May 14, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@yukthagaduputi yukthagaduputi requested a review from lisad May 15, 2026 02:24

@lisad lisad left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks good for robustness

@yukthagaduputi yukthagaduputi merged commit a384d58 into dtinit:master May 19, 2026
6 checks passed
ameya9 pushed a commit to ameya9/data-transfer-project that referenced this pull request Jun 22, 2026
…aims (dtinit#1491)

This PR fixes a critical process-terminating thread crash in the
JobPollingService loop that occurs when a transfer worker attempts to
claim a job that has already been claimed and has credentials stored by
another worker instance.

**Problem**
In a distributed, highly concurrent deployment with multiple worker
instances, a worker can poll a job ID from an eventually consistent
database index that looks free (CREDS_AVAILABLE) but has actually
already been claimed by a faster peer instance.

When our worker performs a strongly consistent read (store.findJob) to
retrieve the job details, it detects the instanceId mismatch (indicating
the job belongs to another worker) and marks the job's state as CANCELED
in memory.

However, in the old implementation:

1. JobPollingService.tryToClaimJob failed to verify whether the
retrieved existingJob was null or canceled, proceeding blindly to build
the updatedJob to claim it.
2. Constructing this job object to transition to state
CREDS_ENCRYPTION_KEY_GENERATED threw a validation exception
(IllegalStateException) because credentials (encryptedAuthData) had
already been set by the peer worker.
3. Because the PortabilityJob builder execution occurred outside the
main try-catch block in tryToClaimJob, this exception was uncaught.
4. This uncaught thread failure propagated up, terminating the periodic
JobPollingService thread and causing the entire container sandbox to
crash.

**Solution**
Implemented two layers of safety in JobPollingService.java to resolve
this:

1. Proactive Prevention (Early Abort): Added null and CANCELED state
checks immediately after retrieving the job from the JobStore. If the
job has been deleted or marked canceled (which happens on instanceId
mismatch), the worker aborts the claim attempt early and returns false
safely.
2. Reactive Safety (Validation Safety Net): Wrapped the PortabilityJob
builder execution in a local try-catch block to safely handle any
unexpected IllegalStateException validation errors during object
construction, returning false (handled failure) instead of propagating
and crashing the thread.

These changes ensure that failing to claim a job (due to losing a race)
is treated as a handled, temporary failure, allowing the worker to
complete the current polling iteration normally and try again in the
next cycle instead of crashing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants