Skip to content

HDDS-15103. OM termination due to race condition between om start and install snapshot#10118

Merged
ChenSammi merged 3 commits into
apache:masterfrom
ChenSammi:HDDS-15103
Apr 27, 2026
Merged

HDDS-15103. OM termination due to race condition between om start and install snapshot#10118
ChenSammi merged 3 commits into
apache:masterfrom
ChenSammi:HDDS-15103

Conversation

@ChenSammi

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

fix the race condition problem between om start and install snapshot

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15103

How was this patch tested?

new unit test

@ChenSammi ChenSammi changed the title HDDS-15103. OM crash due to race condition between om start and install snapshot HDDS-15103. OM termination due to race condition between om start and install snapshot Apr 24, 2026

@adoroszlai adoroszlai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ChenSammi for the patch. The fix looks good, but I don't think the test case is useful or needed. It basically tests (1) behavior with the test-specific flag and (2) normal operation in running state, which is also exercised by other test cases. It does not actually test the real scenario, where OM is not yet running. I understand that testing this for real would require more complex instrumentation, and am not asking for such a test. But I think the current test is unnecessary (both code and execution time) and may even be flaky.

@ChenSammi

ChenSammi commented Apr 24, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @ChenSammi for the patch. The fix looks good, but I don't think the test case is useful or needed. It basically tests (1) behavior with the test-specific flag and (2) normal operation in running state, which is also exercised by other test cases. It does not actually test the real scenario, where OM is not yet running. I understand that testing this for real would require more complex instrumentation, and am not asking for such a test. But I think the current test is unnecessary (both code and execution time) and may even be flaky.

@adoroszlai , the unit test is mainly to test that OM can continue and finish the snapshot installation in case it returns "null" for the first a few installSnapshot requests. I want to make sure the new change will not fail the snapshot installation completely.

Or maybe I can add the case into an existing unit test, so it will not introduce new execution time.

@adoroszlai adoroszlai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @ChenSammi for updating the patch.

Comment on lines 879 to +880
cluster.startInactiveOM(followerNodeId);
OzoneManager.setTestInstallSnapshot(true);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there may be a race condition here, but we can fix it later (by swapping these two statements) if it turns out to be flaky.

@ChenSammi ChenSammi merged commit 86f3df1 into apache:master Apr 27, 2026
45 checks passed
@ChenSammi

Copy link
Copy Markdown
Contributor Author

Thanks @adoroszlai @sadanand48 for the review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants