HDDS-14868. refreshAndValidate ContainerSafemodeRule periodically, not on each applyTransaction

[sadanand48]

What changes were proposed in this pull request?

Periodic refresh — Run refresh on a ~5s (configurable) schedule instead of on every applyTransaction / refresh(false) path.

https://issues.apache.org/jira/browse/HDDS-14868

[szetszwo]

@sadanand48 , thanks for working on this!

How about refreshing the safemode rules every 5s, instead of doing it in applyTransactions?

[sadanand48]

How about refreshing the safemode rules every 5s, instead of doing it in applyTransactions?

Thanks @szetszwo for the input, we could make this behaviour configurable i.e periodic or based on applyTransaction. I'm saying because smaller clusters or cluster's without any pending logs may be impacted by redundant refresh calls.

[szetszwo]

... smaller clusters or cluster's without any pending logs may be impacted by redundant refresh calls.

Refreshing the safemode rules in applyTransaction actually is a big mistake -- applyTransaction is the critical path of the StateMachine, adding unnecessary operations there is going to slow down everything.

In contrast, refreshing the safemode rules every 5s is not going to have any measurable performance impact. Hypothetically, if refreshing every 5s is not okay, then refreshing it applyTransaction is definitely much worse since there are thousands of applyTransaction ops per second.

[szetszwo]

@sadanand48 , thanks for the update

• Since the current code in SCMStateMachine use SCMSafeModeManager to refresh, it is better to do refresh in SCMSafeModeManager.
• When refresh is enabled, SCMStateMachine should not refresh.
• During refreshing, if it is NOT in safemode, we can stop the executor. Then, we don't need any stop method.
• It is better to create a non-mock test using MiniOzoneCluster.

See https://issues.apache.org/jira/secure/attachment/13081501/9953_review.patch

[sadanand48]

Thanks @szetszwo for the review, updated as per your patch

it is better to do refresh in SCMSafeModeManager.

With this, all the safemode rules will have the same behaviour, I guess that should be okay. I will add a non-mock test

[szetszwo]

@sadanand48 , thanks for the update!

Quick question:

• Would it work if we don't make the changes in AbstractContainerSafeModeRule and other code logic changes such as isScmRatisApplyCaughtUpToCommit?

If it works, this PR should only change the refreshing time (i.e. periodic refreshing instead of doing it in SCMStateMachine.) Other code logic changes/improvement can be done in a separate PR.

[szetszwo]

Would it work if we don't make the changes in AbstractContainerSafeModeRule and other code logic changes such as isScmRatisApplyCaughtUpToCommit?

@sadanand48 , any thought?

[sadanand48]

Yes @szetszwo , it should work without other changes. The other change is only about isScmRatisApplyCaughtUpToCommit where we don't refresh if there are no new pending transactions. This is just an optimization . In the current revision of the patch I have removed this

[szetszwo]

@sadanand48 , thanks for the update. Please see the commnets inlined.

[szetszwo]

Just found that there is already a safeModeLogExecutor. We should just use it instead of creating a new ScheduledExecutorService.

@sadanand48 , very sorry that this idea actually is bad since the safeModeLogExecutor is new feature by HDDS-14012. Not sure if it is rock solid. Let don't use it. We may simply start a thread instead; see https://issues.apache.org/jira/secure/attachment/13081633/9953_review2.patch

[szetszwo]

Thanks for fixing the test.
+1

[sumitagrawl]

@sadanand48 Have few questions about the change,

1. Since refreshAndValidate() is triggered every 5 sec, do need have to handle onMessage() -> validate() for all safe mode rule to check and update rules every event ? IMO, other trigger is now un-necessary.

2. If so, SCM will have atleast 5 sec to come out in fresh startup also, right ?

[github-actions]

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

[github-actions]

Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.

[sadanand48]

I plan to merge this PR as it makes sense to run refresh at every 5s (or configurable interval) rather than per applyTransaction in theory.
We are not clubbing this thread with the Safemode Log monitor feature as 1. the interval for both these are different
2. as mentioned above the log monitor might not be tested

cc @szetszwo @sumitagrawl

patch updated

[sumitagrawl]

LGTM