HDDS-12791. Make Retention Service suspendable by xichen01 · Pull Request #9426 · apache/ozone

xichen01 · 2025-12-04T06:49:16Z

What changes were proposed in this pull request?

This PR makes the KeyLifecycleService suspendable at runtime without requiring an OM restart.

Make ozone.lifecycle.service.enabled reconfigurable, the KeyLifecycleService can be stopped by reconfiguring this configuration.
Add KeyLifecycleService status query command
Example:

$ ozone admin om lifecycle status --service-id <omServiceId>
========================================
          Lifecycle Service Status
========================================
IsEnabled: true
IsSuspended: false
Running Buckets:
  - /vol1/bucket1
========================================

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12791

How was this patch tested?

unit test.

Copilot

Pull request overview

This PR makes the KeyLifecycleService suspendable at runtime without requiring an OM restart, adding the ability to dynamically enable/disable the service and query its status.

Makes ozone.lifecycle.service.enabled reconfigurable so the service can be stopped/started via reconfiguration
Adds a new ozone admin om lifecycle status CLI command to query the service state
Implements additional shouldRun() checks in the service to allow graceful suspension during bucket processing

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
hadoop-ozone/interface-client/src/main/proto/OmClientProtocol.proto	Adds `GetLifecycleServiceStatus` request/response proto messages and type enum
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/OmUtils.java	Marks `GetLifecycleServiceStatus` as a read-only operation
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocol/OzoneManagerProtocol.java	Adds `getLifecycleServiceStatus()` method to the OM protocol interface
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/OzoneManagerProtocolClientSideTranslatorPB.java	Implements client-side translation for the new `getLifecycleServiceStatus()` RPC
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java	Adds request handling for `GetLifecycleServiceStatus` command type
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java	Implements `getLifecycleServiceStatus()` and adds reconfiguration handler for lifecycle service enabled flag
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/service/KeyLifecycleService.java	Changes `isServiceEnabled` to `AtomicBoolean`, adds `setServiceEnabled()` and `status()` methods, and adds runtime suspension checks
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om/OMAdmin.java	Registers the new `LifecycleSubCommand`
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om/LifecycleSubCommand.java	New subcommand for lifecycle-related admin operations
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om/LifecycleStatusSubCommand.java	Implements the status query CLI command
hadoop-ozone/tools/src/main/java/org/apache/hadoop/ozone/admin/om/FinalizationStatusSubCommand.java	Refactored to use try-with-resources for proper client cleanup
hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/service/TestKeyLifecycleService.java	Adds test case for lifecycle service status query functionality
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/shell/TestOzoneShellHA.java	Adds integration test for the lifecycle status CLI command
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/reconfig/TestOmReconfiguration.java	Adds lifecycle service enabled configuration to reconfigurable properties test

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ChenSammi · 2025-12-10T06:37:16Z

  }

+  private String reconfKeyLifecycleServiceEnabled(String newVal) {
+    boolean enabled = StringUtils.isEmpty(newVal) ?


Can we throw exception if newVal is null? I think "null" is an invalid input, so it's better response to user with correct info. Will the message in exception be passed to client and show as the output of CLI?

ChenSammi · 2025-12-10T06:37:55Z

+      LOG.warn("Failed reconfiguration for '{}'. KeyLifecycleService is not initialized.",
+          OZONE_KEY_LIFECYCLE_SERVICE_ENABLED);
+    }
+    getConfiguration().setBoolean(OZONE_KEY_LIFECYCLE_SERVICE_ENABLED, enabled);


Let's change the configuration first, and then update the service state.

ChenSammi · 2025-12-10T06:46:08Z

-    UpgradeFinalization.StatusAndMessages progress =
-        client.queryUpgradeFinalizationProgress(upgradeClientID, false, true);
-    System.out.println(progress.status());
+    try (OzoneManagerProtocol client = parent.createOmClient(omServiceId, omHost, false)) {


Is this a relevant change ?

ChenSammi · 2025-12-10T07:04:22Z

+ * Handler of ozone admin status command.
+ */
+@Command(
+    name = "status",


We have new CLI to query the service status, and leverage the reconfiguration framework for the service start/stop, can we use consolidate to one entry, for example, new CLI for both start/stop/status? Current both status CLI and reconfiguration with both ozone-site.xml content change and reconfiguration CLI, feels a little complicated for user.

Added suspend subcommand.

ChenSammi · 2025-12-10T07:06:12Z

               keyTable.iterator(omMetadataManager.getBucketKey(volumeName, bucketName))) {
        while (keyTblItr.hasNext()) {
+          if (!shouldRun()) {
+            return;


Can we add a log message here too about the abort of bucket key evaluation?

ChenSammi · 2025-12-10T07:08:34Z

          return result;
        }

+        if (!shouldRun()) {


When it comes here, the key evaluation is finished, the rest of work is sending the expired remaining key to delete/rename. I think we can let this run to finish, instead of return here.

The most time consuming part of DB key evaluation iteration, send keys to delete/rename doesn't take much time.

ChenSammi · 2025-12-10T07:11:30Z

+    Set<String> runningBuckets = new HashSet<>(inFlight.keySet());
+    return GetLifecycleServiceStatusResponse.newBuilder()
+        .setIsRunning(!runningBuckets.isEmpty())
+        .setIsEnabled(isServiceEnabled.get())


Can we combine the IsRunning and IsEnabled to one state. If IsRuning is the result of whether there is bucket task running or not, this info is already covered by RunningBuckets fields.

Removed the setIsRunning.

ChenSammi · 2025-12-18T06:27:11Z

+   * Set isServiceEnabled.
+   * @param enabled whether enable the lifecycle Service
+   */
+  public void setServiceEnabled(boolean enabled) {


Can we reuse the existing suspended variable, and suspend(), resume() function, instead of this new setServiceEnabled() function, as the current suspended variable and it involved function is actually not used?

I reused the suspended variable and the the isServiceEnabled only based on the Configuration, cannot modify by user

ChenSammi · 2025-12-18T06:29:41Z

+  repeated string runningBuckets = 2;
+}
+
+message SuspendLifecycleServiceRequest {


Can we add an option in the SuspendLifecycleServiceRequest so that we can resume the service once suspend is no longer needed?

Add a resume command. new we can suspend and resume the lifecycle service

ChenSammi · 2025-12-18T06:30:45Z

+}
+
+message GetLifecycleServiceStatusResponse {
+  required bool isEnabled = 1;


Can we add one more state, to indicate that service is suspened/paused or not?

Added the suspened flag

ChenSammi · 2025-12-29T10:41:36Z

Thanks @xichen01 for updating the patch. The last commit looks overall good to me, +1.

HDDS-12791. Make Retention Service suspendable

536b595

ChenSammi requested a review from Copilot December 8, 2025 04:42

Copilot started reviewing on behalf of ChenSammi December 8, 2025 04:42 View session

Copilot AI reviewed Dec 8, 2025

View reviewed changes

jojochuang added the s3-lifecycle HDDS-8342 label Dec 8, 2025

jojochuang requested a review from ChenSammi December 8, 2025 17:27

ChenSammi reviewed Dec 10, 2025

View reviewed changes

pony.chen added 3 commits December 16, 2025 16:51

Add suspend lifecycle service command

b8b5286

Address some comments

1ba79a3

fix test

2bed61b

ChenSammi reviewed Dec 18, 2025

View reviewed changes

Add resume command

f499637

ChenSammi merged commit b16ea44 into apache:HDDS-8342 Dec 29, 2025
41 of 43 checks passed

Conversation

xichen01 commented Dec 4, 2025 • edited by ChenSammi Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChenSammi Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ChenSammi commented Dec 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

xichen01 commented Dec 4, 2025 •

edited by ChenSammi

Loading

ChenSammi Dec 10, 2025 •

edited

Loading