HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated by jojochuang · Pull Request #9580 · apache/ozone

jojochuang · 2026-01-03T14:21:25Z

What changes were proposed in this pull request?

HDDS-13579. [Docs] Explain how Ratis write pipelines are calculated

Please describe your PR in detail:

Similar to HDDS-14199 but for Ratis pipelines.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13579

How was this patch tested?

Doc only.

szetszwo · 2026-01-03T21:08:51Z

@jojochuang , thanks for working on this!

Could you limit the line length to 120 characters like the code? It is easier to comment on it.

Discovered and corrected the documentation for how the number of EC pipelines is calculated. The previous analysis was incorrect. - `ErasureCoding.md` is updated to describe the two new properties `ozone.scm.ec.pipeline.minimum` and `ozone.scm.ec.pipeline.per.volume.factor` and the `max()` logic used to determine the target number of pipelines. - `ProductionDeployment.md` is updated to reference the correct and existing configuration property for tuning EC pipelines. Change-Id: I393dc60d8745da2b2bb7899530665a108956446d

Change-Id: I0ec667c21155436eb6a0654782b43b48636f75d5

jojochuang · 2026-01-04T00:49:41Z

thanks for review. updated.

ashishkumar50

@jojochuang Thanks for writing this up.

ashishkumar50 · 2026-01-05T08:13:01Z

+recommended, as it allows the cluster to scale pipeline capacity naturally with its resources. You can use the
+global limit (`ozone.scm.ratis.pipeline.limit`) as a safety cap if needed. A good starting value for
+`ozone.scm.pipeline.per.metadata.disk` is **2**. Monitor the `NumOpenPipelines` metric in SCM to see if the
+actual number of pipelines aligns with your configured targets.


I think NumOpenPipelines metrics doesn't exist. We can either use admin command to see number of open pipelines or may use recon as well to see the open pipelines.

To clarify, the SCM web UI has a section Pipeline Statistics, and it pulls the metrics from:

{ "name": "Hadoop:service=SCMPipelineManager,name=SCMPipelineManagerInfo", "modelerType": "org.apache.hadoop.hdds.scm.pipeline.PipelineManagerImpl", "PipelineInfo": [ { "key": "CLOSED", "value": 0 }, { "key": "ALLOCATED", "value": 0 }, { "key": "OPEN", "value": 1 }, { "key": "DORMANT", "value": 0 } ] },

Change-Id: I2537905761cf45d23cdb3701b2f0c94e7ff2485a

szetszwo

@jojochuang , thanks for working on this!

The code is quite distributed so it is not easy to understand the exact calculation. Please see the comments inlined (not yet reviewed the example).

szetszwo · 2026-01-07T16:26:03Z


+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum


... FACTOR_THREE ...

Let's use ReplicationFactor.THREE, which is a client API.

... three properties ... that define the maximum

three configuration properties ... that limit the

szetszwo · 2026-01-07T16:31:53Z

+### Calculating Ratis Pipeline Limits
+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.


... a cluster-wide level, datanode level, and metadata disk level, respectively.

a cluster-wide level and a datanode level.

szetszwo · 2026-01-07T16:37:38Z

+1.  **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+    *   **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines
+        across the entire cluster. This acts as a final cap on the total number of pipelines.
+    *   **Default Value**: `0` (which means no global limit is enforced by default).


... no global limit is enforced by default).

no global limit by default).

szetszwo · 2026-01-07T17:07:02Z

+    *   **Calculation**: If this is set, the target is `(<this value> * <number of healthy datanodes>) / 3`.
+
+3.  **Datanode-level Dynamic Limit (`ozone.scm.pipeline.per.metadata.disk`)**
+    *   **Description**: This property is used only when `ozone.scm.datanode.pipeline.limit` is explicitly set to `0`.


... This property is used ... is explicitly set to 0.

This property takes effect ... is not set to a positive number.

szetszwo · 2026-01-07T17:13:41Z

+
+The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum
+number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively.
+SCM will create pipelines until the most restrictive limit is met.


Replace this sentence with

"The number of pipelines created by SCM is restricted by these limits."

szetszwo · 2026-01-07T17:26:56Z

+SCM will create pipelines until the most restrictive limit is met.
+
+1.  **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)**
+    *   **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines


Remove FACTOR_THREE since the code actually includes ReplicationFactor.ONE pipeline; see

ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java

Lines 123 to 128 in 4fd8360

if (pipelineNumberLimit > 0) {

int factorOnePipelineCount = pipelineStateManager

.getPipelines(RatisReplicationConfig.getInstance(ReplicationFactor.ONE)).size();

int allowedOpenPipelines = pipelineNumberLimit - factorOnePipelineCount;

return openPipelines >= allowedOpenPipelines;

}

szetszwo · 2026-01-07T17:29:16Z

+    *   **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+        every datanode. This is one of two ways to calculate a cluster-wide target.
+    *   **Default Value**: `2`
+    *   **Calculation**: If this is set, the target is `(<this value> * <number of healthy datanodes>) / 3`.


* **Cluster-wide Limit Calculation**: If this property is set, the number of pipelines in the cluster is in addition limited by `(<this value> * <number of healthy datanodes>) / 3`.

(revised)

szetszwo · 2026-01-07T17:38:49Z

+    *   **Calculation**: The limit for each datanode is
+        `(<this value> * <number of metadata disks on that datanode>)`.
+        The total cluster-wide target is the sum of all individual datanode limits, divided by 3.


Remove this section since it seems not true (or I cannot find the code enforcing it). I do see that the value is used for filtering datanodes.

ozone/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/RatisPipelineProvider.java

Lines 104 to 131 in 4fd8360

private boolean exceedPipelineNumberLimit(RatisReplicationConfig replicationConfig) {

// Apply limits only for replication factor THREE

if (replicationConfig.getReplicationFactor() != ReplicationFactor.THREE) {

return false;

}

PipelineStateManager pipelineStateManager = getPipelineStateManager();

int totalActivePipelines = pipelineStateManager.getPipelines(replicationConfig).size();

int closedPipelines = pipelineStateManager.getPipelines(replicationConfig, PipelineState.CLOSED).size();

int openPipelines = totalActivePipelines - closedPipelines;

// Check per-datanode pipeline limit

if (maxPipelinePerDatanode > 0) {

int healthyNodeCount = getNodeManager()

.getNodeCount(NodeStatus.inServiceHealthy());

int allowedOpenPipelines = (maxPipelinePerDatanode * healthyNodeCount)

/ replicationConfig.getRequiredNodes();

return openPipelines >= allowedOpenPipelines;

}

// Check global pipeline limit

if (pipelineNumberLimit > 0) {

int factorOnePipelineCount = pipelineStateManager

.getPipelines(RatisReplicationConfig.getInstance(ReplicationFactor.ONE)).size();

int allowedOpenPipelines = pipelineNumberLimit - factorOnePipelineCount;

return openPipelines >= allowedOpenPipelines;

}

// No limits are set

return false;

}

szetszwo · 2026-01-07T17:43:01Z

+
+2.  **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)**
+    *   **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for
+        every datanode. This is one of two ways to calculate a cluster-wide target.


Remove the sentence "This is one of two ways to calculate a cluster-wide target." since the Dynamic Limit
is not used in RatisPipelineProvider.exceedPipelineNumberLimit(..).

jojochuang

@yandrey321

ivandika3 · 2026-01-15T09:24:24Z

+For most production deployments, using the dynamic per-disk limit (`ozone.scm.datanode.pipeline.limit=0`) is
+recommended, as it allows the cluster to scale pipeline capacity naturally with its resources. You can use the
+global limit (`ozone.scm.ratis.pipeline.limit`) as a safety cap if needed. A good starting value for
+`ozone.scm.pipeline.per.metadata.disk` is **2**. Monitor the section **Pipeline Statistics** in SCM web UI, or run
+the command `ozone admin pipeline list` to see if the actual number of pipelines aligns with your configured targets.


I think there should a tradeoff of having a lot of concurrent pipelines. This might be worth documenting.

Here are a few I can thinks of

Each Ratis group takes some resources (e.g. 8MB for the write buffer IIRC)

Larger number of pipelines increase the load on the metadata volume which might cause contention

The higher number of pipelines, there will be higher number concurrent storage containers. If one DN is down and all the pipelines are closed, we might end up with a lot of small containers, which might have overhead long term.

Change-Id: Ieff1a5ffe67c4eed32a7c61c655c19ba9f307d26

jojochuang · 2026-01-17T01:07:14Z

Ok. updated based on Nicholas' comment. I am going to go over this again since it's been awhile.

github-actions · 2026-02-08T00:11:15Z

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

github-actions · 2026-02-16T00:08:07Z

Thank you for your contribution. This PR is being closed due to inactivity. Please contact a maintainer if you would like to reopen it.

jojochuang requested review from ashishkumar50, siddhantsangwan, sodonnel, sumitagrawl and szetszwo January 3, 2026 14:21

jojochuang added 2 commits January 3, 2026 16:49

Limit to 120 characters

c4deea7

Change-Id: I0ec667c21155436eb6a0654782b43b48636f75d5

jojochuang force-pushed the HDDS-13579 branch from 199bd3c to c4deea7 Compare January 4, 2026 00:49

adoroszlai added the documentation Improvements or additions to documentation label Jan 5, 2026

ashishkumar50 reviewed Jan 5, 2026

View reviewed changes

Update text for monitoring number of pipelines.

f8060d9

Change-Id: I2537905761cf45d23cdb3701b2f0c94e7ff2485a

jojochuang requested a review from ashishkumar50 January 7, 2026 17:36

szetszwo reviewed Jan 7, 2026

View reviewed changes

jojochuang mentioned this pull request Jan 12, 2026

HDDS-14369. RatisPipelineProvider does not honor OZONE_DATANODE_PIPELINE_LIMIT_DEFAULT #9609

Merged

jojochuang commented Jan 12, 2026

View reviewed changes

ivandika3 reviewed Jan 15, 2026

View reviewed changes

Update based on Nicholas's comments.

b7eefa2

Change-Id: Ieff1a5ffe67c4eed32a7c61c655c19ba9f307d26

github-actions Bot added the stale label Feb 8, 2026

github-actions Bot closed this Feb 16, 2026


		### Calculating Ratis Pipeline Limits

		The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum

	if (pipelineNumberLimit > 0) {
	int factorOnePipelineCount = pipelineStateManager
	.getPipelines(RatisReplicationConfig.getInstance(ReplicationFactor.ONE)).size();
	int allowedOpenPipelines = pipelineNumberLimit - factorOnePipelineCount;
	return openPipelines >= allowedOpenPipelines;
	}

	private boolean exceedPipelineNumberLimit(RatisReplicationConfig replicationConfig) {
	// Apply limits only for replication factor THREE
	if (replicationConfig.getReplicationFactor() != ReplicationFactor.THREE) {
	return false;
	}

	PipelineStateManager pipelineStateManager = getPipelineStateManager();
	int totalActivePipelines = pipelineStateManager.getPipelines(replicationConfig).size();
	int closedPipelines = pipelineStateManager.getPipelines(replicationConfig, PipelineState.CLOSED).size();
	int openPipelines = totalActivePipelines - closedPipelines;
	// Check per-datanode pipeline limit
	if (maxPipelinePerDatanode > 0) {
	int healthyNodeCount = getNodeManager()
	.getNodeCount(NodeStatus.inServiceHealthy());
	int allowedOpenPipelines = (maxPipelinePerDatanode * healthyNodeCount)
	/ replicationConfig.getRequiredNodes();
	return openPipelines >= allowedOpenPipelines;
	}
	// Check global pipeline limit
	if (pipelineNumberLimit > 0) {
	int factorOnePipelineCount = pipelineStateManager
	.getPipelines(RatisReplicationConfig.getInstance(ReplicationFactor.ONE)).size();
	int allowedOpenPipelines = pipelineNumberLimit - factorOnePipelineCount;
	return openPipelines >= allowedOpenPipelines;
	}
	// No limits are set
	return false;
	}

Conversation

jojochuang commented Jan 3, 2026

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

szetszwo commented Jan 3, 2026

Uh oh!

jojochuang commented Jan 4, 2026

Uh oh!

ashishkumar50 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szetszwo left a comment

Choose a reason for hiding this comment

Uh oh!

szetszwo Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szetszwo Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

szetszwo Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jojochuang left a comment

Choose a reason for hiding this comment

Uh oh!

ivandika3 Jan 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jojochuang commented Jan 17, 2026

Uh oh!

github-actions Bot commented Feb 8, 2026

Uh oh!

github-actions Bot commented Feb 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

szetszwo Jan 7, 2026 •

edited

Loading

szetszwo Jan 7, 2026 •

edited

Loading

szetszwo Jan 7, 2026 •

edited

Loading

ivandika3 Jan 15, 2026 •

edited

Loading