HADOOP-19864. Cut WritableRPCEngine by steveloughran · Pull Request #8433 · apache/hadoop

steveloughran · 2026-04-16T18:32:27Z

Description of PR

Purge WritableRPCEngine from the code.

There's hints of its existence in the code (RpcWritable and ClientCache), but its not an RPC mechanism any more.

How was this patch tested?

tests of immediate methods; remaining homework is left to yetus

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

AI Tooling

If an AI tool was used:

The PR includes the phrase "Contains content generated by "
where is the name of the AI tool used.
My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html

RpcWritable still exists, and ClientCache uses it, but those are vestigal uses. Contains content generated by GitHub Copilot

hadoop-yetus · 2026-04-16T20:49:24Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	8m 39s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+0 🆗	buf	0m 0s		buf was not available.
+0 🆗	buf	0m 0s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	27m 20s		trunk passed
+1 💚	compile	8m 31s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	compile	8m 44s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	checkstyle	0m 51s		trunk passed
+1 💚	mvnsite	1m 8s		trunk passed
+1 💚	javadoc	0m 51s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	0m 52s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	1m 45s		trunk passed
+1 💚	shadedclient	16m 48s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 44s		the patch passed
+1 💚	compile	8m 7s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	cc	8m 7s		the patch passed
+1 💚	javac	8m 7s		the patch passed
+1 💚	compile	8m 48s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	cc	8m 48s		the patch passed
+1 💚	javac	8m 48s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 49s	/results-checkstyle-hadoop-common-project_hadoop-common.txt	hadoop-common-project/hadoop-common: The patch generated 2 new + 323 unchanged - 28 fixed = 325 total (was 351)
+1 💚	mvnsite	1m 10s		the patch passed
+1 💚	javadoc	0m 50s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	0m 54s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	1m 52s		the patch passed
+1 💚	shadedclient	16m 52s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	18m 21s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		136m 0s

Reason	Tests
Failed junit tests	hadoop.ipc.TestMiniRPCBenchmark

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8433/1/artifact/out/Dockerfile
GITHUB PR	#8433
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat
uname	Linux 4bf5c3a3303b 5.15.0-171-generic #181-Ubuntu SMP Fri Feb 6 22:44:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `498f4dd`
Default Java	Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8433/1/testReport/
Max. process+thread count	1306 (vs. ulimit of 10000)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8433/1/console
versions	git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2026-04-17T22:35:15Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 23s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+0 🆗	buf	0m 1s		buf was not available.
+0 🆗	buf	0m 1s		buf was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s		The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 25s		trunk passed
+1 💚	compile	10m 37s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	compile	11m 15s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	checkstyle	0m 52s		trunk passed
+1 💚	mvnsite	1m 12s		trunk passed
+1 💚	javadoc	0m 48s		trunk passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	0m 46s		trunk passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	1m 48s		trunk passed
+1 💚	shadedclient	22m 38s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 46s		the patch passed
+1 💚	compile	11m 0s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	cc	11m 0s		the patch passed
+1 💚	javac	11m 0s		the patch passed
+1 💚	compile	11m 43s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	cc	11m 43s		the patch passed
+1 💚	javac	11m 43s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 53s		hadoop-common-project/hadoop-common: The patch generated 0 new + 323 unchanged - 28 fixed = 323 total (was 351)
+1 💚	mvnsite	1m 16s		the patch passed
+1 💚	javadoc	0m 46s		the patch passed with JDK Ubuntu-21.0.10+7-Ubuntu-124.04
+1 💚	javadoc	0m 45s		the patch passed with JDK Ubuntu-17.0.18+8-Ubuntu-124.04.1
+1 💚	spotbugs	2m 0s		the patch passed
-1 ❌	shadedclient	7m 14s		patch has errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	0m 32s	/patch-unit-hadoop-common-project_hadoop-common.txt	hadoop-common in the patch failed.
+0 🆗	asflicense	0m 33s		ASF License check generated no output?
		121m 16s

Subsystem	Report/Notes
Docker	ClientAPI=1.54 ServerAPI=1.54 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8433/2/artifact/out/Dockerfile
GITHUB PR	#8433
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets cc buflint bufcompat
uname	Linux 4cd6d5ed90de 5.15.0-171-generic #181-Ubuntu SMP Fri Feb 6 22:44:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `8603e30`
Default Java	Ubuntu-17.0.18+8-Ubuntu-124.04.1
Multi-JDK versions	/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.10+7-Ubuntu-124.04 /usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.18+8-Ubuntu-124.04.1
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8433/2/testReport/
Max. process+thread count	610 (vs. ulimit of 10000)
modules	C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8433/2/console
versions	git=2.43.0 maven=3.9.11 spotbugs=4.9.7
Powered by	Apache Yetus 0.14.1 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2026-04-18T17:06:26Z

shaded build failing was vm oom.

I'm going to do a slightly modified PR which touches hdfs, mr and yarn out of diligence

pan3793 · 2026-04-19T15:02:48Z

-// note anything marked public is solely for access by SaslRpcClient
+/**
+ * Marshalling support, for hadoop shaded protobuf and legacy
+ * protobuf 2.5.


do you have a plan to cut protobuf 2.5 thoroughly?

protobuf 3-based ProtobufRpcEngine2 has been introduced in HADOOP-17046 (3.3.0, 6 years ago), also Hadoop 3.2 has reached EOL, I think we have already given downstream projects enough time to allow them to migrate from protobuf 2.5 to hadoop shaded protobuf 3.x seamlessly

good point. the issue was always hbase 1. now we are cleaning up, getting rid of obsolete rpc stuff is good. someone should ping hbase dev list though.

https://issues.apache.org/jira/browse/HBASE-27286

The 1.x release are now fully EOLed.

tez are updating their umbilical, we need to move them to our shaded protobuf before we can say "not used"

steveloughran · 2026-04-20T13:47:32Z

failures are legitimate

org.apache.hadoop.hdfs.security.token.block.TestBlockToken.testBlockTokenRpcLeakLegacy


java.lang.IllegalStateException: No RPC engine configured for org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol
	at org.apache.hadoop.util.Preconditions.checkState(Preconditions.java:298)
	at org.apache.hadoop.ipc.RPC.getProtocolEngine(RPC.java:227)
	at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:677)
	at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:642)
	at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:544)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:520)
	at org.apache.hadoop.hdfs.security.token.block.TestBlockToken.testBlockTokenRpcLeak(TestBlockToken.java:400)
	at org.apache.hadoop.hdfs.security.token.block.TestBlockToken.testBlockTokenRpcLeakLegacy(TestBlockToken.java:435)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

org.apache.hadoop.hdfs.security.token.block.TestBlockToken.testBlockTokenRpcLeakProtobuf


java.lang.IllegalStateException: No RPC engine configured for org.apache.hadoop.hdfs.protocol.ClientDatanodeProtocol
	at org.apache.hadoop.util.Preconditions.checkState(Preconditions.java:298)
	at org.apache.hadoop.ipc.RPC.getProtocolEngine(RPC.java:227)
	at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:677)
	at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:642)
	at org.apache.hadoop.ipc.RPC.getProtocolProxy(RPC.java:544)
	at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:520)
	at org.apache.hadoop.hdfs.security.token.block.TestBlockToken.testBlockTokenRpcLeak(TestBlockToken.java:400)
	at org.apache.hadoop.hdfs.security.token.block.TestBlockToken.testBlockTokenRpcLeakProtobuf(TestBlockToken.java:440)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

final failure less obvious

Failed
org.apache.hadoop.hdfs.server.datanode.TestDataNodeLifeline.testSendLifelineIfHeartbeatBlocked

Failing for the past 1 build (Since [Failed#3](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8433/3/) )
[Took 5.5 sec.](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8433/3/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestDataNodeLifeline/testSendLifelineIfHeartbeatBlocked/history)
Error Message
Could not change property dfs.datanode.data.dir from '' to '/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-8433/ubuntu-noble/src/hadoop-hdfs-project/hadoop-hdfs/target/test/data/2/dfs/data/data-new'
Stacktrace
org.apache.hadoop.conf.ReconfigurationException: Could not change property dfs.datanode.data.dir from '' to '/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-8433/ubuntu-noble/src/hadoop-hdfs-project/hadoop-hdfs/target/test/data/2/dfs/data/data-new'
	at org.apache.hadoop.hdfs.server.datanode.DataNode.reconfigurePropertyImpl(DataNode.java:665)
	at org.apache.hadoop.hdfs.server.datanode.TestDataNodeLifeline.testSendLifelineIfHeartbeatBlocked(TestDataNodeLifeline.java:205)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
Caused by: java.io.IOException: FAILED to ADD: [DISK]file:/home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-8433/ubuntu-noble/src/hadoop-hdfs-project/hadoop-hdfs/target/test/data/2/dfs/data/data-new: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0

	at org.apache.hadoop.hdfs.server.datanode.DataNode.refreshVolumes(DataNode.java:1362)
	at org.apache.hadoop.hdfs.server.datanode.DataNode.reconfigurePropertyImpl(DataNode.java:647)
	... 4 more

But if heartbeats were failing, that could be the trigger.

steveloughran · 2026-04-22T19:50:24Z

test failures are just flaky hdfs tests. github copilot looked @ the logs and says so

steveloughran · 2026-04-22T19:52:12Z

@cnauroth this is ready, not something to backport for the sake of tez compatibility

pan3793 · 2026-04-23T07:34:26Z

+ * that is no removed. All that is retained is the name.
+ * Anything marked public is solely for access by SaslRpcClient
+ */
+// note


nit: restore original note statement

pan3793 · 2026-04-23T07:34:50Z

  }

+
+


nit: revert unnecessary changes

pan3793 · 2026-04-23T07:35:01Z

    }
  }

-


pan3793 · 2026-04-23T07:35:40Z

  <description>Apache Hadoop HDFS Project</description>
  <name>Apache Hadoop HDFS Project</name>
  <packaging>pom</packaging>
+  <!-- ignore this line -->


remove after passing yetus checks

…sts too" This reverts commit 21a982d.

pan3793 · 2026-04-23T15:25:27Z

@steveloughran found an failed case caused by this PR - org.apache.hadoop.ipc.TestMiniRPCBenchmark

proposed a fix #8455

HADOOP-19864. Cut WritableRPCEngine

498f4dd

RpcWritable still exists, and ClientCache uses it, but those are vestigal uses. Contains content generated by GitHub Copilot

github-actions Bot added trunk Common labels Apr 16, 2026

checkstyle

8603e30

add comment lines to poms in mr/hdfs/yarn to trigger their tests too

21a982d

github-actions Bot added YARN MapReduce HDFS build labels Apr 18, 2026

pan3793 reviewed Apr 19, 2026

View reviewed changes

steveloughran added 2 commits April 20, 2026 21:15

HADOOP-19864. HDFS tests to set the proxy.

1888d9e

Merge branch 'trunk' into pr/HADOOP-19864-WritableRPCEngine

735a4d4

pan3793 approved these changes Apr 23, 2026

View reviewed changes

Revert "add comment lines to poms in mr/hdfs/yarn to trigger their te…

d95742f

…sts too" This reverts commit 21a982d.

github-actions Bot removed the YARN label Apr 23, 2026

steveloughran merged commit 998bf0a into apache:trunk Apr 23, 2026
2 of 4 checks passed

pan3793 mentioned this pull request Apr 24, 2026

HADOOP-19864. (Followup) Migrate MiniRPCBenchmark to ProtobufRpcEngine2 #8455

Merged

6 tasks

Conversation

steveloughran commented Apr 16, 2026

Description of PR

How was this patch tested?

For code changes:

AI Tooling

Uh oh!

hadoop-yetus commented Apr 16, 2026

Uh oh!

hadoop-yetus commented Apr 17, 2026

Uh oh!

steveloughran commented Apr 18, 2026

Uh oh!

pan3793 Apr 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steveloughran Apr 19, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 Apr 20, 2026

Choose a reason for hiding this comment

Uh oh!

steveloughran Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Apr 20, 2026

Uh oh!

steveloughran commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steveloughran commented Apr 22, 2026

Uh oh!

pan3793 Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pan3793 commented Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pan3793 Apr 19, 2026 •

edited

Loading

steveloughran commented Apr 22, 2026 •

edited

Loading

pan3793 commented Apr 23, 2026 •

edited

Loading