HDDS-14949 — Migrate gRPC transport to the Ratis-shaded gRPC/Netty/Protobuf stack #10030
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,6 +16,7 @@ | |
| .project | ||
| .settings | ||
| *.factorypath | ||
| *.class | ||
| target | ||
| build | ||
| dependency-reduced-pom.xml | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,195 @@ | ||
| <?xml version="1.0" encoding="UTF-8"?> | ||
| <!-- | ||
| Licensed under the Apache License, Version 2.0 (the "License"); | ||
| you may not use this file except in compliance with the License. | ||
| You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. See accompanying LICENSE file. | ||
| --> | ||
| <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
| <modelVersion>4.0.0</modelVersion> | ||
| <parent> | ||
| <groupId>org.apache.ozone</groupId> | ||
| <artifactId>hdds</artifactId> | ||
| <version>2.2.0-SNAPSHOT</version> | ||
| </parent> | ||
| <artifactId>hdds-datanode-grpc-client</artifactId> | ||
| <version>2.2.0-SNAPSHOT</version> | ||
| <packaging>jar</packaging> | ||
| <name>Apache Ozone HDDS DataNode gRPC Client</name> | ||
| <description>DataNode XCeiver and IntraDatanode gRPC stubs generated from DatanodeClientProtocol.proto. | ||
| The proto file lives in hdds-interface-client (which also generates the shaded ContainerProtos | ||
| message classes); this module generates only the gRPC service stubs. Package names in generated | ||
| sources are rewritten by maven-antrun-plugin (Ant replace task, pure Java) from vanilla | ||
| com.google.protobuf / io.grpc / com.google.common to their ratis-thirdparty counterparts | ||
| before compilation.</description> | ||
|
|
||
| <properties> | ||
| <!-- used in temp dir names; must be filesystem/protoc-safe (no ':') --> | ||
| <maven.build.timestamp.format>yyyyMMddHHmmss</maven.build.timestamp.format> | ||
| <!-- no testable code in this module --> | ||
| <maven.test.skip>true</maven.test.skip> | ||
| <mdep.analyze.skip>true</mdep.analyze.skip> | ||
| <spotbugs.skip>true</spotbugs.skip> | ||
| </properties> | ||
|
|
||
| <dependencies> | ||
| <!-- | ||
| ContainerProtos message classes (shaded) live in hdds-interface-client. The generated | ||
| gRPC stubs reference those types, so hdds-interface-client must be on the compile classpath. | ||
| --> | ||
| <dependency> | ||
| <groupId>org.apache.ozone</groupId> | ||
| <artifactId>hdds-interface-client</artifactId> | ||
| </dependency> | ||
| <!-- | ||
| Shaded gRPC / Netty / protobuf runtime. The generated sources are rewritten to reference | ||
| org.apache.ratis.thirdparty.* directly, so ratis-thirdparty-misc must be on the compile | ||
| classpath and is exposed transitively to all consumers of this module. | ||
| --> | ||
| <dependency> | ||
| <groupId>org.apache.ratis</groupId> | ||
| <artifactId>ratis-thirdparty-misc</artifactId> | ||
| </dependency> | ||
| <dependency> | ||
| <!-- for @Generated annotation in generated sources (Java 11+) --> | ||
| <groupId>javax.annotation</groupId> | ||
| <artifactId>javax.annotation-api</artifactId> | ||
| <scope>provided</scope> | ||
| </dependency> | ||
| </dependencies> | ||
|
|
||
| <build> | ||
| <pluginManagement> | ||
| <plugins> | ||
| <plugin> | ||
| <groupId>com.gradle</groupId> | ||
| <artifactId>develocity-maven-extension</artifactId> | ||
| <configuration> | ||
| <develocity> | ||
| <plugins> | ||
| <plugin> | ||
| <artifactId>maven-compiler-plugin</artifactId> | ||
| <outputs> | ||
| <notCacheableBecause>compiles generated + rewritten sources</notCacheableBecause> | ||
| </outputs> | ||
| </plugin> | ||
| <plugin> | ||
| <artifactId>maven-antrun-plugin</artifactId> | ||
| <outputs> | ||
| <notCacheableBecause>rewrites generated source files</notCacheableBecause> | ||
| </outputs> | ||
| </plugin> | ||
| <plugin> | ||
| <artifactId>protobuf-maven-plugin</artifactId> | ||
| <outputs> | ||
| <notCacheableBecause>generates sources</notCacheableBecause> | ||
| </outputs> | ||
| </plugin> | ||
| </plugins> | ||
| </develocity> | ||
| </configuration> | ||
| </plugin> | ||
| </plugins> | ||
| </pluginManagement> | ||
| <plugins> | ||
| <plugin> | ||
| <groupId>com.salesforce.servicelibs</groupId> | ||
| <artifactId>proto-backwards-compatibility</artifactId> | ||
| </plugin> | ||
| <plugin> | ||
| <groupId>org.apache.maven.plugins</groupId> | ||
| <artifactId>maven-compiler-plugin</artifactId> | ||
| <configuration> | ||
| <proc>none</proc> | ||
| <useIncrementalCompilation>false</useIncrementalCompilation> | ||
| </configuration> | ||
| </plugin> | ||
| <plugin> | ||
| <groupId>org.xolstice.maven.plugins</groupId> | ||
| <artifactId>protobuf-maven-plugin</artifactId> | ||
| <executions> | ||
| <execution> | ||
| <id>compile-proto-datanode</id> | ||
| <goals> | ||
| <!-- | ||
| compile-custom only: generates gRPC service stubs (XceiverClientProtocolServiceGrpc, | ||
| IntraDatanodeProtocolServiceGrpc). The proto messages (ContainerProtos) are compiled | ||
| in hdds-interface-client with the same ratis-compatible protoc; no need to regenerate | ||
| them here. | ||
| --> | ||
| <goal>compile-custom</goal> | ||
| </goals> | ||
| <phase>generate-sources</phase> | ||
| <configuration> | ||
| <!-- | ||
| Use the same protoc/grpc-java versions as ratis-thirdparty so that the generated | ||
| API surface is compatible with the shaded runtime in ratis-thirdparty-misc. | ||
| Output directly to proto-java-ratis so that both Maven and the IDE/LSP compile the | ||
| same ratis-shaded source tree, eliminating any race between Maven's javac and the | ||
| LSP writing vanilla class files to target/classes/. | ||
| DatanodeClientProtocol.proto lives in hdds-interface-client; point protoSourceRoot | ||
| there so protoc finds it without duplicating the file. | ||
| --> | ||
| <protocArtifact>com.google.protobuf:protoc:${ratis-thirdparty.protobuf.version}:exe:${os.detected.classifier}</protocArtifact> | ||
| <outputDirectory>${project.build.directory}/generated-sources/proto-java-ratis</outputDirectory> | ||
| <temporaryProtoFileDirectory>${java.io.tmpdir}/ozone-protoc-dependencies/${project.artifactId}/${maven.build.timestamp}</temporaryProtoFileDirectory> | ||
| <clearOutputDirectory>false</clearOutputDirectory> | ||
| <pluginId>grpc-java</pluginId> | ||
| <pluginArtifact>io.grpc:protoc-gen-grpc-java:${ratis-thirdparty.grpc.version}:exe:${os.detected.classifier}</pluginArtifact> | ||
| <protoSourceRoot>${project.basedir}/../interface-client/src/main/proto</protoSourceRoot> | ||
| <includes> | ||
| <include>DatanodeClientProtocol.proto</include> | ||
| </includes> | ||
| </configuration> | ||
| </execution> | ||
| </executions> | ||
| </plugin> | ||
| <plugin> | ||
| <groupId>org.apache.maven.plugins</groupId> | ||
| <artifactId>maven-antrun-plugin</artifactId> | ||
| <executions> | ||
| <execution> | ||
| <!-- | ||
| mdep.analyze.skip=true for this module so pre-verify-refresh-classes-from-jar | ||
| (inherited from the root POM) is not needed. Binding it to "none" prevents the | ||
| inherited execution from wiping target/classes/ at pre-integration-test, which | ||
| would leave the directory temporarily empty and cause downstream reactor modules | ||
| (e.g. hdds-common) to fail with "class file for ContainerProtos$... not found". | ||
| --> | ||
| <id>pre-verify-refresh-classes-from-jar</id> | ||
| <phase>none</phase> | ||
| </execution> | ||
| <execution> | ||
| <!-- | ||
| Rewrite vanilla package names produced by protoc/grpc-java in-place inside | ||
| proto-java-ratis/ (the protobuf-maven-plugin outputDirectory). Because the | ||
| IDE/LSP also uses proto-java-ratis/ as its Java source root, it will compile | ||
| the same shaded sources and write shaded class files to target/classes/, which | ||
| means both Maven javac and the LSP produce identical bytecode – no more race | ||
| where the LSP overwrites Maven's correctly-shaded class files. | ||
| --> | ||
| <id>rewrite-generated-sources-to-ratis-ns</id> | ||
| <goals> | ||
| <goal>run</goal> | ||
| </goals> | ||
| <phase>process-sources</phase> | ||
| <configuration> | ||
| <target> | ||
| <replace dir="${project.build.directory}/generated-sources/proto-java-ratis" token="com.google.common" value="org.apache.ratis.thirdparty.com.google.common" /> | ||
| <replace dir="${project.build.directory}/generated-sources/proto-java-ratis" token="com.google.protobuf" value="org.apache.ratis.thirdparty.com.google.protobuf" /> | ||
| <replace dir="${project.build.directory}/generated-sources/proto-java-ratis" token="io.grpc" value="org.apache.ratis.thirdparty.io.grpc" /> | ||
| </target> | ||
| </configuration> | ||
| </execution> | ||
| </executions> | ||
| </plugin> | ||
| </plugins> | ||
| </build> | ||
| </project> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| { | ||
| "definitions": [] | ||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"Disabling incremental compilation forces a cleaner, dependency-correct rebuild behavior for that module’s compilation step, so that when an API change happens, Maven/javac are much less likely to leave behind stale .class files that were compiled against old symbols.
In short: it trades some compile speed for build determinism and correctness, which matters a lot in PRs like this one (migrating shaded gRPC/Netty/Ratis dependencies) where API surfaces and classpaths can shift in ways that incremental compilation is particularly bad at handling."