Skip to content

Commit 93780c1

Browse files
Add Sofa RPC instrumentation (Bolt, H2C, REST, Triple) (#11135)
# What Does This Do Adds auto-instrumentation for [Sofa RPC](https://github.com/sofastack/sofa-rpc), a high-perfomance RPC framework. Produces `sofarpc.request` client and server spans with distributed trace propagation for all four supported transports: Bolt, H2C, REST and Triple(gRPC) # Motivation Sofa RPC is widely used in Ant Financial's ecosystem and it's not covered by any existing dd-trace-java integration. WIthout this instrumentaiton, traces break at the service boundary - downstream PRC calls are unlinked from traces # Additional Notes ## Module structure A single module `sofarpc-5.0` covers all protocols and all public version of sofa-rpc. The framework was open-sources at 5.0.0; no earlier versions exists in public repositories ## How server spans are coordinated `ProviderProxyInvoker.invoke()` is a single entry point for all protocols on the server side - but it has noo info about the transport delivered the request. Transport specific instrumentations write the protocol name into ThreadLocal (`SofaRpcProtocolContext`) before `invoke()` is called. `ProviderProxuInvokerInstrumentaiton` reads it to created the span and tag with the protocol. If the ThreadLocal is empty (uninstrumented transport), no span is created. ## Trace propagation by protocol **Bolt** and **H2C** - context travels inside SofaRequest.requestProps (the framework's own key-value bag) <img width="1425" height="212" alt="image" src="https://github.com/user-attachments/assets/bee9f416-7ed7-4342-88fa-65dffd695cce" /> **REST** - `SofaRequest.requestProps` is not serialized as HTTP headers by the REST transport. Context instead travels as standard HTTP headers, injected and extracted by HTTP layer (Apache HttpClient on the client side, Netty on the server side). The `RestServerHandlerInstumentation` sets the protocol context so that `ProviderProxyInvoketInstrumentation` activated and creates a `sofarpc.request[server] span that naturally attaches to the active `netty.request` span <img width="1425" height="360" alt="image" src="https://github.com/user-attachments/assets/7e720a87-18a4-484e-a9ce-8c5db6ce59e2" /> **Triple** - uses gRPC unses the hood. The [grpc-1.5](https://github.com/DataDog/dd-trace-java/tree/master/dd-java-agent/instrumentation/grpc-1.5) instrumentation handles trace propagation via gRPC Metadata. The `TripleServerInstrumentation` sets `protocol="tri"` so that `ProviderProxyInvokerInstrumentaiton` creates a `sofarpc.request[server]` span with child of the active `grpc.server`span <img width="1429" height="357" alt="image" src="https://github.com/user-attachments/assets/05d6a88a-c75a-4414-bbd1-b9d0580fa682" /> ## Span attributes Tag|Value ---|--- sofarpc.protocol|bolt / h2c / rest / tri rpc.system|sofarpc rpc.service|e.g. com.example.HelloServier:1.0 rpc.method|e.g. sayHello span.kind|client / server component|sofarpc-client / sofarpc-server peer.service|derived from rpc.servier Resource name format: `{rpc.service}/{rpc.method}` (e.g. `com.example.HelloService:1.0/sayHello`) ## Known limitations - **REST server**: `SofaRequest.getTaargetServiceUniqueName()` returns null on the server side - the service name is not propagated thought the JAX-RS layer. - H2C: `getTargetServiceUniquieName() does not include the version suffix (e.g. HelloService/sayHello). THis is Sofa RPC behaviour for the H2C transport # Contributor Checklist - Format the title according to [the contribution guidelines](https://github.com/DataDog/dd-trace-java/blob/master/CONTRIBUTING.md#title-format) - Assign the `type:` and (`comp:` or `inst:`) labels in addition to [any other useful labels](https://github.com/DataDog/dd-trace-java/blob/master/CONTRIBUTING.md#labels) - Avoid using `close`, `fix`, or [any linking keywords](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword) when referencing an issue Use `solves` instead, and assign the PR [milestone](https://github.com/DataDog/dd-trace-java/milestones) to the issue - Update the [CODEOWNERS](https://github.com/DataDog/dd-trace-java/blob/master/.github/CODEOWNERS) file on source file addition, migration, or deletion - Update [public documentation](https://docs.datadoghq.com/tracing/trace_collection/library_config/java/) with any new configuration flags or behaviors Jira ticket: [PROJ-IDENT] ***Note:*** **Once your PR is ready to merge, add it to the merge queue by commenting `/merge`.** `/merge -c` cancels the queue request. `/merge -f --reason "reason"` skips all merge queue checks; please use this judiciously, as some checks do not run at the PR-level. For more information, see [this doc](https://datadoghq.atlassian.net/wiki/spaces/DEVX/pages/3121612126/MergeQueue). <!-- # Opening vs Drafting a PR: When opening a pull request, please open it as a draft to not auto assign reviewers before you feel the pull request is in a reviewable state. # Linking a JIRA ticket: Please link your JIRA ticket by adding its identifier between brackets (ex [PROJ-IDENT]) in the PR description, not the title. This requirement only applies to Datadog employees. --> Co-authored-by: valentin.zakharov <valentin.zakharov@datadoghq.com>
1 parent 8569434 commit 93780c1

24 files changed

Lines changed: 1396 additions & 0 deletions

.github/g2j-migrated-modules.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,4 +7,5 @@
77

88
buildSrc/call-site-instrumentation-plugin
99
components/json
10+
dd-java-agent/instrumentation/sofarpc/sofarpc-5.0
1011
dd-trace-api
Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
muzzle {
2+
pass {
3+
group = "com.alipay.sofa"
4+
module = "sofa-rpc-all"
5+
versions = "[5.0.0,)"
6+
assertInverse = true
7+
}
8+
}
9+
10+
apply from: "$rootDir/gradle/java.gradle"
11+
12+
addTestSuiteForDir('latestDepTest', 'test')
13+
14+
configurations.testRuntimeClasspath {
15+
resolutionStrategy.force "com.google.guava:guava:32.1.3-jre"
16+
}
17+
18+
dependencies {
19+
compileOnly group: "com.alipay.sofa", name: "sofa-rpc-all", version: "5.6.0"
20+
21+
testImplementation group: "com.alipay.sofa", name: "sofa-rpc-all", version: "5.14.2"
22+
// JAX-RS annotations required for the REST protocol test interface (@Path, @GET, etc.)
23+
testImplementation group: "javax.ws.rs", name: "javax.ws.rs-api", version: "2.1.1"
24+
// Required so that GrpcServerModule / GrpcClientModule are discovered via ServiceLoader
25+
// in SofaRpcTripleWithGrpcForkedTest.
26+
testImplementation project(':dd-java-agent:instrumentation:grpc-1.5')
27+
testImplementation group: "io.grpc", name: "grpc-netty", version: "1.53.0"
28+
testImplementation group: "io.grpc", name: "grpc-core", version: "1.53.0"
29+
testImplementation group: "io.grpc", name: "grpc-stub", version: "1.53.0"
30+
testImplementation group: "com.google.protobuf", name: "protobuf-java", version: "3.25.3"
31+
testImplementation group: "com.alibaba", name: "fastjson", version: "1.2.83"
32+
testImplementation group: "com.google.guava", name: "guava", version: "32.1.3-jre"
33+
34+
latestDepTestImplementation group: "com.alipay.sofa", name: "sofa-rpc-all", version: "+"
35+
}

dd-java-agent/instrumentation/sofarpc/sofarpc-5.0/gradle.lockfile

Lines changed: 243 additions & 0 deletions
Large diffs are not rendered by default.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
package datadog.trace.instrumentation.sofarpc;
2+
3+
import static datadog.context.propagation.Propagators.defaultPropagator;
4+
import static datadog.trace.agent.tooling.bytebuddy.matcher.NameMatchers.named;
5+
import static datadog.trace.bootstrap.instrumentation.api.AgentTracer.activateSpan;
6+
import static datadog.trace.bootstrap.instrumentation.api.AgentTracer.startSpan;
7+
import static datadog.trace.instrumentation.sofarpc.SofaRpcClientDecorator.DECORATE;
8+
import static datadog.trace.instrumentation.sofarpc.SofaRpcClientDecorator.SOFA_RPC_CLIENT;
9+
import static datadog.trace.instrumentation.sofarpc.SofaRpcInjectAdapter.SETTER;
10+
import static net.bytebuddy.matcher.ElementMatchers.isMethod;
11+
import static net.bytebuddy.matcher.ElementMatchers.takesArgument;
12+
import static net.bytebuddy.matcher.ElementMatchers.takesArguments;
13+
14+
import com.alipay.sofa.rpc.client.AbstractCluster;
15+
import com.alipay.sofa.rpc.config.ConsumerConfig;
16+
import com.alipay.sofa.rpc.core.request.SofaRequest;
17+
import com.alipay.sofa.rpc.core.response.SofaResponse;
18+
import datadog.trace.agent.tooling.Instrumenter;
19+
import datadog.trace.bootstrap.instrumentation.api.AgentScope;
20+
import datadog.trace.bootstrap.instrumentation.api.AgentSpan;
21+
import net.bytebuddy.asm.Advice;
22+
23+
public class AbstractClusterInstrumentation
24+
implements Instrumenter.ForSingleType, Instrumenter.HasMethodAdvice {
25+
26+
@Override
27+
public String instrumentedType() {
28+
return "com.alipay.sofa.rpc.client.AbstractCluster";
29+
}
30+
31+
@Override
32+
public void methodAdvice(MethodTransformer transformer) {
33+
transformer.applyAdvice(
34+
isMethod()
35+
.and(named("invoke"))
36+
.and(takesArguments(1))
37+
.and(takesArgument(0, named("com.alipay.sofa.rpc.core.request.SofaRequest"))),
38+
getClass().getName() + "$InvokeAdvice");
39+
}
40+
41+
public static class InvokeAdvice {
42+
@Advice.OnMethodEnter(suppress = Throwable.class)
43+
public static AgentScope enter(
44+
@Advice.This AbstractCluster self, @Advice.Argument(0) SofaRequest request) {
45+
ConsumerConfig config = self.getConsumerConfig();
46+
String protocol = config != null ? config.getProtocol() : null;
47+
AgentSpan span = startSpan(SOFA_RPC_CLIENT);
48+
DECORATE.afterStart(span);
49+
DECORATE.onRequest(span, request);
50+
if (protocol != null) {
51+
span.setTag("sofarpc.protocol", protocol);
52+
}
53+
AgentScope scope = activateSpan(span);
54+
defaultPropagator().inject(span, request, SETTER);
55+
return scope;
56+
}
57+
58+
@Advice.OnMethodExit(onThrowable = Throwable.class, suppress = Throwable.class)
59+
public static void exit(
60+
@Advice.Enter AgentScope scope,
61+
@Advice.Return SofaResponse response,
62+
@Advice.Thrown Throwable throwable) {
63+
if (scope == null) {
64+
return;
65+
}
66+
AgentSpan span = scope.span();
67+
DECORATE.onResponse(span, response);
68+
DECORATE.onError(span, throwable);
69+
DECORATE.beforeFinish(span);
70+
scope.close();
71+
span.finish();
72+
}
73+
}
74+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
package datadog.trace.instrumentation.sofarpc;
2+
3+
import static datadog.trace.agent.tooling.bytebuddy.matcher.NameMatchers.named;
4+
import static net.bytebuddy.matcher.ElementMatchers.isMethod;
5+
import static net.bytebuddy.matcher.ElementMatchers.takesArgument;
6+
import static net.bytebuddy.matcher.ElementMatchers.takesArguments;
7+
8+
import datadog.trace.agent.tooling.Instrumenter;
9+
import net.bytebuddy.asm.Advice;
10+
11+
public class BoltServerProcessorInstrumentation
12+
implements Instrumenter.ForSingleType, Instrumenter.HasMethodAdvice {
13+
14+
@Override
15+
public String instrumentedType() {
16+
return "com.alipay.sofa.rpc.server.bolt.BoltServerProcessor";
17+
}
18+
19+
@Override
20+
public void methodAdvice(MethodTransformer transformer) {
21+
transformer.applyAdvice(
22+
isMethod()
23+
.and(named("handleRequest"))
24+
.and(takesArguments(3))
25+
.and(takesArgument(2, named("com.alipay.sofa.rpc.core.request.SofaRequest"))),
26+
getClass().getName() + "$HandleRequestAdvice");
27+
}
28+
29+
public static class HandleRequestAdvice {
30+
@Advice.OnMethodEnter(suppress = Throwable.class)
31+
public static void enter() {
32+
SofaRpcProtocolContext.set("bolt");
33+
}
34+
35+
@Advice.OnMethodExit(onThrowable = Throwable.class, suppress = Throwable.class)
36+
public static void exit() {
37+
SofaRpcProtocolContext.clear();
38+
}
39+
}
40+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
package datadog.trace.instrumentation.sofarpc;
2+
3+
import static datadog.trace.agent.tooling.bytebuddy.matcher.NameMatchers.named;
4+
import static net.bytebuddy.matcher.ElementMatchers.isMethod;
5+
import static net.bytebuddy.matcher.ElementMatchers.takesNoArguments;
6+
7+
import datadog.trace.agent.tooling.Instrumenter;
8+
import net.bytebuddy.asm.Advice;
9+
10+
public class H2cServerTaskInstrumentation
11+
implements Instrumenter.ForSingleType, Instrumenter.HasMethodAdvice {
12+
13+
@Override
14+
public String instrumentedType() {
15+
return "com.alipay.sofa.rpc.server.http.AbstractHttpServerTask";
16+
}
17+
18+
@Override
19+
public void methodAdvice(MethodTransformer transformer) {
20+
transformer.applyAdvice(
21+
isMethod().and(named("run")).and(takesNoArguments()), getClass().getName() + "$RunAdvice");
22+
}
23+
24+
public static class RunAdvice {
25+
@Advice.OnMethodEnter(suppress = Throwable.class)
26+
public static void enter() {
27+
SofaRpcProtocolContext.set("h2c");
28+
}
29+
30+
@Advice.OnMethodExit(onThrowable = Throwable.class, suppress = Throwable.class)
31+
public static void exit() {
32+
SofaRpcProtocolContext.clear();
33+
}
34+
}
35+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
package datadog.trace.instrumentation.sofarpc;
2+
3+
import static datadog.trace.agent.tooling.bytebuddy.matcher.NameMatchers.named;
4+
import static datadog.trace.bootstrap.instrumentation.api.AgentPropagation.extractContextAndGetSpanContext;
5+
import static datadog.trace.bootstrap.instrumentation.api.AgentTracer.activateSpan;
6+
import static datadog.trace.bootstrap.instrumentation.api.AgentTracer.startSpan;
7+
import static datadog.trace.instrumentation.sofarpc.SofaRpcExtractAdapter.GETTER;
8+
import static datadog.trace.instrumentation.sofarpc.SofaRpcServerDecorator.DECORATE;
9+
import static datadog.trace.instrumentation.sofarpc.SofaRpcServerDecorator.SOFA_RPC_SERVER;
10+
import static net.bytebuddy.matcher.ElementMatchers.isMethod;
11+
import static net.bytebuddy.matcher.ElementMatchers.takesArgument;
12+
import static net.bytebuddy.matcher.ElementMatchers.takesArguments;
13+
14+
import com.alipay.sofa.rpc.core.request.SofaRequest;
15+
import com.alipay.sofa.rpc.core.response.SofaResponse;
16+
import datadog.trace.agent.tooling.Instrumenter;
17+
import datadog.trace.bootstrap.instrumentation.api.AgentScope;
18+
import datadog.trace.bootstrap.instrumentation.api.AgentSpan;
19+
import datadog.trace.bootstrap.instrumentation.api.AgentSpanContext;
20+
import net.bytebuddy.asm.Advice;
21+
22+
public class ProviderProxyInvokerInstrumentation
23+
implements Instrumenter.ForSingleType, Instrumenter.HasMethodAdvice {
24+
25+
@Override
26+
public String instrumentedType() {
27+
return "com.alipay.sofa.rpc.server.ProviderProxyInvoker";
28+
}
29+
30+
@Override
31+
public void methodAdvice(MethodTransformer transformer) {
32+
transformer.applyAdvice(
33+
isMethod()
34+
.and(named("invoke"))
35+
.and(takesArguments(1))
36+
.and(takesArgument(0, named("com.alipay.sofa.rpc.core.request.SofaRequest"))),
37+
getClass().getName() + "$InvokeAdvice");
38+
}
39+
40+
public static class InvokeAdvice {
41+
@Advice.OnMethodEnter(suppress = Throwable.class)
42+
public static AgentScope enter(@Advice.Argument(0) SofaRequest request) {
43+
// Protocol is set in thread-local by transport-specific instrumentation before this call.
44+
// If null, the transport is not instrumented — skip.
45+
String protocol = SofaRpcProtocolContext.get();
46+
if (protocol == null) {
47+
return null;
48+
}
49+
// For Bolt and H2C the client injects trace context into SofaRequest.requestProps;
50+
// extract it here. For Triple, parentContext will be null and startSpan() without an
51+
// explicit parent naturally attaches to the active grpc.server span. For REST,
52+
// parentContext will also be null and the active netty.request span becomes the parent.
53+
AgentSpanContext parentContext = extractContextAndGetSpanContext(request, GETTER);
54+
AgentSpan span =
55+
parentContext != null
56+
? startSpan(SOFA_RPC_SERVER, parentContext)
57+
: startSpan(SOFA_RPC_SERVER);
58+
DECORATE.afterStart(span);
59+
DECORATE.onRequest(span, request);
60+
span.setTag("sofarpc.protocol", protocol);
61+
return activateSpan(span);
62+
}
63+
64+
@Advice.OnMethodExit(onThrowable = Throwable.class, suppress = Throwable.class)
65+
public static void exit(
66+
@Advice.Enter AgentScope scope,
67+
@Advice.Return SofaResponse response,
68+
@Advice.Thrown Throwable throwable) {
69+
if (scope == null) {
70+
return;
71+
}
72+
AgentSpan span = scope.span();
73+
DECORATE.onResponse(span, response);
74+
DECORATE.onError(span, throwable);
75+
DECORATE.beforeFinish(span);
76+
scope.close();
77+
span.finish();
78+
}
79+
}
80+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
package datadog.trace.instrumentation.sofarpc;
2+
3+
import static datadog.trace.agent.tooling.bytebuddy.matcher.NameMatchers.named;
4+
import static net.bytebuddy.matcher.ElementMatchers.isMethod;
5+
import static net.bytebuddy.matcher.ElementMatchers.takesArguments;
6+
7+
import datadog.trace.agent.tooling.Instrumenter;
8+
import net.bytebuddy.asm.Advice;
9+
10+
public class RestServerHandlerInstrumentation
11+
implements Instrumenter.ForSingleType, Instrumenter.HasMethodAdvice {
12+
13+
@Override
14+
public String instrumentedType() {
15+
return "com.alipay.sofa.rpc.server.rest.SofaRestRequestHandler";
16+
}
17+
18+
@Override
19+
public void methodAdvice(MethodTransformer transformer) {
20+
transformer.applyAdvice(
21+
isMethod().and(named("channelRead0")).and(takesArguments(2)),
22+
getClass().getName() + "$ChannelRead0Advice");
23+
}
24+
25+
public static class ChannelRead0Advice {
26+
@Advice.OnMethodEnter(suppress = Throwable.class)
27+
public static void enter() {
28+
SofaRpcProtocolContext.set("rest");
29+
}
30+
31+
@Advice.OnMethodExit(onThrowable = Throwable.class, suppress = Throwable.class)
32+
public static void exit() {
33+
SofaRpcProtocolContext.clear();
34+
}
35+
}
36+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
package datadog.trace.instrumentation.sofarpc;
2+
3+
import com.alipay.sofa.rpc.core.request.SofaRequest;
4+
import com.alipay.sofa.rpc.core.response.SofaResponse;
5+
import datadog.trace.api.naming.SpanNaming;
6+
import datadog.trace.bootstrap.instrumentation.api.AgentSpan;
7+
import datadog.trace.bootstrap.instrumentation.api.InternalSpanTypes;
8+
import datadog.trace.bootstrap.instrumentation.api.Tags;
9+
import datadog.trace.bootstrap.instrumentation.api.UTF8BytesString;
10+
import datadog.trace.bootstrap.instrumentation.decorator.ClientDecorator;
11+
12+
public class SofaRpcClientDecorator extends ClientDecorator {
13+
14+
public static final CharSequence SOFA_RPC_CLIENT =
15+
UTF8BytesString.create(
16+
SpanNaming.instance().namingSchema().client().operationForProtocol("sofarpc"));
17+
18+
private static final CharSequence COMPONENT_NAME = UTF8BytesString.create("sofarpc-client");
19+
20+
public static final SofaRpcClientDecorator DECORATE = new SofaRpcClientDecorator();
21+
22+
@Override
23+
protected String[] instrumentationNames() {
24+
return new String[] {"sofarpc"};
25+
}
26+
27+
@Override
28+
protected CharSequence component() {
29+
return COMPONENT_NAME;
30+
}
31+
32+
@Override
33+
protected CharSequence spanType() {
34+
return InternalSpanTypes.RPC;
35+
}
36+
37+
@Override
38+
protected String service() {
39+
return null;
40+
}
41+
42+
public AgentSpan onRequest(AgentSpan span, SofaRequest request) {
43+
span.setTag("rpc.system", "sofarpc");
44+
if (request == null) {
45+
return span;
46+
}
47+
String serviceName = request.getTargetServiceUniqueName();
48+
String methodName = request.getMethodName();
49+
span.setTag(Tags.RPC_SERVICE, serviceName);
50+
span.setTag("rpc.method", methodName);
51+
// peer.service is derived automatically by PeerServiceCalculator from rpc.service.
52+
if (serviceName != null && methodName != null) {
53+
span.setResourceName(serviceName + "/" + methodName);
54+
} else if (methodName != null) {
55+
span.setResourceName(methodName);
56+
}
57+
return span;
58+
}
59+
60+
public AgentSpan onResponse(AgentSpan span, SofaResponse response) {
61+
if (response != null && response.isError()) {
62+
span.setError(true);
63+
span.setTag("error.message", response.getErrorMsg());
64+
}
65+
return span;
66+
}
67+
}

0 commit comments

Comments
 (0)