Skip to content

Commit f26c7ac

Browse files
ericfirthclaude
andauthored
Clarify Go DSM context propagation and add goroutine/channel guidance (#37623)
* docs(dsm): clarify Go context propagation and add goroutine/channel guidance Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * docs(dsm): address PR review feedback on Go context propagation section Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent b649429 commit f26c7ac

1 file changed

Lines changed: 36 additions & 1 deletion

File tree

  • content/en/data_streams/setup/language

content/en/data_streams/setup/language/go.md

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,7 @@ producer, err := ddkafka.NewProducer(&kafka.ConfigMap{
9292
}, ddkafka.WithDataStreams())
9393
```
9494

95-
If a service consumes data from one point and produces to another point, propagate context between the two places using the Go context structure:
95+
If a service consumes data from one point and produces to another point, propagate context between the two places using the Go context structure. The `ctx` returned by `ExtractFromBase64Carrier` carries the upstream DSM pathway. Pass it to `SetDataStreamsCheckpointWithParams` when you produce, then inject it into the outbound message with `InjectToBase64Carrier`. Passing `context.Background()` at the produce site creates a new pathway root and breaks end-to-end visibility. This happens, for example, in a worker goroutine that has lost the consume `ctx`.
9696

9797
3. Extract the context from headers
9898

@@ -105,6 +105,41 @@ If a service consumes data from one point and produces to another point, propaga
105105
datastreams.InjectToBase64Carrier(ctx, ddsarama.NewProducerMessageCarrier(message))
106106
```
107107

108+
##### Goroutines and channels
109+
110+
Go channels and goroutines do not carry `context.Context` automatically. If your service fans consumed messages out to worker goroutines before producing, pass the consume `ctx` to the produce site by including it in the work item you send over the channel:
111+
112+
```go
113+
type job struct {
114+
ctx context.Context
115+
payload []byte
116+
}
117+
118+
// consume side
119+
ctx, _ = tracer.SetDataStreamsCheckpointWithParams(
120+
datastreams.ExtractFromBase64Carrier(context.Background(), ddsarama.NewConsumerMessageCarrier(msg)),
121+
options.CheckpointParams{PayloadSize: int64(len(msg.Value))},
122+
"direction:in", "type:kafka", "topic:"+inTopic, "group:"+group,
123+
)
124+
// context.WithoutCancel preserves the pathway if the handler's ctx is canceled before the worker runs (Go 1.21+)
125+
jobs <- job{ctx: context.WithoutCancel(ctx), payload: msg.Value}
126+
127+
// worker goroutine
128+
for j := range jobs {
129+
out := &sarama.ProducerMessage{Topic: outTopic, Value: sarama.ByteEncoder(j.payload)}
130+
ctx, ok := tracer.SetDataStreamsCheckpointWithParams(j.ctx,
131+
options.CheckpointParams{PayloadSize: int64(out.Value.Length())},
132+
"direction:out", "type:kafka", "topic:"+outTopic)
133+
if ok {
134+
datastreams.InjectToBase64Carrier(ctx, ddsarama.NewProducerMessageCarrier(out))
135+
}
136+
producer.SendMessage(out)
137+
}
138+
```
139+
140+
- **Fan-out**: When one consumed message fans out to multiple produce calls, pass the same consume `ctx` to each produce checkpoint. Each call creates its own child node in the pathway.
141+
- **Fan-in**: When many consumed messages merge into one produce call, combine the inbound contexts with `datastreams.MergeContexts(ctxs...)` before producing.
142+
108143
#### Other queuing technologies or protocols
109144

110145
You can also use manual instrumentation. For example, you can propagate context through Kinesis.

0 commit comments

Comments
 (0)