Skip to content

Commit 7fe6ead

Browse files
committed
fix(tracing): direct OTel SDK setup for chain-coherent sampling
Knative's config-observability ConfigMap only exposes a flat tracing-sampling-rate, so at fractional rates each service in the chain rolls independently — PaC can drop a trace while Tekton keeps it, leaving execution spans whose parent_spanID points at nothing. Switching to the OTel SDK opens up OTEL_TRACES_SAMPLER's parentbased_* family, which honors the root span's sample decision in the W3C traceparent flag so the whole chain is kept or dropped together. Controller and watcher call tracing.New() at startup. Tracing is opt-in: both OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_TRACES_SAMPLER must be set, otherwise PaC falls back to noop (matching the prior tracing-sampling-rate "0" default). Resource service.name is pipelines-as-code. Propagator is W3C TraceContext only; Baggage is intentionally not honored per Konflux-CI ADR 0061. otlptracegrpc and otlptracehttp promoted from indirect to direct dependencies. Assisted-by: Claude Code Signed-off-by: Josiah England <jengland@redhat.com>
1 parent 1c0fabc commit 7fe6ead

7 files changed

Lines changed: 324 additions & 39 deletions

File tree

config/305-config-observability.yaml

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -52,20 +52,6 @@ data:
5252
# Only applicable for grpc and http/protobuf protocols.
5353
# metrics-export-interval: "30s"
5454
55-
# tracing-protocol specifies the trace export protocol.
56-
# Supported values: "grpc", "http/protobuf", "none".
57-
# Default is "none" (tracing disabled).
58-
# tracing-protocol: "none"
59-
60-
# tracing-endpoint specifies the OTLP collector endpoint.
61-
# Required when tracing-protocol is "grpc" or "http/protobuf".
62-
# The OTEL_EXPORTER_OTLP_ENDPOINT env var takes precedence if set.
63-
# tracing-endpoint: "http://otel-collector.observability.svc.cluster.local:4317"
64-
65-
# tracing-sampling-rate controls the fraction of traces sampled.
66-
# 0.0 = none, 1.0 = all. Default is 0 (none).
67-
# tracing-sampling-rate: "1.0"
68-
6955
# runtime-profiling enables/disables the pprof profiling server on port 8008.
7056
# Supported values: "enabled", "disabled" (default).
7157
# runtime-profiling: "disabled"

docs/content/docs/operations/tracing.md

Lines changed: 12 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -7,35 +7,24 @@ This page describes how to enable OpenTelemetry distributed tracing for Pipeline
77

88
## Enabling tracing
99

10-
The ConfigMap `pipelines-as-code-config-observability` controls tracing configuration. It must exist in the same namespace as the Pipelines-as-Code controller and watcher deployments. See [config/305-config-observability.yaml](https://github.com/tektoncd/pipelines-as-code/blob/main/config/305-config-observability.yaml) for the full example.
10+
Pipelines-as-Code reads tracing configuration from standard OpenTelemetry environment variables on the controller and watcher pods:
1111

12-
It contains the following tracing fields:
12+
* `OTEL_EXPORTER_OTLP_ENDPOINT`: OTLP collector endpoint URL. Required to enable tracing.
13+
* `OTEL_TRACES_SAMPLER`: Sampler family. Required to enable tracing. Supported values: `always_on`, `always_off`, `traceidratio`, `parentbased_always_on`, `parentbased_always_off`, `parentbased_traceidratio`.
14+
* `OTEL_TRACES_SAMPLER_ARG`: Numeric argument for ratio-based samplers. Set to `0.1` with `OTEL_TRACES_SAMPLER=parentbased_traceidratio` to sample 10% of root traces while keeping the chain coherent on those that are kept.
15+
* `OTEL_EXPORTER_OTLP_PROTOCOL` (or traces-specific `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`): OTLP transport. Supported values: `grpc`, `http/protobuf`. Default: `grpc`.
1316

14-
* `tracing-protocol`: Export protocol. Supported values: `grpc`, `http/protobuf`, `none`. Default is `none` (tracing disabled).
15-
* `tracing-endpoint`: OTLP collector endpoint. Required when protocol is not `none`. The `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable takes precedence if set.
16-
* `tracing-sampling-rate`: Fraction of traces to sample. `0.0` = none, `1.0` = all. Default is `0`.
17+
Both `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_TRACES_SAMPLER` must be set to opt in to tracing. If either is unset PaC falls back to a noop tracer that emits no spans and incurs no exporter cost. Changes to any of these env vars take effect on the next pod restart.
1718

18-
### Example
19+
PaC honors inbound `traceparent` headers on incoming webhook requests via the W3C TraceContext propagator. OTel Baggage is intentionally not honored: every emission point in PaC already has the attributes it needs from the local PipelineRun and webhook event, so no cross-service attribute propagation channel is needed.
1920

20-
```yaml
21-
apiVersion: v1
22-
kind: ConfigMap
23-
metadata:
24-
name: pipelines-as-code-config-observability
25-
namespace: pipelines-as-code
26-
data:
27-
tracing-protocol: grpc
28-
tracing-endpoint: "http://otel-collector.observability.svc.cluster.local:4317"
29-
tracing-sampling-rate: "1.0"
30-
```
21+
### Sampler choice and chain coherency
3122

32-
Changes to `tracing-protocol`, `tracing-endpoint`, and `tracing-sampling-rate` require restarting the controller and watcher pods. The trace exporter is created once at startup from the ConfigMap values at that time. Set `tracing-protocol` to `none` or remove the tracing keys to disable tracing.
33-
34-
The controller and watcher locate this ConfigMap by name via the `CONFIG_OBSERVABILITY_NAME` environment variable set in their deployment manifests. Operator-based installations may manage this differently; consult the operator documentation for details.
23+
The `parentbased_*` sampler family honors the parent span's sample decision carried in the W3C `traceparent` flag bit. When every service in the delivery chain uses parent-based samplers, the root span's sampling decision propagates end to end: each service either keeps its spans or drops them based on what the root chose. Flat-rate samplers (`traceidratio` without parent-based) cause each service to roll independently, which at fractional sampling fragments the chain into orphaned spans whose `parent_spanID` references a span that was dropped. `parentbased_always_on` keeps everything; `parentbased_traceidratio` with a numeric argument samples a coherent fraction.
3524

3625
## Emitted spans
3726

38-
The controller emits a `PipelinesAsCode:ProcessEvent` span for each webhook event. The watcher emits `waitDuration` and `executeDuration` spans for completed PipelineRuns.
27+
The controller emits a `PipelinesAsCode:ProcessEvent` span for each webhook event. The watcher emits `waitDuration` and `executeDuration` spans for completed PipelineRuns. The OTel resource attribute `service.name` on all emitted spans is `pipelines-as-code`.
3928

4029
### Webhook event span (`PipelinesAsCode:ProcessEvent`)
4130

@@ -103,13 +92,13 @@ Unlike the observability ConfigMap above (which requires a pod restart), changes
10392

10493
## Trace context propagation
10594

106-
When Pipelines-as-Code creates a PipelineRun, it sets the `tekton.dev/pipelinerunSpanContext` annotation with a JSON-encoded OTel TextMapCarrier containing the W3C `traceparent`. PaC tracing works independently you get PaC spans regardless of whether Tekton Pipelines has tracing enabled.
95+
When Pipelines-as-Code creates a PipelineRun, it sets the `tekton.dev/pipelinerunSpanContext` annotation with a JSON-encoded OTel TextMapCarrier containing the W3C `traceparent`. PaC tracing works independently - you get PaC spans regardless of whether Tekton Pipelines has tracing enabled.
10796

10897
If Tekton Pipelines is also configured with tracing pointing at the same collector, its reconciler spans appear as children of the PaC span, providing a single end-to-end trace from webhook receipt through task execution. See the [Tekton Pipelines tracing documentation](https://github.com/tektoncd/pipeline/blob/main/docs/developers/tracing.md) for Tekton's independent tracing setup.
10998

11099
## Deploying a trace collector
111100

112-
Pipelines-as-Code exports traces using the standard OpenTelemetry Protocol (OTLP). You need a running OTLP-compatible collector for the `tracing-endpoint` to point to. Common options include:
101+
Pipelines-as-Code exports traces using the standard OpenTelemetry Protocol (OTLP). You need a running OTLP-compatible collector for `OTEL_EXPORTER_OTLP_ENDPOINT` to point to. Common options include:
113102

114103
* [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) -- the vendor-neutral reference collector
115104
* [Jaeger](https://www.jaegertracing.io/docs/latest/getting-started/) -- supports OTLP ingestion natively since v1.35

go.mod

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,8 @@ require (
3030
github.com/tektoncd/pipeline v1.11.1
3131
gitlab.com/gitlab-org/api/client-go v1.46.0
3232
go.opentelemetry.io/otel v1.43.0
33+
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.43.0
34+
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.43.0
3335
go.opentelemetry.io/otel/metric v1.43.0
3436
go.opentelemetry.io/otel/sdk v1.43.0
3537
go.opentelemetry.io/otel/sdk/metric v1.43.0
@@ -90,8 +92,6 @@ require (
9092
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.43.0 // indirect
9193
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.43.0 // indirect
9294
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.43.0 // indirect
93-
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.43.0 // indirect
94-
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.43.0 // indirect
9595
go.opentelemetry.io/otel/exporters/prometheus v0.65.0 // indirect
9696
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.43.0 // indirect
9797
go.opentelemetry.io/proto/otlp v1.10.0 // indirect

pkg/adapter/adapter.go

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,13 @@ func New(run *params.Run, k *kubeinteraction.Interaction) adapter.AdapterConstru
7979
}
8080

8181
func (l *listener) Start(ctx context.Context) error {
82+
tp := tracing.New(l.logger)
83+
defer func() {
84+
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
85+
defer cancel()
86+
_ = tp.Shutdown(shutdownCtx)
87+
}()
88+
8289
adapterPort := globalAdapterPort
8390
envAdapterPort := os.Getenv("PAC_CONTROLLER_PORT")
8491
if envAdapterPort != "" {

pkg/reconciler/controller.go

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ package reconciler
33
import (
44
"context"
55
"path"
6+
"time"
67

78
"github.com/openshift-pipelines/pipelines-as-code/pkg/apis/pipelinesascode"
89
"github.com/openshift-pipelines/pipelines-as-code/pkg/apis/pipelinesascode/keys"
@@ -14,6 +15,7 @@ import (
1415
"github.com/openshift-pipelines/pipelines-as-code/pkg/params/info"
1516
prmetrics "github.com/openshift-pipelines/pipelines-as-code/pkg/pipelinerunmetrics"
1617
queuepkg "github.com/openshift-pipelines/pipelines-as-code/pkg/queue"
18+
"github.com/openshift-pipelines/pipelines-as-code/pkg/tracing"
1719
tektonv1 "github.com/tektoncd/pipeline/pkg/apis/pipeline/v1"
1820
tektonPipelineRunInformerv1 "github.com/tektoncd/pipeline/pkg/client/injection/informers/pipeline/v1/pipelinerun"
1921
tektonPipelineRunReconcilerv1 "github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1/pipelinerun"
@@ -30,6 +32,15 @@ func NewController() func(context.Context, configmap.Watcher) *controller.Impl {
3032
ctx = info.StoreNS(ctx, system.Namespace())
3133
log := logging.FromContext(ctx)
3234

35+
tp := tracing.New(log)
36+
// linter false positive: fresh Background is required because outer ctx is cancelled past <-ctx.Done().
37+
go func() { //nolint:gosec
38+
<-ctx.Done()
39+
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
40+
defer cancel()
41+
_ = tp.Shutdown(shutdownCtx)
42+
}()
43+
3344
run := params.New()
3445
err := run.Clients.NewClients(ctx, &run.Info)
3546
if err != nil {

pkg/tracing/provider.go

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
package tracing
2+
3+
import (
4+
"context"
5+
"os"
6+
"strconv"
7+
8+
"go.opentelemetry.io/otel"
9+
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
10+
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
11+
"go.opentelemetry.io/otel/propagation"
12+
"go.opentelemetry.io/otel/sdk/resource"
13+
sdktrace "go.opentelemetry.io/otel/sdk/trace"
14+
semconv "go.opentelemetry.io/otel/semconv/v1.40.0"
15+
"go.uber.org/zap"
16+
)
17+
18+
const (
19+
EnvOTLPEndpoint = "OTEL_EXPORTER_OTLP_ENDPOINT"
20+
EnvOTLPProtocol = "OTEL_EXPORTER_OTLP_PROTOCOL"
21+
EnvOTLPTracesProtocol = "OTEL_EXPORTER_OTLP_TRACES_PROTOCOL"
22+
EnvTracesSampler = "OTEL_TRACES_SAMPLER"
23+
EnvTracesSamplerArg = "OTEL_TRACES_SAMPLER_ARG"
24+
25+
protocolGRPC = "grpc"
26+
protocolHTTP = "http/protobuf"
27+
)
28+
29+
type TracerProvider struct {
30+
shutdown func(context.Context) error
31+
}
32+
33+
func New(logger *zap.SugaredLogger) *TracerProvider {
34+
if os.Getenv(EnvOTLPEndpoint) == "" {
35+
logger.Info("OTLP endpoint not configured, using noop tracer provider")
36+
return noopProvider()
37+
}
38+
if os.Getenv(EnvTracesSampler) == "" {
39+
logger.Info("OTEL_TRACES_SAMPLER not set, tracing disabled (set explicitly to opt in)")
40+
return noopProvider()
41+
}
42+
43+
proto := protocolFromEnv()
44+
exporter, err := newExporter(context.Background(), logger, proto)
45+
if err != nil {
46+
logger.Errorw("failed to create OTLP exporter, using noop tracer provider", "error", err)
47+
return noopProvider()
48+
}
49+
50+
res, err := resource.Merge(
51+
resource.Default(),
52+
resource.NewWithAttributes(
53+
semconv.SchemaURL,
54+
semconv.ServiceName(TracerName),
55+
),
56+
)
57+
if err != nil {
58+
logger.Errorw("failed to create resource", "error", err)
59+
res = resource.Default()
60+
}
61+
62+
tp := sdktrace.NewTracerProvider(
63+
sdktrace.WithBatcher(exporter),
64+
sdktrace.WithResource(res),
65+
sdktrace.WithSampler(samplerFromEnv(logger)),
66+
)
67+
68+
otel.SetTracerProvider(tp)
69+
otel.SetTextMapPropagator(propagation.TraceContext{})
70+
71+
logger.Infow("tracing initialized", "endpoint", os.Getenv(EnvOTLPEndpoint), "protocol", proto)
72+
73+
return &TracerProvider{shutdown: tp.Shutdown}
74+
}
75+
76+
func noopProvider() *TracerProvider {
77+
return &TracerProvider{shutdown: func(context.Context) error { return nil }}
78+
}
79+
80+
func protocolFromEnv() string {
81+
if v := os.Getenv(EnvOTLPTracesProtocol); v != "" {
82+
return v
83+
}
84+
if v := os.Getenv(EnvOTLPProtocol); v != "" {
85+
return v
86+
}
87+
return protocolGRPC
88+
}
89+
90+
func newExporter(ctx context.Context, logger *zap.SugaredLogger, proto string) (sdktrace.SpanExporter, error) {
91+
endpoint := os.Getenv(EnvOTLPEndpoint)
92+
switch proto {
93+
case protocolHTTP:
94+
return otlptracehttp.New(ctx, otlptracehttp.WithEndpointURL(endpoint))
95+
case protocolGRPC:
96+
return otlptracegrpc.New(ctx, otlptracegrpc.WithEndpointURL(endpoint))
97+
default:
98+
logger.Errorw("unsupported OTLP protocol; falling back to grpc", "protocol", proto)
99+
return otlptracegrpc.New(ctx, otlptracegrpc.WithEndpointURL(endpoint))
100+
}
101+
}
102+
103+
func (tp *TracerProvider) Shutdown(ctx context.Context) error {
104+
if tp.shutdown != nil {
105+
return tp.shutdown(ctx)
106+
}
107+
return nil
108+
}
109+
110+
func samplerFromEnv(logger *zap.SugaredLogger) sdktrace.Sampler {
111+
name := os.Getenv(EnvTracesSampler)
112+
argStr := os.Getenv(EnvTracesSamplerArg)
113+
arg, err := strconv.ParseFloat(argStr, 64)
114+
if err != nil && argStr != "" {
115+
logger.Errorw("ignoring malformed sampler argument", "env", EnvTracesSamplerArg, "value", argStr)
116+
}
117+
if argStr == "" && (name == "traceidratio" || name == "parentbased_traceidratio") {
118+
logger.Infow("ratio sampler selected without "+EnvTracesSamplerArg+"; defaulting to 0% sampling", "env", EnvTracesSampler, "value", name)
119+
}
120+
switch name {
121+
case "always_on":
122+
return sdktrace.AlwaysSample()
123+
case "always_off":
124+
return sdktrace.NeverSample()
125+
case "traceidratio":
126+
return sdktrace.TraceIDRatioBased(arg)
127+
case "parentbased_always_on":
128+
return sdktrace.ParentBased(sdktrace.AlwaysSample())
129+
case "parentbased_always_off":
130+
return sdktrace.ParentBased(sdktrace.NeverSample())
131+
case "parentbased_traceidratio":
132+
return sdktrace.ParentBased(sdktrace.TraceIDRatioBased(arg))
133+
}
134+
logger.Warnw("unrecognized OTEL_TRACES_SAMPLER value; falling back to never sample", "value", name)
135+
return sdktrace.NeverSample()
136+
}

0 commit comments

Comments
 (0)