Skip to content

Commit fd632f9

Browse files
committed
fix(tracing): direct OTel SDK setup for chain-coherent sampling
Knative's config-observability ConfigMap only exposes a flat tracing-sampling-rate, so at fractional rates each service in the chain rolls independently — PaC can drop a trace while Tekton keeps it, leaving execution spans whose parent_spanID points at nothing. Switching to the OTel SDK opens up OTEL_TRACES_SAMPLER's parentbased_* family, which honors the root span's sample decision in the W3C traceparent flag so the whole chain is kept or dropped together. Controller and watcher call tracing.New() at startup. Tracing is opt-in: both OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_TRACES_SAMPLER must be set, otherwise PaC falls back to noop (matching the prior tracing-sampling-rate "0" default). Resource service.name is pipelines-as-code. Propagator is W3C TraceContext only; Baggage is intentionally not honored per Konflux-CI ADR 0061. otlptracegrpc and otlptracehttp promoted from indirect to direct dependencies. Assisted-by: Claude Code Signed-off-by: Josiah England <jengland@redhat.com>
1 parent fff1dac commit fd632f9

7 files changed

Lines changed: 330 additions & 39 deletions

File tree

config/305-config-observability.yaml

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -52,20 +52,6 @@ data:
5252
# Only applicable for grpc and http/protobuf protocols.
5353
# metrics-export-interval: "30s"
5454
55-
# tracing-protocol specifies the trace export protocol.
56-
# Supported values: "grpc", "http/protobuf", "none".
57-
# Default is "none" (tracing disabled).
58-
# tracing-protocol: "none"
59-
60-
# tracing-endpoint specifies the OTLP collector endpoint.
61-
# Required when tracing-protocol is "grpc" or "http/protobuf".
62-
# The OTEL_EXPORTER_OTLP_ENDPOINT env var takes precedence if set.
63-
# tracing-endpoint: "http://otel-collector.observability.svc.cluster.local:4317"
64-
65-
# tracing-sampling-rate controls the fraction of traces sampled.
66-
# 0.0 = none, 1.0 = all. Default is 0 (none).
67-
# tracing-sampling-rate: "1.0"
68-
6955
# runtime-profiling enables/disables the pprof profiling server on port 8008.
7056
# Supported values: "enabled", "disabled" (default).
7157
# runtime-profiling: "disabled"

docs/content/docs/operations/tracing.md

Lines changed: 16 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -3,39 +3,32 @@ title: Distributed Tracing
33
weight: 5
44
---
55

6+
{{< hint warning >}}
7+
The old `tracing-protocol`, `tracing-endpoint`, and `tracing-sampling-rate` ConfigMap keys no longer work. Configure tracing via OpenTelemetry environment variables (see below).
8+
{{< /hint >}}
9+
610
This page describes how to enable OpenTelemetry distributed tracing for Pipelines-as-Code. When enabled, PaC emits trace spans for webhook event processing and PipelineRun lifecycle timing.
711

812
## Enabling tracing
913

10-
The ConfigMap `pipelines-as-code-config-observability` controls tracing configuration. It must exist in the same namespace as the Pipelines-as-Code controller and watcher deployments. See [config/305-config-observability.yaml](https://github.com/tektoncd/pipelines-as-code/blob/main/config/305-config-observability.yaml) for the full example.
11-
12-
It contains the following tracing fields:
14+
Pipelines-as-Code reads tracing configuration from standard OpenTelemetry environment variables on the controller and watcher pods:
1315

14-
* `tracing-protocol`: Export protocol. Supported values: `grpc`, `http/protobuf`, `none`. Default is `none` (tracing disabled).
15-
* `tracing-endpoint`: OTLP collector endpoint. Required when protocol is not `none`. The `OTEL_EXPORTER_OTLP_ENDPOINT` environment variable takes precedence if set.
16-
* `tracing-sampling-rate`: Fraction of traces to sample. `0.0` = none, `1.0` = all. Default is `0`.
16+
* `OTEL_EXPORTER_OTLP_ENDPOINT`: OTLP collector endpoint URL. Required to enable tracing.
17+
* `OTEL_TRACES_SAMPLER`: Sampler family. Required to enable tracing. Supported values: `always_on`, `always_off`, `traceidratio`, `parentbased_always_on`, `parentbased_always_off`, `parentbased_traceidratio`.
18+
* `OTEL_TRACES_SAMPLER_ARG`: Numeric argument for ratio-based samplers. Set to `0.1` with `OTEL_TRACES_SAMPLER=parentbased_traceidratio` to sample 10% of root traces while keeping the chain coherent on those that are kept.
19+
* `OTEL_EXPORTER_OTLP_PROTOCOL` (or traces-specific `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`): OTLP transport. Supported values: `grpc`, `http/protobuf`. Default: `grpc`.
1720

18-
### Example
21+
Both `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_TRACES_SAMPLER` must be set to opt in to tracing. If either is unset PaC falls back to a noop tracer that emits no spans and incurs no exporter cost. Changes to any of these env vars take effect on the next pod restart.
1922

20-
```yaml
21-
apiVersion: v1
22-
kind: ConfigMap
23-
metadata:
24-
name: pipelines-as-code-config-observability
25-
namespace: pipelines-as-code
26-
data:
27-
tracing-protocol: grpc
28-
tracing-endpoint: "http://otel-collector.observability.svc.cluster.local:4317"
29-
tracing-sampling-rate: "1.0"
30-
```
23+
PaC honors inbound `traceparent` headers on incoming webhook requests via the W3C TraceContext propagator. OTel Baggage is intentionally not honored: every emission point in PaC already has the attributes it needs from the local PipelineRun and webhook event, so no cross-service attribute propagation channel is needed.
3124

32-
Changes to `tracing-protocol`, `tracing-endpoint`, and `tracing-sampling-rate` require restarting the controller and watcher pods. The trace exporter is created once at startup from the ConfigMap values at that time. Set `tracing-protocol` to `none` or remove the tracing keys to disable tracing.
25+
### Sampler choice and chain coherency
3326

34-
The controller and watcher locate this ConfigMap by name via the `CONFIG_OBSERVABILITY_NAME` environment variable set in their deployment manifests. Operator-based installations may manage this differently; consult the operator documentation for details.
27+
The `parentbased_*` sampler family honors the parent span's sample decision carried in the W3C `traceparent` flag bit. When every service in the delivery chain uses parent-based samplers, the root span's sampling decision propagates end to end: each service either keeps its spans or drops them based on what the root chose. Flat-rate samplers (`traceidratio` without parent-based) cause each service to roll independently, which at fractional sampling fragments the chain into orphaned spans whose `parent_spanID` references a span that was dropped. `parentbased_always_on` keeps everything; `parentbased_traceidratio` with a numeric argument samples a coherent fraction.
3528

3629
## Emitted spans
3730

38-
The controller emits a `PipelinesAsCode:ProcessEvent` span for each webhook event. The watcher emits `waitDuration` and `executeDuration` spans for completed PipelineRuns.
31+
The controller emits a `PipelinesAsCode:ProcessEvent` span for each webhook event. The watcher emits `waitDuration` and `executeDuration` spans for completed PipelineRuns. The OTel resource attribute `service.name` on all emitted spans is `pipelines-as-code`.
3932

4033
### Webhook event span (`PipelinesAsCode:ProcessEvent`)
4134

@@ -103,13 +96,13 @@ Unlike the observability ConfigMap above (which requires a pod restart), changes
10396

10497
## Trace context propagation
10598

106-
When Pipelines-as-Code creates a PipelineRun, it sets the `tekton.dev/pipelinerunSpanContext` annotation with a JSON-encoded OTel TextMapCarrier containing the W3C `traceparent`. PaC tracing works independently you get PaC spans regardless of whether Tekton Pipelines has tracing enabled.
99+
When Pipelines-as-Code creates a PipelineRun, it sets the `tekton.dev/pipelinerunSpanContext` annotation with a JSON-encoded OTel TextMapCarrier containing the W3C `traceparent`. PaC tracing works independently - you get PaC spans regardless of whether Tekton Pipelines has tracing enabled.
107100

108101
If Tekton Pipelines is also configured with tracing pointing at the same collector, its reconciler spans appear as children of the PaC span, providing a single end-to-end trace from webhook receipt through task execution. See the [Tekton Pipelines tracing documentation](https://github.com/tektoncd/pipeline/blob/main/docs/developers/tracing.md) for Tekton's independent tracing setup.
109102

110103
## Deploying a trace collector
111104

112-
Pipelines-as-Code exports traces using the standard OpenTelemetry Protocol (OTLP). You need a running OTLP-compatible collector for the `tracing-endpoint` to point to. Common options include:
105+
Pipelines-as-Code exports traces using the standard OpenTelemetry Protocol (OTLP). You need a running OTLP-compatible collector for `OTEL_EXPORTER_OTLP_ENDPOINT` to point to. Common options include:
113106

114107
* [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) -- the vendor-neutral reference collector
115108
* [Jaeger](https://www.jaegertracing.io/docs/latest/getting-started/) -- supports OTLP ingestion natively since v1.35

go.mod

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,8 @@ require (
3030
github.com/tektoncd/pipeline v1.13.1
3131
gitlab.com/gitlab-org/api/client-go v1.46.0
3232
go.opentelemetry.io/otel v1.44.0
33+
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.43.0
34+
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.43.0
3335
go.opentelemetry.io/otel/metric v1.44.0
3436
go.opentelemetry.io/otel/sdk v1.43.0
3537
go.opentelemetry.io/otel/sdk/metric v1.43.0
@@ -90,8 +92,6 @@ require (
9092
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.43.0 // indirect
9193
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.43.0 // indirect
9294
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.43.0 // indirect
93-
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.43.0 // indirect
94-
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.43.0 // indirect
9595
go.opentelemetry.io/otel/exporters/prometheus v0.65.0 // indirect
9696
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.43.0 // indirect
9797
go.opentelemetry.io/proto/otlp v1.10.0 // indirect

pkg/adapter/adapter.go

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,13 @@ func New(run *params.Run, k *kubeinteraction.Interaction) adapter.AdapterConstru
7979
}
8080

8181
func (l *listener) Start(ctx context.Context) error {
82+
tp := tracing.New(l.logger)
83+
defer func() {
84+
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
85+
defer cancel()
86+
_ = tp.Shutdown(shutdownCtx)
87+
}()
88+
8289
adapterPort := globalAdapterPort
8390
envAdapterPort := os.Getenv("PAC_CONTROLLER_PORT")
8491
if envAdapterPort != "" {

pkg/reconciler/controller.go

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ package reconciler
33
import (
44
"context"
55
"path"
6+
"time"
67

78
"github.com/openshift-pipelines/pipelines-as-code/pkg/apis/pipelinesascode"
89
"github.com/openshift-pipelines/pipelines-as-code/pkg/apis/pipelinesascode/keys"
@@ -14,6 +15,7 @@ import (
1415
"github.com/openshift-pipelines/pipelines-as-code/pkg/params/info"
1516
prmetrics "github.com/openshift-pipelines/pipelines-as-code/pkg/pipelinerunmetrics"
1617
queuepkg "github.com/openshift-pipelines/pipelines-as-code/pkg/queue"
18+
"github.com/openshift-pipelines/pipelines-as-code/pkg/tracing"
1719
tektonv1 "github.com/tektoncd/pipeline/pkg/apis/pipeline/v1"
1820
tektonPipelineRunInformerv1 "github.com/tektoncd/pipeline/pkg/client/injection/informers/pipeline/v1/pipelinerun"
1921
tektonPipelineRunReconcilerv1 "github.com/tektoncd/pipeline/pkg/client/injection/reconciler/pipeline/v1/pipelinerun"
@@ -30,6 +32,17 @@ func NewController() func(context.Context, configmap.Watcher) *controller.Impl {
3032
ctx = info.StoreNS(ctx, system.Namespace())
3133
log := logging.FromContext(ctx)
3234

35+
tp := tracing.New(log)
36+
// linter false positive: fresh Background is required because outer ctx is cancelled past <-ctx.Done().
37+
go func() { //nolint:gosec
38+
<-ctx.Done()
39+
shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
40+
defer cancel()
41+
if err := tp.Shutdown(shutdownCtx); err != nil {
42+
log.Errorw("failed to shut down tracer provider", "error", err)
43+
}
44+
}()
45+
3346
run := params.New()
3447
err := run.Clients.NewClients(ctx, &run.Info)
3548
if err != nil {

pkg/tracing/provider.go

Lines changed: 136 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,136 @@
1+
package tracing
2+
3+
import (
4+
"context"
5+
"os"
6+
"strconv"
7+
8+
"go.opentelemetry.io/otel"
9+
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
10+
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
11+
"go.opentelemetry.io/otel/propagation"
12+
"go.opentelemetry.io/otel/sdk/resource"
13+
sdktrace "go.opentelemetry.io/otel/sdk/trace"
14+
semconv "go.opentelemetry.io/otel/semconv/v1.40.0"
15+
"go.uber.org/zap"
16+
)
17+
18+
const (
19+
EnvOTLPEndpoint = "OTEL_EXPORTER_OTLP_ENDPOINT"
20+
EnvOTLPProtocol = "OTEL_EXPORTER_OTLP_PROTOCOL"
21+
EnvOTLPTracesProtocol = "OTEL_EXPORTER_OTLP_TRACES_PROTOCOL"
22+
EnvTracesSampler = "OTEL_TRACES_SAMPLER"
23+
EnvTracesSamplerArg = "OTEL_TRACES_SAMPLER_ARG"
24+
25+
protocolGRPC = "grpc"
26+
protocolHTTP = "http/protobuf"
27+
)
28+
29+
type TracerProvider struct {
30+
shutdown func(context.Context) error
31+
}
32+
33+
func New(logger *zap.SugaredLogger) *TracerProvider {
34+
if os.Getenv(EnvOTLPEndpoint) == "" {
35+
logger.Info("OTLP endpoint not configured, using noop tracer provider")
36+
return noopProvider()
37+
}
38+
if os.Getenv(EnvTracesSampler) == "" {
39+
logger.Info("OTEL_TRACES_SAMPLER not set, tracing disabled (set explicitly to opt in)")
40+
return noopProvider()
41+
}
42+
43+
proto := protocolFromEnv()
44+
exporter, err := newExporter(context.Background(), logger, proto)
45+
if err != nil {
46+
logger.Errorw("failed to create OTLP exporter, using noop tracer provider", "error", err)
47+
return noopProvider()
48+
}
49+
50+
res, err := resource.Merge(
51+
resource.Default(),
52+
resource.NewWithAttributes(
53+
semconv.SchemaURL,
54+
semconv.ServiceName(TracerName),
55+
),
56+
)
57+
if err != nil {
58+
logger.Errorw("failed to create resource", "error", err)
59+
res = resource.Default()
60+
}
61+
62+
tp := sdktrace.NewTracerProvider(
63+
sdktrace.WithBatcher(exporter),
64+
sdktrace.WithResource(res),
65+
sdktrace.WithSampler(samplerFromEnv(logger)),
66+
)
67+
68+
otel.SetTracerProvider(tp)
69+
otel.SetTextMapPropagator(propagation.TraceContext{})
70+
71+
logger.Infow("tracing initialized", "endpoint", os.Getenv(EnvOTLPEndpoint), "protocol", proto)
72+
73+
return &TracerProvider{shutdown: tp.Shutdown}
74+
}
75+
76+
func noopProvider() *TracerProvider {
77+
return &TracerProvider{shutdown: func(context.Context) error { return nil }}
78+
}
79+
80+
func protocolFromEnv() string {
81+
if v := os.Getenv(EnvOTLPTracesProtocol); v != "" {
82+
return v
83+
}
84+
if v := os.Getenv(EnvOTLPProtocol); v != "" {
85+
return v
86+
}
87+
return protocolGRPC
88+
}
89+
90+
func newExporter(ctx context.Context, logger *zap.SugaredLogger, proto string) (sdktrace.SpanExporter, error) {
91+
endpoint := os.Getenv(EnvOTLPEndpoint)
92+
switch proto {
93+
case protocolHTTP:
94+
return otlptracehttp.New(ctx, otlptracehttp.WithEndpointURL(endpoint))
95+
case protocolGRPC:
96+
return otlptracegrpc.New(ctx, otlptracegrpc.WithEndpointURL(endpoint))
97+
default:
98+
logger.Errorw("unsupported OTLP protocol; falling back to grpc", "protocol", proto)
99+
return otlptracegrpc.New(ctx, otlptracegrpc.WithEndpointURL(endpoint))
100+
}
101+
}
102+
103+
func (tp *TracerProvider) Shutdown(ctx context.Context) error {
104+
if tp.shutdown != nil {
105+
return tp.shutdown(ctx)
106+
}
107+
return nil
108+
}
109+
110+
func samplerFromEnv(logger *zap.SugaredLogger) sdktrace.Sampler {
111+
name := os.Getenv(EnvTracesSampler)
112+
argStr := os.Getenv(EnvTracesSamplerArg)
113+
arg, err := strconv.ParseFloat(argStr, 64)
114+
if err != nil && argStr != "" {
115+
logger.Errorw("ignoring malformed sampler argument", "env", EnvTracesSamplerArg, "value", argStr)
116+
}
117+
if argStr == "" && (name == "traceidratio" || name == "parentbased_traceidratio") {
118+
logger.Infow("ratio sampler selected without "+EnvTracesSamplerArg+"; defaulting to 0% sampling", "env", EnvTracesSampler, "value", name)
119+
}
120+
switch name {
121+
case "always_on":
122+
return sdktrace.AlwaysSample()
123+
case "always_off":
124+
return sdktrace.NeverSample()
125+
case "traceidratio":
126+
return sdktrace.TraceIDRatioBased(arg)
127+
case "parentbased_always_on":
128+
return sdktrace.ParentBased(sdktrace.AlwaysSample())
129+
case "parentbased_always_off":
130+
return sdktrace.ParentBased(sdktrace.NeverSample())
131+
case "parentbased_traceidratio":
132+
return sdktrace.ParentBased(sdktrace.TraceIDRatioBased(arg))
133+
}
134+
logger.Warnw("unrecognized OTEL_TRACES_SAMPLER value; falling back to never sample", "value", name)
135+
return sdktrace.NeverSample()
136+
}

0 commit comments

Comments
 (0)