Skip to content

Update opentelemetry to 0.32#1698

Open
spacebear21 wants to merge 2 commits into
payjoin:masterfrom
spacebear21:update-otel-0.32
Open

Update opentelemetry to 0.32#1698
spacebear21 wants to merge 2 commits into
payjoin:masterfrom
spacebear21:update-otel-0.32

Conversation

@spacebear21

@spacebear21 spacebear21 commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

This addresses a Dependabot report about unbounded memory allocation GHSA-w9wp-h8wv-79jx

Pull Request Checklist

Please confirm the following before requesting review:

@benalleng

benalleng commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Annoyingly the opentelemetry-http dep drops ring as the default crypto provider. instead it uses aws-lc-rs even though opentelemetry-otlp claims to have a crypto agnostic feature

@coveralls

coveralls commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

Coverage Report for CI Build 29275027577

Coverage increased (+0.1%) to 86.126%

Details

  • Coverage increased (+0.1%) from the base build.
  • Patch coverage: 4 uncovered changes across 1 file (54 of 58 lines covered, 93.1%).
  • No coverage regressions found.

Uncovered Changes

File Changed Covered %
payjoin-mailroom/src/main.rs 4 0 0.0%
Total (2 files) 58 54 93.1%

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 15922
Covered Lines: 13713
Line Coverage: 86.13%
Coverage Strength: 342.8 hits per line

💛 - Coveralls

@benalleng

Copy link
Copy Markdown
Collaborator

I found we could drop reqwest-rustls for the more minimal http-proto. Not sure if there is something missing we would want there but it linted ok.

@spacebear21 spacebear21 left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK @benalleng your change looks good to me, however I can't self-approve this PR.

@xstoicunicornx

xstoicunicornx commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

So I had tried to look at this PR earlier but it was, and still is, a bit over my head. I plugged it into Claude and it had this feedback, maybe you guys can discern whether this concern is legitimate or just hallucination?

Verdict

Do not merge as-is. There's one blocking runtime regression. The dependency bump itself is legitimate (it addresses GHSA-w9wp-h8wv-79jx), but the Cargo.toml feature change breaks OTLP export at runtime.

Blocker: the OTLP exporter has no HTTP client

The commit changes the otlp dep from:
opentelemetry-otlp = { version = "0.31", optional = true, features = ["reqwest-rustls"] }
to:
opentelemetry-otlp = { version = "0.32", optional = true, default-features = false, features = ["http-proto"] }

default-features = false drops otlp's default reqwest-blocking-client, and http-proto only enables the wire encoding — not an HTTP client. The resolved feature graph confirms no reqwest-client / reqwest-blocking-client / hyper-client is enabled anywhere.

In opentelemetry-otlp 0.32, build_client() ends with:
let http_client = http_client.ok_or(ExporterBuildError::NoHttpClient)?;   // exporter/http/mod.rs:268
A client is only populated if (a) a client feature is enabled, or (b) the caller passes .with_http_client(...). Neither is true here — payjoin-mailroom/src/main.rs:52-57 calls .with_http() / .with_endpoint() / .with_headers() but never .with_http_client().

So at startup, whenever telemetry is configured:
.build()
.expect("Failed to build OTLP metric exporter");   // main.rs:57 → panics on Err(NoHttpClient)
the process panics on boot. init_tracing_with_telemetry is only reachable from main.rs:15 (real startup), and no test exercises it, so CI compiles green and misses it entirely.

Fix: re-add an HTTP client feature. To preserve prior behavior (rustls TLS — the endpoint is a user-supplied String that will normally be https), use:
opentelemetry-otlp = { version = "0.32", optional = true, default-features = false, features = ["http-proto", "reqwest-rustls"] }
reqwest-rustls still exists in 0.32 and pulls reqwest + rustls TLS. (Plain reqwest-client would build reqwest with default TLS; reqwest-blocking-client is a blocking client driven from async and is the wrong fit here.)

Other things to weigh

- TLS regression risk even beyond the panic: the old config guaranteed rustls TLS. Any replacement client feature must still provide a TLS backend, or HTTPS export silently fails. reqwest-rustls covers this.
- Upside worth noting: the switch away from otlp's default grpc-tonic correctly drops tonic + tonic-prost from the tree (visible in both lockfiles) — a genuine dependency reduction. That part is good; it just needs the client feature restored.
- Lockfile hygiene: Cargo-minimal.lock pins portable-atomic 1.0.0 and sync_wrapper 1.0.0 (minimal), recent pins 1.13.1. That's expected for the two-lockfile scheme, but confirm contrib/update-lock-files.sh was the source rather than hand-edits.
- No test guards this path. Even after the fix, nothing catches a future regression. Optional but worth suggesting: a small test that builds the exporter (against a dummy endpoint) so NoHttpClient surfaces in CI rather than in production.

Bottom line

The version bump and the grpc→http transport switch are the right direction, but shipping it drops the HTTP client and turns telemetry startup into a guaranteed panic. Add reqwest-rustls (or another client feature) back and it's mergeable.

@benalleng

Copy link
Copy Markdown
Collaborator

I am inclined to believe this. I haven't tested running it with telemetry enabled on this commit but that would certainly answer it immediately. But I guess the issue arises about how we get around the crypto provider default with all of this intertwined if we leave in the reqwest feature

@benalleng

Copy link
Copy Markdown
Collaborator

Here is an example of the errors I was running into while the reqwest-rustls feature was still enabled. https://github.com/payjoin/rust-payjoin/actions/runs/28676821843/job/85051890908 This was due to that feature automatically pulling in opentelemetry/reqwest which uses a different default crypto as of 0.32 https://github.com/open-telemetry/opentelemetry-rust/blob/main/opentelemetry-http/CHANGELOG.md#0320

spacebear21 and others added 2 commits July 13, 2026 14:34
This addresses a Dependabot report about unbounded memory allocation
GHSA-w9wp-h8wv-79jx
Following the update for the opentelemetry to 0.32 the default crypto
provider in a transitive dependency when enabling `reqwest-rustls`
forced the default crypto provider to not align with our default `ring`
in the codebase. However, by manually importing opentelemetry-http and
only enabling `hyper` we are able to avoid this crypto provider
incompatibility.
@benalleng

Copy link
Copy Markdown
Collaborator

@xstoicunicornx I added a http client without pulling in the default crypto provided from opentelemetry. Ready for your re-review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants