Skip to content

feat(data-pipeline): port dd-trace-rs trace buffer implementation#1826

Open
paullegranddc wants to merge 68 commits intomainfrom
paullgdc/data-pipeline/trace_buffer
Open

feat(data-pipeline): port dd-trace-rs trace buffer implementation#1826
paullegranddc wants to merge 68 commits intomainfrom
paullgdc/data-pipeline/trace_buffer

Conversation

@paullegranddc
Copy link
Copy Markdown
Contributor

Motivation

As we implement native spans in ruby in python, we have opportuinities to replace more components of the span pipeline with native code.

One of them i the span buffer which we is before the trace exporter.

Changes

Add a TraceBuffer implementation.
The TraceBuffer is split in two part, a Sender which allows putting trace chunks in the queue and a background task, spawned on the shared runtime which periodically pulls data from the chunk and submit them using the TraceExporter.

With the refactor the stop action is processed right away without
processing all actions sent before it. This can cause some message to be
lost if they were submitted right before shutdown.
@paullegranddc paullegranddc requested a review from a team as a code owner March 31, 2026 09:20
@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 31, 2026

📚 Documentation Check Results

⚠️ 891 documentation warning(s) found

📦 libdd-data-pipeline - 891 warning(s)


Updated: 2026-03-31 17:35:02 UTC | Commit: a963d74 | missing-docs job results

# Motivation

As we implement native spans in ruby in python, we have opportuinities to replace more components of the span pipeline with native code.

One of them i the span buffer which we is before the trace exporter.

# Changes

Add a TraceBuffer implementation.
The TraceBuffer is split in two part, a Sender which allows putting trace chunks in the queue and a background task, spawned on the shared runtime which periodically pulls data from the chunk and submit them using the TraceExporter.
@paullegranddc paullegranddc force-pushed the paullgdc/data-pipeline/trace_buffer branch from 3ede0c0 to 5938b42 Compare March 31, 2026 09:22
@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 31, 2026

Clippy Allow Annotation Report

Comparing clippy allow annotations between branches:

Summary by Rule

Rule Base Branch PR Branch Change
unwrap_used 2 3 ⚠️ +1 (+50.0%)
Total 2 3 ⚠️ +1 (+50.0%)

Annotation Counts by File

File Base Branch PR Branch Change
libdd-data-pipeline/src/trace_buffer/mod.rs 0 1 ⚠️ +1 (N/A)
libdd-data-pipeline/src/trace_exporter/mod.rs 2 2 No change (0%)

Annotation Stats by Crate

Crate Base Branch PR Branch Change
clippy-annotation-reporter 5 5 No change (0%)
datadog-ffe-ffi 1 1 No change (0%)
datadog-ipc 20 20 No change (0%)
datadog-live-debugger 6 6 No change (0%)
datadog-live-debugger-ffi 10 10 No change (0%)
datadog-profiling-replayer 4 4 No change (0%)
datadog-remote-config 3 3 No change (0%)
datadog-sidecar 55 55 No change (0%)
libdd-common 10 10 No change (0%)
libdd-common-ffi 12 12 No change (0%)
libdd-data-pipeline 5 6 ⚠️ +1 (+20.0%)
libdd-ddsketch 2 2 No change (0%)
libdd-dogstatsd-client 1 1 No change (0%)
libdd-profiling 13 13 No change (0%)
libdd-telemetry 19 19 No change (0%)
libdd-tinybytes 4 4 No change (0%)
libdd-trace-normalization 2 2 No change (0%)
libdd-trace-obfuscation 8 8 No change (0%)
libdd-trace-utils 15 15 No change (0%)
Total 195 196 ⚠️ +1 (+0.5%)

About This Report

This report tracks Clippy allow annotations for specific rules, showing how they've changed in this PR. Decreasing the number of these annotations generally improves code quality.

@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 31, 2026

🔒 Cargo Deny Results

⚠️ 1 issue(s) found, showing only errors (advisories, bans, sources)

📦 libdd-data-pipeline - 1 error(s)

Show output
error[vulnerability]: Denial of Service via Stack Exhaustion
    ┌─ /home/runner/work/libdatadog/libdatadog/Cargo.lock:276:1
    │
276 │ time 0.3.41 registry+https://github.com/rust-lang/crates.io-index
    │ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ security vulnerability detected
    │
    ├ ID: RUSTSEC-2026-0009
    ├ Advisory: https://rustsec.org/advisories/RUSTSEC-2026-0009
    ├ ## Impact
      
      When user-provided input is provided to any type that parses with the RFC 2822 format, a denial of
      service attack via stack exhaustion is possible. The attack relies on formally deprecated and
      rarely-used features that are part of the RFC 2822 format used in a malicious manner. Ordinary,
      non-malicious input will never encounter this scenario.
      
      ## Patches
      
      A limit to the depth of recursion was added in v0.3.47. From this version, an error will be returned
      rather than exhausting the stack.
      
      ## Workarounds
      
      Limiting the length of user input is the simplest way to avoid stack exhaustion, as the amount of
      the stack consumed would be at most a factor of the length of the input.
    ├ Announcement: https://github.com/time-rs/time/blob/main/CHANGELOG.md#0347-2026-02-05
    ├ Solution: Upgrade to >=0.3.47 (try `cargo update -p time`)
    ├ time v0.3.41
      └── tracing-appender v0.2.3
          └── libdd-log v1.0.0
              └── (dev) libdd-data-pipeline v3.0.1

advisories FAILED, bans ok, sources ok

Updated: 2026-03-31 17:37:55 UTC | Commit: a963d74 | dependency-check job results

@datadog-datadog-prod-us1
Copy link
Copy Markdown
Contributor

datadog-datadog-prod-us1 bot commented Mar 31, 2026

⚠️ Tests

Fix all issues with BitsAI or with Cursor

⚠️ Other Violations

🧪 1 Test failed

worker::tests::reset::test_reset_clears_buffered_data from libdd-telemetry   View in Datadog   (Fix with Cursor)
Test has failed

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e6f17c2 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

@pr-commenter
Copy link
Copy Markdown

pr-commenter bot commented Mar 31, 2026

Benchmarks

Comparison

Benchmark execution time: 2026-03-31 09:40:19

Comparing candidate commit 5938b42 in PR branch paullgdc/data-pipeline/trace_buffer with baseline commit 41bff26 in branch vianney/implement-shared-runtime.

Found 0 performance improvements and 1 performance regressions! Performance is the same for 61 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:tags/replace_trace_tags

  • 🟥 execution_time [+152.789ns; +159.307ns] or [+6.346%; +6.616%]

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sdk_test_data/rules-based execution_time 145.162µs 147.194µs ± 1.851µs 146.987µs ± 0.608µs 147.504µs 148.894µs 154.489µs 165.669µs 12.71% 6.214 53.332 1.25% 0.131µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sdk_test_data/rules-based execution_time [146.937µs; 147.450µs] or [-0.174%; +0.174%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_trace/test_trace execution_time 244.115ns 254.188ns ± 13.661ns 247.446ns ± 1.207ns 256.824ns 284.718ns 299.800ns 301.808ns 21.97% 1.954 2.976 5.36% 0.966ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_trace/test_trace execution_time [252.294ns; 256.081ns] or [-0.745%; +0.745%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
profile_add_sample2_frames_x1000 execution_time 755.490µs 757.180µs ± 0.605µs 757.134µs ± 0.376µs 757.532µs 758.231µs 758.721µs 758.947µs 0.24% 0.326 0.400 0.08% 0.043µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
profile_add_sample2_frames_x1000 execution_time [757.096µs; 757.264µs] or [-0.011%; +0.011%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
receiver_entry_point/report/2598 execution_time 3.425ms 3.453ms ± 0.018ms 3.448ms ± 0.011ms 3.464ms 3.487ms 3.498ms 3.525ms 2.24% 0.984 0.832 0.51% 0.001ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
receiver_entry_point/report/2598 execution_time [3.450ms; 3.455ms] or [-0.071%; +0.071%] None None None

Group 5

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
profile_add_sample_frames_x1000 execution_time 4.372ms 4.377ms ± 0.002ms 4.377ms ± 0.002ms 4.378ms 4.381ms 4.383ms 4.390ms 0.30% 1.067 3.179 0.06% 0.000ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
profile_add_sample_frames_x1000 execution_time [4.377ms; 4.377ms] or [-0.008%; +0.008%] None None None

Group 6

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
tags/replace_trace_tags execution_time 2.536µs 2.564µs ± 0.017µs 2.560µs ± 0.011µs 2.573µs 2.596µs 2.608µs 2.616µs 2.18% 0.735 -0.111 0.65% 0.001µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
tags/replace_trace_tags execution_time [2.561µs; 2.566µs] or [-0.091%; +0.091%] None None None

Group 7

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
credit_card/is_card_number/ execution_time 3.893µs 3.914µs ± 0.004µs 3.913µs ± 0.002µs 3.916µs 3.919µs 3.922µs 3.928µs 0.38% -0.124 7.013 0.09% 0.000µs 1 200
credit_card/is_card_number/ throughput 254600717.643op/s 255512513.409op/s ± 228583.913op/s 255579476.992op/s ± 124824.793op/s 255664568.087op/s 255737052.045op/s 255777886.821op/s 256886255.168op/s 0.51% 0.148 7.108 0.09% 16163.323op/s 1 200
credit_card/is_card_number/ 3782-8224-6310-005 execution_time 79.549µs 80.012µs ± 0.253µs 79.965µs ± 0.155µs 80.140µs 80.549µs 80.616µs 80.707µs 0.93% 0.751 -0.044 0.32% 0.018µs 1 200
credit_card/is_card_number/ 3782-8224-6310-005 throughput 12390503.878op/s 12498179.354op/s ± 39470.673op/s 12505514.939op/s ± 24184.513op/s 12527126.558op/s 12550250.639op/s 12562153.382op/s 12570921.367op/s 0.52% -0.738 -0.064 0.32% 2790.998op/s 1 200
credit_card/is_card_number/ 378282246310005 execution_time 73.143µs 73.591µs ± 0.229µs 73.555µs ± 0.142µs 73.725µs 74.000µs 74.257µs 74.528µs 1.32% 0.743 0.958 0.31% 0.016µs 1 200
credit_card/is_card_number/ 378282246310005 throughput 13417733.508op/s 13588742.076op/s ± 42218.429op/s 13595181.912op/s ± 26183.206op/s 13618234.480op/s 13650845.305op/s 13665220.412op/s 13671818.060op/s 0.56% -0.721 0.894 0.31% 2985.294op/s 1 200
credit_card/is_card_number/37828224631 execution_time 3.894µs 3.917µs ± 0.005µs 3.916µs ± 0.003µs 3.919µs 3.925µs 3.942µs 3.947µs 0.79% 1.898 12.307 0.13% 0.000µs 1 200
credit_card/is_card_number/37828224631 throughput 253347052.827op/s 255293143.117op/s ± 331215.054op/s 255342918.091op/s ± 170479.472op/s 255491747.091op/s 255620352.412op/s 255701130.017op/s 256832818.930op/s 0.58% -1.856 12.169 0.13% 23420.441op/s 1 200
credit_card/is_card_number/378282246310005 execution_time 69.605µs 69.979µs ± 0.230µs 69.940µs ± 0.142µs 70.124µs 70.418µs 70.558µs 70.840µs 1.29% 0.778 0.470 0.33% 0.016µs 1 200
credit_card/is_card_number/378282246310005 throughput 14116301.742op/s 14290119.381op/s ± 46778.607op/s 14297951.818op/s ± 29155.792op/s 14323720.442op/s 14354096.083op/s 14365366.281op/s 14366723.373op/s 0.48% -0.759 0.423 0.33% 3307.747op/s 1 200
credit_card/is_card_number/37828224631000521389798 execution_time 51.561µs 52.017µs ± 0.192µs 52.039µs ± 0.140µs 52.163µs 52.296µs 52.358µs 52.389µs 0.67% -0.308 -0.741 0.37% 0.014µs 1 200
credit_card/is_card_number/37828224631000521389798 throughput 19087893.420op/s 19224691.039op/s ± 71022.559op/s 19216312.118op/s ± 51643.365op/s 19277026.130op/s 19345199.103op/s 19377081.744op/s 19394638.076op/s 0.93% 0.321 -0.731 0.37% 5022.053op/s 1 200
credit_card/is_card_number/x371413321323331 execution_time 5.746µs 5.884µs ± 0.054µs 5.882µs ± 0.035µs 5.915µs 5.977µs 6.011µs 6.015µs 2.27% 0.035 -0.230 0.91% 0.004µs 1 200
credit_card/is_card_number/x371413321323331 throughput 166238848.022op/s 169964129.026op/s ± 1548728.618op/s 170004704.233op/s ± 1015979.168op/s 171110156.247op/s 172437366.587op/s 173319885.316op/s 174046044.711op/s 2.38% 0.013 -0.229 0.91% 109511.651op/s 1 200
credit_card/is_card_number_no_luhn/ execution_time 3.894µs 3.918µs ± 0.004µs 3.917µs ± 0.003µs 3.920µs 3.925µs 3.927µs 3.931µs 0.35% -0.469 4.445 0.11% 0.000µs 1 200
credit_card/is_card_number_no_luhn/ throughput 254401296.829op/s 255250206.334op/s ± 269425.943op/s 255286703.763op/s ± 171973.284op/s 255432114.207op/s 255586331.382op/s 255684132.656op/s 256796304.225op/s 0.59% 0.489 4.547 0.11% 19051.291op/s 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time 64.193µs 64.393µs ± 0.117µs 64.363µs ± 0.057µs 64.435µs 64.631µs 64.801µs 64.873µs 0.79% 1.493 2.753 0.18% 0.008µs 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput 15414672.476op/s 15529724.925op/s ± 28134.668op/s 15536833.450op/s ± 13794.405op/s 15547075.037op/s 15562010.099op/s 15568728.678op/s 15578011.637op/s 0.27% -1.479 2.700 0.18% 1989.421op/s 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time 57.487µs 57.615µs ± 0.092µs 57.599µs ± 0.057µs 57.659µs 57.786µs 57.851µs 57.995µs 0.69% 1.149 1.872 0.16% 0.006µs 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 throughput 17242920.443op/s 17356621.217op/s ± 27547.458op/s 17361332.840op/s ± 17136.325op/s 17376897.436op/s 17391244.900op/s 17393536.631op/s 17395174.579op/s 0.19% -1.136 1.824 0.16% 1947.899op/s 1 200
credit_card/is_card_number_no_luhn/37828224631 execution_time 3.901µs 3.917µs ± 0.004µs 3.916µs ± 0.003µs 3.919µs 3.924µs 3.928µs 3.931µs 0.39% 0.536 0.951 0.11% 0.000µs 1 200
credit_card/is_card_number_no_luhn/37828224631 throughput 254381684.678op/s 255329191.025op/s ± 274053.523op/s 255373173.071op/s ± 174269.807op/s 255522235.569op/s 255677287.135op/s 255792287.145op/s 256329373.409op/s 0.37% -0.528 0.950 0.11% 19378.510op/s 1 200
credit_card/is_card_number_no_luhn/378282246310005 execution_time 54.606µs 54.791µs ± 0.135µs 54.762µs ± 0.061µs 54.827µs 55.113µs 55.255µs 55.452µs 1.26% 2.060 5.210 0.25% 0.010µs 1 200
credit_card/is_card_number_no_luhn/378282246310005 throughput 18033460.050op/s 18251216.346op/s ± 44717.898op/s 18260881.540op/s ± 20178.563op/s 18279949.638op/s 18298739.401op/s 18309197.391op/s 18313075.536op/s 0.29% -2.038 5.102 0.24% 3162.033op/s 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time 51.404µs 52.145µs ± 0.239µs 52.158µs ± 0.186µs 52.359µs 52.443µs 52.482µs 52.532µs 0.72% -0.713 0.140 0.46% 0.017µs 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput 19035976.714op/s 19177654.413op/s ± 88263.922op/s 19172533.331op/s ± 68386.954op/s 19231939.616op/s 19346709.831op/s 19437837.774op/s 19453846.857op/s 1.47% 0.736 0.199 0.46% 6241.202op/s 1 200
credit_card/is_card_number_no_luhn/x371413321323331 execution_time 5.706µs 5.887µs ± 0.055µs 5.898µs ± 0.035µs 5.922µs 5.969µs 6.001µs 6.050µs 2.57% -0.367 0.332 0.94% 0.004µs 1 200
credit_card/is_card_number_no_luhn/x371413321323331 throughput 165296818.766op/s 169884924.669op/s ± 1606249.539op/s 169552986.733op/s ± 996677.214op/s 170742116.737op/s 172520793.526op/s 174112874.907op/s 175244568.347op/s 3.36% 0.429 0.392 0.94% 113578.994op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
credit_card/is_card_number/ execution_time [3.913µs; 3.914µs] or [-0.012%; +0.012%] None None None
credit_card/is_card_number/ throughput [255480833.877op/s; 255544192.941op/s] or [-0.012%; +0.012%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 execution_time [79.977µs; 80.048µs] or [-0.044%; +0.044%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 throughput [12492709.099op/s; 12503649.610op/s] or [-0.044%; +0.044%] None None None
credit_card/is_card_number/ 378282246310005 execution_time [73.559µs; 73.623µs] or [-0.043%; +0.043%] None None None
credit_card/is_card_number/ 378282246310005 throughput [13582891.008op/s; 13594593.144op/s] or [-0.043%; +0.043%] None None None
credit_card/is_card_number/37828224631 execution_time [3.916µs; 3.918µs] or [-0.018%; +0.018%] None None None
credit_card/is_card_number/37828224631 throughput [255247239.896op/s; 255339046.338op/s] or [-0.018%; +0.018%] None None None
credit_card/is_card_number/378282246310005 execution_time [69.947µs; 70.011µs] or [-0.045%; +0.045%] None None None
credit_card/is_card_number/378282246310005 throughput [14283636.316op/s; 14296602.446op/s] or [-0.045%; +0.045%] None None None
credit_card/is_card_number/37828224631000521389798 execution_time [51.991µs; 52.044µs] or [-0.051%; +0.051%] None None None
credit_card/is_card_number/37828224631000521389798 throughput [19214847.995op/s; 19234534.082op/s] or [-0.051%; +0.051%] None None None
credit_card/is_card_number/x371413321323331 execution_time [5.877µs; 5.892µs] or [-0.126%; +0.126%] None None None
credit_card/is_card_number/x371413321323331 throughput [169749490.134op/s; 170178767.917op/s] or [-0.126%; +0.126%] None None None
credit_card/is_card_number_no_luhn/ execution_time [3.917µs; 3.918µs] or [-0.015%; +0.015%] None None None
credit_card/is_card_number_no_luhn/ throughput [255212866.489op/s; 255287546.178op/s] or [-0.015%; +0.015%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time [64.377µs; 64.409µs] or [-0.025%; +0.025%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput [15525825.730op/s; 15533624.119op/s] or [-0.025%; +0.025%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time [57.602µs; 57.628µs] or [-0.022%; +0.022%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 throughput [17352803.404op/s; 17360439.030op/s] or [-0.022%; +0.022%] None None None
credit_card/is_card_number_no_luhn/37828224631 execution_time [3.916µs; 3.917µs] or [-0.015%; +0.015%] None None None
credit_card/is_card_number_no_luhn/37828224631 throughput [255291209.843op/s; 255367172.208op/s] or [-0.015%; +0.015%] None None None
credit_card/is_card_number_no_luhn/378282246310005 execution_time [54.772µs; 54.810µs] or [-0.034%; +0.034%] None None None
credit_card/is_card_number_no_luhn/378282246310005 throughput [18245018.876op/s; 18257413.817op/s] or [-0.034%; +0.034%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time [52.112µs; 52.178µs] or [-0.064%; +0.064%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput [19165421.883op/s; 19189886.944op/s] or [-0.064%; +0.064%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 execution_time [5.879µs; 5.895µs] or [-0.131%; +0.131%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 throughput [169662313.931op/s; 170107535.407op/s] or [-0.131%; +0.131%] None None None

Group 8

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time 534.633µs 535.331µs ± 0.508µs 535.168µs ± 0.278µs 535.624µs 536.259µs 536.822µs 537.367µs 0.41% 1.130 1.071 0.09% 0.036µs 1 200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput 1860925.180op/s 1868003.766op/s ± 1770.247op/s 1868573.345op/s ± 970.945op/s 1869242.578op/s 1870152.146op/s 1870372.134op/s 1870441.555op/s 0.10% -1.125 1.052 0.09% 125.175op/s 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time 381.913µs 382.847µs ± 0.462µs 382.768µs ± 0.306µs 383.111µs 383.632µs 384.109µs 384.633µs 0.49% 0.821 1.012 0.12% 0.033µs 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput 2599877.693op/s 2612015.951op/s ± 3149.675op/s 2612545.626op/s ± 2091.495op/s 2614298.777op/s 2615896.217op/s 2617295.626op/s 2618394.105op/s 0.22% -0.812 0.987 0.12% 222.716op/s 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time 194.544µs 195.514µs ± 0.494µs 195.557µs ± 0.254µs 195.778µs 196.104µs 196.446µs 199.155µs 1.84% 1.821 13.424 0.25% 0.035µs 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput 5021211.978op/s 5114754.570op/s ± 12878.646op/s 5113610.637op/s ± 6643.531op/s 5121482.960op/s 5135117.765op/s 5138075.153op/s 5140234.387op/s 0.52% -1.731 12.663 0.25% 910.658op/s 1 200
normalization/normalize_service/normalize_service/[empty string] execution_time 37.963µs 38.315µs ± 0.130µs 38.306µs ± 0.082µs 38.406µs 38.534µs 38.604µs 38.736µs 1.12% 0.080 0.277 0.34% 0.009µs 1 200
normalization/normalize_service/normalize_service/[empty string] throughput 25815922.272op/s 26099776.312op/s ± 88862.624op/s 26105454.639op/s ± 56048.329op/s 26153468.819op/s 26245179.056op/s 26322381.104op/s 26341483.420op/s 0.90% -0.057 0.269 0.34% 6283.536op/s 1 200
normalization/normalize_service/normalize_service/test_ASCII execution_time 45.801µs 45.943µs ± 0.080µs 45.923µs ± 0.039µs 45.982µs 46.106µs 46.191µs 46.281µs 0.78% 1.323 2.228 0.17% 0.006µs 1 200
normalization/normalize_service/normalize_service/test_ASCII throughput 21607035.251op/s 21766200.000op/s ± 37815.879op/s 21775786.351op/s ± 18332.460op/s 21790339.048op/s 21810108.906op/s 21827075.042op/s 21833399.036op/s 0.26% -1.310 2.181 0.17% 2673.986op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time [535.261µs; 535.402µs] or [-0.013%; +0.013%] None None None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput [1867758.427op/s; 1868249.105op/s] or [-0.013%; +0.013%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time [382.783µs; 382.911µs] or [-0.017%; +0.017%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput [2611579.437op/s; 2612452.466op/s] or [-0.017%; +0.017%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time [195.446µs; 195.583µs] or [-0.035%; +0.035%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput [5112969.714op/s; 5116539.427op/s] or [-0.035%; +0.035%] None None None
normalization/normalize_service/normalize_service/[empty string] execution_time [38.297µs; 38.333µs] or [-0.047%; +0.047%] None None None
normalization/normalize_service/normalize_service/[empty string] throughput [26087460.807op/s; 26112091.817op/s] or [-0.047%; +0.047%] None None None
normalization/normalize_service/normalize_service/test_ASCII execution_time [45.932µs; 45.954µs] or [-0.024%; +0.024%] None None None
normalization/normalize_service/normalize_service/test_ASCII throughput [21760959.083op/s; 21771440.917op/s] or [-0.024%; +0.024%] None None None

Group 9

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 14.987µs 15.554µs ± 0.201µs 15.537µs ± 0.120µs 15.665µs 15.953µs 16.031µs 16.179µs 4.13% 0.347 0.427 1.29% 0.014µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [15.526µs; 15.581µs] or [-0.179%; +0.179%] None None None

Group 10

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching serializing traces from their internal representation to msgpack execution_time 14.724ms 14.797ms ± 0.030ms 14.793ms ± 0.017ms 14.813ms 14.846ms 14.863ms 14.944ms 1.01% 1.270 4.631 0.20% 0.002ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching serializing traces from their internal representation to msgpack execution_time [14.793ms; 14.801ms] or [-0.028%; +0.028%] None None None

Group 11

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
single_flag_killswitch/rules-based execution_time 191.119ns 193.821ns ± 2.150ns 193.349ns ± 1.179ns 194.677ns 197.961ns 201.344ns 203.578ns 5.29% 1.519 3.216 1.11% 0.152ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
single_flag_killswitch/rules-based execution_time [193.523ns; 194.119ns] or [-0.154%; +0.154%] None None None

Group 12

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 5.222µs 5.418µs ± 0.037µs 5.416µs ± 0.026µs 5.445µs 5.477µs 5.492µs 5.500µs 1.54% -0.476 2.764 0.68% 0.003µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [5.413µs; 5.424µs] or [-0.095%; +0.095%] None None None

Group 13

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 14.862ms 14.910ms ± 0.017ms 14.909ms ± 0.010ms 14.919ms 14.936ms 14.960ms 14.985ms 0.51% 0.492 2.000 0.12% 0.001ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [14.907ms; 14.912ms] or [-0.016%; +0.016%] None None None

Group 14

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sql/obfuscate_sql_string execution_time 288.634µs 289.526µs ± 0.576µs 289.438µs ± 0.190µs 289.612µs 290.412µs 291.836µs 293.627µs 1.45% 3.600 20.030 0.20% 0.041µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sql/obfuscate_sql_string execution_time [289.446µs; 289.605µs] or [-0.028%; +0.028%] None None None

Group 15

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time 185.650µs 186.487µs ± 0.660µs 186.242µs ± 0.295µs 186.951µs 187.750µs 188.363µs 188.613µs 1.27% 1.102 0.272 0.35% 0.047µs 1 200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput 5301872.025op/s 5362364.084op/s ± 18900.401op/s 5369366.076op/s ± 8519.453op/s 5376292.601op/s 5381953.072op/s 5385146.412op/s 5386485.013op/s 0.32% -1.091 0.237 0.35% 1336.460op/s 1 200
normalization/normalize_name/normalize_name/bad-name execution_time 17.850µs 17.979µs ± 0.065µs 17.977µs ± 0.035µs 18.007µs 18.102µs 18.168µs 18.229µs 1.40% 0.898 1.787 0.36% 0.005µs 1 200
normalization/normalize_name/normalize_name/bad-name throughput 54857142.857op/s 55620139.313op/s ± 200755.788op/s 55626646.928op/s ± 107684.306op/s 55755093.011op/s 55924799.103op/s 55994131.933op/s 56020865.905op/s 0.71% -0.866 1.698 0.36% 14195.578op/s 1 200
normalization/normalize_name/normalize_name/good execution_time 9.864µs 9.929µs ± 0.072µs 9.885µs ± 0.015µs 9.978µs 10.073µs 10.122µs 10.129µs 2.47% 1.090 -0.146 0.72% 0.005µs 1 200
normalization/normalize_name/normalize_name/good throughput 98726648.921op/s 100717184.066op/s ± 725984.940op/s 101166649.247op/s ± 148775.520op/s 101249768.524op/s 101322506.300op/s 101361822.768op/s 101379910.161op/s 0.21% -1.075 -0.189 0.72% 51334.887op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time [186.396µs; 186.579µs] or [-0.049%; +0.049%] None None None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput [5359744.670op/s; 5364983.498op/s] or [-0.049%; +0.049%] None None None
normalization/normalize_name/normalize_name/bad-name execution_time [17.970µs; 17.988µs] or [-0.050%; +0.050%] None None None
normalization/normalize_name/normalize_name/bad-name throughput [55592316.492op/s; 55647962.135op/s] or [-0.050%; +0.050%] None None None
normalization/normalize_name/normalize_name/good execution_time [9.919µs; 9.939µs] or [-0.101%; +0.101%] None None None
normalization/normalize_name/normalize_name/good throughput [100616569.535op/s; 100817798.596op/s] or [-0.100%; +0.100%] None None None

Group 16

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
profile_add_sample_timestamped_x1000 execution_time 4.374ms 4.381ms ± 0.009ms 4.379ms ± 0.002ms 4.382ms 4.385ms 4.393ms 4.501ms 2.78% 11.444 145.811 0.21% 0.001ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
profile_add_sample_timestamped_x1000 execution_time [4.379ms; 4.382ms] or [-0.029%; +0.029%] None None None

Group 17

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
redis/obfuscate_redis_string execution_time 33.647µs 34.231µs ± 0.984µs 33.772µs ± 0.056µs 33.889µs 36.332µs 36.419µs 37.277µs 10.38% 1.703 1.004 2.87% 0.070µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
redis/obfuscate_redis_string execution_time [34.094µs; 34.367µs] or [-0.399%; +0.399%] None None None

Group 18

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
profile_serialize_compressed_pprof_timestamped_x1000 execution_time 934.320µs 937.751µs ± 2.304µs 937.143µs ± 1.085µs 938.459µs 941.935µs 945.090µs 951.054µs 1.48% 1.823 5.759 0.25% 0.163µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
profile_serialize_compressed_pprof_timestamped_x1000 execution_time [937.431µs; 938.070µs] or [-0.034%; +0.034%] None None None

Group 19

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
ip_address/quantize_peer_ip_address_benchmark execution_time 4.929µs 5.007µs ± 0.052µs 5.013µs ± 0.055µs 5.046µs 5.091µs 5.093µs 5.097µs 1.68% 0.231 -1.377 1.03% 0.004µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
ip_address/quantize_peer_ip_address_benchmark execution_time [5.000µs; 5.014µs] or [-0.143%; +0.143%] None None None

Group 20

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching string interning on wordpress profile execution_time 160.220µs 161.100µs ± 0.614µs 161.078µs ± 0.227µs 161.283µs 161.677µs 162.973µs 165.895µs 2.99% 4.142 27.671 0.38% 0.043µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching string interning on wordpress profile execution_time [161.015µs; 161.185µs] or [-0.053%; +0.053%] None None None

Group 21

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz 5938b42 1774948956 paullgdc/data-pipeline/trace_buffer
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching deserializing traces from msgpack to their internal representation execution_time 47.873ms 48.280ms ± 0.975ms 48.127ms ± 0.095ms 48.260ms 48.438ms 53.542ms 58.876ms 22.33% 8.559 79.737 2.01% 0.069ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching deserializing traces from msgpack to their internal representation execution_time [48.145ms; 48.416ms] or [-0.280%; +0.280%] None None None

Baseline

Omitted due to size.

@paullegranddc paullegranddc requested a review from a team as a code owner March 31, 2026 10:03
if res.timed_out() {
return Err(TraceBufferError::TimedOut(timeout));
}
Ok(())
Copy link
Copy Markdown
Contributor

@bantonsson bantonsson Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or am I missing something?

Doesn't this always return Ok, even if the last flush didn't happen because of shut down?

I think we need to look at the state.last_flush_generation

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

res is a WaitTimeoutResult so I cannot just return it

Copy link
Copy Markdown
Contributor

@bantonsson bantonsson Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the confusion. I was editing my question while you answered.

Doesn't this always return Ok, even if the last flush didn't happen because of shut down?

I think we need to look at the state.last_flush_generation

@@ -0,0 +1,851 @@
// Copyright 2025-Present Datadog, Inc. https://www.datadoghq.com/
// SPDX-License-Identifier: Apache-2.0
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: a module-level doc, even if short, is always nice to get an idea of what happens here quickly.

#[derive(Debug)]
pub enum TraceBufferError {
AlreadyShutdown,
TimedOut(std::time::Duration),
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
TimedOut(std::time::Duration),
TimedOut(Duration),

}

fn reset(&mut self) {
*self = Self::new(self.max_buffered_spans);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it is worth it (is this remotely perf-sensitive?), but clear()ing chunks instead of rea might avoid dropping the old vector to just re-allocate a new one.

return Ok(());
}

let chunk_len: usize = chunk.len();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let chunk_len: usize = chunk.len();
let chunk_len = chunk.len();


struct Batch<T> {
chunks: Vec<TraceChunk<T>>,
last_flush: std::time::Instant,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
last_flush: std::time::Instant,
last_flush: Instant,

Comment on lines +251 to +252
let chunk_len = trace_chunk.len();
if chunk_len == 0 {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let chunk_len = trace_chunk.len();
if chunk_len == 0 {
if trace_chunk.is_empty() {

pub spans_queued: usize,
}

fn channel<T>(
Copy link
Copy Markdown
Contributor

@yannham yannham Mar 31, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We discussed that orally, but I'm writing it down for the record: I'm not sure about the trade-off of going with a custom channel implementation here. It seems what we do is simple enough that we should be able to piggy-back on existing, well-tested, well-optimized channel impls. It's also a non trivial part of the code of this module. It might be the case that the custom channel is actually the right solution, but using a standard bounded channel or whatnot should be the default approach IMHO, only moving to a custom solution if we find good reasons to do so in the process. And in this case, the trade-offs should be properly documented here.

let gen = state.batch.batch_gen;
if state.batch.span_count() > self.flush_trigger_number_of_spans || self.synchronous_write {
state.flush_needed = true;
self.waiter.notify_receiver(state);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we can avoid notifying if state.flush_needed was already set to true here. I think, but it must be double checked that this won't let a waiter be stuck forever. I'm saying that because notifying is basically a futex syscall, and a syscall is costly, so usually we try to avoid unnecessary wait or notify. That being said, it's a bit of a detail and it might be for something to think about later, or be ignored entirely if this part is not perf sensitive.

Comment on lines +459 to +460
// Enable the notify future BEFORE acquiring the lock to avoid lost wakeups:
// any notify_waiters() call that fires between enable() and .await is captured.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL you can lose a notify_one() notification between notified() and enable(). Tokio/async-rust footgun #1989495 I guess...

/// This allows mapping from the buffered spans to another type, and
/// calling any method on the trace exporter to send traces
pub trait Export<T>: Send + Debug {
fn export_trace_chunks<'a: 'c, 'b: 'c, 'c>(
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do 'a and 'b need to be different here?

Base automatically changed from vianney/implement-shared-runtime to main April 8, 2026 15:17
@gh-worker-dd-mergequeue-cf854d gh-worker-dd-mergequeue-cf854d bot requested a review from a team as a code owner April 8, 2026 15:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants