feat(data-pipeline): port dd-trace-rs trace buffer implementation#1826
feat(data-pipeline): port dd-trace-rs trace buffer implementation#1826paullegranddc wants to merge 68 commits intomainfrom
Conversation
With the refactor the stop action is processed right away without processing all actions sent before it. This can cause some message to be lost if they were submitted right before shutdown.
📚 Documentation Check Results📦
|
# Motivation As we implement native spans in ruby in python, we have opportuinities to replace more components of the span pipeline with native code. One of them i the span buffer which we is before the trace exporter. # Changes Add a TraceBuffer implementation. The TraceBuffer is split in two part, a Sender which allows putting trace chunks in the queue and a background task, spawned on the shared runtime which periodically pulls data from the chunk and submit them using the TraceExporter.
3ede0c0 to
5938b42
Compare
Clippy Allow Annotation ReportComparing clippy allow annotations between branches:
Summary by Rule
Annotation Counts by File
Annotation Stats by Crate
About This ReportThis report tracks Clippy allow annotations for specific rules, showing how they've changed in this PR. Decreasing the number of these annotations generally improves code quality. |
🔒 Cargo Deny Results📦
|
|
✨ Fix all issues with BitsAI or with Cursor
|
BenchmarksComparisonBenchmark execution time: 2026-03-31 09:40:19 Comparing candidate commit 5938b42 in PR branch Found 0 performance improvements and 1 performance regressions! Performance is the same for 61 metrics, 0 unstable metrics.
|
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | 145.162µs | 147.194µs ± 1.851µs | 146.987µs ± 0.608µs | 147.504µs | 148.894µs | 154.489µs | 165.669µs | 12.71% | 6.214 | 53.332 | 1.25% | 0.131µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| sdk_test_data/rules-based | execution_time | [146.937µs; 147.450µs] or [-0.174%; +0.174%] | None | None | None |
Group 2
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| normalization/normalize_trace/test_trace | execution_time | 244.115ns | 254.188ns ± 13.661ns | 247.446ns ± 1.207ns | 256.824ns | 284.718ns | 299.800ns | 301.808ns | 21.97% | 1.954 | 2.976 | 5.36% | 0.966ns | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| normalization/normalize_trace/test_trace | execution_time | [252.294ns; 256.081ns] or [-0.745%; +0.745%] | None | None | None |
Group 3
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| profile_add_sample2_frames_x1000 | execution_time | 755.490µs | 757.180µs ± 0.605µs | 757.134µs ± 0.376µs | 757.532µs | 758.231µs | 758.721µs | 758.947µs | 0.24% | 0.326 | 0.400 | 0.08% | 0.043µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| profile_add_sample2_frames_x1000 | execution_time | [757.096µs; 757.264µs] or [-0.011%; +0.011%] | None | None | None |
Group 4
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| receiver_entry_point/report/2598 | execution_time | 3.425ms | 3.453ms ± 0.018ms | 3.448ms ± 0.011ms | 3.464ms | 3.487ms | 3.498ms | 3.525ms | 2.24% | 0.984 | 0.832 | 0.51% | 0.001ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| receiver_entry_point/report/2598 | execution_time | [3.450ms; 3.455ms] or [-0.071%; +0.071%] | None | None | None |
Group 5
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| profile_add_sample_frames_x1000 | execution_time | 4.372ms | 4.377ms ± 0.002ms | 4.377ms ± 0.002ms | 4.378ms | 4.381ms | 4.383ms | 4.390ms | 0.30% | 1.067 | 3.179 | 0.06% | 0.000ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| profile_add_sample_frames_x1000 | execution_time | [4.377ms; 4.377ms] or [-0.008%; +0.008%] | None | None | None |
Group 6
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| tags/replace_trace_tags | execution_time | 2.536µs | 2.564µs ± 0.017µs | 2.560µs ± 0.011µs | 2.573µs | 2.596µs | 2.608µs | 2.616µs | 2.18% | 0.735 | -0.111 | 0.65% | 0.001µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| tags/replace_trace_tags | execution_time | [2.561µs; 2.566µs] or [-0.091%; +0.091%] | None | None | None |
Group 7
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| credit_card/is_card_number/ | execution_time | 3.893µs | 3.914µs ± 0.004µs | 3.913µs ± 0.002µs | 3.916µs | 3.919µs | 3.922µs | 3.928µs | 0.38% | -0.124 | 7.013 | 0.09% | 0.000µs | 1 | 200 |
| credit_card/is_card_number/ | throughput | 254600717.643op/s | 255512513.409op/s ± 228583.913op/s | 255579476.992op/s ± 124824.793op/s | 255664568.087op/s | 255737052.045op/s | 255777886.821op/s | 256886255.168op/s | 0.51% | 0.148 | 7.108 | 0.09% | 16163.323op/s | 1 | 200 |
| credit_card/is_card_number/ 3782-8224-6310-005 | execution_time | 79.549µs | 80.012µs ± 0.253µs | 79.965µs ± 0.155µs | 80.140µs | 80.549µs | 80.616µs | 80.707µs | 0.93% | 0.751 | -0.044 | 0.32% | 0.018µs | 1 | 200 |
| credit_card/is_card_number/ 3782-8224-6310-005 | throughput | 12390503.878op/s | 12498179.354op/s ± 39470.673op/s | 12505514.939op/s ± 24184.513op/s | 12527126.558op/s | 12550250.639op/s | 12562153.382op/s | 12570921.367op/s | 0.52% | -0.738 | -0.064 | 0.32% | 2790.998op/s | 1 | 200 |
| credit_card/is_card_number/ 378282246310005 | execution_time | 73.143µs | 73.591µs ± 0.229µs | 73.555µs ± 0.142µs | 73.725µs | 74.000µs | 74.257µs | 74.528µs | 1.32% | 0.743 | 0.958 | 0.31% | 0.016µs | 1 | 200 |
| credit_card/is_card_number/ 378282246310005 | throughput | 13417733.508op/s | 13588742.076op/s ± 42218.429op/s | 13595181.912op/s ± 26183.206op/s | 13618234.480op/s | 13650845.305op/s | 13665220.412op/s | 13671818.060op/s | 0.56% | -0.721 | 0.894 | 0.31% | 2985.294op/s | 1 | 200 |
| credit_card/is_card_number/37828224631 | execution_time | 3.894µs | 3.917µs ± 0.005µs | 3.916µs ± 0.003µs | 3.919µs | 3.925µs | 3.942µs | 3.947µs | 0.79% | 1.898 | 12.307 | 0.13% | 0.000µs | 1 | 200 |
| credit_card/is_card_number/37828224631 | throughput | 253347052.827op/s | 255293143.117op/s ± 331215.054op/s | 255342918.091op/s ± 170479.472op/s | 255491747.091op/s | 255620352.412op/s | 255701130.017op/s | 256832818.930op/s | 0.58% | -1.856 | 12.169 | 0.13% | 23420.441op/s | 1 | 200 |
| credit_card/is_card_number/378282246310005 | execution_time | 69.605µs | 69.979µs ± 0.230µs | 69.940µs ± 0.142µs | 70.124µs | 70.418µs | 70.558µs | 70.840µs | 1.29% | 0.778 | 0.470 | 0.33% | 0.016µs | 1 | 200 |
| credit_card/is_card_number/378282246310005 | throughput | 14116301.742op/s | 14290119.381op/s ± 46778.607op/s | 14297951.818op/s ± 29155.792op/s | 14323720.442op/s | 14354096.083op/s | 14365366.281op/s | 14366723.373op/s | 0.48% | -0.759 | 0.423 | 0.33% | 3307.747op/s | 1 | 200 |
| credit_card/is_card_number/37828224631000521389798 | execution_time | 51.561µs | 52.017µs ± 0.192µs | 52.039µs ± 0.140µs | 52.163µs | 52.296µs | 52.358µs | 52.389µs | 0.67% | -0.308 | -0.741 | 0.37% | 0.014µs | 1 | 200 |
| credit_card/is_card_number/37828224631000521389798 | throughput | 19087893.420op/s | 19224691.039op/s ± 71022.559op/s | 19216312.118op/s ± 51643.365op/s | 19277026.130op/s | 19345199.103op/s | 19377081.744op/s | 19394638.076op/s | 0.93% | 0.321 | -0.731 | 0.37% | 5022.053op/s | 1 | 200 |
| credit_card/is_card_number/x371413321323331 | execution_time | 5.746µs | 5.884µs ± 0.054µs | 5.882µs ± 0.035µs | 5.915µs | 5.977µs | 6.011µs | 6.015µs | 2.27% | 0.035 | -0.230 | 0.91% | 0.004µs | 1 | 200 |
| credit_card/is_card_number/x371413321323331 | throughput | 166238848.022op/s | 169964129.026op/s ± 1548728.618op/s | 170004704.233op/s ± 1015979.168op/s | 171110156.247op/s | 172437366.587op/s | 173319885.316op/s | 174046044.711op/s | 2.38% | 0.013 | -0.229 | 0.91% | 109511.651op/s | 1 | 200 |
| credit_card/is_card_number_no_luhn/ | execution_time | 3.894µs | 3.918µs ± 0.004µs | 3.917µs ± 0.003µs | 3.920µs | 3.925µs | 3.927µs | 3.931µs | 0.35% | -0.469 | 4.445 | 0.11% | 0.000µs | 1 | 200 |
| credit_card/is_card_number_no_luhn/ | throughput | 254401296.829op/s | 255250206.334op/s ± 269425.943op/s | 255286703.763op/s ± 171973.284op/s | 255432114.207op/s | 255586331.382op/s | 255684132.656op/s | 256796304.225op/s | 0.59% | 0.489 | 4.547 | 0.11% | 19051.291op/s | 1 | 200 |
| credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 | execution_time | 64.193µs | 64.393µs ± 0.117µs | 64.363µs ± 0.057µs | 64.435µs | 64.631µs | 64.801µs | 64.873µs | 0.79% | 1.493 | 2.753 | 0.18% | 0.008µs | 1 | 200 |
| credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 | throughput | 15414672.476op/s | 15529724.925op/s ± 28134.668op/s | 15536833.450op/s ± 13794.405op/s | 15547075.037op/s | 15562010.099op/s | 15568728.678op/s | 15578011.637op/s | 0.27% | -1.479 | 2.700 | 0.18% | 1989.421op/s | 1 | 200 |
| credit_card/is_card_number_no_luhn/ 378282246310005 | execution_time | 57.487µs | 57.615µs ± 0.092µs | 57.599µs ± 0.057µs | 57.659µs | 57.786µs | 57.851µs | 57.995µs | 0.69% | 1.149 | 1.872 | 0.16% | 0.006µs | 1 | 200 |
| credit_card/is_card_number_no_luhn/ 378282246310005 | throughput | 17242920.443op/s | 17356621.217op/s ± 27547.458op/s | 17361332.840op/s ± 17136.325op/s | 17376897.436op/s | 17391244.900op/s | 17393536.631op/s | 17395174.579op/s | 0.19% | -1.136 | 1.824 | 0.16% | 1947.899op/s | 1 | 200 |
| credit_card/is_card_number_no_luhn/37828224631 | execution_time | 3.901µs | 3.917µs ± 0.004µs | 3.916µs ± 0.003µs | 3.919µs | 3.924µs | 3.928µs | 3.931µs | 0.39% | 0.536 | 0.951 | 0.11% | 0.000µs | 1 | 200 |
| credit_card/is_card_number_no_luhn/37828224631 | throughput | 254381684.678op/s | 255329191.025op/s ± 274053.523op/s | 255373173.071op/s ± 174269.807op/s | 255522235.569op/s | 255677287.135op/s | 255792287.145op/s | 256329373.409op/s | 0.37% | -0.528 | 0.950 | 0.11% | 19378.510op/s | 1 | 200 |
| credit_card/is_card_number_no_luhn/378282246310005 | execution_time | 54.606µs | 54.791µs ± 0.135µs | 54.762µs ± 0.061µs | 54.827µs | 55.113µs | 55.255µs | 55.452µs | 1.26% | 2.060 | 5.210 | 0.25% | 0.010µs | 1 | 200 |
| credit_card/is_card_number_no_luhn/378282246310005 | throughput | 18033460.050op/s | 18251216.346op/s ± 44717.898op/s | 18260881.540op/s ± 20178.563op/s | 18279949.638op/s | 18298739.401op/s | 18309197.391op/s | 18313075.536op/s | 0.29% | -2.038 | 5.102 | 0.24% | 3162.033op/s | 1 | 200 |
| credit_card/is_card_number_no_luhn/37828224631000521389798 | execution_time | 51.404µs | 52.145µs ± 0.239µs | 52.158µs ± 0.186µs | 52.359µs | 52.443µs | 52.482µs | 52.532µs | 0.72% | -0.713 | 0.140 | 0.46% | 0.017µs | 1 | 200 |
| credit_card/is_card_number_no_luhn/37828224631000521389798 | throughput | 19035976.714op/s | 19177654.413op/s ± 88263.922op/s | 19172533.331op/s ± 68386.954op/s | 19231939.616op/s | 19346709.831op/s | 19437837.774op/s | 19453846.857op/s | 1.47% | 0.736 | 0.199 | 0.46% | 6241.202op/s | 1 | 200 |
| credit_card/is_card_number_no_luhn/x371413321323331 | execution_time | 5.706µs | 5.887µs ± 0.055µs | 5.898µs ± 0.035µs | 5.922µs | 5.969µs | 6.001µs | 6.050µs | 2.57% | -0.367 | 0.332 | 0.94% | 0.004µs | 1 | 200 |
| credit_card/is_card_number_no_luhn/x371413321323331 | throughput | 165296818.766op/s | 169884924.669op/s ± 1606249.539op/s | 169552986.733op/s ± 996677.214op/s | 170742116.737op/s | 172520793.526op/s | 174112874.907op/s | 175244568.347op/s | 3.36% | 0.429 | 0.392 | 0.94% | 113578.994op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| credit_card/is_card_number/ | execution_time | [3.913µs; 3.914µs] or [-0.012%; +0.012%] | None | None | None |
| credit_card/is_card_number/ | throughput | [255480833.877op/s; 255544192.941op/s] or [-0.012%; +0.012%] | None | None | None |
| credit_card/is_card_number/ 3782-8224-6310-005 | execution_time | [79.977µs; 80.048µs] or [-0.044%; +0.044%] | None | None | None |
| credit_card/is_card_number/ 3782-8224-6310-005 | throughput | [12492709.099op/s; 12503649.610op/s] or [-0.044%; +0.044%] | None | None | None |
| credit_card/is_card_number/ 378282246310005 | execution_time | [73.559µs; 73.623µs] or [-0.043%; +0.043%] | None | None | None |
| credit_card/is_card_number/ 378282246310005 | throughput | [13582891.008op/s; 13594593.144op/s] or [-0.043%; +0.043%] | None | None | None |
| credit_card/is_card_number/37828224631 | execution_time | [3.916µs; 3.918µs] or [-0.018%; +0.018%] | None | None | None |
| credit_card/is_card_number/37828224631 | throughput | [255247239.896op/s; 255339046.338op/s] or [-0.018%; +0.018%] | None | None | None |
| credit_card/is_card_number/378282246310005 | execution_time | [69.947µs; 70.011µs] or [-0.045%; +0.045%] | None | None | None |
| credit_card/is_card_number/378282246310005 | throughput | [14283636.316op/s; 14296602.446op/s] or [-0.045%; +0.045%] | None | None | None |
| credit_card/is_card_number/37828224631000521389798 | execution_time | [51.991µs; 52.044µs] or [-0.051%; +0.051%] | None | None | None |
| credit_card/is_card_number/37828224631000521389798 | throughput | [19214847.995op/s; 19234534.082op/s] or [-0.051%; +0.051%] | None | None | None |
| credit_card/is_card_number/x371413321323331 | execution_time | [5.877µs; 5.892µs] or [-0.126%; +0.126%] | None | None | None |
| credit_card/is_card_number/x371413321323331 | throughput | [169749490.134op/s; 170178767.917op/s] or [-0.126%; +0.126%] | None | None | None |
| credit_card/is_card_number_no_luhn/ | execution_time | [3.917µs; 3.918µs] or [-0.015%; +0.015%] | None | None | None |
| credit_card/is_card_number_no_luhn/ | throughput | [255212866.489op/s; 255287546.178op/s] or [-0.015%; +0.015%] | None | None | None |
| credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 | execution_time | [64.377µs; 64.409µs] or [-0.025%; +0.025%] | None | None | None |
| credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 | throughput | [15525825.730op/s; 15533624.119op/s] or [-0.025%; +0.025%] | None | None | None |
| credit_card/is_card_number_no_luhn/ 378282246310005 | execution_time | [57.602µs; 57.628µs] or [-0.022%; +0.022%] | None | None | None |
| credit_card/is_card_number_no_luhn/ 378282246310005 | throughput | [17352803.404op/s; 17360439.030op/s] or [-0.022%; +0.022%] | None | None | None |
| credit_card/is_card_number_no_luhn/37828224631 | execution_time | [3.916µs; 3.917µs] or [-0.015%; +0.015%] | None | None | None |
| credit_card/is_card_number_no_luhn/37828224631 | throughput | [255291209.843op/s; 255367172.208op/s] or [-0.015%; +0.015%] | None | None | None |
| credit_card/is_card_number_no_luhn/378282246310005 | execution_time | [54.772µs; 54.810µs] or [-0.034%; +0.034%] | None | None | None |
| credit_card/is_card_number_no_luhn/378282246310005 | throughput | [18245018.876op/s; 18257413.817op/s] or [-0.034%; +0.034%] | None | None | None |
| credit_card/is_card_number_no_luhn/37828224631000521389798 | execution_time | [52.112µs; 52.178µs] or [-0.064%; +0.064%] | None | None | None |
| credit_card/is_card_number_no_luhn/37828224631000521389798 | throughput | [19165421.883op/s; 19189886.944op/s] or [-0.064%; +0.064%] | None | None | None |
| credit_card/is_card_number_no_luhn/x371413321323331 | execution_time | [5.879µs; 5.895µs] or [-0.131%; +0.131%] | None | None | None |
| credit_card/is_card_number_no_luhn/x371413321323331 | throughput | [169662313.931op/s; 170107535.407op/s] or [-0.131%; +0.131%] | None | None | None |
Group 8
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... | execution_time | 534.633µs | 535.331µs ± 0.508µs | 535.168µs ± 0.278µs | 535.624µs | 536.259µs | 536.822µs | 537.367µs | 0.41% | 1.130 | 1.071 | 0.09% | 0.036µs | 1 | 200 |
| normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... | throughput | 1860925.180op/s | 1868003.766op/s ± 1770.247op/s | 1868573.345op/s ± 970.945op/s | 1869242.578op/s | 1870152.146op/s | 1870372.134op/s | 1870441.555op/s | 0.10% | -1.125 | 1.052 | 0.09% | 125.175op/s | 1 | 200 |
| normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて | execution_time | 381.913µs | 382.847µs ± 0.462µs | 382.768µs ± 0.306µs | 383.111µs | 383.632µs | 384.109µs | 384.633µs | 0.49% | 0.821 | 1.012 | 0.12% | 0.033µs | 1 | 200 |
| normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて | throughput | 2599877.693op/s | 2612015.951op/s ± 3149.675op/s | 2612545.626op/s ± 2091.495op/s | 2614298.777op/s | 2615896.217op/s | 2617295.626op/s | 2618394.105op/s | 0.22% | -0.812 | 0.987 | 0.12% | 222.716op/s | 1 | 200 |
| normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters | execution_time | 194.544µs | 195.514µs ± 0.494µs | 195.557µs ± 0.254µs | 195.778µs | 196.104µs | 196.446µs | 199.155µs | 1.84% | 1.821 | 13.424 | 0.25% | 0.035µs | 1 | 200 |
| normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters | throughput | 5021211.978op/s | 5114754.570op/s ± 12878.646op/s | 5113610.637op/s ± 6643.531op/s | 5121482.960op/s | 5135117.765op/s | 5138075.153op/s | 5140234.387op/s | 0.52% | -1.731 | 12.663 | 0.25% | 910.658op/s | 1 | 200 |
| normalization/normalize_service/normalize_service/[empty string] | execution_time | 37.963µs | 38.315µs ± 0.130µs | 38.306µs ± 0.082µs | 38.406µs | 38.534µs | 38.604µs | 38.736µs | 1.12% | 0.080 | 0.277 | 0.34% | 0.009µs | 1 | 200 |
| normalization/normalize_service/normalize_service/[empty string] | throughput | 25815922.272op/s | 26099776.312op/s ± 88862.624op/s | 26105454.639op/s ± 56048.329op/s | 26153468.819op/s | 26245179.056op/s | 26322381.104op/s | 26341483.420op/s | 0.90% | -0.057 | 0.269 | 0.34% | 6283.536op/s | 1 | 200 |
| normalization/normalize_service/normalize_service/test_ASCII | execution_time | 45.801µs | 45.943µs ± 0.080µs | 45.923µs ± 0.039µs | 45.982µs | 46.106µs | 46.191µs | 46.281µs | 0.78% | 1.323 | 2.228 | 0.17% | 0.006µs | 1 | 200 |
| normalization/normalize_service/normalize_service/test_ASCII | throughput | 21607035.251op/s | 21766200.000op/s ± 37815.879op/s | 21775786.351op/s ± 18332.460op/s | 21790339.048op/s | 21810108.906op/s | 21827075.042op/s | 21833399.036op/s | 0.26% | -1.310 | 2.181 | 0.17% | 2673.986op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... | execution_time | [535.261µs; 535.402µs] or [-0.013%; +0.013%] | None | None | None |
| normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... | throughput | [1867758.427op/s; 1868249.105op/s] or [-0.013%; +0.013%] | None | None | None |
| normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて | execution_time | [382.783µs; 382.911µs] or [-0.017%; +0.017%] | None | None | None |
| normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて | throughput | [2611579.437op/s; 2612452.466op/s] or [-0.017%; +0.017%] | None | None | None |
| normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters | execution_time | [195.446µs; 195.583µs] or [-0.035%; +0.035%] | None | None | None |
| normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters | throughput | [5112969.714op/s; 5116539.427op/s] or [-0.035%; +0.035%] | None | None | None |
| normalization/normalize_service/normalize_service/[empty string] | execution_time | [38.297µs; 38.333µs] or [-0.047%; +0.047%] | None | None | None |
| normalization/normalize_service/normalize_service/[empty string] | throughput | [26087460.807op/s; 26112091.817op/s] or [-0.047%; +0.047%] | None | None | None |
| normalization/normalize_service/normalize_service/test_ASCII | execution_time | [45.932µs; 45.954µs] or [-0.024%; +0.024%] | None | None | None |
| normalization/normalize_service/normalize_service/test_ASCII | throughput | [21760959.083op/s; 21771440.917op/s] or [-0.024%; +0.024%] | None | None | None |
Group 9
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| two way interface | execution_time | 14.987µs | 15.554µs ± 0.201µs | 15.537µs ± 0.120µs | 15.665µs | 15.953µs | 16.031µs | 16.179µs | 4.13% | 0.347 | 0.427 | 1.29% | 0.014µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| two way interface | execution_time | [15.526µs; 15.581µs] or [-0.179%; +0.179%] | None | None | None |
Group 10
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| benching serializing traces from their internal representation to msgpack | execution_time | 14.724ms | 14.797ms ± 0.030ms | 14.793ms ± 0.017ms | 14.813ms | 14.846ms | 14.863ms | 14.944ms | 1.01% | 1.270 | 4.631 | 0.20% | 0.002ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| benching serializing traces from their internal representation to msgpack | execution_time | [14.793ms; 14.801ms] or [-0.028%; +0.028%] | None | None | None |
Group 11
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | 191.119ns | 193.821ns ± 2.150ns | 193.349ns ± 1.179ns | 194.677ns | 197.961ns | 201.344ns | 203.578ns | 5.29% | 1.519 | 3.216 | 1.11% | 0.152ns | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| single_flag_killswitch/rules-based | execution_time | [193.523ns; 194.119ns] or [-0.154%; +0.154%] | None | None | None |
Group 12
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| write only interface | execution_time | 5.222µs | 5.418µs ± 0.037µs | 5.416µs ± 0.026µs | 5.445µs | 5.477µs | 5.492µs | 5.500µs | 1.54% | -0.476 | 2.764 | 0.68% | 0.003µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| write only interface | execution_time | [5.413µs; 5.424µs] or [-0.095%; +0.095%] | None | None | None |
Group 13
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| concentrator/add_spans_to_concentrator | execution_time | 14.862ms | 14.910ms ± 0.017ms | 14.909ms ± 0.010ms | 14.919ms | 14.936ms | 14.960ms | 14.985ms | 0.51% | 0.492 | 2.000 | 0.12% | 0.001ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| concentrator/add_spans_to_concentrator | execution_time | [14.907ms; 14.912ms] or [-0.016%; +0.016%] | None | None | None |
Group 14
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sql/obfuscate_sql_string | execution_time | 288.634µs | 289.526µs ± 0.576µs | 289.438µs ± 0.190µs | 289.612µs | 290.412µs | 291.836µs | 293.627µs | 1.45% | 3.600 | 20.030 | 0.20% | 0.041µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| sql/obfuscate_sql_string | execution_time | [289.446µs; 289.605µs] or [-0.028%; +0.028%] | None | None | None |
Group 15
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... | execution_time | 185.650µs | 186.487µs ± 0.660µs | 186.242µs ± 0.295µs | 186.951µs | 187.750µs | 188.363µs | 188.613µs | 1.27% | 1.102 | 0.272 | 0.35% | 0.047µs | 1 | 200 |
| normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... | throughput | 5301872.025op/s | 5362364.084op/s ± 18900.401op/s | 5369366.076op/s ± 8519.453op/s | 5376292.601op/s | 5381953.072op/s | 5385146.412op/s | 5386485.013op/s | 0.32% | -1.091 | 0.237 | 0.35% | 1336.460op/s | 1 | 200 |
| normalization/normalize_name/normalize_name/bad-name | execution_time | 17.850µs | 17.979µs ± 0.065µs | 17.977µs ± 0.035µs | 18.007µs | 18.102µs | 18.168µs | 18.229µs | 1.40% | 0.898 | 1.787 | 0.36% | 0.005µs | 1 | 200 |
| normalization/normalize_name/normalize_name/bad-name | throughput | 54857142.857op/s | 55620139.313op/s ± 200755.788op/s | 55626646.928op/s ± 107684.306op/s | 55755093.011op/s | 55924799.103op/s | 55994131.933op/s | 56020865.905op/s | 0.71% | -0.866 | 1.698 | 0.36% | 14195.578op/s | 1 | 200 |
| normalization/normalize_name/normalize_name/good | execution_time | 9.864µs | 9.929µs ± 0.072µs | 9.885µs ± 0.015µs | 9.978µs | 10.073µs | 10.122µs | 10.129µs | 2.47% | 1.090 | -0.146 | 0.72% | 0.005µs | 1 | 200 |
| normalization/normalize_name/normalize_name/good | throughput | 98726648.921op/s | 100717184.066op/s ± 725984.940op/s | 101166649.247op/s ± 148775.520op/s | 101249768.524op/s | 101322506.300op/s | 101361822.768op/s | 101379910.161op/s | 0.21% | -1.075 | -0.189 | 0.72% | 51334.887op/s | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... | execution_time | [186.396µs; 186.579µs] or [-0.049%; +0.049%] | None | None | None |
| normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... | throughput | [5359744.670op/s; 5364983.498op/s] or [-0.049%; +0.049%] | None | None | None |
| normalization/normalize_name/normalize_name/bad-name | execution_time | [17.970µs; 17.988µs] or [-0.050%; +0.050%] | None | None | None |
| normalization/normalize_name/normalize_name/bad-name | throughput | [55592316.492op/s; 55647962.135op/s] or [-0.050%; +0.050%] | None | None | None |
| normalization/normalize_name/normalize_name/good | execution_time | [9.919µs; 9.939µs] or [-0.101%; +0.101%] | None | None | None |
| normalization/normalize_name/normalize_name/good | throughput | [100616569.535op/s; 100817798.596op/s] or [-0.100%; +0.100%] | None | None | None |
Group 16
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| profile_add_sample_timestamped_x1000 | execution_time | 4.374ms | 4.381ms ± 0.009ms | 4.379ms ± 0.002ms | 4.382ms | 4.385ms | 4.393ms | 4.501ms | 2.78% | 11.444 | 145.811 | 0.21% | 0.001ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| profile_add_sample_timestamped_x1000 | execution_time | [4.379ms; 4.382ms] or [-0.029%; +0.029%] | None | None | None |
Group 17
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| redis/obfuscate_redis_string | execution_time | 33.647µs | 34.231µs ± 0.984µs | 33.772µs ± 0.056µs | 33.889µs | 36.332µs | 36.419µs | 37.277µs | 10.38% | 1.703 | 1.004 | 2.87% | 0.070µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| redis/obfuscate_redis_string | execution_time | [34.094µs; 34.367µs] or [-0.399%; +0.399%] | None | None | None |
Group 18
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| profile_serialize_compressed_pprof_timestamped_x1000 | execution_time | 934.320µs | 937.751µs ± 2.304µs | 937.143µs ± 1.085µs | 938.459µs | 941.935µs | 945.090µs | 951.054µs | 1.48% | 1.823 | 5.759 | 0.25% | 0.163µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| profile_serialize_compressed_pprof_timestamped_x1000 | execution_time | [937.431µs; 938.070µs] or [-0.034%; +0.034%] | None | None | None |
Group 19
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ip_address/quantize_peer_ip_address_benchmark | execution_time | 4.929µs | 5.007µs ± 0.052µs | 5.013µs ± 0.055µs | 5.046µs | 5.091µs | 5.093µs | 5.097µs | 1.68% | 0.231 | -1.377 | 1.03% | 0.004µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| ip_address/quantize_peer_ip_address_benchmark | execution_time | [5.000µs; 5.014µs] or [-0.143%; +0.143%] | None | None | None |
Group 20
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| benching string interning on wordpress profile | execution_time | 160.220µs | 161.100µs ± 0.614µs | 161.078µs ± 0.227µs | 161.283µs | 161.677µs | 162.973µs | 165.895µs | 2.99% | 4.142 | 27.671 | 0.38% | 0.043µs | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| benching string interning on wordpress profile | execution_time | [161.015µs; 161.185µs] or [-0.053%; +0.053%] | None | None | None |
Group 21
| cpu_model | git_commit_sha | git_commit_date | git_branch |
|---|---|---|---|
| Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | 5938b42 | 1774948956 | paullgdc/data-pipeline/trace_buffer |
| scenario | metric | min | mean ± sd | median ± mad | p75 | p95 | p99 | max | peak_to_median_ratio | skewness | kurtosis | cv | sem | runs | sample_size |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| benching deserializing traces from msgpack to their internal representation | execution_time | 47.873ms | 48.280ms ± 0.975ms | 48.127ms ± 0.095ms | 48.260ms | 48.438ms | 53.542ms | 58.876ms | 22.33% | 8.559 | 79.737 | 2.01% | 0.069ms | 1 | 200 |
| scenario | metric | 95% CI mean | Shapiro-Wilk pvalue | Ljung-Box pvalue (lag=1) | Dip test pvalue |
|---|---|---|---|---|---|
| benching deserializing traces from msgpack to their internal representation | execution_time | [48.145ms; 48.416ms] or [-0.280%; +0.280%] | None | None | None |
Baseline
Omitted due to size.
| if res.timed_out() { | ||
| return Err(TraceBufferError::TimedOut(timeout)); | ||
| } | ||
| Ok(()) |
There was a problem hiding this comment.
Or am I missing something?
Doesn't this always return Ok, even if the last flush didn't happen because of shut down?
I think we need to look at the state.last_flush_generation
There was a problem hiding this comment.
res is a WaitTimeoutResult so I cannot just return it
There was a problem hiding this comment.
Sorry for the confusion. I was editing my question while you answered.
Doesn't this always return Ok, even if the last flush didn't happen because of shut down?
I think we need to look at the
state.last_flush_generation
| @@ -0,0 +1,851 @@ | |||
| // Copyright 2025-Present Datadog, Inc. https://www.datadoghq.com/ | |||
| // SPDX-License-Identifier: Apache-2.0 | |||
There was a problem hiding this comment.
Nit: a module-level doc, even if short, is always nice to get an idea of what happens here quickly.
| #[derive(Debug)] | ||
| pub enum TraceBufferError { | ||
| AlreadyShutdown, | ||
| TimedOut(std::time::Duration), |
There was a problem hiding this comment.
| TimedOut(std::time::Duration), | |
| TimedOut(Duration), |
| } | ||
|
|
||
| fn reset(&mut self) { | ||
| *self = Self::new(self.max_buffered_spans); |
There was a problem hiding this comment.
Not sure if it is worth it (is this remotely perf-sensitive?), but clear()ing chunks instead of rea might avoid dropping the old vector to just re-allocate a new one.
| return Ok(()); | ||
| } | ||
|
|
||
| let chunk_len: usize = chunk.len(); |
There was a problem hiding this comment.
| let chunk_len: usize = chunk.len(); | |
| let chunk_len = chunk.len(); |
|
|
||
| struct Batch<T> { | ||
| chunks: Vec<TraceChunk<T>>, | ||
| last_flush: std::time::Instant, |
There was a problem hiding this comment.
| last_flush: std::time::Instant, | |
| last_flush: Instant, |
| let chunk_len = trace_chunk.len(); | ||
| if chunk_len == 0 { |
There was a problem hiding this comment.
| let chunk_len = trace_chunk.len(); | |
| if chunk_len == 0 { | |
| if trace_chunk.is_empty() { |
| pub spans_queued: usize, | ||
| } | ||
|
|
||
| fn channel<T>( |
There was a problem hiding this comment.
We discussed that orally, but I'm writing it down for the record: I'm not sure about the trade-off of going with a custom channel implementation here. It seems what we do is simple enough that we should be able to piggy-back on existing, well-tested, well-optimized channel impls. It's also a non trivial part of the code of this module. It might be the case that the custom channel is actually the right solution, but using a standard bounded channel or whatnot should be the default approach IMHO, only moving to a custom solution if we find good reasons to do so in the process. And in this case, the trade-offs should be properly documented here.
| let gen = state.batch.batch_gen; | ||
| if state.batch.span_count() > self.flush_trigger_number_of_spans || self.synchronous_write { | ||
| state.flush_needed = true; | ||
| self.waiter.notify_receiver(state); |
There was a problem hiding this comment.
I wonder if we can avoid notifying if state.flush_needed was already set to true here. I think, but it must be double checked that this won't let a waiter be stuck forever. I'm saying that because notifying is basically a futex syscall, and a syscall is costly, so usually we try to avoid unnecessary wait or notify. That being said, it's a bit of a detail and it might be for something to think about later, or be ignored entirely if this part is not perf sensitive.
| // Enable the notify future BEFORE acquiring the lock to avoid lost wakeups: | ||
| // any notify_waiters() call that fires between enable() and .await is captured. |
There was a problem hiding this comment.
TIL you can lose a notify_one() notification between notified() and enable(). Tokio/async-rust footgun #1989495 I guess...
| /// This allows mapping from the buffered spans to another type, and | ||
| /// calling any method on the trace exporter to send traces | ||
| pub trait Export<T>: Send + Debug { | ||
| fn export_trace_chunks<'a: 'c, 'b: 'c, 'c>( |
There was a problem hiding this comment.
Do 'a and 'b need to be different here?
Motivation
As we implement native spans in ruby in python, we have opportuinities to replace more components of the span pipeline with native code.
One of them i the span buffer which we is before the trace exporter.
Changes
Add a TraceBuffer implementation.
The TraceBuffer is split in two part, a Sender which allows putting trace chunks in the queue and a background task, spawned on the shared runtime which periodically pulls data from the chunk and submit them using the TraceExporter.