Skip to content

chore: Batch aggregator & range aggregation pipeline#21998

Draft
benclive wants to merge 9 commits into
mainfrom
benclive/batch-aggregator-add
Draft

chore: Batch aggregator & range aggregation pipeline#21998
benclive wants to merge 9 commits into
mainfrom
benclive/batch-aggregator-add

Conversation

@benclive
Copy link
Copy Markdown
Contributor

What this PR does / why we need it:
Switches the aggregator to operate on batches in order to be more performant.

I achieved a 2-3x improvement to range aggregation using this approach (+ operating on UnixNano instead of time.Time was significant)

RangeAggregation:

goos: darwin
goarch: arm64
pkg: github.com/grafana/loki/v3/pkg/engine/internal/executor
cpu: Apple M3 Max
                                             │ range_agg_before.txt │        range_agg_after.txt         │
                                             │        sec/op        │   sec/op     vs base               │
RangeAggregationPipeline/case=instant-14               189.17µ ± 1%   84.50µ ± 2%  -55.33% (p=0.002 n=6)
RangeAggregationPipeline/case=aligned-14                82.36µ ± 6%   34.54µ ± 3%  -58.06% (p=0.002 n=6)
RangeAggregationPipeline/case=gapped-14                 80.97µ ± 2%   30.97µ ± 1%  -61.75% (p=0.002 n=6)
RangeAggregationPipeline/case=overlapping-14            86.98µ ± 3%   38.96µ ± 2%  -55.21% (p=0.002 n=6)
geomean                                                 102.3µ        43.32µ       -57.68%

                                             │ range_agg_before.txt │        range_agg_after.txt         │
                                             │         B/op         │     B/op      vs base              │
RangeAggregationPipeline/case=instant-14               21.92Ki ± 0%   20.64Ki ± 0%  -5.81% (p=0.002 n=6)
RangeAggregationPipeline/case=aligned-14               23.14Ki ± 0%   21.86Ki ± 0%  -5.51% (p=0.002 n=6)
RangeAggregationPipeline/case=gapped-14                21.18Ki ± 0%   20.03Ki ± 0%  -5.43% (p=0.002 n=6)
RangeAggregationPipeline/case=overlapping-14           24.26Ki ± 0%   25.38Ki ± 0%  +4.61% (p=0.002 n=6)
geomean                                                22.59Ki        21.89Ki       -3.13%

                                             │ range_agg_before.txt │        range_agg_after.txt        │
                                             │      allocs/op       │ allocs/op   vs base               │
RangeAggregationPipeline/case=instant-14                 274.0 ± 0%   234.0 ± 0%  -14.60% (p=0.002 n=6)
RangeAggregationPipeline/case=aligned-14                 296.0 ± 0%   256.0 ± 0%  -13.51% (p=0.002 n=6)
RangeAggregationPipeline/case=gapped-14                  265.0 ± 0%   234.0 ± 0%  -11.70% (p=0.002 n=6)
RangeAggregationPipeline/case=overlapping-14             301.0 ± 0%   278.0 ± 0%   -7.64% (p=0.002 n=6)
geomean                                                  283.6        249.8       -11.90%

Aggregator itself:

goos: darwin
goarch: arm64
pkg: github.com/grafana/loki/v3/pkg/engine/internal/executor
cpu: Apple M3 Max
                                    │ agg_before.txt │           agg_after.txt            │
                                    │     sec/op     │   sec/op     vs base               │
AggregatorAdd/case=interleaved-14       119.57µ ± 1%   69.39µ ± 1%  -41.97% (p=0.002 n=6)
AggregatorAdd/case=single_series-14     118.47µ ± 1%   69.01µ ± 2%  -41.75% (p=0.002 n=6)
geomean                                  119.0µ        69.20µ       -41.86%

                                    │ agg_before.txt │           agg_after.txt           │
                                    │      B/op      │    B/op     vs base               │
AggregatorAdd/case=interleaved-14        382.00 ± 1%   26.00 ± 0%  -93.19% (p=0.002 n=6)
AggregatorAdd/case=single_series-14      373.00 ± 0%   21.50 ± 2%  -94.24% (p=0.002 n=6)
geomean                                   377.5        23.64       -93.74%

                                    │ agg_before.txt │             agg_after.txt              │
                                    │   allocs/op    │ allocs/op   vs base                    │
AggregatorAdd/case=interleaved-14         2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.002 n=6)
AggregatorAdd/case=single_series-14       2.000 ± 0%   0.000 ± 0%  -100.00% (p=0.002 n=6)
geomean                                   2.000                    ?                      ¹ ²
¹ summaries must be >0 to compute geomean
² ratios must be >0 to compute geomean

Note: I used AI to make most of the changes and committed after checking each major change.

Best commits to review:
Support without
Use unixnano everywhere instead of time.Time
and the final Clean up

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant