Skip to content

Commit 4312232

Browse files
MryangeMryange
authored
[env](compiler)Optimize the compilation time of aggregate_function_orthogonal_bitmap.cpp (#61606)
### What problem does this PR solve? Optimize the compilation time of `aggregate_function_orthogonal_bitmap.cpp` by reducing template instantiation bloat and improving build parallelism. ### Changes **1. Split single translation unit into multiple files** The original `aggregate_function_orthogonal_bitmap.cpp` instantiated all 6 aggregate functions (~36 template instances) in a single file, forcing serial compilation. Now split into: - `aggregate_function_orthogonal_bitmap.cpp` — forward declarations + registration only (compiles fast) - `aggregate_function_orth_bitmap_intersect.cpp` - `aggregate_function_orth_bitmap_intersect_count.cpp` - `aggregate_function_orth_bitmap_union_count.cpp` - `aggregate_function_orth_intersect_count.cpp` - `aggregate_function_orth_bitmap_expr_cal.cpp` - `aggregate_function_orth_bitmap_expr_cal_count.cpp` **2. Remove unnecessary `PrimitiveType` template parameter from `OrthBitmapUnionCountData`** `OrthBitmapUnionCountData<T>` never uses `T` — it only operates on `ColumnBitmap` directly. Removed the template parameter, reducing 6 useless instantiations to 1. **3. Eliminate type dispatch for `ExprCal` / `ExprCalCount`** FE guarantees these functions receive exactly 3 parameters with VARCHAR filter columns. Replaced the generic 6-type dispatch with direct `TYPE_STRING` instantiation, reducing 12 instantiations to 2. **4. Use precise expression tags matching FE definitions** Added `ExprTag` template parameter to `AggFunctionOrthBitmapFunc` (default: `VarargsExpression`): | Function | FE Arity | Old BE Tag | New BE Tag | |----------|----------|------------|------------| | `orthogonal_bitmap_union_count` | `UnaryExpression` (1 arg) | `VarargsExpression` | `UnaryExpression` | | `orthogonal_bitmap_expr_calculate` | Fixed 3 args | `VarargsExpression` | `MultiExpression` | | `orthogonal_bitmap_expr_calculate_count` | Fixed 3 args | `VarargsExpression` | `MultiExpression` | | Others (intersect/intersect_count) | Varargs (≥3) | `VarargsExpression` | `VarargsExpression` (unchanged) | **5. Add `inline` to template specializations in `bitmap_intersect.h`** The `Helper::write_to` / `serialize_size` / `read_from` template specializations in the header lacked `inline`, causing duplicate symbol linker errors after splitting into multiple translation units. ### Summary | Metric | Before | After | |--------|--------|-------| | Translation units | 1 (serial) | 7 (parallel) | | Template instantiations | ~36 | ~21 (~42% reduction) | | `ExprCal`/`ExprCalCount` instantiations | 12 | 2 | | `OrthBitmapUnionCountData` instantiations | 6 | 1 |
1 parent 0a4f3a9 commit 4312232

10 files changed

Lines changed: 312 additions & 65 deletions

‎be/src/exprs/aggregate/aggregate_function_orth_bitmap_expr_cal.cpp‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#include "core/data_type/data_type.h"
19+
#include "exprs/aggregate/aggregate_function_orthogonal_bitmap.h"
20+
#include "exprs/aggregate/helpers.h"
21+
22+
namespace doris {
23+
#include "common/compile_check_begin.h"
24+
25+
// FE guarantees exactly 3 params: (bitmap, varchar_filter, varchar_expr)
26+
AggregateFunctionPtr create_aggregate_function_orth_bitmap_expr_cal(
27+
const std::string& name, const DataTypes& argument_types, const DataTypePtr& result_type,
28+
const bool result_is_nullable, const AggregateFunctionAttr& attr) {
29+
if (argument_types.size() != 3) {
30+
LOG(WARNING) << "Incorrect number of arguments for aggregate function " << name;
31+
return nullptr;
32+
}
33+
return creator_without_type::create<
34+
AggFunctionOrthBitmapFunc<AggOrthBitMapExprCal<TYPE_STRING>, MultiExpression>>(
35+
argument_types, result_is_nullable, attr);
36+
}
37+
38+
} // namespace doris

‎be/src/exprs/aggregate/aggregate_function_orth_bitmap_expr_cal_count.cpp‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#include "core/data_type/data_type.h"
19+
#include "exprs/aggregate/aggregate_function_orthogonal_bitmap.h"
20+
#include "exprs/aggregate/helpers.h"
21+
22+
namespace doris {
23+
#include "common/compile_check_begin.h"
24+
25+
// FE guarantees exactly 3 params: (bitmap, varchar_filter, varchar_expr)
26+
AggregateFunctionPtr create_aggregate_function_orth_bitmap_expr_cal_count(
27+
const std::string& name, const DataTypes& argument_types, const DataTypePtr& result_type,
28+
const bool result_is_nullable, const AggregateFunctionAttr& attr) {
29+
if (argument_types.size() != 3) {
30+
LOG(WARNING) << "Incorrect number of arguments for aggregate function " << name;
31+
return nullptr;
32+
}
33+
return creator_without_type::create<
34+
AggFunctionOrthBitmapFunc<AggOrthBitMapExprCalCount<TYPE_STRING>, MultiExpression>>(
35+
argument_types, result_is_nullable, attr);
36+
}
37+
38+
} // namespace doris

‎be/src/exprs/aggregate/aggregate_function_orth_bitmap_intersect.cpp‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#include "exprs/aggregate/aggregate_function_orthogonal_bitmap_impl.h"
19+
20+
namespace doris {
21+
#include "common/compile_check_begin.h"
22+
23+
AggregateFunctionPtr create_aggregate_function_orth_bitmap_intersect(
24+
const std::string& name, const DataTypes& argument_types, const DataTypePtr& result_type,
25+
const bool result_is_nullable, const AggregateFunctionAttr& attr) {
26+
return create_aggregate_function_orthogonal<AggOrthBitMapIntersect>(
27+
name, argument_types, result_type, result_is_nullable, attr);
28+
}
29+
30+
} // namespace doris

‎be/src/exprs/aggregate/aggregate_function_orth_bitmap_intersect_count.cpp‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#include "exprs/aggregate/aggregate_function_orthogonal_bitmap_impl.h"
19+
20+
namespace doris {
21+
#include "common/compile_check_begin.h"
22+
23+
AggregateFunctionPtr create_aggregate_function_orth_bitmap_intersect_count(
24+
const std::string& name, const DataTypes& argument_types, const DataTypePtr& result_type,
25+
const bool result_is_nullable, const AggregateFunctionAttr& attr) {
26+
return create_aggregate_function_orthogonal<AggOrthBitMapIntersectCount>(
27+
name, argument_types, result_type, result_is_nullable, attr);
28+
}
29+
30+
} // namespace doris

‎be/src/exprs/aggregate/aggregate_function_orth_bitmap_union_count.cpp‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#include "core/data_type/data_type.h"
19+
#include "exprs/aggregate/aggregate_function_orthogonal_bitmap.h"
20+
#include "exprs/aggregate/helpers.h"
21+
22+
namespace doris {
23+
#include "common/compile_check_begin.h"
24+
25+
AggregateFunctionPtr create_aggregate_function_orth_bitmap_union_count(
26+
const std::string& name, const DataTypes& argument_types, const DataTypePtr& result_type,
27+
const bool result_is_nullable, const AggregateFunctionAttr& attr) {
28+
if (argument_types.empty()) {
29+
LOG(WARNING) << "Incorrect number of arguments for aggregate function " << name;
30+
return nullptr;
31+
}
32+
return creator_without_type::create<
33+
AggFunctionOrthBitmapFunc<OrthBitmapUnionCountData, UnaryExpression>>(
34+
argument_types, result_is_nullable, attr);
35+
}
36+
37+
} // namespace doris

‎be/src/exprs/aggregate/aggregate_function_orth_intersect_count.cpp‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#include "exprs/aggregate/aggregate_function_orthogonal_bitmap_impl.h"
19+
20+
namespace doris {
21+
#include "common/compile_check_begin.h"
22+
23+
AggregateFunctionPtr create_aggregate_function_orth_intersect_count(
24+
const std::string& name, const DataTypes& argument_types, const DataTypePtr& result_type,
25+
const bool result_is_nullable, const AggregateFunctionAttr& attr) {
26+
return create_aggregate_function_orthogonal<AggIntersectCount>(
27+
name, argument_types, result_type, result_is_nullable, attr);
28+
}
29+
30+
} // namespace doris

‎be/src/exprs/aggregate/aggregate_function_orthogonal_bitmap.cpp‎

Lines changed: 30 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -15,70 +15,52 @@
1515
// specific language governing permissions and limitations
1616
// under the License.
1717

18-
#include "exprs/aggregate/aggregate_function_orthogonal_bitmap.h"
19-
20-
#include <map>
21-
#include <ostream>
18+
#include <string>
2219

2320
#include "core/data_type/data_type.h"
24-
#include "core/data_type/data_type_nullable.h"
21+
#include "exprs/aggregate/aggregate_function.h"
2522
#include "exprs/aggregate/aggregate_function_simple_factory.h"
26-
#include "exprs/aggregate/helpers.h"
2723

2824
namespace doris {
2925
#include "common/compile_check_begin.h"
30-
struct StringRef;
31-
} // namespace doris
3226

33-
namespace doris {
27+
AggregateFunctionPtr create_aggregate_function_orth_bitmap_intersect(
28+
const std::string& name, const DataTypes& argument_types, const DataTypePtr& result_type,
29+
const bool result_is_nullable, const AggregateFunctionAttr& attr);
3430

35-
template <template <PrimitiveType> class Impl>
36-
AggregateFunctionPtr create_aggregate_function_orthogonal(const std::string& name,
37-
const DataTypes& argument_types,
38-
const DataTypePtr& result_type,
39-
const bool result_is_nullable,
40-
const AggregateFunctionAttr& attr) {
41-
if (argument_types.empty()) {
42-
LOG(WARNING) << "Incorrect number of arguments for aggregate function " << name;
43-
return nullptr;
44-
} else if (argument_types.size() == 1) {
45-
return creator_without_type::create<AggFunctionOrthBitmapFunc<Impl<TYPE_STRING>>>(
46-
argument_types, result_is_nullable, attr);
47-
} else {
48-
AggregateFunctionPtr res(
49-
creator_with_type_list_base<1, TYPE_TINYINT, TYPE_SMALLINT, TYPE_INT, TYPE_BIGINT,
50-
TYPE_LARGEINT>::create<AggFunctionOrthBitmapFunc,
51-
Impl>(argument_types,
52-
result_is_nullable, attr));
53-
if (res) {
54-
return res;
55-
} else if (is_string_type(argument_types[1]->get_primitive_type())) {
56-
res = creator_without_type::create<AggFunctionOrthBitmapFunc<Impl<TYPE_STRING>>>(
57-
argument_types, result_is_nullable, attr);
58-
return res;
59-
}
31+
AggregateFunctionPtr create_aggregate_function_orth_bitmap_intersect_count(
32+
const std::string& name, const DataTypes& argument_types, const DataTypePtr& result_type,
33+
const bool result_is_nullable, const AggregateFunctionAttr& attr);
6034

61-
const IDataType& argument_type = *argument_types[1];
62-
LOG(WARNING) << "Incorrect Type " << argument_type.get_name()
63-
<< " of arguments for aggregate function " << name;
64-
return nullptr;
65-
}
66-
}
35+
AggregateFunctionPtr create_aggregate_function_orth_bitmap_union_count(
36+
const std::string& name, const DataTypes& argument_types, const DataTypePtr& result_type,
37+
const bool result_is_nullable, const AggregateFunctionAttr& attr);
38+
39+
AggregateFunctionPtr create_aggregate_function_orth_intersect_count(
40+
const std::string& name, const DataTypes& argument_types, const DataTypePtr& result_type,
41+
const bool result_is_nullable, const AggregateFunctionAttr& attr);
42+
43+
AggregateFunctionPtr create_aggregate_function_orth_bitmap_expr_cal(
44+
const std::string& name, const DataTypes& argument_types, const DataTypePtr& result_type,
45+
const bool result_is_nullable, const AggregateFunctionAttr& attr);
46+
47+
AggregateFunctionPtr create_aggregate_function_orth_bitmap_expr_cal_count(
48+
const std::string& name, const DataTypes& argument_types, const DataTypePtr& result_type,
49+
const bool result_is_nullable, const AggregateFunctionAttr& attr);
6750

6851
void register_aggregate_function_orthogonal_bitmap(AggregateFunctionSimpleFactory& factory) {
6952
factory.register_function_both("orthogonal_bitmap_intersect",
70-
create_aggregate_function_orthogonal<AggOrthBitMapIntersect>);
71-
factory.register_function_both(
72-
"orthogonal_bitmap_intersect_count",
73-
create_aggregate_function_orthogonal<AggOrthBitMapIntersectCount>);
53+
create_aggregate_function_orth_bitmap_intersect);
54+
factory.register_function_both("orthogonal_bitmap_intersect_count",
55+
create_aggregate_function_orth_bitmap_intersect_count);
7456
factory.register_function_both("orthogonal_bitmap_union_count",
75-
create_aggregate_function_orthogonal<OrthBitmapUnionCountData>);
57+
create_aggregate_function_orth_bitmap_union_count);
7658
factory.register_function_both("intersect_count",
77-
create_aggregate_function_orthogonal<AggIntersectCount>);
59+
create_aggregate_function_orth_intersect_count);
7860
factory.register_function_both("orthogonal_bitmap_expr_calculate",
79-
create_aggregate_function_orthogonal<AggOrthBitMapExprCal>);
61+
create_aggregate_function_orth_bitmap_expr_cal);
8062
factory.register_function_both("orthogonal_bitmap_expr_calculate_count",
81-
create_aggregate_function_orthogonal<AggOrthBitMapExprCalCount>);
63+
create_aggregate_function_orth_bitmap_expr_cal_count);
8264
}
8365

8466
} // namespace doris

‎be/src/exprs/aggregate/aggregate_function_orthogonal_bitmap.h‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -336,7 +336,6 @@ struct AggOrthBitMapExprCalCount : public AggOrthBitmapExprCalBaseData<T> {
336336
int64_t result = 0;
337337
};
338338

339-
template <PrimitiveType T>
340339
struct OrthBitmapUnionCountData {
341340
static constexpr auto name = "orthogonal_bitmap_union_count";
342341

@@ -373,16 +372,17 @@ struct OrthBitmapUnionCountData {
373372
int64_t result = 0;
374373
};
375374

376-
template <typename Impl>
375+
template <typename Impl, typename ExprTag = VarargsExpression>
377376
class AggFunctionOrthBitmapFunc final
378-
: public IAggregateFunctionDataHelper<Impl, AggFunctionOrthBitmapFunc<Impl>>,
379-
VarargsExpression,
377+
: public IAggregateFunctionDataHelper<Impl, AggFunctionOrthBitmapFunc<Impl, ExprTag>>,
378+
ExprTag,
380379
NullableAggregateFunction {
381380
public:
382381
String get_name() const override { return Impl::name; }
383382

384383
AggFunctionOrthBitmapFunc(const DataTypes& argument_types_)
385-
: IAggregateFunctionDataHelper<Impl, AggFunctionOrthBitmapFunc<Impl>>(argument_types_),
384+
: IAggregateFunctionDataHelper<Impl, AggFunctionOrthBitmapFunc<Impl, ExprTag>>(
385+
argument_types_),
386386
// The number of arguments will not exceed the size of an int
387387
_argument_size(int(argument_types_.size())) {}
388388

0 commit comments

Comments
 (0)