Skip to content

Commit aea642b

Browse files
committed
fix: aggregate collection-mode widget pipes correctly (IN-1195)
Addresses Copilot/Cursor Bugbot findings on PR #4348: - activities_filtered / activities_filtered_retention: collection requests now read from activityRelations_deduplicated_cleaned_bucket_union instead of the single-bucket activityRelations_bucket_routing pipe. The routing pipe defaults to bucket 0 when bucketId isn't supplied (only injected for single-project requests), so collection-scoped segments in other hash buckets were silently dropped. Project-scope path is unchanged. - package_metrics: added a per-project aggregation step before the existing collapse, so collection mode sums each project's own max/latest metrics instead of taking a single max() across all projects' raw rows (which returned only the largest project's values). - search_volume: collection mode now groups by dataTimestamp and sums volume into one series; project mode is unchanged. Previously collection requests returned duplicate dates with per-project volumes. - member_roles: added a collectionSlug branch scoped via segments_filtered_by_collection. Previously only `project` was checked, so collection-scoped contributors_leaderboard requests joined roles from every project in the system, not just the collection. Signed-off-by: Gašper Grom <gasper.grom@gmail.com>
1 parent c1cc168 commit aea642b

5 files changed

Lines changed: 64 additions & 12 deletions

File tree

services/libs/tinybird/pipes/activities_filtered.pipe

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,10 @@ NODE activities_filtered_bucket_routing
2424
SQL >
2525
%
2626
SELECT activityId as id, timestamp, type, platform, memberId, organizationId, segmentId
27-
FROM activityRelations_bucket_routing a
27+
FROM
28+
{% if defined(collectionSlug) %} activityRelations_deduplicated_cleaned_bucket_union
29+
{% else %} activityRelations_bucket_routing
30+
{% end %} as a
2831
where
2932
{% if defined(collectionSlug) %}
3033
segmentId IN (SELECT segmentId FROM segments_filtered_by_collection)

services/libs/tinybird/pipes/activities_filtered_retention.pipe

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,10 @@ NODE activities_filtered_retention_bucket_routing
2323
SQL >
2424
%
2525
SELECT activityId as id, timestamp, type, platform, memberId, organizationId, segmentId
26-
FROM activityRelations_bucket_routing a
26+
FROM
27+
{% if defined(collectionSlug) %} activityRelations_deduplicated_cleaned_bucket_union
28+
{% else %} activityRelations_bucket_routing
29+
{% end %} as a
2730
where
2831
{% if defined(collectionSlug) %}
2932
segmentId IN (SELECT segmentId FROM segments_filtered_by_collection)

services/libs/tinybird/pipes/member_roles.pipe

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,9 @@ SQL >
55
FROM maintainers_roles_copy_ds
66
WHERE
77
1 = 1
8-
{% if defined(project) %}
8+
{% if defined(collectionSlug) %}
9+
AND insightsProjectId IN (SELECT insightsProjectId FROM segments_filtered_by_collection)
10+
{% elif defined(project) %}
911
AND insightsProjectId = (SELECT insightsProjectId FROM segments_filtered)
1012
{% end %}
1113
{% if defined(repos) %}

services/libs/tinybird/pipes/package_metrics.pipe

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -144,11 +144,12 @@ SQL >
144144
) package_downloads_timeseries_bounds
145145
WHERE "startDate" >= actual_start_date AND "startDate" < actual_end_date
146146

147-
NODE package_downloads_initial_aggregation
147+
NODE package_downloads_per_project_aggregation
148148
SQL >
149149
%
150150
{% if defined(granularity) %}
151151
SELECT
152+
p.insightsProjectId,
152153
p.date,
153154
sum(p.downloadsCount) as downloadsCount,
154155
sum(p.dockerDownloadsCount) as "dockerDownloadsCount",
@@ -159,6 +160,7 @@ SQL >
159160
group by p.insightsProjectId, p.date
160161
{% else %}
161162
SELECT
163+
p.insightsProjectId,
162164
max(p.downloadsCount) as downloadsCount,
163165
max(p.dockerDownloadsCount) as "dockerDownloadsCount",
164166
max(p.dockerDependentsCount) as "dockerDependentsCount",
@@ -168,6 +170,29 @@ SQL >
168170
group by p.insightsProjectId
169171
{% end %}
170172

173+
NODE package_downloads_initial_aggregation
174+
SQL >
175+
%
176+
{% if defined(granularity) %}
177+
SELECT
178+
date,
179+
sum(downloadsCount) as downloadsCount,
180+
sum(dockerDownloadsCount) as "dockerDownloadsCount",
181+
sum(dockerDependentsCount) as "dockerDependentsCount",
182+
sum(dependentPackagesCount) as "dependentPackagesCount",
183+
sum(dependentReposCount) as "dependentReposCount"
184+
FROM package_downloads_per_project_aggregation
185+
group by date
186+
{% else %}
187+
SELECT
188+
sum(downloadsCount) as downloadsCount,
189+
sum(dockerDownloadsCount) as "dockerDownloadsCount",
190+
sum(dockerDependentsCount) as "dockerDependentsCount",
191+
sum(dependentPackagesCount) as "dependentPackagesCount",
192+
sum(dependentReposCount) as "dependentReposCount"
193+
FROM package_downloads_per_project_aggregation
194+
{% end %}
195+
171196
NODE package_downloads_timeseries_merge
172197
SQL >
173198
%

services/libs/tinybird/pipes/search_volume.pipe

Lines changed: 27 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -9,17 +9,14 @@ DESCRIPTION >
99
- `collectionSlug`: Collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`. Required unless `project` is given.
1010
- `startDate`: Optional DateTime filter for search volume data after timestamp (e.g., '2024-01-01 00:00:00')
1111
- `endDate`: Optional DateTime filter for search volume data before timestamp (e.g., '2024-12-31 23:59:59')
12-
- Response: `insightsProjectId`, `project`, `dataTimestamp` (formatted as YYYY-MM-DD), `volume`, `updatedAt`
12+
- Response: `insightsProjectId`, `project`, `dataTimestamp` (formatted as YYYY-MM-DD), `volume`, `updatedAt`.
13+
When `collectionSlug` is given, rows are aggregated to one row per `dataTimestamp` with `volume` summed
14+
across every project in the collection; `insightsProjectId`/`project`/`updatedAt` are empty in that mode.
1315

14-
NODE searchVolume_pipe
16+
NODE searchVolume_filtered
1517
SQL >
1618
%
17-
SELECT
18-
insightsProjectId,
19-
project,
20-
formatDateTime(dataTimestamp, '%Y-%m-%d') AS dataTimestamp,
21-
volume,
22-
updatedAt
19+
SELECT insightsProjectId, project, dataTimestamp, volume, updatedAt
2320
FROM searchVolume FINAL
2421
WHERE
2522
{% if defined(collectionSlug) %}
@@ -48,3 +45,25 @@ SQL >
4845
}}
4946
)
5047
{% end %}
48+
49+
NODE searchVolume_pipe
50+
SQL >
51+
%
52+
{% if defined(collectionSlug) %}
53+
SELECT
54+
toUUID('00000000-0000-0000-0000-000000000000') AS insightsProjectId,
55+
'' AS project,
56+
formatDateTime(dataTimestamp, '%Y-%m-%d') AS dataTimestamp,
57+
sum(volume) AS volume,
58+
max(updatedAt) AS updatedAt
59+
FROM searchVolume_filtered
60+
GROUP BY formatDateTime(dataTimestamp, '%Y-%m-%d')
61+
{% else %}
62+
SELECT
63+
insightsProjectId,
64+
project,
65+
formatDateTime(dataTimestamp, '%Y-%m-%d') AS dataTimestamp,
66+
volume,
67+
updatedAt
68+
FROM searchVolume_filtered
69+
{% end %}

0 commit comments

Comments
 (0)