Skip to content

Commit 938f542

Browse files
committed
feat: add dedicated collection_contributors_leaderboard pipe (IN-1195)
contributors_leaderboard.pipe's collectionSlug support (added earlier this branch) made contribution percentage a SUM(...) OVER () window function, which forces ClickHouse to fully materialize the per-member GROUP BY before ORDER BY/LIMIT can apply - measured 7-16s at a 242-project collection's scale, timing out entirely once at Tinybird's 20s limit. Every other collection-scoped widget pipe tested fine (2-7s), so per the narrower fix: only this one gets a separate, purpose-built pipe rather than reworking the shared one. collection_contributors_leaderboard.pipe computes the total contribution count as a separate scalar (same cheap shape as the existing count-mode query) instead of a window function, then divides against it directly - avoiding the full-materialization cost. Full parameter parity with the single-project pipe (date range, platform, activity type, includeCollaborations, pagination). contributors_leaderboard.pipe is reverted back to its original, project-only form - every other collection-scoped pipe from earlier in this branch (segments_filtered_by_collection, activities_filtered, active_contributors, organizations_leaderboard, etc.) is unchanged and continues to serve both project and collection scope from the same pipe, since they perform acceptably as-is. Signed-off-by: Gašper Grom <gasper.grom@gmail.com>
1 parent aea642b commit 938f542

2 files changed

Lines changed: 92 additions & 11 deletions

File tree

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
DESCRIPTION >
2+
- `collection_contributors_leaderboard.pipe` serves the contributors leaderboard widget for a
3+
collection's Contributors tab, aggregated across every project in the collection.
4+
- Separate from `contributors_leaderboard.pipe` (the single-project version) because a naive
5+
collection-scoped port of that pipe's `ROUND(COUNT(af.id) * 100.0 / SUM(COUNT(af.id)) OVER (), 2)`
6+
contributionPercentage forces ClickHouse to fully materialize the per-member GROUP BY (potentially
7+
hundreds of thousands of distinct members at collection scale) before it can apply ORDER BY/LIMIT -
8+
confirmed via direct timing: the equivalent single-project pipe took 7-16s (once timing out
9+
entirely at Tinybird's 20s limit) on a 242-project collection, vs 2-3s for the count-only path
10+
with no window function. This pipe computes the total contribution count as a separate scalar
11+
subquery instead of a window function, avoiding that full-materialization cost.
12+
- Supports both count mode (`count=true`) and data mode for pagination-friendly leaderboard display.
13+
- Always shows `displayName` (not `publicName`) since a collection spans many projects with mixed
14+
LF/non-LF status - there's no single project-type answer to substitute on, unlike the single-project
15+
pipe's `is_request_from_non_lf_project` check.
16+
- Primary use case: powering the collection-scoped contributor ranking widget on Collection detail
17+
pages' Contributors tab.
18+
- Parameters:
19+
- `collectionSlug`: Required collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`.
20+
- `repos`: Optional array of repository URLs for filtering (inherited from `segments_filtered_by_collection`)
21+
- `startDate`: Optional DateTime filter for activities after timestamp (e.g., '2024-01-01 00:00:00')
22+
- `endDate`: Optional DateTime filter for activities before timestamp (e.g., '2024-12-31 23:59:59')
23+
- `platform`: Optional string filter for source platform (e.g., 'github', 'discord', 'slack')
24+
- `activity_type`: Optional string filter for single activity type (e.g., 'authored-commit')
25+
- `activity_types`: Optional array of activity types (e.g., ['authored-commit', 'co-authored-commit'])
26+
- `includeCollaborations`: Optional boolean, defaults to false - same toggle as the single-project pipe.
27+
- `count`: Optional boolean, when true returns contributor count instead of leaderboard data
28+
- `limit`: Optional integer for result pagination, defaults to 10
29+
- `offset`: Optional integer for result pagination, defaults to 0
30+
- Response:
31+
- Count mode (`count=true`): `count` (total number of contributors)
32+
- Data mode (default): `id`, `avatar`, `displayName`, `githubHandleArray`, `contributionCount`, `contributionPercentage`, `roles`
33+
34+
TAGS "Insights, Widget", "Collection", "Contributors"
35+
36+
NODE collection_contributors_total
37+
DESCRIPTION >
38+
Total contribution count across the collection - a plain aggregate with no GROUP BY, so it's
39+
cheap regardless of how many distinct members contributed. Computed once and reused as a scalar
40+
for the percentage calculation instead of a SUM(...) OVER () window function.
41+
42+
SQL >
43+
%
44+
SELECT count(af.id) AS total
45+
FROM activities_filtered af
46+
47+
NODE collection_member_aggregates
48+
DESCRIPTION >
49+
Per-member contribution counts, ranked and paginated. No window function here - the percentage
50+
is computed in the next node against the scalar total, so ORDER BY/LIMIT can apply directly
51+
after this GROUP BY without waiting on a second full-set pass.
52+
53+
SQL >
54+
%
55+
{% if Boolean(count, false) %} SELECT count(distinct af.memberId) FROM activities_filtered af
56+
{% else %}
57+
SELECT af.memberId, count(af.id) AS contributionCount
58+
FROM activities_filtered af
59+
GROUP BY af.memberId
60+
ORDER BY contributionCount DESC, af.memberId DESC
61+
LIMIT {{ Int32(limit, 10) }}
62+
OFFSET {{ Int32(offset, 0) }}
63+
{% end %}
64+
65+
NODE collection_contributors_leaderboard_result
66+
SQL >
67+
%
68+
{% if Boolean(count, false) %}
69+
SELECT count(distinct af.memberId) AS count FROM activities_filtered af
70+
{% else %}
71+
SELECT
72+
m.id,
73+
m.avatar,
74+
m.displayName,
75+
m.githubHandleArray,
76+
ma.contributionCount,
77+
ROUND(ma.contributionCount * 100.0 / total.total, 2) AS contributionPercentage,
78+
mr.roles
79+
FROM members_sorted AS m any
80+
INNER JOIN collection_member_aggregates ma ON ma.memberId = m.id
81+
LEFT JOIN member_roles mr ON mr.memberId = m.id
82+
CROSS JOIN collection_contributors_total total
83+
WHERE m.id IN (SELECT memberId FROM collection_member_aggregates)
84+
ORDER BY contributionCount DESC
85+
{% end %}

services/libs/tinybird/pipes/contributors_leaderboard.pipe

Lines changed: 7 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,7 @@ DESCRIPTION >
44
- Calculates contribution counts and percentages, showing different display names based on project type (LF vs non-LF).
55
- Primary use case: powering contributor ranking widgets in development insights.
66
- Parameters:
7-
- `project`: Required string for project slug (e.g., 'k8s', 'tensorflow') - inherited from `segments_filtered`. Required unless `collectionSlug` is given.
8-
- `collectionSlug`: Collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`. Required unless `project` is given. A collection spans many projects with mixed LF/non-LF status, so the non-LF `publicName` display-name substitution below only applies in single-project mode; collection-scoped requests always use `displayName`.
7+
- `project`: Required string for project slug (e.g., 'k8s', 'tensorflow') - inherited from `segments_filtered`
98
- `repos`: Optional array of repository URLs for filtering (inherited from `segments_filtered`)
109
- `startDate`: Optional DateTime filter for activities after timestamp (e.g., '2024-01-01 00:00:00')
1110
- `endDate`: Optional DateTime filter for activities before timestamp (e.g., '2024-12-31 23:59:59')
@@ -40,15 +39,12 @@ SQL >
4039
NODE is_request_from_non_lf_project
4140
SQL >
4241
%
43-
{% if defined(collectionSlug) %} SELECT false AS result
44-
{% else %}
45-
SELECT not isLF AS result
46-
FROM insightsProjects final
47-
WHERE
48-
isNull (deletedAt)
49-
AND enabled = 1
50-
and slug = {{ String(project, description="Filter by project slug", required=True) }}
51-
{% end %}
42+
SELECT not isLF AS result
43+
FROM insightsProjects final
44+
WHERE
45+
isNull (deletedAt)
46+
AND enabled = 1
47+
and slug = {{ String(project, description="Filter by project slug", required=True) }}
5248

5349
NODE contributors_leaderboard_2
5450
SQL >

0 commit comments

Comments
 (0)