Skip to content

Commit 7782241

Browse files
gaspergromgaspergrom
committed
feat: add collections v2 aggregate Tinybird pipes and toggle column
Adds the showAggregateTabs toggle column on collections, and new Tinybird pipes aggregating project_insights and activity-relations data across an arbitrary list of projects for the LFX Insights Collections v2 UI (IN-1193, IN-1195). Signed-off-by: Gašper Grom <gasper.grom@gmail.com>
1 parent a4d2422 commit 7782241

5 files changed

Lines changed: 172 additions & 0 deletions

‎backend/src/database/migrations/V1782500000__addShowAggregateTabsColumnToCollectionsTable.sql‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
-- Add the showAggregateTabs column to the collections table
2+
ALTER TABLE collections
3+
ADD COLUMN "showAggregateTabs" boolean DEFAULT true NOT NULL;

‎services/libs/tinybird/pipes/collection_contributors_leaderboard_aggregate.pipe‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
DESCRIPTION >
2+
- `collection_contributors_leaderboard_aggregate.pipe` serves the contributors leaderboard widget for a collection (an arbitrary list of projects).
3+
- Returns the top 20 contributors ranked by total contribution count, aggregated/deduplicated across all projects in the list (a contributor active on multiple projects is summed once, not listed per project).
4+
- Parameters:
5+
- `ids`: Required array of project ids (as returned by `project_insights.pipe`'s `id`/`ids`) to aggregate over.
6+
- Response:
7+
- `id`: member id.
8+
- `avatar`: member avatar URL.
9+
- `displayName`: member display name.
10+
- `githubHandleArray`: member's GitHub usernames.
11+
- `contributionCount`: total contribution count for the member, summed across all matched projects.
12+
13+
TAGS "Insights, Widget", "Collection"
14+
15+
NODE collection_contributors_leaderboard_aggregate_projects
16+
DESCRIPTION >
17+
Resolves the input project ids to their segment ids, since activity data is keyed by segmentId, not project id
18+
19+
SQL >
20+
%
21+
SELECT id, segmentId
22+
FROM project_insights_copy_ds
23+
WHERE
24+
type = 'project'
25+
AND id IN {{ Array(ids, 'String', description="Filter by project id list", required=True) }}
26+
27+
NODE collection_contributors_leaderboard_aggregate_members
28+
DESCRIPTION >
29+
Sums contribution counts per member across the matched projects' segments
30+
31+
SQL >
32+
SELECT ar.memberId AS memberId, count(ar.activityId) AS contributionCount
33+
FROM activityRelations_deduplicated_cleaned_bucket_union AS ar
34+
WHERE
35+
ar.memberId != ''
36+
AND ar.segmentId IN (SELECT segmentId FROM collection_contributors_leaderboard_aggregate_projects)
37+
GROUP BY ar.memberId
38+
ORDER BY contributionCount DESC, memberId DESC
39+
LIMIT 20
40+
41+
NODE collection_contributors_leaderboard_aggregate_results
42+
DESCRIPTION >
43+
Joins the top members with their profile data
44+
45+
SQL >
46+
SELECT
47+
m.id AS id,
48+
m.avatar AS avatar,
49+
m.displayName AS displayName,
50+
m.githubHandleArray AS githubHandleArray,
51+
ma.contributionCount AS contributionCount
52+
FROM collection_contributors_leaderboard_aggregate_members AS ma
53+
INNER JOIN members_sorted AS m ON m.id = ma.memberId
54+
ORDER BY contributionCount DESC, id DESC

‎services/libs/tinybird/pipes/collection_development_aggregate.pipe‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
DESCRIPTION >
2+
- `collection_development_aggregate.pipe` serves aggregate development activity metrics for a collection (an arbitrary list of projects).
3+
- Returns a single row with active contributors and active organizations aggregated across all projects in the list, deduplicated (a contributor or organization active on multiple projects in the list is counted once, not once per project), restricted to the last 365 days to mirror the per-project `activeContributorsLast365Days` / `activeOrganizationsLast365Days` semantics.
4+
- Parameters:
5+
- `ids`: Required array of project ids (as returned by `project_insights.pipe`'s `id`/`ids`) to aggregate over.
6+
- Response:
7+
- `activeContributorsLast365Days`: total unique contributors across all matched projects' segments in the last 365 days.
8+
- `activeOrganizationsLast365Days`: total unique organizations across all matched projects' segments in the last 365 days.
9+
10+
TAGS "Insights, Widget", "Collection"
11+
12+
NODE collection_development_aggregate_projects
13+
DESCRIPTION >
14+
Resolves the input project ids to their segment ids, since activity data is keyed by segmentId, not project id
15+
16+
SQL >
17+
%
18+
SELECT id, segmentId
19+
FROM project_insights_copy_ds
20+
WHERE
21+
type = 'project'
22+
AND id IN {{ Array(ids, 'String', description="Filter by project id list", required=True) }}
23+
24+
NODE collection_development_aggregate_results
25+
DESCRIPTION >
26+
Counts unique contributors and unique organizations across matched projects' segments in the last 365 days
27+
28+
SQL >
29+
SELECT
30+
uniq(case when ar.memberId != '' then ar.memberId else null end) AS activeContributorsLast365Days,
31+
uniq(case when ar.organizationId != '' then ar.organizationId else null end) AS activeOrganizationsLast365Days
32+
FROM activityRelations_deduplicated_cleaned_bucket_union AS ar
33+
WHERE
34+
ar.segmentId IN (SELECT segmentId FROM collection_development_aggregate_projects)
35+
AND ar.timestamp >= now() - INTERVAL 365 DAY
36+
AND ar.timestamp < now()

‎services/libs/tinybird/pipes/collection_insights_aggregate.pipe‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
DESCRIPTION >
2+
- `collection_insights_aggregate.pipe` serves aggregate insights metrics for a collection (an arbitrary list of projects).
3+
- Returns a single row with metrics aggregated across all projects in the list, not one row per project.
4+
- Parameters:
5+
- `ids`: Required array of project ids (as returned by `project_insights.pipe`'s `id`/`ids`) to aggregate over.
6+
- Response:
7+
- `projectCount`: count of distinct project ids from the input list that matched an existing project.
8+
- `uniqueContributorCount`: total unique contributors across all matched projects, deduplicated (a contributor active on multiple projects in the list is counted once).
9+
- `avgHealthScore`: average of each matched project's health score, rounded.
10+
11+
TAGS "Insights, Widget", "Collection"
12+
13+
NODE collection_insights_aggregate_projects
14+
DESCRIPTION >
15+
Resolves the input project ids to their segment ids, since activity data is keyed by segmentId, not project id
16+
17+
SQL >
18+
%
19+
SELECT id, segmentId, healthScore
20+
FROM project_insights_copy_ds
21+
WHERE
22+
type = 'project'
23+
AND id IN {{ Array(ids, 'String', description="Filter by project id list", required=True) }}
24+
25+
NODE collection_insights_aggregate_results
26+
DESCRIPTION >
27+
Aggregates health score across matched projects and counts unique contributors across their segments in one pass
28+
29+
SQL >
30+
SELECT
31+
(SELECT count(distinct id) FROM collection_insights_aggregate_projects) AS projectCount,
32+
uniq(ar.memberId) AS uniqueContributorCount,
33+
(SELECT round(avg(healthScore)) FROM collection_insights_aggregate_projects) AS avgHealthScore
34+
FROM activityRelations_deduplicated_cleaned_bucket_union AS ar
35+
WHERE
36+
ar.memberId != ''
37+
AND ar.segmentId IN (SELECT segmentId FROM collection_insights_aggregate_projects)

‎services/libs/tinybird/pipes/collection_popularity_aggregate.pipe‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
DESCRIPTION >
2+
- `collection_popularity_aggregate.pipe` serves aggregate popularity metrics for a collection (an arbitrary list of projects).
3+
- Returns a single row with stars and forks summed across all projects in the list, for both the current and previous 365-day windows, so the frontend can compute a trend delta.
4+
- Stars and forks are not deduplication-sensitive (a star on one repo is independent of a star on another), so a plain SUM across matched projects is correct.
5+
- Parameters:
6+
- `ids`: Required array of project ids (as returned by `project_insights.pipe`'s `id`/`ids`) to aggregate over.
7+
- Response:
8+
- `totalStars`: sum of `starsLast365Days` across all matched projects.
9+
- `totalForks`: sum of `forksLast365Days` across all matched projects.
10+
- `starsPrevious365Days`: sum of `starsPrevious365Days` across all matched projects.
11+
- `forksPrevious365Days`: sum of `forksPrevious365Days` across all matched projects.
12+
13+
TAGS "Insights, Widget", "Collection"
14+
15+
NODE collection_popularity_aggregate_projects
16+
DESCRIPTION >
17+
Resolves the input project ids to their per-project popularity fields
18+
19+
SQL >
20+
%
21+
SELECT
22+
id,
23+
starsLast365Days,
24+
forksLast365Days,
25+
starsPrevious365Days,
26+
forksPrevious365Days
27+
FROM project_insights_copy_ds
28+
WHERE
29+
type = 'project'
30+
AND id IN {{ Array(ids, 'String', description="Filter by project id list", required=True) }}
31+
32+
NODE collection_popularity_aggregate_results
33+
DESCRIPTION >
34+
Sums stars and forks across matched projects for the current and previous 365-day windows
35+
36+
SQL >
37+
SELECT
38+
sum(starsLast365Days) AS totalStars,
39+
sum(forksLast365Days) AS totalForks,
40+
sum(starsPrevious365Days) AS starsPrevious365Days,
41+
sum(forksPrevious365Days) AS forksPrevious365Days
42+
FROM collection_popularity_aggregate_projects

0 commit comments

Comments
 (0)