Skip to content

Commit 0e22e13

Browse files
gaspergromjoanagmaia
gaspergrom
and
joanagmaia
authored
feat: add Tinybird pipes for organization page (#4128)
Signed-off-by: Gašper Grom <gasper.grom@gmail.com> Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org> Co-authored-by: Joana Maia <jmaia@contractor.linuxfoundation.org>
1 parent 7ad5b89 commit 0e22e13

15 files changed

Lines changed: 629 additions & 3 deletions

‎services/libs/tinybird/datasources/org_page_activities_timeseries_copy_ds.datasource‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
DESCRIPTION >
2+
Precomputed yearly activity counts per organization for the org page. Rebuilt nightly by org_page_activities_timeseries_copy_pipe.
3+
One row per (organizationId, startDate). Used by org_page_activities_timeseries.pipe for cheap request-time lookups.
4+
5+
SCHEMA >
6+
`organizationId` String,
7+
`startDate` Date,
8+
`endDate` Date,
9+
`activityCount` UInt64,
10+
`computedAt` DateTime
11+
12+
ENGINE ReplacingMergeTree
13+
ENGINE_SORTING_KEY organizationId, startDate
14+
ENGINE_VER computedAt

‎services/libs/tinybird/datasources/org_page_contributors_timeseries_copy_ds.datasource‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
DESCRIPTION >
2+
Precomputed yearly unique contributor counts per organization for the org page. Rebuilt nightly by org_page_contributors_timeseries_copy_pipe.
3+
One row per (organizationId, startDate). Used by org_page_contributors_timeseries.pipe for cheap request-time lookups.
4+
5+
SCHEMA >
6+
`organizationId` String,
7+
`startDate` Date,
8+
`endDate` Date,
9+
`contributorCount` UInt64,
10+
`computedAt` DateTime
11+
12+
ENGINE ReplacingMergeTree
13+
ENGINE_SORTING_KEY organizationId, startDate
14+
ENGINE_VER computedAt

‎services/libs/tinybird/datasources/org_page_kpis_copy_ds.datasource‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
DESCRIPTION >
2+
Precomputed organization-level KPIs for the org page. Rebuilt nightly by org_page_kpis_copy_pipe.
3+
One row per organizationId. Used by org_page_kpis.pipe for cheap request-time lookups.
4+
5+
SCHEMA >
6+
`organizationId` String,
7+
`activeContributors` UInt64,
8+
`activeContributorsPrevious` UInt64,
9+
`maintainerRoles` UInt64,
10+
`criticalProjects` UInt64,
11+
`computedAt` DateTime
12+
13+
ENGINE ReplacingMergeTree
14+
ENGINE_SORTING_KEY organizationId
15+
ENGINE_VER computedAt

‎services/libs/tinybird/datasources/org_page_projects_copy_ds.datasource‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
DESCRIPTION >
2+
Precomputed per-org per-project metrics for the org page. Rebuilt nightly by org_page_projects_copy_pipe.
3+
One row per (organizationId, segmentId). Used by org_page_projects.pipe.
4+
5+
SCHEMA >
6+
`organizationId` String,
7+
`segmentId` String,
8+
`projectSlug` String,
9+
`projectName` String,
10+
`projectLogo` String,
11+
`activityCount` UInt64,
12+
`contributorCount` UInt64,
13+
`maintainersCount` UInt64,
14+
`totalContributors` UInt64,
15+
`orgContributors` UInt64,
16+
`totalCommits` UInt64,
17+
`orgCommits` UInt64,
18+
`totalPrsOpened` UInt64,
19+
`orgPrsOpened` UInt64,
20+
`technicalScore` Float64,
21+
`computedAt` DateTime
22+
23+
ENGINE ReplacingMergeTree
24+
ENGINE_SORTING_KEY organizationId, segmentId
25+
ENGINE_VER computedAt

‎services/libs/tinybird/pipes/org_page_activities_timeseries.pipe‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
DESCRIPTION >
2+
Activity timeseries for a given organization, bucketed by year (all-time).
3+
4+
TAGS "Organization page"
5+
6+
NODE org_slug_lookup
7+
SQL >
8+
%
9+
SELECT id
10+
FROM organizations_populated_slug
11+
WHERE slug = {{ String(orgSlug, '', description="Organization slug", required=True) }}
12+
13+
NODE org_page_activities_timeseries_data
14+
SQL >
15+
SELECT startDate, endDate, activityCount
16+
FROM org_page_activities_timeseries_copy_ds FINAL
17+
WHERE organizationId = (SELECT id FROM org_slug_lookup)
18+
ORDER BY startDate

‎services/libs/tinybird/pipes/org_page_activities_timeseries_copy_pipe.pipe‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
DESCRIPTION >
2+
Nightly copy pipe that precomputes yearly activity counts per organization for the org page.
3+
Writes one row per (organizationId, startDate) into org_page_activities_timeseries_copy_ds.
4+
5+
TAGS "Organization page"
6+
7+
NODE org_page_activities_timeseries_copy_pipe_data
8+
SQL >
9+
SELECT
10+
organizationId,
11+
toStartOfYear(timestamp) AS startDate,
12+
toDate(toStartOfYear(timestamp) + INTERVAL 1 YEAR - INTERVAL 1 DAY) AS endDate,
13+
count() AS activityCount,
14+
now() AS computedAt
15+
FROM activityRelations_deduplicated_cleaned_bucket_union
16+
WHERE organizationId != '' AND timestamp >= '2016-01-01'
17+
GROUP BY organizationId, startDate, endDate
18+
19+
TYPE COPY
20+
TARGET_DATASOURCE org_page_activities_timeseries_copy_ds
21+
COPY_MODE replace
22+
COPY_SCHEDULE 30 1 * * *

‎services/libs/tinybird/pipes/org_page_contributors.pipe‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
DESCRIPTION >
2+
Top contributors for a given organization leaderboard.
3+
Returns members sorted by contribution count within the specified date range.
4+
5+
TAGS "Organization page"
6+
7+
NODE org_slug_lookup
8+
SQL >
9+
%
10+
SELECT id
11+
FROM organizations_populated_slug
12+
WHERE slug = {{ String(orgSlug, '', description="Organization slug", required=True) }}
13+
14+
NODE org_page_contributors_activity_aggregates
15+
SQL >
16+
%
17+
{% if Boolean(count, false) %}
18+
SELECT count(distinct memberId)
19+
FROM activityRelations_deduplicated_cleaned_bucket_union
20+
WHERE
21+
organizationId = (SELECT id FROM org_slug_lookup)
22+
{% if defined(startDate) %}
23+
AND timestamp
24+
>= {{ DateTime(startDate, description="Filter activity timestamp after") }}
25+
{% end %}
26+
{% if defined(endDate) %}
27+
AND timestamp < {{ DateTime(endDate, description="Filter activity timestamp before") }}
28+
{% end %}
29+
{% else %}
30+
SELECT
31+
memberId,
32+
count() as "contributionCount",
33+
ROUND(COUNT(*) * 100.0 / SUM(COUNT(*)) OVER (), 2) as "contributionPercentage"
34+
FROM activityRelations_deduplicated_cleaned_bucket_union
35+
WHERE
36+
organizationId = (SELECT id FROM org_slug_lookup)
37+
{% if defined(startDate) %}
38+
AND timestamp
39+
>= {{ DateTime(startDate, description="Filter activity timestamp after") }}
40+
{% end %}
41+
{% if defined(endDate) %}
42+
AND timestamp < {{ DateTime(endDate, description="Filter activity timestamp before") }}
43+
{% end %}
44+
GROUP BY memberId
45+
ORDER BY contributionCount DESC, memberId DESC
46+
LIMIT {{ Int32(limit, 10) }}
47+
OFFSET {{ Int32(offset, 0) }}
48+
{% end %}
49+
50+
NODE org_page_contributors_leaderboard
51+
SQL >
52+
%
53+
{% if Boolean(count, false) %}
54+
SELECT count(distinct memberId) as count
55+
FROM activityRelations_deduplicated_cleaned_bucket_union
56+
WHERE
57+
organizationId = (SELECT id FROM org_slug_lookup)
58+
{% if defined(startDate) %} AND timestamp >= {{ DateTime(startDate) }} {% end %}
59+
{% if defined(endDate) %} AND timestamp < {{ DateTime(endDate) }} {% end %}
60+
{% else %}
61+
SELECT
62+
m.id,
63+
m.avatar,
64+
m.displayName,
65+
m.githubHandleArray,
66+
agg.contributionCount,
67+
agg.contributionPercentage,
68+
mr.roles
69+
FROM members_sorted AS m ANY
70+
INNER JOIN org_page_contributors_activity_aggregates agg ON agg.memberId = m.id
71+
LEFT JOIN member_roles mr ON mr.memberId = m.id
72+
WHERE m.id IN (SELECT memberId FROM org_page_contributors_activity_aggregates)
73+
ORDER BY agg.contributionCount DESC
74+
{% end %}

‎services/libs/tinybird/pipes/org_page_contributors_timeseries.pipe‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
DESCRIPTION >
2+
Contributor count timeseries for a given organization, bucketed by year (all-time).
3+
4+
TAGS "Organization page"
5+
6+
NODE org_slug_lookup
7+
SQL >
8+
%
9+
SELECT id
10+
FROM organizations_populated_slug
11+
WHERE slug = {{ String(orgSlug, '', description="Organization slug", required=True) }}
12+
13+
NODE org_page_contributors_timeseries_data
14+
SQL >
15+
SELECT startDate, endDate, contributorCount
16+
FROM org_page_contributors_timeseries_copy_ds FINAL
17+
WHERE organizationId = (SELECT id FROM org_slug_lookup)
18+
ORDER BY startDate

‎services/libs/tinybird/pipes/org_page_contributors_timeseries_copy_pipe.pipe‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
DESCRIPTION >
2+
Nightly copy pipe that precomputes yearly unique contributor counts per organization for the org page.
3+
Writes one row per (organizationId, startDate) into org_page_contributors_timeseries_copy_ds.
4+
5+
TAGS "Organization page"
6+
7+
NODE org_page_contributors_timeseries_copy_pipe_data
8+
SQL >
9+
SELECT
10+
organizationId,
11+
toStartOfYear(timestamp) AS startDate,
12+
toDate(toStartOfYear(timestamp) + INTERVAL 1 YEAR - INTERVAL 1 DAY) AS endDate,
13+
uniq(memberId) AS contributorCount,
14+
now() AS computedAt
15+
FROM activityRelations_deduplicated_cleaned_bucket_union
16+
WHERE organizationId != '' AND timestamp >= '2016-01-01'
17+
GROUP BY organizationId, startDate, endDate
18+
19+
TYPE COPY
20+
TARGET_DATASOURCE org_page_contributors_timeseries_copy_ds
21+
COPY_MODE replace
22+
COPY_SCHEDULE 45 1 * * *

‎services/libs/tinybird/pipes/org_page_kpis.pipe‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
DESCRIPTION >
2+
Returns KPIs for a given organization from the precomputed org_page_kpis_copy_ds.
3+
Includes trend calculations comparing current to previous 365-day period.
4+
5+
TAGS "Organization page"
6+
7+
NODE org_slug_lookup
8+
SQL >
9+
%
10+
SELECT id
11+
FROM organizations_populated_slug
12+
WHERE slug = {{ String(orgSlug, '', description="Organization slug", required=True) }}
13+
14+
NODE org_page_kpis_main
15+
SQL >
16+
SELECT
17+
activeContributors,
18+
if(
19+
activeContributorsPrevious = 0,
20+
0,
21+
round(
22+
(toInt64(activeContributors) - toInt64(activeContributorsPrevious))
23+
/ activeContributorsPrevious
24+
* 100,
25+
1
26+
)
27+
) AS activeContributorsTrend,
28+
toInt64(activeContributors)
29+
- toInt64(activeContributorsPrevious) AS activeContributorsTrendAbsolute,
30+
activeContributorsPrevious AS activeContributorsTrendPrevious,
31+
maintainerRoles,
32+
criticalProjects
33+
FROM org_page_kpis_copy_ds FINAL
34+
WHERE organizationId = (SELECT id FROM org_slug_lookup)

0 commit comments

Comments
 (0)