Skip to content

Commit 84531db

Browse files
gaspergromgaspergrom
committed
feat: add copy pipes and datasources for org page timeseries
Signed-off-by: Gašper Grom <gasper.grom@gmail.com>
1 parent 85aed81 commit 84531db

9 files changed

Lines changed: 86 additions & 23 deletions

‎services/libs/tinybird/datasources/org_page_activities_timeseries_copy_ds.datasource‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
DESCRIPTION >
2+
Precomputed yearly activity counts per organization for the org page. Rebuilt nightly by org_page_activities_timeseries_copy_pipe.
3+
One row per (organizationId, startDate). Used by org_page_activities_timeseries.pipe for cheap request-time lookups.
4+
5+
SCHEMA >
6+
`organizationId` String,
7+
`startDate` Date,
8+
`endDate` Date,
9+
`activityCount` UInt64,
10+
`computedAt` DateTime
11+
12+
ENGINE ReplacingMergeTree
13+
ENGINE_SORTING_KEY organizationId, startDate
14+
ENGINE_VER computedAt

‎services/libs/tinybird/datasources/org_page_contributors_timeseries_copy_ds.datasource‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
DESCRIPTION >
2+
Precomputed yearly unique contributor counts per organization for the org page. Rebuilt nightly by org_page_contributors_timeseries_copy_pipe.
3+
One row per (organizationId, startDate). Used by org_page_contributors_timeseries.pipe for cheap request-time lookups.
4+
5+
SCHEMA >
6+
`organizationId` String,
7+
`startDate` Date,
8+
`endDate` Date,
9+
`contributorCount` UInt64,
10+
`computedAt` DateTime
11+
12+
ENGINE ReplacingMergeTree
13+
ENGINE_SORTING_KEY organizationId, startDate
14+
ENGINE_VER computedAt

‎services/libs/tinybird/pipes/org_page_activities_timeseries.pipe‎

Lines changed: 3 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,7 @@ TAGS "Organization page"
66
NODE org_page_activities_timeseries_data
77
SQL >
88
%
9-
SELECT
10-
toStartOfYear(timestamp) AS startDate,
11-
toDate(toStartOfYear(timestamp) + INTERVAL 1 YEAR - INTERVAL 1 DAY) AS endDate,
12-
count() AS activityCount
13-
FROM activityRelations_deduplicated_cleaned_bucket_union
14-
WHERE
15-
organizationId = {{ String(orgId, '', description="Organization ID", required=True) }}
16-
AND timestamp >= '2005-01-01'
17-
GROUP BY startDate, endDate
9+
SELECT startDate, endDate, activityCount
10+
FROM org_page_activities_timeseries_copy_ds FINAL
11+
WHERE organizationId = {{ String(orgId, '', description="Organization ID", required=True) }}
1812
ORDER BY startDate

‎services/libs/tinybird/pipes/org_page_activities_timeseries_copy_pipe.pipe‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
DESCRIPTION >
2+
Nightly copy pipe that precomputes yearly activity counts per organization for the org page.
3+
Writes one row per (organizationId, startDate) into org_page_activities_timeseries_copy_ds.
4+
5+
TAGS "Organization page"
6+
7+
NODE org_page_activities_timeseries_copy_pipe_data
8+
SQL >
9+
SELECT
10+
organizationId,
11+
toStartOfYear(timestamp) AS startDate,
12+
toDate(toStartOfYear(timestamp) + INTERVAL 1 YEAR - INTERVAL 1 DAY) AS endDate,
13+
count() AS activityCount,
14+
now() AS computedAt
15+
FROM activityRelations_deduplicated_cleaned_bucket_union
16+
WHERE organizationId != '' AND timestamp >= '2005-01-01'
17+
GROUP BY organizationId, startDate, endDate
18+
19+
TYPE COPY
20+
TARGET_DATASOURCE org_page_activities_timeseries_copy_ds
21+
COPY_MODE replace
22+
COPY_SCHEDULE 30 1 * * *

‎services/libs/tinybird/pipes/org_page_contributors.pipe‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ SQL >
4747
SELECT count(distinct memberId) as count
4848
FROM activityRelations_deduplicated_cleaned_bucket_union
4949
WHERE
50-
organizationId = {{ String(orgId, '') }}
50+
organizationId = {{ String(orgId, '', description="Organization ID", required=True) }}
5151
{% if defined(startDate) %} AND timestamp >= {{ DateTime(startDate) }} {% end %}
5252
{% if defined(endDate) %} AND timestamp < {{ DateTime(endDate) }} {% end %}
5353
{% else %}

‎services/libs/tinybird/pipes/org_page_contributors_timeseries.pipe‎

Lines changed: 3 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,7 @@ TAGS "Organization page"
66
NODE org_page_contributors_timeseries_data
77
SQL >
88
%
9-
SELECT
10-
toStartOfYear(timestamp) AS startDate,
11-
toDate(toStartOfYear(timestamp) + INTERVAL 1 YEAR - INTERVAL 1 DAY) AS endDate,
12-
uniqExact(memberId) AS contributorCount
13-
FROM activityRelations_deduplicated_cleaned_bucket_union
14-
WHERE
15-
organizationId = {{ String(orgId, '', description="Organization ID", required=True) }}
16-
AND timestamp >= '2005-01-01'
17-
GROUP BY startDate, endDate
9+
SELECT startDate, endDate, contributorCount
10+
FROM org_page_contributors_timeseries_copy_ds FINAL
11+
WHERE organizationId = {{ String(orgId, '', description="Organization ID", required=True) }}
1812
ORDER BY startDate

‎services/libs/tinybird/pipes/org_page_contributors_timeseries_copy_pipe.pipe‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
DESCRIPTION >
2+
Nightly copy pipe that precomputes yearly unique contributor counts per organization for the org page.
3+
Writes one row per (organizationId, startDate) into org_page_contributors_timeseries_copy_ds.
4+
5+
TAGS "Organization page"
6+
7+
NODE org_page_contributors_timeseries_copy_pipe_data
8+
SQL >
9+
SELECT
10+
organizationId,
11+
toStartOfYear(timestamp) AS startDate,
12+
toDate(toStartOfYear(timestamp) + INTERVAL 1 YEAR - INTERVAL 1 DAY) AS endDate,
13+
uniq(memberId) AS contributorCount,
14+
now() AS computedAt
15+
FROM activityRelations_deduplicated_cleaned_bucket_union
16+
WHERE organizationId != '' AND timestamp >= '2005-01-01'
17+
GROUP BY organizationId, startDate, endDate
18+
19+
TYPE COPY
20+
TARGET_DATASOURCE org_page_contributors_timeseries_copy_ds
21+
COPY_MODE replace
22+
COPY_SCHEDULE 45 1 * * *

‎services/libs/tinybird/pipes/org_page_kpis_copy_pipe.pipe‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -70,8 +70,12 @@ SQL >
7070
now() AS computedAt
7171
FROM org_page_kpis_current_contributors c
7272
FULL OUTER JOIN org_page_kpis_previous_contributors p ON c.organizationId = p.organizationId
73-
FULL OUTER JOIN org_page_kpis_maintainer_roles m ON c.organizationId = m.organizationId
74-
FULL OUTER JOIN org_page_kpis_critical_projects cp ON c.organizationId = cp.organizationId
73+
FULL OUTER JOIN
74+
org_page_kpis_maintainer_roles m
75+
ON coalesce(c.organizationId, p.organizationId) = m.organizationId
76+
FULL OUTER JOIN
77+
org_page_kpis_critical_projects cp
78+
ON coalesce(c.organizationId, p.organizationId, m.organizationId) = cp.organizationId
7579
WHERE organizationId != ''
7680

7781
TYPE COPY

‎services/libs/tinybird/pipes/org_page_profile.pipe‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,7 @@ SQL >
1818
FROM organizationIdentities FINAL
1919
WHERE
2020
organizationId = {{ String(orgId, '', description="Organization ID", required=True) }}
21-
AND platform = 'website'
22-
AND type = 'primary'
21+
AND type = 'primary-domain'
2322
GROUP BY organizationId
2423

2524
NODE org_page_profile_final

0 commit comments

Comments
 (0)