diff --git a/services/libs/tinybird/pipes/active_contributors.pipe b/services/libs/tinybird/pipes/active_contributors.pipe index bee53a0e11..ac108dd1f0 100644 --- a/services/libs/tinybird/pipes/active_contributors.pipe +++ b/services/libs/tinybird/pipes/active_contributors.pipe @@ -4,7 +4,8 @@ DESCRIPTION > - **When `granularity` is NOT provided, returns total unique contributor count** for the specified filters. - **When `granularity` is provided, returns time-series data** showing unique contributor counts for each time period. - Parameters: - - `project`: Required string for project slug (e.g., 'k8s', 'tensorflow') - inherited from `segments_filtered` + - `project`: Required string for project slug (e.g., 'k8s', 'tensorflow') - inherited from `segments_filtered`. Required unless `collectionSlug` is given. + - `collectionSlug`: Collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`. Required unless `project` is given. - `repos`: Optional array of repository URLs for filtering (inherited from `segments_filtered`) - `startDate`: Optional DateTime filter for activities after timestamp (e.g., '2024-01-01 00:00:00') - `endDate`: Optional DateTime filter for activities before timestamp (e.g., '2024-12-31 23:59:59') @@ -55,7 +56,10 @@ SQL > SELECT uniqIf(memberId, memberId != '') AS maintainerCount FROM maintainers_roles_copy_ds WHERE - insightsProjectId = (SELECT insightsProjectId FROM segments_filtered) AND role = 'maintainer' + {% if defined(collectionSlug) %} + insightsProjectId IN (SELECT insightsProjectId FROM segments_filtered_by_collection) + {% else %} insightsProjectId = (SELECT insightsProjectId FROM segments_filtered) + {% end %} AND role = 'maintainer' {% if defined(repos) %} AND repoUrl IN {{ Array(repos, 'String', description="Filter maintainer repo list", required=False) }} diff --git a/services/libs/tinybird/pipes/activities_filtered.pipe b/services/libs/tinybird/pipes/activities_filtered.pipe index a4cbaa7b18..f7bec9eab2 100644 --- a/services/libs/tinybird/pipes/activities_filtered.pipe +++ b/services/libs/tinybird/pipes/activities_filtered.pipe @@ -2,9 +2,12 @@ DESCRIPTION > - `activities_filtered.pipe` is the core filtering infrastructure pipe for activity data across the entire analytics platform. - This pipe serves as the foundation for most activity-related widgets, by providing a consistent, filtered view of contribution activities. - It filters activities from `activityRelations_deduplicated_cleaned_ds` datasource based on project segment, time ranges, repositories, platforms, and activity types. - - The pipe automatically scopes data to the current project using `segments_filtered` pipe for security and data isolation. + - The pipe scopes data either to a single project (via `segments_filtered`) or to every project in a collection + (via `segments_filtered_by_collection`), for security and data isolation. Exactly one of `project`/`collectionSlug` + must be provided by the caller. - Parameters: - - `project`: Inherited from `segments_filtered`, project slug (e.g., 'k8s', 'tensorflow') + - `project`: Project slug (e.g., 'k8s', 'tensorflow') - inherited from `segments_filtered`. Required unless `collectionSlug` is given. + - `collectionSlug`: Collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`. Required unless `project` is given. - `repos`: Optional array of repository URLs for filtering (e.g., ['https://github.com/kubernetes/kubernetes']). Inherited from `segments_filtered`. - `startDate`: Optional DateTime filter for activities after timestamp (e.g., '2024-01-01 00:00:00') - `endDate`: Optional DateTime filter for activities before timestamp (e.g., '2024-12-31 23:59:59') @@ -21,9 +24,15 @@ NODE activities_filtered_bucket_routing SQL > % SELECT activityId as id, timestamp, type, platform, memberId, organizationId, segmentId - FROM activityRelations_bucket_routing a + FROM + {% if defined(collectionSlug) %} activityRelations_deduplicated_cleaned_bucket_union + {% else %} activityRelations_bucket_routing + {% end %} as a where - segmentId = (SELECT segmentId FROM segments_filtered) + {% if defined(collectionSlug) %} + segmentId IN (SELECT segmentId FROM segments_filtered_by_collection) + {% else %} segmentId = (SELECT segmentId FROM segments_filtered) + {% end %} {% if defined(startDate) %} AND a.timestamp > {{ DateTime(startDate, description="Filter activity timestamp after", required=False) }} diff --git a/services/libs/tinybird/pipes/activities_filtered_retention.pipe b/services/libs/tinybird/pipes/activities_filtered_retention.pipe index 820d90cd1e..c919df4003 100644 --- a/services/libs/tinybird/pipes/activities_filtered_retention.pipe +++ b/services/libs/tinybird/pipes/activities_filtered_retention.pipe @@ -5,7 +5,8 @@ DESCRIPTION > - This time extension ensures retention rate calculations have the necessary baseline data from the preceding period for comparing distinct member activity for the first given period. - Primary use case: providing data foundation for member retention analysis and cohort studies. - Parameters: - - `project`: Inherited from `segments_filtered`, project slug (e.g., 'k8s', 'tensorflow') + - `project`: Inherited from `segments_filtered`, project slug (e.g., 'k8s', 'tensorflow'). Required unless `collectionSlug` is given. + - `collectionSlug`: Inherited from `segments_filtered_by_collection`, collection slug (e.g., 'cncf'). Required unless `project` is given. - `repos`: Inherited from `segments_filtered`, array of repository URLs for filtering - `startDate`: Optional DateTime filter, automatically extended backwards by one granularity period - `endDate`: Optional DateTime filter for activities before timestamp (e.g., '2024-12-31 23:59:59') @@ -22,9 +23,15 @@ NODE activities_filtered_retention_bucket_routing SQL > % SELECT activityId as id, timestamp, type, platform, memberId, organizationId, segmentId - FROM activityRelations_bucket_routing a + FROM + {% if defined(collectionSlug) %} activityRelations_deduplicated_cleaned_bucket_union + {% else %} activityRelations_bucket_routing + {% end %} as a where - segmentId = (SELECT segmentId FROM segments_filtered) + {% if defined(collectionSlug) %} + segmentId IN (SELECT segmentId FROM segments_filtered_by_collection) + {% else %} segmentId = (SELECT segmentId FROM segments_filtered) + {% end %} {% if defined(startDate) %} AND a.timestamp > {% if defined(granularity) and granularity == "daily" %} diff --git a/services/libs/tinybird/pipes/activityTypes_by_project.pipe b/services/libs/tinybird/pipes/activityTypes_by_project.pipe index 18b162623f..7074602911 100644 --- a/services/libs/tinybird/pipes/activityTypes_by_project.pipe +++ b/services/libs/tinybird/pipes/activityTypes_by_project.pipe @@ -1,9 +1,15 @@ DESCRIPTION > - - `activityTypes_by_project.pipe` returns activity types that actually exist for a given project. + - `activityTypes_by_project.pipe` returns activity types that actually exist for a given project, or + across every project in a collection. - Unlike `activityTypes_filtered` which returns all possible activity types, this pipe only returns types with actual activities in the project. - Useful for populating dropdowns, filters, or analytics that should only show activity types with data. - Parameters: - - `project`: **Required** string for project slug (e.g., 'k8s', 'tensorflow'). Passed to `segments_filtered`. + - `project`: String for project slug (e.g., 'k8s', 'tensorflow'). Passed to `segments_filtered`. Required unless `collectionSlug` is given. + - `collectionSlug`: Collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`. Required unless `project` is given. + Collection requests read from `activityRelations_deduplicated_cleaned_bucket_union` (all 10 buckets) instead + of `activityRelations_bucket_routing` (a single bucket, defaulting to bucket 0 when `bucketId` is not + injected), since a collection's member projects' segments can be routed to any of the 10 buckets - same + pattern as `activities_filtered.pipe`. - `repos`: Optional array of repository URLs for filtering (e.g., ['https://github.com/kubernetes/kubernetes']). Filters activities by repository. - `includeCodeContributions`: Optional boolean to include code contribution activities. Defaults to 1. Set to 0 to exclude. Passed to `activityTypes_filtered`. - `includeCollaborations`: Optional boolean to include or exclude collaboration activities. Defaults to 0. Passed to `activityTypes_filtered`. @@ -15,10 +21,16 @@ NODE activityTypes_by_project_0 SQL > % SELECT DISTINCT a.type as activityType, a.platform, at.label - FROM activityRelations_bucket_routing a + FROM + {% if defined(collectionSlug) %} activityRelations_deduplicated_cleaned_bucket_union + {% else %} activityRelations_bucket_routing + {% end %} a INNER JOIN activityTypes at ON a.type = at.activityType AND a.platform = at.platform WHERE - a.segmentId = (SELECT segmentId FROM segments_filtered) + {% if defined(collectionSlug) %} + a.segmentId IN (SELECT segmentId FROM segments_filtered_by_collection) + {% else %} a.segmentId = (SELECT segmentId FROM segments_filtered) + {% end %} {% if defined(repos) %} AND a.channel IN (SELECT channel FROM repos_to_channels) {% end %} AND (a.type, a.platform) IN (SELECT activityType, platform FROM activityTypes_filtered) ORDER BY activityType, platform diff --git a/services/libs/tinybird/pipes/activity_heatmap_by_weekday_and_2hours_blocks.pipe b/services/libs/tinybird/pipes/activity_heatmap_by_weekday_and_2hours_blocks.pipe index 4ea60f8489..916299670e 100644 --- a/services/libs/tinybird/pipes/activity_heatmap_by_weekday_and_2hours_blocks.pipe +++ b/services/libs/tinybird/pipes/activity_heatmap_by_weekday_and_2hours_blocks.pipe @@ -5,7 +5,8 @@ DESCRIPTION > - Returns a complete grid of all weekday/time combinations with zero counts for periods with no activity, ensuring consistent heatmap visualization. - Primary use case: identifying contribution patterns outside traditional work hours for community health analysis. - Parameters: - - `project`: Inherited from `segments_filtered`, project slug (e.g., 'k8s', 'tensorflow') + - `project`: Inherited from `segments_filtered`, project slug (e.g., 'k8s', 'tensorflow'). Required unless `collectionSlug` is given. + - `collectionSlug`: Inherited from `segments_filtered_by_collection`, collection slug (e.g., 'cncf'). Required unless `project` is given. - `repos`: Inherited from `segments_filtered`, array of repository URLs for filtering - `startDate`: Optional DateTime filter for activities after timestamp (e.g., '2024-01-01 00:00:00') - `endDate`: Optional DateTime filter for activities before timestamp (e.g., '2024-12-31 23:59:59') @@ -18,7 +19,10 @@ SQL > select count(id) as activityCount, weekday, two_hours_block from contributions_with_local_time_ds a where - segmentId = (SELECT segmentId FROM segments_filtered) + {% if defined(collectionSlug) %} + segmentId IN (SELECT segmentId FROM segments_filtered_by_collection) + {% else %} segmentId = (SELECT segmentId FROM segments_filtered) + {% end %} {% if defined(startDate) %} AND a.timestamp > {{ DateTime(startDate, description="Filter activity timestamp after", required=False) }} diff --git a/services/libs/tinybird/pipes/collection_contributor_dependency.pipe b/services/libs/tinybird/pipes/collection_contributor_dependency.pipe new file mode 100644 index 0000000000..0f2e8b168b --- /dev/null +++ b/services/libs/tinybird/pipes/collection_contributor_dependency.pipe @@ -0,0 +1,52 @@ +DESCRIPTION > + - `collection_contributor_dependency.pipe` serves the contributor dependency widget for a + collection's Contributors tab - identifies contributors whose combined contributions reach or + exceed 51% of total collection activity (bus factor analysis), aggregated across every project + in the collection. + - Collection-scoped counterpart to `contributor_dependency.pipe` - separate because that pipe + references `contributors_leaderboard` (project-only) by name, and the collection-scale + replacement for that table is `collection_contributors_leaderboard` (a differently-shaped + pipe - see its own DESCRIPTION for why a naive collectionSlug branch on the single-project + leaderboard pipe isn't used there either). + - Combines data from `collection_contributors_leaderboard` and `active_contributors` (both + already collectionSlug-aware) to provide dependency risk assessment at collection scale. + - Parameters: + - `collectionSlug`: Required collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`. + - `repos`: Optional array of repository URLs for filtering (inherited from `segments_filtered_by_collection`) + - `startDate`: Optional DateTime filter for activities after timestamp (e.g., '2024-01-01 00:00:00') + - `endDate`: Optional DateTime filter for activities before timestamp (e.g., '2024-12-31 23:59:59') + - `platform`: Optional string filter for source platform (e.g., 'github', 'discord', 'slack') + - `activity_type`: Optional string filter for single activity type (e.g., 'authored-commit') + - `includeCollaborations`: Optional boolean, defaults to false - same toggle as the single-project pipe. + - `limit`: Optional integer for the number of top contributors to rank before finding the 51% + cutoff. Inherited by `collection_contributors_leaderboard` (this pipe has no default of its + own), whose own default is 10 if omitted - callers doing bus-factor analysis should pass + limit=100 explicitly (same convention as the single-project `contributor_dependency.pipe` / + `contributors_leaderboard.pipe` pair) to reliably reach the 51% cutoff. + - Response: `id`, `displayName`, `githubHandleArray`, `contributionCount`, `contributionPercentage`, `roles`, `contributionPercentageRunningTotal`, `totalContributorCount` + +TAGS "Insights, Widget", "Collection", "Contributors" + +NODE collection_contributions_percentage_running_total +SQL > + % + SELECT t.*, active_contributors.contributorCount as "totalContributorCount" + FROM + ( + SELECT + id, + displayName, + githubHandleArray, + contributionCount, + contributionPercentage, + roles, + sum(contributionPercentage) OVER ( + ORDER BY contributionPercentage DESC, id + ) AS contributionPercentageRunningTotal + FROM collection_contributors_leaderboard + ) t + cross join active_contributors + WHERE + contributionPercentageRunningTotal <= 51 + OR (contributionPercentageRunningTotal - contributionPercentage < 51) + ORDER BY contributionPercentageRunningTotal ASC diff --git a/services/libs/tinybird/pipes/collection_contributors_leaderboard.pipe b/services/libs/tinybird/pipes/collection_contributors_leaderboard.pipe new file mode 100644 index 0000000000..71fd463003 --- /dev/null +++ b/services/libs/tinybird/pipes/collection_contributors_leaderboard.pipe @@ -0,0 +1,84 @@ +DESCRIPTION > + - `collection_contributors_leaderboard.pipe` serves the contributors leaderboard widget for a + collection's Contributors tab, aggregated across every project in the collection. + - Separate from `contributors_leaderboard.pipe` (the single-project version) because a naive + collection-scoped port of that pipe's `ROUND(COUNT(af.id) * 100.0 / SUM(COUNT(af.id)) OVER (), 2)` + contributionPercentage forces ClickHouse to fully materialize the per-member GROUP BY (potentially + hundreds of thousands of distinct members at collection scale) before it can apply ORDER BY/LIMIT - + confirmed via direct timing: the equivalent single-project pipe took 7-16s (once timing out + entirely at Tinybird's 20s limit) on a 242-project collection, vs 2-3s for the count-only path + with no window function. This pipe computes the total contribution count as a separate scalar + subquery instead of a window function, avoiding that full-materialization cost. + - Supports both count mode (`count=true`) and data mode for pagination-friendly leaderboard display. + - Always shows `displayName` (not `publicName`) since a collection spans many projects with mixed + LF/non-LF status - there's no single project-type answer to substitute on, unlike the single-project + pipe's `is_request_from_non_lf_project` check. + - Primary use case: powering the collection-scoped contributor ranking widget on Collection detail + pages' Contributors tab. + - Parameters: + - `collectionSlug`: Required collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`. + - `repos`: Optional array of repository URLs for filtering (inherited from `segments_filtered_by_collection`) + - `startDate`: Optional DateTime filter for activities after timestamp (e.g., '2024-01-01 00:00:00') + - `endDate`: Optional DateTime filter for activities before timestamp (e.g., '2024-12-31 23:59:59') + - `platform`: Optional string filter for source platform (e.g., 'github', 'discord', 'slack') + - `activity_type`: Optional string filter for single activity type (e.g., 'authored-commit') + - `activity_types`: Optional array of activity types (e.g., ['authored-commit', 'co-authored-commit']) + - `includeCollaborations`: Optional boolean, defaults to false - same toggle as the single-project pipe. + - `count`: Optional boolean, when true returns contributor count instead of leaderboard data + - `limit`: Optional integer for result pagination, defaults to 10 + - `offset`: Optional integer for result pagination, defaults to 0 + - Response: + - Count mode (`count=true`): `count` (total number of contributors) + - Data mode (default): `id`, `avatar`, `displayName`, `githubHandleArray`, `contributionCount`, `contributionPercentage`, `roles` + +TAGS "Insights, Widget", "Collection", "Contributors" + +NODE collection_contributors_total +DESCRIPTION > + Total contribution count across the collection - a plain aggregate with no GROUP BY, so it's + cheap regardless of how many distinct members contributed. Computed once and reused as a scalar + for the percentage calculation instead of a SUM(...) OVER () window function. + +SQL > + % + SELECT count(af.id) AS total FROM activities_filtered af + +NODE collection_member_aggregates +DESCRIPTION > + Per-member contribution counts, ranked and paginated. No window function here - the percentage + is computed in the next node against the scalar total, so ORDER BY/LIMIT can apply directly + after this GROUP BY without waiting on a second full-set pass. + +SQL > + % + {% if Boolean(count, false) %} SELECT count(distinct af.memberId) FROM activities_filtered af + {% else %} + SELECT af.memberId, count(af.id) AS contributionCount + FROM activities_filtered af + GROUP BY af.memberId + ORDER BY contributionCount DESC, af.memberId DESC + LIMIT {{ Int32(limit, 10) }} + OFFSET {{ Int32(offset, 0) }} + {% end %} + +NODE collection_contributors_leaderboard_result +SQL > + % + {% if Boolean(count, false) %} + SELECT count(distinct af.memberId) AS count FROM activities_filtered af + {% else %} + SELECT + m.id, + m.avatar, + m.displayName, + m.githubHandleArray, + ma.contributionCount, + ROUND(ma.contributionCount * 100.0 / total.total, 2) AS contributionPercentage, + mr.roles + FROM members_sorted AS m any + INNER JOIN collection_member_aggregates ma ON ma.memberId = m.id + LEFT JOIN member_roles mr ON mr.memberId = m.id + CROSS JOIN collection_contributors_total total + WHERE m.id IN (SELECT memberId FROM collection_member_aggregates) + ORDER BY contributionCount DESC + {% end %} diff --git a/services/libs/tinybird/pipes/collection_insights_aggregate.pipe b/services/libs/tinybird/pipes/collection_insights_aggregate.pipe index 15a5c1f2c6..0edaa91cf0 100644 --- a/services/libs/tinybird/pipes/collection_insights_aggregate.pipe +++ b/services/libs/tinybird/pipes/collection_insights_aggregate.pipe @@ -9,7 +9,11 @@ DESCRIPTION > - Response: - `projectCount`: count of distinct projects in the collection. - `uniqueContributorCount`: total unique contributors across all projects in the collection, deduplicated (a contributor active on multiple projects in the collection is counted once). - - `avgHealthScore`: average of each project's health score in the collection, rounded. + - `avgHealthScore`: average health score across the collection's onboarded projects, rounded. A project + is "onboarded" when it has at least one contributor or organization attributed to it (same definition + used by insightsProjects_filtered.pipe's `onboarded` filter and the frontend's project-row `isOnboarded` + check) — projects with no attributed contributors/organizations have no meaningful health signal yet + and would otherwise pull the average down artificially. TAGS "Insights, Widget", "Collection" @@ -21,7 +25,7 @@ DESCRIPTION > SQL > % - SELECT id, segmentId, healthScore + SELECT id, segmentId, healthScore, organizationCount, contributorCount FROM insights_projects_populated_ds WHERE enabled = 1 @@ -38,7 +42,11 @@ SQL > SELECT (SELECT count(distinct id) FROM collection_insights_aggregate_projects) AS projectCount, uniq(ar.memberId) AS uniqueContributorCount, - (SELECT round(avg(healthScore)) FROM collection_insights_aggregate_projects) AS avgHealthScore + ( + SELECT round(avg(healthScore)) + FROM collection_insights_aggregate_projects + WHERE NOT (organizationCount = 0 AND contributorCount = 0) + ) AS avgHealthScore FROM activityRelations_deduplicated_cleaned_bucket_union AS ar WHERE ar.memberId != '' diff --git a/services/libs/tinybird/pipes/member_roles.pipe b/services/libs/tinybird/pipes/member_roles.pipe index f613b22ac5..0ebf849f1f 100644 --- a/services/libs/tinybird/pipes/member_roles.pipe +++ b/services/libs/tinybird/pipes/member_roles.pipe @@ -5,7 +5,9 @@ SQL > FROM maintainers_roles_copy_ds WHERE 1 = 1 - {% if defined(project) %} + {% if defined(collectionSlug) %} + AND insightsProjectId IN (SELECT insightsProjectId FROM segments_filtered_by_collection) + {% elif defined(project) %} AND insightsProjectId = (SELECT insightsProjectId FROM segments_filtered) {% end %} {% if defined(repos) %} diff --git a/services/libs/tinybird/pipes/package_metrics.pipe b/services/libs/tinybird/pipes/package_metrics.pipe index 3cca2958cc..4024a12478 100644 --- a/services/libs/tinybird/pipes/package_metrics.pipe +++ b/services/libs/tinybird/pipes/package_metrics.pipe @@ -6,7 +6,8 @@ DESCRIPTION > - Uses custom time-series generation logic specifically designed for package download data patterns. - Primary use case: powers package downloads and package dependency widgets. - Parameters: - - `project`: Required string for project slug (e.g., 'k8s', 'tensorflow') - inherited from `segments_filtered` + - `project`: Required string for project slug (e.g., 'k8s', 'tensorflow') - inherited from `segments_filtered`. Required unless `collectionSlug` is given. + - `collectionSlug`: Collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`. Required unless `project` is given. - `startDate`: Optional DateTime filter for package data after timestamp (e.g., '2024-01-01 00:00:00') - `endDate`: Optional DateTime filter for package data before timestamp (e.g., '2024-12-31 23:59:59') - `ecosystem`: Optional string to filter by package ecosystem (e.g., 'npm', 'pypi', 'maven') @@ -23,7 +24,10 @@ SQL > SELECT * FROM packageDownloads p FINAL where - insightsProjectId = (select insightsProjectId from segments_filtered) + {% if defined(collectionSlug) %} + insightsProjectId IN (select insightsProjectId from segments_filtered_by_collection) + {% else %} insightsProjectId = (select insightsProjectId from segments_filtered) + {% end %} {% if defined(startDate) %} AND p.date > toDate( {{ DateTime(startDate, description="Filter package downloads after", required=False) }} @@ -140,11 +144,12 @@ SQL > ) package_downloads_timeseries_bounds WHERE "startDate" >= actual_start_date AND "startDate" < actual_end_date -NODE package_downloads_initial_aggregation +NODE package_downloads_per_project_aggregation SQL > % {% if defined(granularity) %} SELECT + p.insightsProjectId, p.date, sum(p.downloadsCount) as downloadsCount, sum(p.dockerDownloadsCount) as "dockerDownloadsCount", @@ -155,6 +160,7 @@ SQL > group by p.insightsProjectId, p.date {% else %} SELECT + p.insightsProjectId, max(p.downloadsCount) as downloadsCount, max(p.dockerDownloadsCount) as "dockerDownloadsCount", max(p.dockerDependentsCount) as "dockerDependentsCount", @@ -164,6 +170,29 @@ SQL > group by p.insightsProjectId {% end %} +NODE package_downloads_initial_aggregation +SQL > + % + {% if defined(granularity) %} + SELECT + date, + sum(downloadsCount) as downloadsCount, + sum(dockerDownloadsCount) as "dockerDownloadsCount", + sum(dockerDependentsCount) as "dockerDependentsCount", + sum(dependentPackagesCount) as "dependentPackagesCount", + sum(dependentReposCount) as "dependentReposCount" + FROM package_downloads_per_project_aggregation + group by date + {% else %} + SELECT + sum(downloadsCount) as downloadsCount, + sum(dockerDownloadsCount) as "dockerDownloadsCount", + sum(dockerDependentsCount) as "dockerDependentsCount", + sum(dependentPackagesCount) as "dependentPackagesCount", + sum(dependentReposCount) as "dependentReposCount" + FROM package_downloads_per_project_aggregation + {% end %} + NODE package_downloads_timeseries_merge SQL > % diff --git a/services/libs/tinybird/pipes/packages.pipe b/services/libs/tinybird/pipes/packages.pipe index 5f146fae8d..0fd65ba2bd 100644 --- a/services/libs/tinybird/pipes/packages.pipe +++ b/services/libs/tinybird/pipes/packages.pipe @@ -5,7 +5,8 @@ DESCRIPTION > - Uses `segments_filtered` for automatic project scoping and optional repository filtering. - Primary use case: powers package selection dropdown in the package downloads widget. - Parameters: - - `project`: Required string for project slug (e.g., 'k8s', 'tensorflow') - inherited from `segments_filtered` + - `project`: Required string for project slug (e.g., 'k8s', 'tensorflow') - inherited from `segments_filtered`. Required unless `collectionSlug` is given. + - `collectionSlug`: Collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`. Required unless `project` is given. - `search`: Optional string for searching package names or ecosystems (case-insensitive with wildcards) - `repos`: Optional array of repository URLs to filter packages by specific repositories (e.g., ['https://github.com/kubernetes/kubernetes']) - Response: `repo`, `name`, `ecosystem` (distinct packages matching the filters) @@ -16,7 +17,10 @@ SQL > SELECT distinct p.repo, p.name, p.ecosystem FROM packageDownloads p where - p.insightsProjectId = (select insightsProjectId from segments_filtered) + {% if defined(collectionSlug) %} + p.insightsProjectId IN (select insightsProjectId from segments_filtered_by_collection) + {% else %} p.insightsProjectId = (select insightsProjectId from segments_filtered) + {% end %} {% if defined(search) %} AND ( p.name diff --git a/services/libs/tinybird/pipes/pull_requests_filtered.pipe b/services/libs/tinybird/pipes/pull_requests_filtered.pipe index 5f93a73b49..2e1d8121ae 100644 --- a/services/libs/tinybird/pipes/pull_requests_filtered.pipe +++ b/services/libs/tinybird/pipes/pull_requests_filtered.pipe @@ -2,18 +2,25 @@ DESCRIPTION > - `pull_requests_filtered.pipe` is the core filtering infrastructure for pull request analytics across the platform. - Provides consistent, reusable PR filtering logic for all pull request-related widgets and analytics. - Uses `pull_requests_analyzed` datasource which contains pre-calculated PR metrics and timing data. - - Automatically scopes data to the current project using `segments_filtered` pipe for security and data isolation. + - Scopes data either to a single project (via `segments_filtered`) or to every project in a collection (via + `segments_filtered_by_collection`), for security and data isolation. Exactly one of `project`/`collectionSlug` + must be provided by the caller. - Supports repository-level filtering for targeted PR analysis within projects. - Primary use case: serving as the foundation for all PR analytics pipes including lead times, review metrics, and velocity analysis. - Parameters: - - `project`: Required string for project slug (e.g., 'k8s', 'tensorflow') - inherited from `segments_filtered` + - `project`: Project slug (e.g., 'k8s', 'tensorflow') - inherited from `segments_filtered`. Required unless `collectionSlug` is given. + - `collectionSlug`: Collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`. Required unless `project` is given. - `repos`: Optional array of repository URLs for filtering PRs by specific repositories (e.g., ['https://github.com/kubernetes/kubernetes']) - Response: All fields from `pull_requests_analyzed` including PR metadata, timing metrics, and calculated fields NODE pull_requests_filtered_0 SQL > + % SELECT * FROM pull_requests_analyzed pra WHERE - pra.segmentId = (SELECT segmentId FROM segments_filtered) - AND pra.channel IN (SELECT channel FROM repos_to_channels) + pra.channel IN (SELECT channel FROM repos_to_channels) + {% if defined(collectionSlug) %} + AND pra.segmentId IN (SELECT segmentId FROM segments_filtered_by_collection) + {% else %} AND pra.segmentId = (SELECT segmentId FROM segments_filtered) + {% end %} diff --git a/services/libs/tinybird/pipes/search_volume.pipe b/services/libs/tinybird/pipes/search_volume.pipe index bb49056ee5..f7946b14ca 100644 --- a/services/libs/tinybird/pipes/search_volume.pipe +++ b/services/libs/tinybird/pipes/search_volume.pipe @@ -5,23 +5,24 @@ DESCRIPTION > - Formats timestamps for consistent date representation in visualizations. - Primary use case: displaying search trend analytics and project popularity metrics in dashboard widgets. - Parameters: - - `project`: Required string for project slug (e.g., 'k8s', 'tensorflow') - inherited from `segments_filtered` + - `project`: Required string for project slug (e.g., 'k8s', 'tensorflow') - inherited from `segments_filtered`. Required unless `collectionSlug` is given. + - `collectionSlug`: Collection slug (e.g., 'cncf') - inherited from `segments_filtered_by_collection`. Required unless `project` is given. - `startDate`: Optional DateTime filter for search volume data after timestamp (e.g., '2024-01-01 00:00:00') - `endDate`: Optional DateTime filter for search volume data before timestamp (e.g., '2024-12-31 23:59:59') - - Response: `insightsProjectId`, `project`, `dataTimestamp` (formatted as YYYY-MM-DD), `volume`, `updatedAt` + - Response: `insightsProjectId`, `project`, `dataTimestamp` (formatted as YYYY-MM-DD), `volume`, `updatedAt`. + When `collectionSlug` is given, rows are aggregated to one row per `dataTimestamp` with `volume` summed + across every project in the collection; `insightsProjectId`/`project`/`updatedAt` are empty in that mode. -NODE searchVolume_pipe +NODE searchVolume_filtered SQL > % - SELECT - insightsProjectId, - project, - formatDateTime(dataTimestamp, '%Y-%m-%d') AS dataTimestamp, - volume, - updatedAt + SELECT insightsProjectId, project, dataTimestamp, volume, updatedAt FROM searchVolume FINAL WHERE - insightsProjectId = toUUID((select insightsProjectId from segments_filtered)) + {% if defined(collectionSlug) %} + insightsProjectId IN (select toUUID(insightsProjectId) from segments_filtered_by_collection) + {% else %} insightsProjectId = toUUID((select insightsProjectId from segments_filtered)) + {% end %} {% if defined(startDate) %} AND searchVolume.dataTimestamp >= parseDateTimeBestEffort( {{ @@ -44,3 +45,25 @@ SQL > }} ) {% end %} + +NODE searchVolume_pipe +SQL > + % + {% if defined(collectionSlug) %} + SELECT + toUUID('00000000-0000-0000-0000-000000000000') AS insightsProjectId, + '' AS project, + formatDateTime(dataTimestamp, '%Y-%m-%d') AS dataTimestamp, + sum(volume) AS volume, + max(updatedAt) AS updatedAt + FROM searchVolume_filtered + GROUP BY formatDateTime(dataTimestamp, '%Y-%m-%d') + {% else %} + SELECT + insightsProjectId, + project, + formatDateTime(dataTimestamp, '%Y-%m-%d') AS dataTimestamp, + volume, + updatedAt + FROM searchVolume_filtered + {% end %} diff --git a/services/libs/tinybird/pipes/segments_filtered_by_collection.pipe b/services/libs/tinybird/pipes/segments_filtered_by_collection.pipe new file mode 100644 index 0000000000..c354609a70 --- /dev/null +++ b/services/libs/tinybird/pipes/segments_filtered_by_collection.pipe @@ -0,0 +1,35 @@ +DESCRIPTION > + - `segments_filtered_by_collection.pipe` is the collection-scoped counterpart to `segments_filtered.pipe`: + it establishes project scope for the widget pipes that back the Collection detail page's Contributors, + Popularity, and Development tabs, resolving a collection to every one of its member projects instead of + a single project. + - **Requires `collectionSlug` parameter** - without it, returns no results (WHERE clause becomes AND false). + - Returns one row per project in the collection (not a single aggregate row) - `insights_projects_populated_ds.collectionsSlugs` + already lists every collection each project belongs to (see `insightsProjects_filtered.pipe` and + `collection_insights_aggregate.pipe` for the same `has(collectionsSlugs, ...)` filter pattern), so this + pipe resolves collection membership directly rather than requiring the caller to first look up a project + id list. + - Downstream pipes that today do `segmentId = (SELECT segmentId FROM segments_filtered)` (a single-value + scalar subquery) must switch to `segmentId IN (SELECT segmentId FROM segments_filtered_by_collection)` + when scoping by collection instead of by project, since this pipe can return multiple rows. + - Ensures only enabled projects are accessible. + - Parameters: + - `collectionSlug`: Required string for collection slug (e.g., 'cncf', 'agentic-ai-foundation') + - Response: `insightsProjectId`, `segmentId`, `repositories` (array of the project's own repositories) - one row per project in the collection + +TAGS "Infrastructure", "Core", "Project scoping", "Security", "Collection" + +NODE segments_filtered_by_collection_0 +SQL > + % + SELECT id as insightsProjectId, segmentId, repositories + FROM insights_projects_populated_ds + WHERE + enabled = 1 + {% if defined(collectionSlug) %} + AND has( + collectionsSlugs, + {{ String(collectionSlug, description="Filter by collection slug", required=True) }} + ) + {% else %} AND false + {% end %}