Skip to content

Commit 14c7dc4

Browse files
epipavepipav
authored
feat: packages tb enrichment (#4243)
Signed-off-by: anilb <epipav@gmail.com>
1 parent 1aea913 commit 14c7dc4

5 files changed

Lines changed: 598 additions & 0 deletions

File tree

‎backend/src/osspckgs/migrations/V1781539311__packages_tables_sequin_updates.sql‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
-- ─── 1. package_dependencies (created_at, id) index ─────────────────────────
2+
CREATE INDEX IF NOT EXISTS package_dependencies_created_at_id_idx
3+
ON package_dependencies (created_at, id);
4+
5+
-- ─── 2. repo_activity_snapshot replication ──────────────────────────────────
6+
DO $$
7+
BEGIN
8+
IF EXISTS (
9+
SELECT 1 FROM pg_publication WHERE pubname = 'sequin_pub'
10+
) AND NOT EXISTS (
11+
SELECT 1 FROM pg_publication_tables
12+
WHERE pubname = 'sequin_pub'
13+
AND schemaname = 'public'
14+
AND tablename = 'repo_activity_snapshot'
15+
) THEN
16+
ALTER PUBLICATION sequin_pub ADD TABLE repo_activity_snapshot;
17+
END IF;
18+
END$$;
19+
20+
ALTER TABLE public.repo_activity_snapshot REPLICA IDENTITY FULL;
21+
22+
-- ─── 3. packages enriched health/lifecycle fields (synced back from Tinybird) ───
23+
ALTER TABLE packages
24+
ADD COLUMN IF NOT EXISTS lifecycle_label text,
25+
ADD COLUMN IF NOT EXISTS health_score smallint,
26+
ADD COLUMN IF NOT EXISTS health_label text,
27+
ADD COLUMN IF NOT EXISTS maintainer_health_score smallint,
28+
ADD COLUMN IF NOT EXISTS security_supply_chain_score smallint,
29+
ADD COLUMN IF NOT EXISTS development_activity_score smallint,
30+
ADD COLUMN IF NOT EXISTS signal_coverage_health jsonb;

‎services/libs/tinybird/datasources/ossPackages_enriched_ds.datasource‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
DESCRIPTION >
2+
- `ossPackages_enriched_ds` contains all `ossPackages` metadata plus derived enrichment fields, materialized by `ossPackages_enriched.pipe` (daily COPY, replace mode).
3+
- Carries every column from the `ossPackages` datasource verbatim, so it can serve as a drop-in enriched replacement for package analytics.
4+
- `lifecycleLabel` is the categorical maintenance state of the package's primary repository, derived from repo + activity-snapshot signals: 'active', 'stable', 'declining', 'abandoned', or 'archived'.
5+
- `healthScore` is the composite 0–100 health score = `maintainerHealthScore` (≤40) + `securitySupplyChainScore` (≤35) + `developmentActivityScore` (≤25).
6+
- `healthLabel` buckets `healthScore`: 'excellent' (85+), 'healthy' (70–84), 'fair' (50–69), 'concerning' (30–49), 'critical' (<30).
7+
- `maintainerHealthScore`, `securitySupplyChainScore`, `developmentActivityScore` are the three health sub-totals.
8+
- `signalCoverageHealth` is a JSON object mapping each health sub-signal to 'available' | 'partial' | 'blocked', so the UI can flag which scores reflect real data vs. gaps.
9+
- `impact` (carried verbatim from `ossPackages`) is the criticality impact score and serves as the Impact total score.
10+
- All other fields mirror `ossPackages`; see `ossPackages.datasource` for per-field documentation.
11+
12+
TAGS "OSS packages", "Enrichment"
13+
14+
SCHEMA >
15+
`id` UInt64,
16+
`purl` String,
17+
`ecosystem` String,
18+
`namespace` Nullable(String),
19+
`name` String,
20+
`registryUrl` Nullable(String),
21+
`status` Nullable(String),
22+
`description` Nullable(String),
23+
`homepage` Nullable(String),
24+
`declaredRepositoryUrl` Nullable(String),
25+
`repositoryUrl` Nullable(String),
26+
`licenses` Array(String),
27+
`licensesRaw` Nullable(String),
28+
`keywords` Array(String),
29+
`distTagsLatest` Nullable(String),
30+
`distTagsNext` Nullable(String),
31+
`distTagsBeta` Nullable(String),
32+
`versionsCount` Nullable(UInt32),
33+
`latestVersion` Nullable(String),
34+
`firstReleaseAt` Nullable(DateTime64(3)),
35+
`latestReleaseAt` Nullable(DateTime64(3)),
36+
`dependentCount` Nullable(UInt32),
37+
`transitiveDependentCount` Nullable(UInt64),
38+
`dependentReposCount` Nullable(UInt32),
39+
`hasCriticalVulnerability` UInt8,
40+
`impact` Nullable(String),
41+
`isCritical` UInt8,
42+
`lastRankPassAt` Nullable(DateTime64(3)),
43+
`downloadsLast30d` Nullable(UInt64),
44+
`centralityScore` Nullable(String),
45+
`rankInEcosystem` Nullable(UInt32),
46+
`ingestionSource` Nullable(String),
47+
`lastSyncedAt` DateTime64(3),
48+
`createdAt` DateTime64(3),
49+
`lifecycleLabel` String,
50+
`healthScore` UInt8,
51+
`healthLabel` String,
52+
`maintainerHealthScore` UInt8,
53+
`securitySupplyChainScore` UInt8,
54+
`developmentActivityScore` UInt8,
55+
`signalCoverageHealth` String
56+
57+
ENGINE MergeTree
58+
ENGINE_SORTING_KEY ecosystem, purl

‎services/libs/tinybird/datasources/repoActivitySnapshot.datasource‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
DESCRIPTION >
2+
- `repoActivitySnapshot` holds per-repository activity health metrics over a rolling window (default 12 months).
3+
- Replicated from Postgres packages-db — computed by the github-repos-enricher worker from the GitHub API (commits, PRs, issues).
4+
- One row per repo; a row is absent until the enricher has reached that repo, so consumers must treat a missing row as "no signal" (degraded path).
5+
- Used to derive package lifecycle (active / stable / declining / abandoned) by joining repos -> packageRepos -> packages.
6+
- `repoId` links to the parent repos row — the primary key.
7+
- `snapshotAt` is when this snapshot was computed — also the updatedAt watermark for Tinybird sync.
8+
- `windowMonths` is the rolling window length the metrics were computed over (default 12).
9+
- `commitsLast12m`, `commitsLast6m`, `commitsPrior6m` are commit counts for the trailing 12m, the most recent 6m, and the 6m before that (NULL if not computed).
10+
- `prsOpenedLast12m`, `prsMergedLast12m`, `prsClosedUnmerged12m` are PR counts opened / merged / closed-without-merge in the trailing 12m (NULL if not computed).
11+
- `prMedianTimeToMergeHours` is the median hours from PR open to merge (NULL if no merged PRs in window).
12+
- `prMedianTimeToFirstResponseHours` is the median hours to first non-author response on a PR (NULL if no PRs or no responses in window).
13+
- `issuesOpenedLast12m`, `issuesClosedLast12m` are issue counts opened / closed in the trailing 12m (NULL if not computed).
14+
- `issuesOpenedLast6m`, `issuesOpenedPrior6m` are issues opened in the most recent 6m and the 6m before that (NULL if not computed).
15+
- `issuesOpenNow` is the current count of open issues (NULL if not computed).
16+
- `issueMedianTimeToCloseHours` is the median hours from issue open to close (NULL if no closed issues in window).
17+
- `issueMedianTimeToFirstResponseHours` is the median hours to first non-author response on an issue (NULL if no issues or no responses in window).
18+
19+
SCHEMA >
20+
`repoId` UInt64 `json:$.record.repo_id`,
21+
`snapshotAt` DateTime64(3) `json:$.record.snapshot_at`,
22+
`windowMonths` UInt16 `json:$.record.window_months` DEFAULT 12,
23+
`commitsLast12m` Nullable(Int32) `json:$.record.commits_last_12m`,
24+
`commitsLast6m` Nullable(Int32) `json:$.record.commits_last_6m`,
25+
`commitsPrior6m` Nullable(Int32) `json:$.record.commits_prior_6m`,
26+
`prsOpenedLast12m` Nullable(Int32) `json:$.record.prs_opened_last_12m`,
27+
`prsMergedLast12m` Nullable(Int32) `json:$.record.prs_merged_last_12m`,
28+
`prsClosedUnmerged12m` Nullable(Int32) `json:$.record.prs_closed_unmerged_12m`,
29+
`prMedianTimeToMergeHours` Nullable(Int32) `json:$.record.pr_median_time_to_merge_hours`,
30+
`prMedianTimeToFirstResponseHours` Nullable(Int32) `json:$.record.pr_median_time_to_first_response_hours`,
31+
`issuesOpenedLast12m` Nullable(Int32) `json:$.record.issues_opened_last_12m`,
32+
`issuesClosedLast12m` Nullable(Int32) `json:$.record.issues_closed_last_12m`,
33+
`issuesOpenedLast6m` Nullable(Int32) `json:$.record.issues_opened_last_6m`,
34+
`issuesOpenedPrior6m` Nullable(Int32) `json:$.record.issues_opened_prior_6m`,
35+
`issuesOpenNow` Nullable(Int32) `json:$.record.issues_open_now`,
36+
`issueMedianTimeToCloseHours` Nullable(Int32) `json:$.record.issue_median_time_to_close_hours`,
37+
`issueMedianTimeToFirstResponseHours` Nullable(Int32) `json:$.record.issue_median_time_to_first_response_hours`
38+
39+
ENGINE ReplacingMergeTree
40+
ENGINE_SORTING_KEY repoId
41+
ENGINE_VER snapshotAt

0 commit comments

Comments
 (0)