feat: bucketed lambda merger for activityRelations_enriched_deduplicated datasources

[epipav]

No description provided.

[cursor]

PR Summary

High Risk
This is a core Tinybird data-plane migration affecting CDP dashboards, PR analytics, and monitoring; a mis-deploy or incomplete bucket bootstrap could skew or drop activity data until initial snapshots are re-run.

Overview
Replaces the single activityRelations_enriched_deduplicated_ds lambda serving table with three cityHash64(segmentId) % 3 buckets (_bucket_0..2_ds), exposed via activityRelations_enriched_deduplicated_bucket_union. The old activityRelations_snapshot_merger_copy append merger is removed in favor of daily staggered replace copies (activityRelations_snapshot_merger_copy_0..2) that merge MV deltas with per-bucket carry-forward; bootstrap is split into activityRelations_enrich_initial_snapshot_0..2.

Bucket datasources drop snapshot-based TTL and partition by toYear(timestamp) instead of snapshotId. MV delta retention on activityRelations_enrich_snapshot_MV_ds increases from 1 to 3 days.

Downstream pipes (CDP metrics, activities_daily_counts, activities_relations_filtered, PR initial snapshot, monitoring) now read the union and omit max(snapshotId) filters, since each bucket holds one snapshot. monitoring_entities counts rows per bucket and sums to avoid union count() timeouts.

README and lambda-architecture.md document the new topology, failure modes, and query rules (no global snapshot filter on the union).

^{Reviewed by Cursor Bugbot for commit 747071b. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[github-actions]

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

• feat: add user authentication (CM-123)
• feat: add user authentication (IN-123)

Projects:

• CM: Community Data Platform
• IN: Insights

Please add a Jira issue key to your PR title.

[Copilot]

Pull request overview

Introduces a three-bucket, replace-mode serving layer for unfiltered activity relations.

Changes:

• Adds bucket datasources, bootstrap pipes, daily mergers, and a union pipe.
• Migrates downstream queries from the legacy snapshot datasource.
• Extends delta retention and updates architecture documentation.

Review note: The PR title is missing the required trailing JIRA key (CM-XXX).

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
services/libs/tinybird/README.md	Documents bucketed lambda output.
services/libs/tinybird/pipes/pull_request_analysis_initial_snapshot.pipe	Reads from the bucket union.
services/libs/tinybird/pipes/monitoring_entities.pipe	Counts rows per bucket.
services/libs/tinybird/pipes/cdp_segment_metrics_total_sink.pipe	Migrates total metrics source.
services/libs/tinybird/pipes/cdp_segment_metrics_copy_pipe.pipe	Migrates segment metrics source.
services/libs/tinybird/pipes/cdp_organization_segment_aggregates_initial_snapshot.pipe	Migrates organization aggregates.
services/libs/tinybird/pipes/cdp_member_segment_aggregates_initial_snapshot.pipe	Migrates member aggregates.
services/libs/tinybird/pipes/activityRelations_snapshot_merger_copy.pipe	Removes the legacy merger.
services/libs/tinybird/pipes/activityRelations_snapshot_merger_copy_2.pipe	Adds bucket 2 merger.
services/libs/tinybird/pipes/activityRelations_snapshot_merger_copy_1.pipe	Adds bucket 1 merger.
services/libs/tinybird/pipes/activityRelations_snapshot_merger_copy_0.pipe	Adds bucket 0 merger.
services/libs/tinybird/pipes/activityRelations_enriched_deduplicated_bucket_union.pipe	Unions serving buckets.
services/libs/tinybird/pipes/activityRelations_enrich_initial_snapshot_2.pipe	Bootstraps bucket 2.
services/libs/tinybird/pipes/activityRelations_enrich_initial_snapshot_1.pipe	Bootstraps bucket 1.
services/libs/tinybird/pipes/activityRelations_enrich_initial_snapshot_0.pipe	Bootstraps bucket 0.
services/libs/tinybird/pipes/activities_relations_filtered.pipe	Migrates filtered activity queries.
services/libs/tinybird/pipes/activities_daily_counts.pipe	Migrates daily activity counts.
services/libs/tinybird/lambda-architecture.md	Documents the new architecture.
services/libs/tinybird/datasources/activityRelations_enriched_deduplicated_bucket_2_ds.datasource	Defines bucket 2 storage.
services/libs/tinybird/datasources/activityRelations_enriched_deduplicated_bucket_1_ds.datasource	Defines bucket 1 storage.
services/libs/tinybird/datasources/activityRelations_enriched_deduplicated_bucket_0_ds.datasource	Updates bucket 0 storage.
services/libs/tinybird/datasources/activityRelations_enrich_snapshot_MV_ds.datasource	Extends delta retention.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

services/libs/tinybird/pipes/activityRelations_enriched_deduplicated_bucket_union.pipe:18

• This union becomes the source for all migrated consumers in the same release that creates buckets 1 and 2, but those datasources remain empty until the @on-demand initial-snapshot pipes are run. The scheduled mergers can recover only the MV's three-day retention, so deploying these files before completing the full bootstrap serves an incomplete historical dataset (at least the two new buckets) for the duration of the bootstrap. Please use a staged rollout that seeds and validates all buckets before switching consumers/enabling the union.

    FROM activityRelations_enriched_deduplicated_bucket_0_ds
    UNION ALL
    SELECT *
    FROM activityRelations_enriched_deduplicated_bucket_1_ds

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 747071b. Configure here.}

[cursor]

Stale MV delta overwrites newer rows

Medium Severity

Inclusive replay (snapshotId >= lastSnapshotId) plus always preferring realtime_delta over historical_carryforward can replace a newer bucket row with an older MV version for the same dedup key. This shows up when a merger runs on the same calendar day as an initial snapshot, because today's updates sit at snapshotId = tomorrow and are outside the MV window.

Additional Locations (2)

• services/libs/tinybird/pipes/activityRelations_snapshot_merger_copy_1.pipe#L18-L30
• services/libs/tinybird/pipes/activityRelations_snapshot_merger_copy_2.pipe#L18-L30

 

^{Reviewed by Cursor Bugbot for commit 747071b. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

services/libs/tinybird/pipes/activityRelations_enriched_deduplicated_bucket_union.pipe:21

• All live consumers are switched to this union in the same change, but these three targets are populated only by @on-demand initial-snapshot pipes. A one-shot deployment will therefore serve empty or partially bootstrapped results until all three full copies finish; if a scheduled merger runs first, an empty target is seeded from only the three-day MV retention window. Please stage the rollout: deploy and bootstrap/validate all buckets first, then switch consumers and enable the scheduled mergers.

    SELECT *
    FROM activityRelations_enriched_deduplicated_bucket_0_ds
    UNION ALL
    SELECT *
    FROM activityRelations_enriched_deduplicated_bucket_1_ds
    UNION ALL
    SELECT *
    FROM activityRelations_enriched_deduplicated_bucket_2_ds