Skip to content

fix: remove useless cols (CM-731)#3528

Merged
ulemons merged 10 commits into
mainfrom
feat/purge-activities-deduplicated-ds
Oct 28, 2025
Merged

fix: remove useless cols (CM-731)#3528
ulemons merged 10 commits into
mainfrom
feat/purge-activities-deduplicated-ds

Conversation

@ulemons
Copy link
Copy Markdown
Contributor

@ulemons ulemons commented Oct 17, 2025

PR: Cleanup of redundant fields in Tinybird and addition of a cleaned backup MV

Description

This PR focuses on simplifying the Tinybird data pipeline by removing redundant fields and introducing a new materialized view (MV) for a lightweight backup of the activities datasource.


What Has Been Changed

  • Removed unused or redundant columns from the activities.datasource.
  • Updated activities_deduplicated_ds.datasource and its copy-pipe (activities_deduplicated_copy_pipe.pipe) to align with the reduced schema.
  • Added a new datasource activities_backup_cleaned.datasource with a simplified structure for backup purposes.
  • Created a materialized view activities_backup_cleaned_mv.pipe that populates the cleaned backup from the original activities table.
  • Ensured that the backup uses the same engine setup (ReplacingMergeTree, partitioned by year of createdAt, sorted by id) but retains only essential fields for archival and analytics.

@ulemons ulemons self-assigned this Oct 17, 2025
@ulemons ulemons force-pushed the feat/purge-activities-deduplicated-ds branch from 5b22857 to e5ccf80 Compare October 21, 2025 15:37
@ulemons ulemons changed the title fix: remove useless cols fix: remove useless cols (CM-731) Oct 21, 2025
@ulemons ulemons force-pushed the feat/purge-activities-deduplicated-ds branch from e5ccf80 to cad93d2 Compare October 24, 2025 08:32
@ulemons ulemons marked this pull request as ready for review October 24, 2025 08:50
@ulemons ulemons added the Feature Created by Linear-GitHub Sync label Oct 27, 2025
@ulemons ulemons requested a review from borfast October 27, 2025 13:48
Copy link
Copy Markdown
Contributor

@borfast borfast left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good but there are a few places where you're removing the isContribution flag, which has already been removed (those changes have already been merged), so I would suggest you rebase your branch and check for conflicts that may arise from that. It shouldn't be a big deal, as it's only removing stuff.

@ulemons ulemons merged commit 06d7032 into main Oct 28, 2025
15 of 17 checks passed
@ulemons ulemons deleted the feat/purge-activities-deduplicated-ds branch October 28, 2025 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Feature Created by Linear-GitHub Sync

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants