Skip to content

feat: Add metronome source implementation#324

Merged
ferenc-csaky merged 2 commits into
mainfrom
feat/metronome
May 6, 2026
Merged

feat: Add metronome source implementation#324
ferenc-csaky merged 2 commits into
mainfrom
feat/metronome

Conversation

@ferenc-csaky

@ferenc-csaky ferenc-csaky commented May 6, 2026

Copy link
Copy Markdown
Collaborator

Key Changes

  • Added a new metronome connector that emits a monotonically increasing BIGINT sequence per second based on a wall-clock timestamp column
  • Parallelism is forced to 1 so the metronome acts as a single global clock source
  • Emits catch-up records when delayed by comparing current epoch seconds against the persisted start epoch second and last emitted sequence number
  • Uses intended event timestamps for catch-up records, so delayed emissions still map to the correct event-time seconds
  • State is carried through MetronomeSplit, preserving lastEmittedNumber and startTimestampSec
  • MetronomeEnumeratorState also tracks any pending split, so returned splits are reassigned without losing progress
  • Supports limit pushdown by creating a bounded runtime source when LIMIT is pushed into the table source
  • Added schema validation requiring exactly two physical columns: BIGINT and TIMESTAMP/TIMESTAMP_LTZ

@ferenc-csaky ferenc-csaky added the enhancement New feature or request label May 6, 2026
@ferenc-csaky
ferenc-csaky requested a review from mateczagany May 6, 2026 12:37
@ferenc-csaky ferenc-csaky added this to the 0.10.1 milestone May 6, 2026

@mateczagany mateczagany left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good for now, but we should add tests for this later

@ferenc-csaky
ferenc-csaky merged commit f0bda49 into main May 6, 2026
13 checks passed
@ferenc-csaky
ferenc-csaky deleted the feat/metronome branch May 6, 2026 20:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants