Skip to content

Add time based flush + add force telemetry flag#880

Merged
samikshya-db merged 2 commits into
databricks:mainfrom
samikshya-db:samikshya-chand_data/timerPush
Jul 8, 2025
Merged

Add time based flush + add force telemetry flag#880
samikshya-db merged 2 commits into
databricks:mainfrom
samikshya-db:samikshya-chand_data/timerPush

Conversation

@samikshya-db
Copy link
Copy Markdown
Collaborator

@samikshya-db samikshya-db commented Jul 6, 2025

Description

  • We should be sending out telemetry in 3 cases :
    • When batch size reaches a threshold (Already implemented ✅ )
    • When connection/statement is closed (Already implemented ✅ )
    • When time since last telemetry export reaches threshold (In this PR 🚧 )
  • Add a force telemetry flag : this will turn on telemetry even when server flag say otherwise. This will be useful when :
    • Internal use-cases need to be laced with telemetry (eg: benchmarking)
    • Force telemetry for customers facing issues : for us to be able to debug better.

Testing

  • Added unit test to ensure they are working

Additional Notes to the Reviewer

NO_CHANGELOG=TRUE

@samikshya-db samikshya-db requested a review from gopalldb July 6, 2025 13:41
private void schedulePeriodicFlush() {
// Ensure minimum 1 second interval to avoid over-calling flush
int intervalMillis = Math.max(1000, context.getTelemetryFlushIntervalInMilliseconds());
scheduledExecutorService.scheduleAtFixedRate(
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's reset the last flushedTime on flush, so that timer also gets reset

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense, addressed ✅

@samikshya-db samikshya-db requested a review from gopalldb July 7, 2025 09:16
@samikshya-db samikshya-db merged commit a644512 into databricks:main Jul 8, 2025
8 of 12 checks passed
@samikshya-db samikshya-db deleted the samikshya-chand_data/timerPush branch July 8, 2025 17:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants