Skip to content

refactor(connectors): extract row-processing logic into reusable process_row method#3198

Open
atharvalade wants to merge 1 commit into
apache:masterfrom
atharvalade:fix/deduplicate-row-processing-logic
Open

refactor(connectors): extract row-processing logic into reusable process_row method#3198
atharvalade wants to merge 1 commit into
apache:masterfrom
atharvalade:fix/deduplicate-row-processing-logic

Conversation

@atharvalade
Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #3173

Rationale

Row-processing logic lived inline in poll_tables, making it impossible to reuse across future parallel/chunked polling paths without duplication.

What changed?

The column-iteration, value-extraction, JSON-payload construction, and ProducedMessage assembly were tightly coupled inside poll_tables's inner loop (~80 lines). Any new polling variant (parallel, chunked) would need to copy-paste this block.

Extracted a process_row method that accepts a RowProcessingConfig (table, tracking/pk columns, format flags) and returns a ProcessedRow (message + offset + pk). The poll_tables loop now delegates to this single-responsibility method in ~12 lines.

Local Execution

  • Passed
  • Pre-commit hooks ran

AI Usage

  1. Opus 4.6
  2. used for exploration and implementation guidance
  3. Verified via cargo check, clippy -D warnings, fmt --check, all 19 unit tests passing, CI lint scripts passing
  4. Yes, all code can be explained

@codecov
Copy link
Copy Markdown

codecov Bot commented Apr 29, 2026

Codecov Report

❌ Patch coverage is 0% with 87 lines in your changes missing coverage. Please review.
✅ Project coverage is 19.22%. Comparing base (611fca0) to head (9cc08ad).
⚠️ Report is 36 commits behind head on master.

Files with missing lines Patch % Lines
core/connectors/sources/postgres_source/src/lib.rs 0.00% 87 Missing ⚠️

❌ Your patch check has failed because the patch coverage (0.00%) is below the target coverage (50.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@              Coverage Diff              @@
##             master    #3198       +/-   ##
=============================================
- Coverage     74.10%   19.22%   -54.88%     
  Complexity      943      943               
=============================================
  Files          1159     1157        -2     
  Lines        102033    90344    -11689     
  Branches      79083    67394    -11689     
=============================================
- Hits          75607    17371    -58236     
- Misses        23765    72575    +48810     
+ Partials       2661      398     -2263     
Components Coverage Δ
Rust Core 0.95% <0.00%> (-74.38%) ⬇️
Java SDK 60.14% <ø> (ø)
C# SDK 69.38% <ø> (ø)
Python SDK 81.43% <ø> (ø)
Node SDK 91.53% <ø> (ø)
Go SDK 39.43% <ø> (ø)
Files with missing lines Coverage Δ
core/connectors/sources/postgres_source/src/lib.rs 40.78% <0.00%> (-26.40%) ⬇️

... and 647 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 7, 2026

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

If you need a review, please ensure CI is green and the PR is rebased on the latest master. Don't hesitate to ping the maintainers - either @core on Discord or by mentioning them directly here on the PR.

Thank you for your contribution!

@github-actions github-actions Bot added stale Inactive issue or pull request and removed stale Inactive issue or pull request labels May 7, 2026
@hubcio
Copy link
Copy Markdown
Contributor

hubcio commented May 14, 2026

/ready

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label May 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Row processing logic duplicated three times

2 participants