Skip to content

feat(connectors): add Redshift sink connector#3654

Open
realonbebeto wants to merge 9 commits into
apache:masterfrom
realonbebeto:feat/redshift-sink-connector
Open

feat(connectors): add Redshift sink connector#3654
realonbebeto wants to merge 9 commits into
apache:masterfrom
realonbebeto:feat/redshift-sink-connector

Conversation

@realonbebeto

Copy link
Copy Markdown

Summary

Adds a Redshift sink connector for writing Apache Iggy messages into Amazon Redshift using S3-staged Parquet files and Redshift's COPY command.

The connector batches messages into Parquet files, uploads them to a configurable S3 bucket/prefix, and performs bulk COPY loads into the target Redshift table. It supports configurable batch sizing, connection pooling, optional metadata/checksum/origin timestamp fields, payload encoding modes, retry/backoff handling for failed loads, optional archival of staged Parquet files, and runtime metrics logging.

The implementation treats S3 purely as a staging layer for efficient bulk ingestion rather than a destination system. Staged files can either be deleted after a successful load or retained under an archive prefix for replay, auditing, or downstream processing.

It also integrates the connector throughout the project, including workspace membership, connector documentation, release and binary artifact pipelines, version bump automation, and Docker-based integration test infrastructure.

Disclosures

  1. pgwire, sqlparser, and tokio-postgres are introduced as workspace dependencies solely to support the RedshiftFixture used by the integration test suite.
  2. The Redshift integration tests are inspired by and closely follow the existing PostgreSQL test setup to maintain consistency across sink connector implementations.

Closes #2540

AI Usage

A blend of ChatGPT and Claude was used to help build, refactor the connector and associated test suite

@github-actions

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Jul 12, 2026
@codecov

codecov Bot commented Jul 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.03%. Comparing base (b7c87d8) to head (de9c97f).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #3654      +/-   ##
============================================
- Coverage     73.61%   72.03%   -1.59%     
  Complexity      937      937              
============================================
  Files          1297     1234      -63     
  Lines        143770   139083    -4687     
  Branches     119328   119241      -87     
============================================
- Hits         105842   100193    -5649     
- Misses        34562    35607    +1045     
+ Partials       3366     3283      -83     
Components Coverage Δ
Rust Core 73.85% <ø> (+<0.01%) ⬆️
Java SDK 62.44% <ø> (ø)
C# SDK 72.19% <ø> (+0.01%) ⬆️
Python SDK 92.17% <ø> (ø)
PHP SDK 84.29% <ø> (ø)
Node SDK 87.27% <ø> (-4.08%) ⬇️
Go SDK 16.94% <ø> (-25.78%) ⬇️
see 131 files with indirect coverage changes
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@krishvishal

Copy link
Copy Markdown
Contributor

@realonbebeto can you please fix the CI failures? After that we can review the code.

@realonbebeto

Copy link
Copy Markdown
Author

@hubcio resolved the CI failures.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement Redshift Sink Connector with S3 staging

3 participants