compress S3 uploads with GZIP before COPY to Redshift

[tobiascadee]

Summary

• Changes the S3 file extension from .csv to .csv.gz — smart_open auto-compresses on write when the URI ends in .gz
• Adds GZIP keyword to the Redshift COPY statement so Redshift decompresses on load

Reduces S3 upload size and transfer time, which is typically the bottleneck for large batches.

Note: Users with custom copy_options in their config should ensure they don't already include GZIP to avoid a duplicate keyword error.

Test plan

• Run target against a real Redshift cluster and verify records load correctly
• Check S3 that uploaded files are .csv.gz and are valid gzip
• Verify existing custom copy_options configs still work

🤖 Generated with Claude Code