Enable CURL response decompression

[rustyconover]

Summary

• Change CURLOPT_ACCEPT_ENCODING from NULL to "" so CURL advertises all supported compression encodings (gzip, deflate, br, zstd) and automatically decompresses responses
• Reduces bandwidth usage for HTTP transfers with no behavioral change for callers

Fixes #277

Test plan

• Verify CURL sends Accept-Encoding header with supported encodings
• Verify compressed responses are automatically decompressed
• Run existing httpfs test suite to confirm no regressions

🤖 Generated with Claude Code

[carlopi]

Thanks for the PR, that would make sense, but before enabling this behaviour we would first need to fix duckdb/duckdb to handle receiving those pre-compressed streams.

Internal timeline is "nice for 1.5.1, but unclear"

[rustyconover]

But curl decompresses it for you

…

On Sat, Mar 7, 2026 at 06:22 Carlo Piovesan ***@***.***> wrote: *carlopi* left a comment (duckdb/duckdb-httpfs#278) <#278 (comment)> Thanks for the PR, that would make sense, but before enabling this behaviour we would first need to fix duckdb/duckdb to handle receiving those pre-compressed streams. Internal timeline is "nice for 1.5.1, but unclear" — Reply to this email directly, view it on GitHub <#278 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFSWJOPMGNFSIXZKAK3TAT4PQA6XAVCNFSM6AAAAACWKCJGACVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DAMJWGI4DEMRRGI> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[carlopi]

I would be OK with adding an option, (that defaults to using NULL) but allows to opt in to "". Would that work for you?

The option we might need in any case to avoid regressions.
On defaulting to "" vs NULL, issue was that (as mentioned in the PR at #263 (comment)) uploading gzip-ped JSON files to S3 will have them doubly decompressed.
Given it's a niche but relevant data-source, I'd say first we need a fix before changing the default.

[rustyconover]

Hey @carlopi,

You should let curl handle the decompression.

The whole argument around:

aws s3 cp file.json.gz s3://<bucket>/file.json.gz --content-type gzip

is flawed. HTTP Compression is a content encoding for the transmission over the HTTP request, not a content-type. The content type here should still be application/octet-stream not gzip.

I realize how it can be confusing for people, but we need to be pedantic about this.

I'd be happy to talk this through more. But I think this original motivation for #263 is flawed.

Rusty