investigate pyiceberg-core size (currently ~100mb)

[kevinjqliu]

Apache Iceberg Rust version

None

Describe the bug

In the latest nightly run for pyiceberg-core, the wheels are 100MB each...
https://github.com/apache/iceberg-rust/actions/runs/19274277069

From #1458, It was

Python wheels for 0.4.0: between ~ 0.7 and 1.5 MiB

Python wheels for 0.5.1: 25 to 55MiB

Let's investigate if we can shrink this.

For reference, pyarrow 22 wheels are ~50MB each
https://pypi.org/project/pyarrow/22.0.0/#files

To Reproduce

No response

Expected behavior

No response

Willingness to contribute

None

[kevinjqliu]

opendal and datafusion both use profile.release flags to optimize artifact size. we can give that a try

https://github.com/apache/opendal/blob/e03f40a3f33856506c5f04c149f52bcc48253872/bindings/python/Cargo.toml#L219-L224

https://github.com/apache/datafusion-python/blob/89d8930625c7a903fb0e5f1d4f5310341e0b4fea/Cargo.toml#L89-L91

[kevinjqliu]

#1841 dropped the largest artifact to around 70MB (-30%)

Name	Old Size	New Size	% Decrease
wheels-macos-latest-universal2-apple-darwin	63.2 MB	44.4 MB	29.75%
wheels-ubuntu-latest-aarch64	101 MB	66.1 MB	34.55%
wheels-ubuntu-latest-armv7l	104 MB	70.5 MB	32.21%
wheels-ubuntu-latest-x86_64	34.8 MB	23.8 MB	31.61%
wheels-windows-latest	30.3 MB	25.9 MB	14.52%
wheels-sdist	574 KB	575 KB

70MB is still relatively large, let's see if there are more improvements

[kevinjqliu]

More optimization in #1844

Github action run: https://github.com/apache/iceberg-rust/actions/runs/19285828822

Name	Old Size	New Size	% Decrease
wheels-macos-latest-universal2-apple-darwin	44.4 MB	23.5 MB	47.07%
wheels-ubuntu-latest-aarch64	66.1 MB	40.9 MB	38.12%
wheels-ubuntu-latest-armv7l	70.5 MB	33.1 MB	53.05%
wheels-ubuntu-latest-x86_64	23.8 MB	13 MB	45.38%
wheels-windows-latest	25.9 MB	12.7 MB	45.38%
wheels-sdist	575 KB	574 KB

This is on top of #1841 😄

[liurenjie1024]

More optimization in #1844

Github action run: https://github.com/apache/iceberg-rust/actions/runs/19285828822

Name Old Size New Size % Decrease
wheels-macos-latest-universal2-apple-darwin 44.4 MB 23.5 MB 47.07%
wheels-ubuntu-latest-aarch64 66.1 MB 40.9 MB 38.12%
wheels-ubuntu-latest-armv7l 70.5 MB 33.1 MB 53.05%
wheels-ubuntu-latest-x86_64 23.8 MB 13 MB 45.38%
wheels-windows-latest 25.9 MB 12.7 MB 45.38%
wheels-sdist 575 KB 574 KB
This is on top of #1841 😄

Awesome!

[kevinjqliu]

Ah these are zipped files of the wheels and may contain many wheels inside. The actual wheel size is actually smaller

Latest run with both #1841 and #1841 (https://test.pypi.org/project/pyiceberg-core/0.7.0.dev20251112154529/#files)

Largest wheel size is only 24.8 MB, rest are around 11-15 MB 😄