[3.14] compression.zstd is slow

[Rogdham]

Hello, I'm noticing that the compression.zstd in CPython 3.14.0rc2 module has slow performance when using the release assets from this repository.

Local tests show between 4 and 7 times slowdown when compared to building CPython from source, depending on the Zstandard level, and operation made (compression/decompression).

Maybe related: #327

Reproducer

First, download enwik8.zip (official source), uncompress it at /tmp/enwik8 (should be 100Mo).

Then, run:

python -m timeit -s 'from compression import zstd; DATA = open("/tmp/enwik8","rb").read()' 'zstd.compress(DATA)'

Version	Timing
Local build	1 loop, best of 5: 663 msec per loop
cpython-3.14.0rc2+20250814-x86_64-unknown-linux-gnu-install_only_stripped	1 loop, best of 5: 3.21 sec per loop
cpython-3.14.0rc2+20250814-x86_64_v2-unknown-linux-gnu-install_only_stripped	1 loop, best of 5: 3.2 sec per loop
cpython-3.14.0rc2+20250814-x86_64_v3-unknown-linux-gnu-install_only_stripped	1 loop, best of 5: 3.25 sec per loop
cpython-3.14.0rc2+20250814-x86_64-unknown-linux-gnu-pgo+lto-full	1 loop, best of 5: 3.23 sec per loop
cpython-3.14.0rc2+20250814-x86_64-unknown-linux-gnu-freethreaded+pgo+lto-full	1 loop, best of 5: 3.27 sec per loop

Tests performed under Archlinux.

[zanieb]

Thanks for the report!

I ran

python -c "from compression import zstd; print(zstd.CompressionParameter.nb_workers.bounds())"

and the result is (0, 0) which I think means multithreading is disabled? Can you confirm the bounds are different for your from-source build?

[zanieb]

This would do it...

https://github.com/facebook/zstd/blob/e128976193546dceb24249206a02ff8f444f7120/lib/README.md?plain=1#L22

When building with make, by default the dynamic library is multithreaded and static library is single-threaded (for compatibility reasons).

[Rogdham]

and the result is (0, 0) which I think means multithreading is disabled?

Yes you are right. This is what we use in CPython as well for detection.

Can you confirm the bounds are different for your from-source build?

I can confirm: I have (0, 256).

[indygreg]

If the CPython stdlib tries to use multithreaded compression by default, that's questionable behavior. I may need to get involved upstream. (I'm the maintainer of the zstandard Python package.)

I think the correct behavior is for libzstd to be compiled with multithreaded support (it requires setting a flag at build time) and for the use of multithreaded compression to require opt-in at compression time.

[Rogdham]

Hello @indygreg, nice to see you chime in here! I am myself the maintainer of pyzstd (and backports.zstd) and tried to help Emma including Zstandard into the stdlib.

In CPython, I think that libzstd is likely to be linked dynamically (at least it is the case when I compile it locally with default config). This may be different on Windows; maybe @emmatyping you could comment on this?

For runtime support, CompressionParameter.nb_workers needs to be specified. I can confirm that for example it takes half the time when running with 2 as the value:

python -m timeit -s 'from compression import zstd; DATA = open("/tmp/enwik8","rb").read()' 'zstd.compress(DATA, options={zstd.CompressionParameter.nb_workers: 2})'

[zanieb]

It sounds like there's probably another root cause here then.

[Rogdham]

It sounds like there's probably another root cause here then.

Yes, I'm having similar timings with the changes in #762 (using cpython-3.14.0rc2-x86_64_v3-unknown-linux-gnu-debug-20250816T1313.tar.zst from GitHub action build artefacts of that PR):

1 loop, best of 5: 3.36 sec per loop

[indygreg]

Yeah, I just looked at the CPython source code and it (reasonably) defaults to single threaded compression.

So something else is at play.

Just eyeballing the numbers, 3.21s to compress a 100 MB file (~31MB/s) feels low to me by ~5x. The 663ms timing reported seems appropriate.

Accidentally compiling libzstd in debug mode is a possible explanation. Not engaging e.g. AVX2 intrinsics is another possibility. I'd look at how PBS is compiling libzstd. Compare the raw compiler arguments between CPython and PBS for the library and extension module.

[emmatyping]

If the CPython stdlib tries to use multithreaded compression by default, that's questionable behavior.

Yeah we definitely do single-threaded by default.

In CPython, I think that libzstd is likely to be linked dynamically (at least it is the case when I compile it locally with default config). This may be different on Windows

On Windows things are dynamically linked by default as well.

Accidentally compiling libzstd in debug mode is a possible explanation.

It looks like on this line PBS is compiling for a debug target (you want libzstd.a-release for a non-debug build). You can see here that appending -release removes the debug flags:
https://github.com/facebook/zstd/blob/e128976193546dceb24249206a02ff8f444f7120/lib/Makefile#L205-L207

Will make a PR and verify this fixes the issue.

[emmatyping]

Huh, #764 isn't fixing the difference either, even though I compiled for x86_64-v4 (I have a Ryzen 9 9950X with AVX 512 support).

I guess my diagnosis might be incorrect, the debug flags may not be set by default so -release may not be needed after all (probably good to add anyway).

[indygreg]

We should pass VERBOSE=1 into zstd's make invocation so it actually prints the full compiler command. Without it, the build logs are next to worthless since compiler flags aren't getting logged.

[indygreg]

I think I found it.

https://github.com/facebook/zstd/blob/e128976193546dceb24249206a02ff8f444f7120/lib/libzstd.mk#L94C1-L106C6

CFLAGS ?= -O3

Our build system exports CFLAGS. That ?= Makefile syntax says to assign the value if the variable is not defined or has an empty value. So essentially our setting of CFLAGS is suppressing the default compiler flags of -O3.

We should export CFLAGS="${CFLAGS} -O3" in our build script and also change to make VERBOSE=1 so we can debug future issues easier.