XZ Support#6721
Closed
Shivansps wants to merge 5 commits into
Closed
Conversation
Fix some warnings, finish with the TODOs, fix asignation error in cf_clear_compression_info and remove the optimised single block decoder as it was not any faster.
Contributor
Author
|
The reason for retire the PR is here. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a WIP. And im not even sure if it is desired or viable yet.
This work adds xz compressed files support, both single files and compressed packs (.vp.xz).
XZ is no anywhere near as fast as LZ4 in ST decompression, so it strongely depends on MT to keep loading times as short as possible.
The idea behind it as i commented on #6718 is that if we could get use the files that are uploaded to nebula directly, this on top of saving disk space (and saving the process of re-compress later for anyone who wants it), it would also remove the need to extract the files after download and maybe even implement something like zsync.
XZ files are also the regular ones, as random access is in its design, there is no need for specialised compressors. The only real requeriment is to use 1MB solid blocks to try to get as much multi-threading as possible.
My tests so far with the current code on MVP 4.7.3:
Nebula Upload (7z) is 7.95GB.
Extracted (with vps): 17.61GB -
Trailer mission loading time: 0:42 seconds on a NVME
Compressed VPs (.vp.xz) 8.41GB
Trailer mission loading time: 1:01m with my 5800X. (yes speed is more than likely impacted by CPU, a CPU with better ST perf is likely to be faster) Also loading uses multiples threads.
By comparison the VPC version on LZ4 is 10GB and loads on 44s.
Still a lot of to do, still looking if this could be speed up somehow, there are a few warnings to fix and several TODOS around the code, mostly debug checks missings. And this will not compile, i dont know how to include liblzma into the cmake build process, i tried for hours to no avail. One big TODO that will try to commit soon is to disable the decoder buffer optimisation when blocksizes over 4MB are used. What will certanly lead to absurd memory usages.
The corrent most optimised compressor settings are:
1MB solid blocks
1 or 2 MB Dictionary
64 wordsize
Level 9 (Ultra) or Level 8 (max)
So im attaching the header files, the lib and dll needed to compile and run in case anyone wants to try it and im also attaching a script file to compress an entire folder vps with XZ,, you will need to point it to the 7z executable and the folder.
liblzma.zip
7z script.zip
Im attaching compiled builds (this is actually a 7z file)
user_build_202505071036420949.zip