perf: enhance LZMA decompression performance and improve resource management#471
Open
unxed wants to merge 2 commits into
Open
perf: enhance LZMA decompression performance and improve resource management#471unxed wants to merge 2 commits into
unxed wants to merge 2 commits into
Conversation
…port and improved resource management. This change replaces the standard LZMA library with a performance-optimized fork to enable multi-threaded block decompression when the input stream supports random access via Seek and ReadAt interfaces. The Close methods in the lzma and lzma2 packages have been updated to explicitly close the internal decoder using errors.Join, ensuring that buffers are properly returned to the pool and background goroutines are terminated to prevent memory leaks and test timeouts. Furthermore, the LZMA2 constructor now includes logic to detect seekable streams and safely wrap them in section readers for parallel processing, accompanied by new unit tests to verify the interface detection and cleanup logic. Fix bodgit#470
unxed
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description:
This pull request introduces significant performance optimizations for LZMA/LZMA2 decompression and fixes a resource management issue where internal decoder buffers and goroutines were not being properly released.
Key Changes:
Closemethods in thelzmaandlzma2packages to explicitly close the internal decoder. This is crucial for returning large dictionary buffers to thesync.Pooland ensuring background goroutines are terminated, preventing memory leaks and potential hangs in high-concurrency environments.github.com/unxed/xz). This fork provides faster sequential decoding through register caching and manual inlining, and supports multi-threaded block decompression.io.ReaderAtandio.Seekerinterfaces within the LZMA2 constructor. When a seekable stream is detected (such as aSectionReadercommon in 7z archives), the decoder now attempts to use theParallelReaderfor concurrent block processing, significantly increasing throughput on multi-core systems.io.NewSectionReaderto ensure parallel reads stay within the bounds of the specific archive folder. If initialization of the parallel reader fails or the stream is not seekable, the system transparently falls back to the standard sequential mode.reader_test.goto verify the interface detection logic, ensure proper cleanup duringClose, and confirm that the fallback mechanism works correctly for different types of input streams.Performance Impact:
Sequential decoding speed is improved due to internal optimizations, and parallel-capable streams (where independent XZ blocks are present) can see a substantial increase in decompression throughput, scaling with the number of available CPU cores.
Fix #470