fix: RNTuple footer leak on extend, broken WritableNTuple.compression, directory __delitem__ KeyError#1650
Open
henryiii wants to merge 2 commits into
Open
fix: RNTuple footer leak on extend, broken WritableNTuple.compression, directory __delitem__ KeyError#1650henryiii wants to merge 2 commits into
henryiii wants to merge 2 commits into
Conversation
…, directory __delitem__ KeyError - _cascadentuple.py extend(): release full key+payload region (num_bytes+compressed_bytes) instead of just the key header (allocation/num_bytes), preventing footer-sized FreeSegments leak on every extend call - WritableNTuple.compression getter/setter: replace copied-from-WritableTree code that referenced nonexistent _branch_data; getter now returns actual file-level compression; setter raises NotImplementedError with a clear message - WritableDirectory._del: check get_key() result for None and raise KeyInFileError, fixing AttributeError on nonexistent key deletion (MutableMapping contract) - Remove dead/broken NTuple code: Ntuple_PageLink class (unused, broken serialize), NTuple.branch_types and NTuple.field_name properties (referencing unset attributes), NTuple_Footer.__repr__ (referenced nonexistent metadata_block_envelope_links), NTuple.write_updates (stub never called) - Remove Tree.branch_types property in _cascadetree.py (referencing unset _branch_types) Assisted-by: ClaudeCode:claude-sonnet-4-6
Add tests/test_1650_rntuple_writing_bugfixes.py covering: - RNTuple extend() FreeSegments leak: verify file stays small and is readable after 10 extends - WritableNTuple.compression getter: returns file-level compression without AttributeError - WritableNTuple.compression setter: raises NotImplementedError as expected - WritableDirectory.__delitem__ on nonexistent key: raises KeyError (was AttributeError) - WritableDirectory.__delitem__ on nonexistent key: specifically raises KeyInFileError Assisted-by: ClaudeCode:claude-sonnet-4-6
Codecov Report❌ Patch coverage is
❌ Your patch check has failed because the patch coverage (80.00%) is below the target coverage (98.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
What was broken
1. RNTuple
extend()leaked footer-sized regions into FreeSegments on every callNTuple.extendin_cascadentuple.pycalledfreesegments.release(start, start + old_footer_key.allocation), butKey.allocationis justKey.num_bytes(the ~70-byte key record header). The full allocated region isnum_bytes + compressed_bytes. Every call toextend()left the footer payload stranded in FreeSegments, causing permanent file bloat proportional to the number of extends. This parallels the TTree path in_cascadetree.py:487which correctly usesnum_bytes + compressed_bytes.Reproducer: Create an RNTuple and call
extend()10 times — the file is larger than it needs to be and FreeSegments entries accumulate. After the fix the freed region is fully reclaimed.2.
WritableNTuple.compressiongetter/setter raisedAttributeErrorThe property was copy-pasted verbatim from
WritableTreebutNTuplehas no_branch_data; it always raisedAttributeError: 'NTuple' object has no attribute '_branch_data'. The getter now returns the actual file-level compression fromself._cascading._freesegments.fileheader.compression. The setter raisesNotImplementedErrorwith a clear message since per-RNTuple compression is not currently supported by the cascading implementation.Reproducer:
3.
WritableDirectory.__delitem__raisedAttributeErrorfor nonexistent keys_delcalledkey.seek_locationwithout checking whetherget_key()returnedNone. Deleting a nonexistent key raisedAttributeError: 'NoneType' object has no attribute 'seek_location'instead of aKeyErrorsubclass as theMutableMappingcontract requires. Now raisesuproot.exceptions.KeyInFileError(aKeyErrorsubclass), exactly like_get.Reproducer:
4. Dead/broken code removed
Ntuple_PageLinkclass: unreferenced anywhere; itsserializepacked two values into a single-field struct → guaranteedstruct.error.NTuple.branch_typesandNTuple.field_nameproperties: referencedself._branch_types/self._field_namewhich are never set onNTuple→ guaranteedAttributeError.NTuple_Footer.__repr__: referenced nonexistentself.metadata_block_envelope_links→ guaranteedAttributeError.NTuple.write_updates: a one-line stub (sink.flush()) that is never called from anywhere.Tree.branch_typesin_cascadetree.py: referencedself._branch_typeswhich is never set (branch types are stored in_branch_data).What changed
src/uproot/writing/_cascadentuple.py: fix footer release extent; remove 5 dead/broken code itemssrc/uproot/writing/_cascadetree.py: removeTree.branch_typesdead propertysrc/uproot/writing/writable.py: fixWritableNTuple.compression; fixWritableDirectory._delnull checkSkipped
Nothing was skipped. All 4 categories of findings were verified and fixed.
Test results
29 tests passed, 6 skipped (ROOT not installed) across the relevant test files.
Part of #1646.