fix: interpretation-layer bugs (pandas NaN fallback, typename parsing, strided objects, bitset)#1660
Open
henryiii wants to merge 2 commits into
Open
fix: interpretation-layer bugs (pandas NaN fallback, typename parsing, strided objects, bitset)#1660henryiii wants to merge 2 commits into
henryiii wants to merge 2 commits into
Conversation
…, strided objects, bitset)
Fixes a batch of verified bugs in the interpretation layer:
- library.py: `_process_array_for_pandas` fell off the end returning
None when `awkward_form` raised CannotBeAwkward, so `library="pd"`
silently produced an all-NaN Series. Now returns the object array.
- library.py: `_strided_to_awkward` uses `reshape(-1)` (view) instead of
`flatten()` (copy) at each recursion level.
- identify.py: `parse_typename("long long*")` /
`"unsigned long long*"` were dead (shadowed by the scalar branches)
and constructed `AsArray` with the wrong number of arguments. Reordered
and pass the speedbump argument.
- identify.py: replaced a truncated/dead assert (its f-string message was
a separate statement) with a proper ValueError.
- objects.py: `AsStridedObjects.__init__` no longer deletes from a list
while iterating its original range (IndexError) and no longer drops the
sole member; rewritten as a filter that drops interior (None, None)
header markers.
- objects.py: `AsStridedObjects` no longer mutates shared `self._to_dtype`
in `basket_array`; the output dtype is derived locally in `final_array`,
making the interpretation safe to share across basket-worker threads.
- objects.py: hoisted the per-basket awkward-form recomputation out of the
loop, reused `has_any_awkward_types`, and removed a dead assignment.
- strings.py: `final_array` now binds `output` on all paths for mixed
StringArray/awkward baskets (was UnboundLocalError).
- containers.py: `STLBitSet.__str__`/`__len__` now reference the real
attribute and return an int; `AsBitSet` keeps its bit count and renders
`std::bitset<N>`; simplified always-true `KeysView`/`ValuesView` checks.
- known_forth/__init__.py: fallback typename now takes `classname_decode(...)[0]`
(a string) so the known-forth fast path can match.
- custom.py: `AsBinary` numpy uniformity check now tests the per-entry
sizes themselves (`numpy.unique(counts).size <= 1`), fixing both the
single-entry AssertionError and the arithmetic-progression false pass.
- python.py: a branch named like a function (e.g. `sqrt`) used in call
position now resolves to the function and emits NameConflictWarning,
instead of breaking `sqrt(x)`.
Part of #1646
Assisted-by: ClaudeCode:claude-opus-4-8
Covers: parse_typename("long long*"); pandas object-branch returns data
not NaN; AsBinary single-entry basket and non-uniform rejection; STLBitSet
repr/str/len; AsBitSet typename bit count; expression with a branch named
like a function; AsStridedObjects shared-interpretation has no to_dtype
state leak.
Assisted-by: ClaudeCode:claude-opus-4-8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Part of #1646.
Fixes a batch of verified bugs in the interpretation layer, found during review.
Fixed
_process_array_for_pandasfell off the end returningNonewheninterpretation.awkward_form(...)raisedCannotBeAwkward, solibrary="pd"silently builtpandas.Series(None, index=...)(all-NaN) instead of the object array. Now returns the array.parse_typenameforlong long*/unsigned long long*. These branches were dead (shadowed by the scalarlong/unsigned longbranches, which consume the token) and constructedAsArray({header}, dtype)with the wrong arity. Reordered so the pointer forms are checked first and pass the missingspeedbumpargument.-O). Replaced with araise ValueError(...).AsStridedObjects:__init__deleted from a list while iteratingrange()of its original length (IndexError with mid-list(None, None)header markers from multiple base classes) and could drop the sole member. Rewritten as a filter that drops interior header markers;all_headers_prependedsemantics preserved.basket_arraymutated sharedself._to_dtypebased on a basket's byte offsets — a race / state leak across basket-worker threads. The header dtype is now computed locally, andfinal_arrayderives the output dtype from the actual basket arrays.awkward_form(...)recomputation out of the loop, reusedhas_any_awkward_types, removed a deadcontainer = {}.final_array.outputcould be referenced unbound for mixed StringArray/awkward baskets; now bound on all paths.STLBitSet.__str__referenced a nonexistentself._valuesand__len__returned an ndarray (bothTypeError); fixed to useself._numbytesand return an int.AsBitSetnow keeps its bit count and rendersstd::bitset<N>instead ofstd::bitset<bool>.KeysView is not None/ValuesView is not None(Py2 leftovers).typename = classname_decode(model.__name__)returned a(classname, version)tuple that could never match the string keys inKNOWN_FORTH_DICT; now takes[0].AsBinary(library="np"). The uniformity checknumpy.unique(counts[1:] - counts[:-1]).size == 1failed on single-entry baskets (empty diff → AssertionError) and passed for any arithmetic progression. Changed tonumpy.unique(counts).size <= 1on the per-entry sizes.sqrt) used in call position was rewritten toget('sqrt'), breakingsqrt(x). Names infuncposition that exist in the functions table now resolve to the function and emitNameConflictWarningwhen shadowed by a branch._strided_to_awkwardnow usesdata.reshape(-1)(view) instead ofdata.flatten()(copy) at every recursion level.Skipped
None — all ten findings held up under verification.
Tests
Regression tests added in
tests/test_<N>_*.py(named after the PR number) and the targeted/interpretation-heavy suites pass locally.