Skip to content

[DEV] (2.5.0 rc) Bitfield revamp#34

Merged
MatrixEditor merged 8 commits into
dev/2.5.0-rcfrom
dev/2.5.0-rc_bitfield
Jun 27, 2025
Merged

[DEV] (2.5.0 rc) Bitfield revamp#34
MatrixEditor merged 8 commits into
dev/2.5.0-rcfrom
dev/2.5.0-rc_bitfield

Conversation

@MatrixEditor

@MatrixEditor MatrixEditor commented Jun 26, 2025

Copy link
Copy Markdown
Owner

New Concept

Each Bitfield instance maintains a sequence of bitfield groups, where each group contains a collection of sized fields. A bitfield group may consist of either multiple entries (i.e., any types that can be converted to an integral type) or a single _StructLike object. For example, consider the following bitfield definition:

  @bitfield
  class Format:
      a1 : 1
      a2 : 2
      _  : 0
      b1 : char
      c1 : uint32

This Bitfield definition will generate three distinct bitfield groups (labeled here as groups a, b, and c). By default, bitfields use 8-bit alignment, leading to the following layout:

      Group      Pos       Bits
      a          0x00      8
      b          0x01      8
      c          0x02      32

Internally, only the first group requires special bit-level parsing. The remaining groups (b and c) are treated as standard structures since they span full bytes or words without sub-byte alignment. This dynamic grouping mechanism allows leveraging full struct-like class definitions within bitfields.

Example Code:

@bitfield
class Format:
    # bitwise fields can be defined as before
    a1: 1
    a2: 2
    # the underlying integer size will be adjusted
    # automatically. Default alignment is 8bits.
    a3: 12

    # same as before, adds padding until the end based
    # on the current alignment (8bits -> aligned to 16bits,
    # since 12+2+1 is already bigger than 8)
    _: 0 

    # The new concept allows us to specify arbitrary structs
    # or fields.
    b1: Bytes(10)
    c1: CString(...)

    # The updated concept introduces a new extended syntax 
    # based on tuples
    d1: 10
    # ends the current group (implied) and sets the alignment
    # to 24bits for all following groups.
    _: (0, EndGroup, SetAlignment(24))

    # 5bits of 32, because the current alignment will be replaced
    # with the size of the current struct (here: uint32)
    e1: 5 - uint32 | B_OVERWRITE_ALIGNMENT

    # Actions are supported too

Additional changes introduced:

caterpillar.abc
  • _ActionLike was removed and split into two separate protocols: _SupportsActionUnpack and _SupportsActionPack
  • Int and UInt field structs now implement a reasonable __repr__ format

---

abc.py:
+ _ActionLike was removed and splitted into two separate protocols:_SupportsActionUnpack and _SupportsActionPack
---

+ BitField was renamed to Bitfield
+ Each Bitfield now stores a list of groups, which either represent a field or a list of entries
+ bitfield_example was updated
---

+ New option: B_NO_AUTO_BOOL to disable automatically assigning fields with one bit width a boolean factory
---

+ Updated imports in __init__
+ Added two new value factories: CharFactory and EnumFactory
+ Bitfields now corrently populate documentation types if S_REPLACE_TYPES is active
@MatrixEditor MatrixEditor marked this pull request as ready for review June 27, 2025 08:07
@MatrixEditor MatrixEditor merged commit 89fc6be into dev/2.5.0-rc Jun 27, 2025
12 checks passed
@MatrixEditor MatrixEditor deleted the dev/2.5.0-rc_bitfield branch June 27, 2025 13:33
MatrixEditor added a commit that referenced this pull request Jun 29, 2025
---

## Changes made

The code of all modules were refactored to include as little type hints as possible now. All typing related information should be taken from `.pyi` files.

##### `caterpillar.abc`

- Removed `_Action` and split into two separate Protocols `_ActionLike` := `_SupportsActionUnpack` | `_SupportsActionPack`
- Renamed `_Switch` to `_SwitchLike`
- Removed `_EnumLike`
- Added two new protocols: '_SupportsBits' and '_ContainsBits'
- The following attributes and methods were moved into [caterpillar.shared](#caterpillarshared): `STRUCT_FIELD` -> `ATTR_STRUCT`, `hasstruct`, `getstruct` and `typeof`
-  Removed unused '_ContextPathStr'
- Removed `__type__()` requirement from '_StructLike'
- Added new protocol: `_SupportsType`

##### `caterpillar.byteorder`

- Moved 'BYTEORDER_FIELD' to [caterpillar.shared](#caterpillarshared) as `ATTR_BYTEORDER`

##### `caterpillar.shortcuts`

- Shortcuts now include `typeof`, `to_struct`, `hasstruct`, `getstruct` and `sizeof`

##### `caterpillar.shared`

- New constants moved from other modules: `ATTR_BYTEORDER`, `ATTR_TYPE`, `ATTR_BITS`, `ATTR_SIGNED`, `ATTR_TEMPLATE`

##### `caterpillar.context`

- New context attribute: '_root' can be set to point to the root context instance. Internally, instead of a for-loop that iterates through parent context instances, a simple self.get(...) call is made. (see #35)

##### `caterpillar.model._base`

- Fixed an issue when parsing union objects with an unbound stream object
- Fixed an issue where field options defined in Sequences were not populated when creating fields.

##### `caterpillar.model._struct`

- `sizeof` now checks if the provided object implements the `_SupportsSize` protocol

##### `caterpillar.model._bitfield`

- **Completely reworked the bitfield mechanism to make it even more powerful. For details refer to #34.**
- Fixed an issue where field options defined in BitFields were not populated when creating fields.
- Moved `BITS_ATTR` and `SIGNED_ATTR` into [caterpillar.shared](#caterpillarshared)
- Removed unused `getformat` function

##### `caterpillar.fields.common`

- `Transformer`: removed `__fmt__` method

##### `caterpillar.fields.compression`

- Updated public compression methods to use lazy imports

#### Documentation

- Updated library documentation to reflect changes made to function signatures
- Updated the reference to explicitly state the Protocols defined in `caterpillar.abc`
- Created a change log to reflect changes made up to v2.5.0

## Associated Issues and Pull Requests

- #30: Stub files for Caterpillar allowing to use generics and more detailed return types
- #27: Python3.14 does NOT break this project. However, some features (with-statements) can't be used
- #34: Introducing a new Bitfield concept to make it even more flexible
- #35: Optimized resolving the root context instance
- #37: Various small fixes related to the documentation
- #24: Bitfield docs
- #33: General documentation updates
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant