Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
8195860
Type stubs for shared.py and abc.py
MatrixEditor Jun 24, 2025
0a49a4e
Added type stubs for _common.py
MatrixEditor Jun 24, 2025
6b77117
Added type stubs for byteorder.py
MatrixEditor Jun 24, 2025
383af66
Created type stubs for context.py
MatrixEditor Jun 24, 2025
a6b3abc
Added type stubs for exception.py and options..py
MatrixEditor Jun 24, 2025
07a1fe9
Updated shortcuts and registry stubs
MatrixEditor Jun 24, 2025
8e06f96
Type hints for model._base.py (Sequence) class
MatrixEditor Jun 24, 2025
ba38ee9
Added model._struct python type hints
MatrixEditor Jun 25, 2025
928ffcd
Added type hints for model._bitfield.
MatrixEditor Jun 25, 2025
af088ae
Added model._template type stubs
MatrixEditor Jun 25, 2025
5707d0a
Added stubs file for fields._base
MatrixEditor Jun 25, 2025
bab081e
Stubs file for fields._mixin
MatrixEditor Jun 25, 2025
d89a1e0
Added stubs for field.common
MatrixEditor Jun 25, 2025
21820a6
Stub files for fields.{crypto,digest,hook}
MatrixEditor Jun 25, 2025
fc6f65d
Stub files for fields.{net,pointer,varint}
MatrixEditor Jun 25, 2025
66848a4
Updated imports and removed unnecessary imports
MatrixEditor Jun 25, 2025
8e2510c
Added stubs for _C module and c.py
MatrixEditor Jun 25, 2025
a8d1655
Updated action to include recent Python versions
MatrixEditor Jun 25, 2025
413a9c5
Update caterpilllar C installatin candidate to reflect 3.14 updates
MatrixEditor Jun 25, 2025
e12f13b
[DEV] (2.5.0 rc) Python type hints
MatrixEditor Jun 25, 2025
9e3cf71
Split _ActionLike into separate protocols
MatrixEditor Jun 26, 2025
1c27b4f
Int and UInt field structs now implement a reasonable __repr__ format
MatrixEditor Jun 26, 2025
a87b71b
New bitfield concept based in int.{from,to}_bytes
MatrixEditor Jun 26, 2025
ceecbc7
Added documentation comments and new option for Bitfield class
MatrixEditor Jun 27, 2025
d7f9752
Updated bitfield behaviour to match concept and added tests
MatrixEditor Jun 27, 2025
391f4cb
Fix inonsistencies with automatically sizing a bitfield-group
MatrixEditor Jun 27, 2025
291d761
Fixed bitfield issues with alignment calculation
MatrixEditor Jun 27, 2025
9d84f02
Removed Manifest file and version to 2.5.0-rc
MatrixEditor Jun 27, 2025
89fc6be
[DEV] (2.5.0 rc) Bitfield revamp
MatrixEditor Jun 27, 2025
6732134
Made root context an attribute of the current context
MatrixEditor Jun 27, 2025
a1dbd6a
Update context stub file to reflect changes
MatrixEditor Jun 27, 2025
d958910
[DEV] (2.5.0-rc) Updates to resolving the root-context
MatrixEditor Jun 27, 2025
f212a36
Update project.toml definition
MatrixEditor Jun 27, 2025
cca9892
Added bitfield library and reference docs
MatrixEditor Jun 28, 2025
1b1cd5f
Add changelog and update library docs
MatrixEditor Jun 29, 2025
b7a1de1
Add caterpillar's protocols to the reference
MatrixEditor Jun 29, 2025
651e042
Update library docs
MatrixEditor Jun 29, 2025
c7c913c
Fix bitfield behaviour for B_GROUP_KEEP Option.
MatrixEditor Jun 29, 2025
2cc030d
Update bitfield tutorial
MatrixEditor Jun 29, 2025
e3f0500
Fix SetAlignment option and option parsing in bitfield
MatrixEditor Jun 29, 2025
08d5892
Update version badges in README
MatrixEditor Jun 29, 2025
79e7ac0
Rename _Switch to _SwitchLike in stub files
MatrixEditor Jun 29, 2025
178e6eb
[DEV] (2.5.0-rc) Documentation Updates
MatrixEditor Jun 29, 2025
9984e9f
[DEV] (2.5.0-rc) Cleanup and Improvements
MatrixEditor Jun 29, 2025
f13ddd5
Fix bitfield alignment on last group
MatrixEditor Jun 29, 2025
a10fff4
2.5.0 Minor Release
MatrixEditor Jun 29, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/python-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ jobs:
fail-fast: true
matrix:
os: ["ubuntu-latest", "windows-latest", "macos-latest"]
python-version: ["3.12"]
python-version: ["3.12", "3.13"]

steps:
- name: Checkout source
Expand Down
56 changes: 56 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Changelog

## [2.5.0] - Minor Release

### Added
- **caterpillar.abc**
- Add `_SupportsBits` protocol.
- Add `_ContainsBits` protocol.
- Add `_SupportsType` protocol.

- **caterpillar.shortcuts**
- Add new shortcuts: `typeof`, `to_struct`, `hasstruct`, `getstruct`, and `sizeof`.

- **caterpillar.shared**
- Add new constants from other modules: `ATTR_BYTEORDER`, `ATTR_TYPE`, `ATTR_BITS`, `ATTR_SIGNED`, `ATTR_TEMPLATE`.

- **caterpillar.context**
- Add context attribute `_root` to point to the root context instance. Internal parent iteration now uses `self.get(...)`.

---

### Removed
- **caterpillar.abc**
- Remove `_Action` protocol; split into `_ActionLike = _SupportsActionUnpack | _SupportsActionPack`.
- Remove `__type__` requirement from `_StructLike`.
- **Breaking:** Remove `_EnumLike` and `_ContextPathStr`.

- **caterpillar.model**
- Remove unused `getformat` function.

- **caterpillar.fields.common**
- Remove unused `__fmt__` function in `Transformer`.

---

### Changed
- **caterpillar.abc**
- Rename `_Switch` protocol to `_SwitchLike`.
- Move `STRUCT_FIELD` to `caterpillar.shared` as `ATTR_STRUCT`; move `hasstruct`, `getstruct`, and `typeof` to `caterpillar.shared`.

- **caterpillar.byteorder**
- Move `BYTEORDER_FIELD` to `caterpillar.shared` as `ATTR_BYTEORDER`.

- **caterpillar.model**
- Update `sizeof` to check if the object implements `_SupportsSize` protocol.
- Renew `Bitfield` concept with enhanced syntax.

- **Documentation**
- Update reference and library docs; improve section numbering.

---

### Fixed
- **caterpillar.model**
- Fix parsing of union objects with an unbound stream.
- Fix field options in Sequences and Structs not being populated when creating fields.
9 changes: 0 additions & 9 deletions MANIFEST.in

This file was deleted.

62 changes: 36 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Caterpillar - 🐛

[![python](https://img.shields.io/python/required-version-toml?tomlFilePath=https%3A%2F%2Fraw.githubusercontent.com%2FMatrixEditor%2Fcaterpillar%2Fmaster%2Fpyproject.toml&logo=python)](https://www.python.org/downloads/)
[![python](https://img.shields.io/badge/Python-3.12+-blue?logo=python&logoColor=yellow)](https://www.python.org/downloads/)
![![Latest Version](https://pypi.org/project/caterpillar-py/)](https://img.shields.io/github/v/release/MatrixEditor/caterpillar.svg?logo=github&label=Latest+Version)
[![Build and Deploy Docs](https://github.com/MatrixEditor/caterpillar/actions/workflows/python-sphinx.yml/badge.svg)](https://github.com/MatrixEditor/caterpillar/actions/workflows/python-sphinx.yml)
[![Run Tests](https://github.com/MatrixEditor/caterpillar/actions/workflows/python-test.yml/badge.svg)](https://github.com/MatrixEditor/caterpillar/actions/workflows/python-test.yml)
![GitHub issues](https://img.shields.io/github/issues/MatrixEditor/caterpillar?logo=github)
Expand All @@ -10,15 +11,8 @@

> [!WARNING]
> This project is still in beta/testing phase. Expect bugs, naming changes and errors while using this
> library. C API Reference is WIP, C extensions are supported since v2.1.0.

> [!NOTE]
> Python 3.14 breaks `with` statements in class definitions since `__annotations__` are added at the end
> of a class definition. Therefore, `Digest` and conditional statements **ARE NOT SUPPORTED** in using the `with` syntax Python 3.14+.
> As of version `2.4.5` the `Digest` class has a counterpart (`DigestField`), which can be used to manually specify a digest without
> the need of a `ẁith` statement.


> library. C API Reference is WIP, C extensions are supported since v2.1.0. The latest stable release
> is available at PyPI.

Caterpillar is a Python 3.12+ library to pack and unpack structurized binary data. It
enhances the capabilities of [Python Struct](https://docs.python.org/3/library/struct.html)
Expand All @@ -27,7 +21,7 @@ options will be added in the future. Documentation is [here >](https://matrixedi

*Caterpillar* is able to:

* Pack and unpack data just from processing Python class definitions (including support for bitfields, c++-like templates and c-like unions!),
* Pack and unpack data just from processing Python class definitions (including support for powerful bitfields, c++-like templates and c-like unions!),
* apply a wide range of data types (with endianess and architecture configuration),
* dynamically adapt structs based on their inheritance layout,
* reduce the used memory space using `__slots__`,
Expand All @@ -36,36 +30,52 @@ options will be added in the future. Documentation is [here >](https://matrixedi
* it helps you to create cleaner and more compact code.
* You can even extend Caterpillar and write your parsing logic in C or C++!!

> [!NOTE]
> Python 3.14 breaks `with` statements in class definitions since `__annotations__` are added at the end
> of a class definition. Therefore, `Digest` and conditional statements **ARE NOT SUPPORTED** using the `with` syntax in Python 3.14+.
> As of version `2.4.5` the `Digest` class has a counterpart (`DigestField`), which can be used to manually specify a digest without
> the need of a `ẁith` statement.


## Give me some code!

```python
from caterpillar.py import *
@bitfield(order=LittleEndian)
class Header:
version : 4 # 4bit integer
valid : 1 # 1bit flag (boolean)
ident : (8, CharFactory) # 8bit char
# automatic alignment to 16bits

@struct(order=LittleEndian)
class Format:
magic: b"ITS MAGIC" # Supports string and byte constants directly
a: uint8 # Primitive data types
b: int32
length: uint8 # String fields with computed lengths
name: String(this.length) # -> you can also use Prefixed(uint8)

# wraps all following fields and creates a new attr
# only for Python <= 3.13
with Md5(name="hash", verify=True):
# Sequences with prefixed, computed lengths
names: CString[uint8::]
magic : b"ITS MAGIC" # Supports string and byte constants directly
header : Header
a : uint8 # Primitive data types
b : int32
length : uint8 # String fields with computed lengths
name : String(this.length) # -> you can also use Prefixed(uint8)

_hash_begin : DigestField.begin("hash", Md5_Algo) # custom actions, e.g. for hashes
names : CString[uint8::] # Sequences with prefixed, computed lengths
hash : Md5_Field("hash", verify=True) = None # automatic hash creation and verification + default value

# Instantiation (keyword-only arguments, magic is auto-inferred):
obj = Format(a=1, b=2, length=3, name="foo", names=["a", "b"])
obj = Format(
header=Header(version=2, valid=True, ident="F"),
a=1,
b=2,
length=3,
name="foo",
names=["a", "b"]
)
# Packing the object, reads as 'PACK obj FROM Format'
# objects of struct classes can be packed right away
blob = pack(obj, Format)
# results in: b'ITS MAGIC\x01\x02\x00\x00\x00\x03foo\x02a\x00b\x00\xf55...
# results in: b'ITS MAGIC0*\x01\x02\x00\x00\x00\x03foo\x02a\x00b\x00)\x9a...'

# Unpacking the binary data, reads as 'UNPACK Format FROM blob'
obj2 = unpack(Format, blob)
assert obj2.hash == obj.hash
```

This library offers extensive functionality beyond basic struct handling. For further details
Expand Down
51 changes: 51 additions & 0 deletions docs/sphinx/source/_static/css/custom.css
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
table.t-stbl {
padding: 0em 1em 0.5em 1em;
flex-direction: column;
display: flex;
}

.t-stbl td.t-hr {
border-top: 1px solid var(--pst-color-surface);
padding: 0
}

.t-stbl span.t-decl-id {
color: var(--pst-color-on-surface);
font-style: italic;
}

.t-stbl span.t-decl-opt {
color: var(--pst-color-primary);
font-style: italic;
padding-left: 0;
font-size: small;
}

.t-stbl td.t-no {
padding-left: 2em;
}

table.t-par-begin {
flex-direction: column;
display: flex;
}

tr.t-bar {
vertical-align: top;
}

tr.t-par>td {
vertical-align: top;
padding: 0 1em 0 0;
}

tr.t-par>td:nth-child(1) {
white-space: nowrap;
text-align: right;
font-weight: bold;
font-family: var(--pst-font-family-monospace-system);
}

tr.t-par>td:nth-child(3) {
width: 100%
}
10 changes: 5 additions & 5 deletions docs/sphinx/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
"sphinx_design",
"breathe",
"c_annotations",
"sphinx_copybutton"
"sphinx_copybutton",
]

templates_path = ["_templates"]
Expand Down Expand Up @@ -63,13 +63,14 @@
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False

refcount_file = '../../../src/capi.dat'
autodoc_member_order = 'bysource'
refcount_file = "../../../src/capi.dat"
autodoc_member_order = "bysource"

# -- Options for HTML output -------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
html_theme = "pydata_sphinx_theme"
html_static_path = ["_static"]
html_css_files = ["css/custom.css"]
html_theme_options = {
"show_nav_level": 4,
"navbar_end": ["navbar-icon-links", "theme-switcher"],
Expand All @@ -78,10 +79,9 @@
"logo": {
"text": f"Caterpillar {version}",
},
"announcement": "https://raw.githubusercontent.com/MatrixEditor/caterpillar/master/docs/sphinx/source/_templates/announcement.html",
}
html_sidebars = {
"installing/index": [],
"installing/index": [],
}

# -- Options for C++ Docs -----------------------------------------------------
Expand Down
107 changes: 106 additions & 1 deletion docs/sphinx/source/development/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,109 @@
Changelog
*********

*Entries will be added in the future.*
*More entries will be added in the future.*

.. _changelog_2.5.0:

[2.5.0] - Minor Release
=======================

This version introduces massive changes due to the addition of stub files. Most of the type hints in the Python
file are ported into several stub files. Additionally, the bitfield concept was completely renewed to be more
flexible and dynamic.

Added
-----

*caterpillar.abc*
^^^^^^^^^^^^^^^^^

- :class:`_SupportsBits` protocol
- :class:`_ContainsBits` protocol
- :class:`_SupportsType` protocol

*caterpillar.shortcuts*
^^^^^^^^^^^^^^^^^^^^^^^

- New shortcuts: :func:`typeof`, :func:`to_struct`, :func:`hasstruct`, :func:`getstruct` and :func:`sizeof`

*caterpillar.shared*
^^^^^^^^^^^^^^^^^^^^

- New constants from other modules: :attr:`ATTR_BYTEORDER`, :attr:`ATTR_TYPE`, :attr:`ATTR_BITS`, :attr:`ATTR_SIGNED`, :attr:`ATTR_TEMPLATE`

*caterpillar.context*
^^^^^^^^^^^^^^^^^^^^^

- New context attribute: `_root` can be set to point to the root context instance. Internally, instead of a for-loop that iterates through parent context instances, a simple :code:`self.get(...)` call is made.

.. raw:: html

<hr>

Removed
-------

*caterpillar.abc*
^^^^^^^^^^^^^^^^^

- ``_Action`` protocol and create two separate Protocols that form::

_ActionLike = _SupportsActionUnpack | _SupportsActionPack

- ``__type__`` requirement from :class:`_StructLike`
- **Breaking:** ``_EnumLike``, ``_ContextPathStr``

*caterpillar.model*
^^^^^^^^^^^^^^^^^^^

- Unused ``getformat`` function

*caterpillar.fields.common*
^^^^^^^^^^^^^^^^^^^^^^^^^^^

- Unused ``__fmt__`` function in :class:`Transformer`

.. raw:: html

<hr>

Changed
-------

*caterpillar.abc*
^^^^^^^^^^^^^^^^^

- Rename ``_Switch`` protocol to :attr:`_SwitchLike`
- Move the following attributes and methods into *caterpillar.shared*: rename ``STRUCT_FIELD`` to :attr:`ATTR_STRUCT`, :func:`hasstruct`, :func:`getstruct` and :func:`typeof`

*caterpillar.byteorder*
^^^^^^^^^^^^^^^^^^^^^^^

- Move ``BYTEORDER_FIELD`` to *caterpillar.shared* as :attr:`ATTR_BYTEORDER`


*caterpillar.model*
^^^^^^^^^^^^^^^^^^^

- :func:`sizeof` now checks if the provided object implements the :class:`_SupportsSize` protocol
- New :class:`Bitfield` concept with enhanced syntax


*Documentation*
^^^^^^^^^^^^^^^

- Update reference and library docs as well as section numbering

.. raw:: html

<hr>

Fixed
-----

*caterpillar.model*
^^^^^^^^^^^^^^^^^^^

- when parsing union objects with an unbound stream object
- field options defined in Sequences and Structs were not populated when creating fields.
1 change: 0 additions & 1 deletion docs/sphinx/source/development/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@ Development
*TODO*

.. toctree::
:numbered:
:maxdepth: 2

roadmap.rst
Expand Down
Loading