Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
04f9d7b
Prepare C API and remove all components
MatrixEditor Jul 5, 2025
1972a58
Changed Ar ch and Endian CAPI compatibility
MatrixEditor Jul 5, 2025
96c9145
Add __set_byteorder__ test
MatrixEditor Jul 5, 2025
549cc4e
Add new CAPI functions to capi.dat
MatrixEditor Jul 5, 2025
f1ec801
[DEV] (2.6.0-rc) CAPI - Endian and Arch
MatrixEditor Jul 5, 2025
2d7d9db
CAPI - c_Option (Flag counterpart)
MatrixEditor Jul 6, 2025
11ebc6d
[DEV] (2.6.0-rc) CAPI - c_Option (Flag counterpart)
MatrixEditor Jul 6, 2025
bb8163c
Bump release to 2.6.0-rc
MatrixEditor Jul 6, 2025
e9cd46b
Revamp CAPI Context implementation
MatrixEditor Jul 19, 2025
d5fd14b
Update documentation and remove imports
MatrixEditor Jul 19, 2025
c2ecb3a
[DEV] (2.6.0-rc) CAPI Context
MatrixEditor Jul 19, 2025
b452c3b
Update license headers
MatrixEditor Jul 19, 2025
20cea9b
Remove module and private API header
MatrixEditor Jul 19, 2025
b6db1d6
Add Atom and LengthInfo to CAPI
MatrixEditor Jul 20, 2025
91844e4
[DEV] (2.6.0-rc) CAPI Atom and LengthInfo
MatrixEditor Jul 20, 2025
377066d
Add new arbitrary length reference struct
MatrixEditor Jul 20, 2025
77bce84
Update CAPI code generation
MatrixEditor Jul 24, 2025
78d10ef
Default operations for Atoms (CAPI)
MatrixEditor Jul 24, 2025
bf8cc92
Utility methods for CAPI context objects
MatrixEditor Jul 24, 2025
e5f91d2
Fix LengthInfo CAPI initialization
MatrixEditor Jul 24, 2025
0bd0a04
Builtin atoms (CAPI)
MatrixEditor Jul 24, 2025
5121141
Add tests for new builtin atoms
MatrixEditor Jul 24, 2025
75a03a5
Fix PyObject head in struct definitions
MatrixEditor Jul 24, 2025
63225f2
C API builtin offset atom
MatrixEditor Jul 27, 2025
6e77346
Add tests for conditional atom
MatrixEditor Jul 27, 2025
f22ed87
[DEV] (2.6.0-rc) CAPI Builtin-Atoms
MatrixEditor Jul 30, 2025
013c3af
[DEV] (2.6.0-rc) New Struct: LengthFor / AsLengthRef
MatrixEditor Jul 30, 2025
f55b0b8
Significantly reduced overhead in Field class
MatrixEditor Jul 30, 2025
b2c6a9d
Update examples (checksum and pointer)
MatrixEditor Aug 1, 2025
c1c1cfd
Update Field implementation and reduced runtime on initialization
MatrixEditor Aug 1, 2025
1e9dc78
Add new method to _ContextLike protocol
MatrixEditor Aug 1, 2025
91de439
Update stubs to include little endian order identifier
MatrixEditor Aug 1, 2025
ca44aa8
Improvde PyStructFormattedField class to support CAPI atoms
MatrixEditor Aug 1, 2025
df16a62
Update Sequence implementation and reduced time when packing and unpa…
MatrixEditor Aug 1, 2025
1774e9d
Remove old refcounts file
MatrixEditor Aug 1, 2025
e3133e9
Update documentation tutorial
MatrixEditor Aug 2, 2025
851635b
Update CAPI reference
MatrixEditor Aug 2, 2025
33ad964
Update library docs
MatrixEditor Aug 2, 2025
4bbf070
Add changelog to docs
MatrixEditor Aug 13, 2025
bf92497
[DEV] (2.6.0-rc) Documentation updates
MatrixEditor Aug 13, 2025
89fbbdf
BUmp version to 2.6.0
MatrixEditor Aug 13, 2025
90f7080
Update logic in PyStructFormattedField to support greedy length
MatrixEditor Aug 13, 2025
2f93a93
Fix MSVC errors with double ;; in struct definitions
MatrixEditor Aug 13, 2025
709902c
Update README
MatrixEditor Aug 13, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ venv-matrix/
# C extensions
*.so
include/caterpillar/caterpillar.h
src/ccaterpillar/caterpillarapi.c
src/ccaterpillar/module.c
src/ccaterpillar/private.h

# Distribution / packaging
.Python
Expand All @@ -29,6 +32,8 @@ share/python-wheels/
*.egg
MANIFEST
bench/
pypi/
caterpillarapi.h

# PyInstaller
# Usually these files are written by a python script from a template
Expand Down Expand Up @@ -164,4 +169,6 @@ cython_debug/
#.idea/

# VSCode
.vscode/
.vscode/
Makefile
.clangd
54 changes: 54 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,59 @@
# Changelog

## [2.6.0] - Minor Release

### Added
- **caterpillar.fields._base**
- Add support for non-context lambda switch fields.

- **caterpillar.fields.common**
- Add compatibility support for CAPI atoms in `Int`, `UInt` and `PyStructFormattedField`.

- **caterpillar.options**
- Add custom `Flag.__eq__` implementation to support equality check with `c_Option` objects.

- **caterpillar.abc**
- Add new typing-only `_OptionLike` protocol.
- Add missing `_SupportsType` protocol to the stubs file.
- Add new method `get` to the `_ContextLike` protocol.

- **caterpillar.context**
- Add new option `O_CONTEXT_FACTORY` that controls the global context type.
Value must be a method or another type implementing the `_ContextLike` protocol.
- Add new global context path: `root` (exported as `G` in shortcuts).

- **caterpillar.shortcuts**
- Add new shortcuts `C` for `ctx`, `P` for `parent` and `G` for the `root` context as `ContextPath` objects.

- **CAPI**
- New index assignment system when generating CAPI code — a running number is now applied instead of a hard-coded index.
- Add complete Context implementation in C (`c_Context`) that conforms to the `_ContextLike` protocol.
- Add `Atom` for C-based struct-like classes (previously known as `catom`).
- Add native support for `__bits__` in `Atom`.
- Add special class `LengthInfo` for packing or unpacking multiple objects.
- New builtin atoms (CAPI): `Repeated`, `Conditional` and `Switch`.
- Add new shared objects and exception types to the native implementation (`Cp_ContextFactory`, `Cp_ArrayFactory`, `CpExc_Stop` and `Cp_DefaultOption`).

---

### Changed
- **caterpillar.fields._base**
- Rework `Field` implementation to reduce overhead when packing and unpacking elements.
- Use pre-computed states instead of calculating everything on the fly; states will be adjusted automatically when setting new values (via `@property` attributes).

---

### Fixed
- **caterpillar.fields.common**
- Fix issue in `Prefixed` that occurred when the internal struct packs a sequence of elements.

---

### Removed
- **CAPI**
- Remove old CAPI and completely revamp the CAPI concept to make it compatible with the Python equivalent.


## [2.5.1] - Hot-Fix Release

### Fixed
Expand Down
50 changes: 15 additions & 35 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -17,42 +17,17 @@ python_add_library(
MODULE

src/ccaterpillar/arch.c
src/ccaterpillar/atomobj.c
src/ccaterpillar/context.c
src/ccaterpillar/option.c
src/ccaterpillar/module.c
src/ccaterpillar/context.c
src/ccaterpillar/state.c
src/ccaterpillar/struct.c
src/ccaterpillar/layer.c
src/ccaterpillar/default.c

src/ccaterpillar/parsing_typeof.c
src/ccaterpillar/parsing_sizeof.c
src/ccaterpillar/parsing_pack.c
src/ccaterpillar/parsing_unpack.c

src/ccaterpillar/atomimpl/int.c
src/ccaterpillar/atomimpl/float.c
src/ccaterpillar/atomimpl/bool.c
src/ccaterpillar/atomimpl/char.c
src/ccaterpillar/atomimpl/pad.c
src/ccaterpillar/atomimpl/string.c
src/ccaterpillar/atomimpl/const.c
src/ccaterpillar/atomimpl/bytes.c
src/ccaterpillar/atomimpl/pstring.c
src/ccaterpillar/atomimpl/enum.c
src/ccaterpillar/atomimpl/varint.c
src/ccaterpillar/atomimpl/computed.c
src/ccaterpillar/atomimpl/lazy.c
src/ccaterpillar/atomimpl/cstring.c

src/ccaterpillar/atomimpl/builtins/builtin.c
src/ccaterpillar/atomimpl/builtins/repeated.c
src/ccaterpillar/atomimpl/builtins/condition.c
src/ccaterpillar/atomimpl/builtins/switch.c
src/ccaterpillar/atomimpl/builtins/atoffset.c
src/ccaterpillar/atomimpl/builtins/primitive.c
src/ccaterpillar/atom.c
src/ccaterpillar/lengthinfo.c
src/ccaterpillar/shared.c
src/ccaterpillar/atoms/builtin/builtin.c
src/ccaterpillar/atoms/builtin/repeated.c
src/ccaterpillar/atoms/builtin/switch.c
src/ccaterpillar/atoms/builtin/conditional.c
src/ccaterpillar/atoms/builtin/offset.c

WITH_SOABI
)
Expand All @@ -65,15 +40,20 @@ message(STATUS "Python ${Python_VERSION_MAJOR}.${Python_VERSION_MINOR}")
set (CP_GENAPI_SCRIPT ${CMAKE_CURRENT_SOURCE_DIR}/src/code_gen/genapi.py)
set (CP_CAPI_HEADER ${CMAKE_CURRENT_SOURCE_DIR}/src/caterpillar/include/caterpillar/caterpillarapi.h.in)
set (CP_CAPI_CSRC ${CMAKE_CURRENT_SOURCE_DIR}/src/ccaterpillar/caterpillarapi.c.in)
set (CP_CAPI_MOD_CSRC ${CMAKE_CURRENT_SOURCE_DIR}/src/ccaterpillar/module.c.in)
set (CP_CAPI_PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/src/ccaterpillar/private.h.in)

set (CP_CAPI_HEADER_OUT ${CMAKE_CURRENT_SOURCE_DIR}/src/caterpillar/include/caterpillar/caterpillarapi.h)
set (CP_CAPI_CSRC_OUT ${CMAKE_CURRENT_SOURCE_DIR}/src/ccaterpillar/caterpillarapi.c)
set (CP_CAPI_MOD_CSRC_OUT ${CMAKE_CURRENT_SOURCE_DIR}/src/ccaterpillar/module.c)
set (CP_CAPI_PRIVATE_OUT ${CMAKE_CURRENT_SOURCE_DIR}/src/ccaterpillar/private.h)

add_custom_target(
genapi ALL
# execute python ./src/code_gen/genapi.py ./src/caterpillar/include/caterpillar/caterpillarapi.h.in ./src/ccaterpillar/caterpillarapi.c.in
# in the root directory
COMMAND ${PYTHON_EXECUTABLE} ${CP_GENAPI_SCRIPT} ${CP_CAPI_HEADER} ${CP_CAPI_CSRC}
BYPRODUCTS ${CP_CAPI_HEADER_OUT} ${CP_CAPI_CSRC_OUT}
COMMAND ${PYTHON_EXECUTABLE} ${CP_GENAPI_SCRIPT} ${CP_CAPI_HEADER} ${CP_CAPI_CSRC} ${CP_CAPI_MOD_CSRC} ${CP_CAPI_PRIVATE}
BYPRODUCTS ${CP_CAPI_HEADER_OUT} ${CP_CAPI_CSRC_OUT} ${CP_CAPI_MOD_CSRC_OUT} ${CP_CAPI_PRIVATE_OUT}
COMMENT "Generating Public Caterpillar API"
)

Expand Down
8 changes: 1 addition & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,6 @@
![GitHub License](https://img.shields.io/github/license/MatrixEditor/caterpillar?logo=github)



> [!WARNING]
> This project is still in beta/testing phase. Expect bugs, naming changes and errors while using this
> library. C API Reference is WIP, C extensions are supported since v2.1.0. The latest stable release
> is available at PyPI.

Caterpillar is a Python 3.12+ library to pack and unpack structurized binary data. It
enhances the capabilities of [Python Struct](https://docs.python.org/3/library/struct.html)
by enabling direct class declaration. More information about the different configuration
Expand Down Expand Up @@ -100,7 +94,7 @@ pip install caterpillar-py
pip install "caterpillar[all]@git+https://github.com/MatrixEditor/caterpillar"
```

### C-extension installation
### Installation + C-extension

```bash
pip install "caterpillar[all]@git+https://github.com/MatrixEditor/caterpillar/#subdirectory=src/ccaterpillar"
Expand Down
2 changes: 1 addition & 1 deletion docs/doxygen/Doxyfile
Original file line number Diff line number Diff line change
Expand Up @@ -949,7 +949,7 @@ WARN_LOGFILE =
# spaces. See also FILE_PATTERNS and EXTENSION_MAPPING
# Note: If this tag is empty the current directory is searched.

INPUT = ../../include
INPUT = ../../src/caterpillar/include

# This tag can be used to specify the character encoding of the source files
# that doxygen parses. Internally doxygen uses the UTF-8 encoding. Doxygen uses
Expand Down
2 changes: 1 addition & 1 deletion docs/sphinx/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@

project = "Caterpillar"
author = caterpillar.__author__
copyright = f"2024, {author}"
copyright = f"2025, {author}"
release = version = caterpillar.__version__

# -- General configuration ---------------------------------------------------
Expand Down
99 changes: 99 additions & 0 deletions docs/sphinx/source/development/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,105 @@ Changelog

*More entries will be added in the future.*

.. _changelog_2.6.0:

[2.6.0] - Minor Release
=======================

Added
-----

*caterpillar.fields._base*
^^^^^^^^^^^^^^^^^^^^^^^^^^

- Add support for non-context lambda switch fields


*caterpillar.fields.common*
^^^^^^^^^^^^^^^^^^^^^^^^^^^

- Add compatibility support for CAPI atoms in ``Int``, ``UInt`` and ``PyStructFormattedField``

*caterpillar.options*
^^^^^^^^^^^^^^^^^^^^^

- Add custom ``Flag.__eq__`` implementation to support equality check with ``c_Option`` objects


*caterpillar.abc*
^^^^^^^^^^^^^^^^^

- Add new typing-only `_OptionLike` protocol
- Add missing `_SupportsType` protocol to the stubs file
- Add new method `get` to the `_ContextLike` protocol


*caterpillar.context*
^^^^^^^^^^^^^^^^^^^^^

- Add new option :py:attr:`~caterpillar.context.O_CONTEXT_FACTORY` that controls
the global context type. Value must be a method or another type implementing
the :class:`_ContextLike` protocol
- Add new global context path: :py:attr:`~caterpillar.context.root` (exported as ``G`` in shortcuts)

*caterpillar.shortcuts*
^^^^^^^^^^^^^^^^^^^^^^^

- Add new shortcuts ``C`` for :py:attr:`~caterpillar.context.ctx`, ``P`` for
:py:attr:`~caterpillar.context.parent` and ``G`` for the
:py:attr:`~caterpillar.context.root` context as
:class:`~caterpillar.context.ContextPath` objects


*CAPI*
^^^^^^

- New index assignment system when generating CAPI code. A running number is now
applied instead of a hard coded index.
- Add *complete* Context implementation in C (:class:`c_Context`) that conforms
to the :class:`_ContextLike` protocol.
- Add :class:`Atom` for C-based struct-like classes. This class was previously
known as :class:`catom`
- Add native support for `__bits__` in :class:`Atom`
- Add special class :class:`LengthInfo` for packing or unpacking multiple objects
- New builtin atoms (CAPI): :class:`Repeated`, :class:`Conditional` and :class:`Switch`
- Add new shared objects and exception types to the native implementation
(:c:var:`Cp_ContextFactory`, :c:var:`Cp_ArrayFactory`, :c:var:`CpExc_Stop` and
:c:var:`Cp_DefaultOption`)


Changed
-------

*caterpillar.fields._base*
^^^^^^^^^^^^^^^^^^^^^^^^^^

- Rework :class:`~caterpillar.fields._base.Field` implementation to reduce
overhead when packing and unpacking elements
- Use pre-computed states instead of calculating everything on-the-fly. States
will be adjusted when setting new values (automatically updated through
:code:`@property` attributes)


Fixed
-----

*caterpillar.fields.common*
^^^^^^^^^^^^^^^^^^^^^^^^^^^

+ Fix issue in Prefixed that occurred when the internal struct packs a sequence
of elements

Removed
-------

*CAPI*
^^^^^^

- Remove old CAPI and completely revamp the CAPI concept to make it compatible
to the Python equivalent.


.. _changelog_2.5.1:

[2.5.1] - Hot-Fix Release
Expand Down
Loading