[2.7.0] Dynamic Endian

Merge pull request #52 from MatrixEditor/dev/dyn-endian

Author: MatrixEditor

.github/workflows/python-publish.yml

@@ -31,9 +31,15 @@ jobs:
           pip install build
 
       - name: Build Package
+        # This is a small hach since the 'caterpillar' project is already
+        # published on PyPi and we can't use that name.
         run: |
-          python -m build .
+          python3 -m build pypi || true
+          sed -i 's/\.\.\/src/\.\/src/g' pypi/pyproject.toml
+          python3 -m build pypi
 
       - name: Publish package to Prod PyPi
-        # if: github.event_name == 'release' && startsWith(github.ref, 'refs/tags') && !github.event.release.prerelease
-        uses: pypa/gh-action-pypi-publish@release/v1
+        if: github.event_name == 'release' && startsWith(github.ref, 'refs/tags') && !github.event.release.prerelease
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          package-dir: pypi/dist

CHANGELOG.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## [2.7.0] - Dynamic Byteorder
+
+###  Added
+
+- New concept: *dynamic* byte order (explained in documentation "Dynamic Byte Order")
+
+### Fixed
+
+**caterpillar.field.net**
+
+- Fixed parsing of :class:`~caterpillar.fields.net.MACAddress`
+- Fixed mixed endian in a struct, sequence or bit-field
+
+
+
 ## [2.6.3] - Patch Release
 
 ### Fixed

CMakeLists.txt

@@ -1,17 +1,17 @@
 cmake_minimum_required(VERSION 3.15...3.26)
 project(${SKBUILD_PROJECT_NAME} LANGUAGES C)
 
+# TODO: document this change
+if (DEFINED ENV{CP_ENABLE_NATIVE} OR DEFINED CP_ENABLE_NATIVE)
+message(STATUS "Building Caterpillar C extension")
+
 find_package(
   Python
   COMPONENTS Interpreter Development.Module
   REQUIRED)
 
 add_compile_definitions(_CPMODULE)
 
-# TODO: document this change
-if (DEFINED ENV{CP_ENABLE_NATIVE} OR DEFINED CP_ENABLE_NATIVE)
-message(STATUS "Building Caterpillar C extension")
-
 python_add_library(
   _C
   MODULE

README.md

@@ -22,6 +22,7 @@ options will be added in the future. Documentation is [here >](https://matrixedi
 * allowing you to place conditional statements into class definitions,
 * insert proper types into the class definition to support documentation and
 * it helps you to create cleaner and more compact code.
+* There is also a feature that lets you dynamically change the endian within a struct!
 * You can even extend Caterpillar and write your parsing logic in C or C++!!
 
 > [!NOTE]
@@ -34,6 +35,8 @@ options will be added in the future. Documentation is [here >](https://matrixedi
 ## Give me some code!
 
 ```python
+from caterpillar.py import *
+
 @bitfield(order=LittleEndian)
 class Header:
     version : 4                # 4bit integer
@@ -46,7 +49,7 @@ class Format:
     magic  : b"ITS MAGIC"        # Supports string and byte constants directly
     header : Header
     a      : uint8               # Primitive data types
-    b      : int32
+    b      : Dynamic + int32     # dynamic endian based on global config
     length : uint8               # String fields with computed lengths
     name   : String(this.length) #  -> you can also use Prefixed(uint8)
 
@@ -70,6 +73,9 @@ blob = pack(obj, Format)
 
 # Unpacking the binary data, reads as 'UNPACK Format FROM blob'
 obj2 = unpack(Format, blob)
+
+# to pack with a different endian for field 'b', use _order
+data = pack(obj, Format, _order=BigEndian)
 ```
 
 This library offers extensive functionality beyond basic struct handling. For further details

docs/sphinx/source/development/changelog.rst

@@ -6,6 +6,26 @@ Changelog
 
 *More entries will be added in the future.*
 
+.. _changelog_2.7.0:
+
+[2.7.0] - Dynamic Byteorder
+===========================
+
+Added
+-----
+
+- New concept: *dynamic* byte order (explained in :ref:`tutorial-dyn_byteorder`)
+
+Fixed
+-----
+
+*caterpillar.field.net*
+^^^^^^^^^^^^^^^^^^^^^^^
+
+- Fixed parsing of :class:`~caterpillar.fields.net.MACAddress`
+- Fixed mixed endian in a struct, sequence or bit-field
+
+
 .. _changelog_2.6.3:
 
 [2.6.3] - Patch
@@ -60,8 +80,8 @@ Fixed
 
 .. _changelog_2.6.0:
 
-[2.6.0] - Minor Release
-=======================
+[2.6.0] - CAPI Compatibility
+============================
 
 Added
 -----
@@ -175,8 +195,8 @@ Fixed
 
 .. _changelog_2.5.0:
 
-[2.5.0] - Minor Release
-=======================
+[2.5.0] - Stub Files
+====================
 
 This version introduces massive changes due to the addition of stub files. Most of the type hints in the Python
 file are ported into several stub files. Additionally, the bitfield concept was completely renewed to be more

docs/sphinx/source/library/byteorder.rst

@@ -12,7 +12,11 @@ Byteorder
 .. autoclass:: ByteOrder
     :members:
 
+.. autoclass:: DynByteOrder
+    :members:
+
 .. autofunction:: byteorder(obj, default: Optional[ByteOrder] = None) -> ByteOrder
+.. autofunction:: byteorder_is_little
 
 Standard Byteorder Instances
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -27,6 +31,7 @@ Standard Byteorder Instances
 
 .. autoattribute:: caterpillar.byteorder.SysNative
 
+.. autoattribute:: caterpillar.byteorder.Dynamic
 
 
 

docs/sphinx/source/tutorial/advanced/byteorder.rst

@@ -0,0 +1,134 @@
+.. _tutorial-dyn_byteorder:
+
+Dynamic Byte Order
+==================
+
+In addition to traditional byte order types, *caterpillar* supports a dynamic byte order
+based on the current pack or unpack context.
+
+.. versionadded:: 2.6.4
+    This feature is available in starting from version ``2.6.4``
+
+
+There are various use-cases that require a struct to handle both big-endian and
+little-endian. In order to reduce the amount of code for these structs, *caterpillar*
+introduces a special byte order type: :class:`~caterpillar.byteorder.DynByteOrder`. It
+supports two different configuration levels:
+
+1. struct-wide: endianess is configured when calling :func:`~caterpillar.model.pack` or
+   :func:`~caterpillar.model.unpack`.
+2. field-level: byte order is applied per field
+
+Each of these configuration levels support various methods of selecting the target endian:
+
+* global configuration using an additional keyword argument in :func:`~caterpillar.model.pack` or
+  :func:`~caterpillar.model.unpack`.
+* context-key: configuration based on a value within the current context
+* custom function: endian is derived from a custom function
+
+Example: struct-wide dynamic byte order
+---------------------------------------
+
+Let's consider the following struct definition. The dynamic endian configuration will be applied
+to all fields that haven't got an endian already set.
+
+.. code-block:: python
+    :linenos:
+
+    from caterpillar.context import CTX_ORDER
+
+    @struct(order=Dynamic)
+    class Format:
+        a: uint16           # litte endian or big endian is decided using
+        b: uint32           # a global context variable
+
+    config = {CTX_ORDER: BigEndian}
+    obj = Format(a=0x1234, b=0x56789ABC)
+
+    # pack the object using BigEndian
+    pack(obj, **config)
+
+    # now pack with little endian
+    config[CTX_ORDER] = LittleEndian
+    pack(obj, **config)
+
+Here we pass an additional global context variable named :attr:`~caterpillar.context.CTX_ORDER` (``"_order"``)
+to the packing and unpacking process. The dynamic endian will automatically infer the order based on this
+global variable.
+
+Example: field-level dynamic byte order
+---------------------------------------
+
+The same concept as shown above can be applied to single fields too. By default, the endian to use must be
+given as a global context variable as described before.
+
+.. code-block:: python
+    :linenos:
+
+    from caterpillar.context import CTX_ORDER
+
+    @struct
+    class Format:
+        a: uint16
+        b: Dynamic + uint32   # only this field will be affected
+
+    # packing and unpacking is the same as in the previous example
+
+
+Example: context key reference
+------------------------------
+
+Sometimes format specifications use a special field indicating whether all following
+fields are using big or little endian. To implement this kind of endian selection, a
+so called *context key* can be specified, which can take one of the following forms:
+
+*   direct reference: just a string reference
+
+    .. code-block:: python
+
+        # ...
+        spec: uint8
+        number: Dynamic(key="spec")
+        # ...
+
+*   context-lambda: a function that takes the current context as its first parameter and
+    returns the target endian configuration value.
+
+    .. code-block:: python
+
+        # ...
+        spec: uint8
+        number: Dynamic(key=this.spec)
+        # ...
+
+The target endianess is decided based on the context value:
+
+* :code:`str` will be applied directly as the format character
+* any object storing a :code:`ch` string value
+* any other object is converted to a :code:`bool` and the following mapping is applied:
+
+  * :code:`True`: :attr:`~caterpillar.byteorder.LittleEndian`
+  * :code:`False`: :attr:`~caterpillar.byteorder.BigEndian`
+
+As an example, consider the following definition:
+
+.. code-block:: python
+    :linenos:
+
+    @struct(order=BigEndian)
+    class Format:
+        spec: uint8 = 0
+        a: DynByteOrder(key=this.spec) + uint16
+        # alternatively
+        # a: Dynamic(this.spec) + uint16
+        b: uint32
+
+    # packing and unpacking does not require the extra endian value
+    obj = Format(spec=0, a=0x1234, b=0x56789ABC)
+    # 0 -> False, results in BigEndian
+    data_be = pack(obj)
+
+    # 1 -> True, results in LittleEndian
+    obj.spec = 1
+    data_le = pack(obj)
+

docs/sphinx/source/tutorial/advanced/index.rst

@@ -21,6 +21,7 @@ create and manipulate your own custom struct classes in Python.
     :maxdepth: 2
 
     operators
+    byteorder
     pointers
     chaining
     conditional

docs/sphinx/source/tutorial/first_steps/configuration.rst

@@ -32,6 +32,12 @@ In both cases, the :code:`pixels_per_unit_x` and :code:`pixels_per_unit_y` field
 so they will be interpreted using big-endian encoding. The :code:`unit` field is only 1 byte, so
 endianess doesn't affect it.
 
+.. tip::
+    It is also possible to mixup various byte orders within the same struct by applying the
+    endianess directly to the field types. Additionally, there is a special byte order type
+    that allows dynamic selection of the endian during runtime. See the
+    :ref:`advanced configuration <tutorial-dyn_byteorder>` section for more details.
+
 In addition to configuring the endianess, you can also specify the **architecture** associated
 with the struct using the :class:`~caterpillar.byteorder.Arch` class with the :code:`arch` keyword.
 

examples/comparison/comparison_1_caterpillar.py

@@ -4,6 +4,7 @@
 
 # The __slots__ options does not affect the packing or
 # unpacking process.
+# from caterpillar import options as opt
 # opt.set_struct_flags(opt.S_SLOTS, opt.S_REPLACE_TYPES)
 
 # The following lines will reduce the time around 10% for