v2.8.0: Extended syntax & typing fixes

Merge pull request #55 from MatrixEditor/dev/2.8.0-extended-syntax

Author: MatrixEditor

CHANGELOG.md

@@ -1,5 +1,40 @@
 # Changelog
 
+## [2.8.0] - Extended Syntax
+
+### Added
+
+- New concept: *extended* syntax featuring typing compliance; introduces shortcut `f` to annotate fields and `Invisible()` to hide fields from the constructor
+- `struct_factory.mixin` and `bitfield_factory.mixin` both provide default function wrappers for packing and unpacking data as well as typing fixes for operating on types directly.
+- New `parentctx` context path
+- Dynamic endian is now used by default (i. e. can be changed via order parameter in pack() or unpack())
+- New generic `Timestamp` class
+- A number of new test cases
+- New module: `caterpillar.types`: defines default types that can be used as annotations within struct definitions
+- `O_DEFAULT_ENDIAN` as a global option to set a global default byteorder
+- `O_DEFAULT_ARCH` same concept for arch-like objects
+- new 'strict' option to `Enum` struct
+- `ATTR_PACK` and `ATTR_UNPACK` to *caterpillar.shared*
+- `'order'` and `'arch'` options to pack() and unpack(), which temporarily change the global endianess or arch (compatible with Dynamic byteorder)
+
+### Changes
+
+- Merge all stub files with their corresponding Python files. All typing is now inline.
+- `padding` struct now has its own dedicated class
+- sizeof() now always returns an integer
+- ContextPath: dropped support for the call action
+- Field.get_name() now always returns a string
+- Rename `ssize` and `size` to `pssize` and `psize` in caterpillar.fields exports
+- options.get_flags() always returns a list
+
+### Fixes
+
+- IntFlag support for `Enum` structs
+- Fix incorrect Sha1 Digest length
+- Add missing global exports in `caterpillar.py`
+- Bitfield, Struct and Sequence now respect fields with already configured byteorder
+- Prefixed struct now does not require `as_field=True` when calling pack() or unpack()
+
 ## [2.7.0] - Dynamic Byteorder
 
 ###  Added

README.md

@@ -23,41 +23,83 @@ options will be added in the future. Documentation is [here >](https://matrixedi
 * insert proper types into the class definition to support documentation and
 * it helps you to create cleaner and more compact code.
 * There is also a feature that lets you dynamically change the endian within a struct!
-* You can even extend Caterpillar and write your parsing logic in C or C++!!
+* You can even extend Caterpillar and write your parsing logic in C or C++
+* All struct definitions can be typing compliant!!! (tested with pyright)
 
-> [!NOTE]
-> Python 3.14 breaks `with` statements in class definitions since `__annotations__` are added at the end
-> of a class definition. Therefore, `Digest` and conditional statements **ARE NOT SUPPORTED** using the `with` syntax in Python 3.14+.
-> As of version `2.4.5` the `Digest` class has a counterpart (`DigestField`), which can be used to manually specify a digest without
-> the need of a `ẁith` statement.
+## Give me some code!
 
+*The following code is typing compliant, meaning your static type checker won't*
+*scream at you when developing with this code*.
 
-## Give me some code!
+<details>
+<summary><i>If you want to check out the default syntax, open this block.</i></summary>
 
 ```python
 from caterpillar.py import *
+from caterpillar.types import *
 
 @bitfield(order=LittleEndian)
 class Header:
-    version : 4                # 4bit integer
-    valid   : 1                # 1bit flag (boolean)
-    ident   : (8, CharFactory) # 8bit char
+    version : 4                   # 4bit integer
+    valid   : 1                   # 1bit flag (boolean)
+    ident   : (8, CharFactory)    # 8bit char
     # automatic alignment to 16bits
 
-@struct(order=LittleEndian)
+THE_KEY = b"ITS MAGIC"
+
+@struct(order=LittleEndian, kw_only=True)
 class Format:
-    magic  : b"ITS MAGIC"        # Supports string and byte constants directly
+    magic  : THE_KEY                      # Supports string and byte constants directly
     header : Header
-    a      : uint8               # Primitive data types
-    b      : Dynamic + int32     # dynamic endian based on global config
-    length : uint8               # String fields with computed lengths
-    name   : String(this.length) #  -> you can also use Prefixed(uint8)
+    a      : uint8                        # Primitive data types
+    b      : Dynamic + int32              # dynamic endian based on global config
+    length : uint8                        # String fields with computed lengths
+    name   : String(this.length)          #  -> you can also use Prefixed(uint8)
+
+    # custom actions, e.g. for hashes
+    _hash_begin : DigestField.begin("hash", Md5_Algo)
+    # Sequences with prefixed, computed lengths    -+ part of the MD5 hash
+    names       : CString[uint8::]               #  |
+    #                                              -+
+    # automatic hash creation and verification + default value
+    hash        : Md5_Field("hash", verify=True)
+
+# Creation, packing and unpacking remains the same
+```
 
-    _hash_begin : DigestField.begin("hash", Md5_Algo)   # custom actions, e.g. for hashes
-    names       : CString[uint8::]                      # Sequences with prefixed, computed lengths
-    hash        : Md5_Field("hash", verify=True) = None # automatic hash creation and verification + default value
+</details>
 
-# Instantiation (keyword-only arguments, magic is auto-inferred):
+```python
+from caterpillar.py import *
+from caterpillar.types import *
+
+@bitfield(order=LittleEndian)
+class Header:
+    version : int4_t                   # 4bit integer
+    valid   : int1_t                   # 1bit flag (boolean)
+    ident   : f[str, (8, CharFactory)] # 8bit char
+    # automatic alignment to 16bits
+
+THE_KEY = b"ITS MAGIC"
+
+@struct(order=LittleEndian, kw_only=True)
+class Format:
+    magic  : f[bytes, THE_KEY] = THE_KEY  # Supports string and byte constants directly
+    header : Header
+    a      : uint8_t                      # Primitive data types
+    b      : f[int, Dynamic + int32]      # dynamic endian based on global config
+    length : uint8_t                      # String fields with computed lengths
+    name   : f[str, String(this.length)]  #  -> you can also use Prefixed(uint8)
+
+    # custom actions, e.g. for hashes
+    _hash_begin : f[None, DigestField.begin("hash", Md5_Algo)] = None
+    # Sequences with prefixed, computed lengths    -+ part of the MD5 hash
+    names       : f[list[str], CString[uint8::]] #  |
+    #                                              -+
+    # automatic hash creation and verification + default value
+    hash        : f[bytes, Md5_Field("hash", verify=True)] = b""
+
+# Creation (keyword-only arguments, magic is auto-inferred):
 obj = Format(
     header=Header(version=2, valid=True, ident="F"),
     a=1,
@@ -66,22 +108,32 @@ obj = Format(
     name="foo",
     names=["a", "b"]
 )
-# Packing the object, reads as 'PACK obj FROM Format'
+
+# Packing the object; reads as 'PACK obj FROM Format'
 # objects of struct classes can be packed right away
-blob = pack(obj, Format)
+data_le = pack(obj, Format)
 # results in: b'ITS MAGIC0*\x01\x02\x00\x00\x00\x03foo\x02a\x00b\x00)\x9a...'
 
 # Unpacking the binary data, reads as 'UNPACK Format FROM blob'
-obj2 = unpack(Format, blob)
+obj2 = unpack(Format, data_le)
+assert obj2.names == obj.names
 
-# to pack with a different endian for field 'b', use _order
-data = pack(obj, Format, _order=BigEndian)
+# to pack with a different endian for fields 'a' and 'b', use 'order'
+data_be = pack(obj, Format, order=BigEndian)
+assert data_le != data_be
 ```
 
-This library offers extensive functionality beyond basic struct handling. For further details
+> [!NOTE]
+> Python 3.14 breaks `with` statements in class definitions since `__annotations__` are added at the end
+> of a class definition. Therefore, `Digest` and conditional statements **ARE NOT SUPPORTED** using the `with` syntax in Python 3.14+.
+> As of version `2.4.5` the `Digest` class has a counterpart (`DigestField`), which can be used to manually specify a digest without
+> the need of a `ẁith` statement.
+
+This library offers extensive functionality beyond basic struct definitions. For further details
 on its powerful features, explore the official [documentation](https://matrixeditor.github.io/caterpillar/),
 [examples](./examples/), and [test cases](./test/).
 
+
 ## Installation
 
 > [!NOTE]

docs/sphinx/source/development/changelog.rst

@@ -6,6 +6,48 @@ Changelog
 
 *More entries will be added in the future.*
 
+.. _changelog_2.8.0:
+
+[2.8.0] - Extended Syntax
+=========================
+
+Added
+-----
+
+- New concept: *extended* syntax featuring typing compliance; introduces shortcut :data:`f` to annotate fields and :func:`~caterpillar.model.Invisible` to hide fields from the constructor
+- ``struct_factory.mixin`` and ``bitfield_factory.mixin`` both provide default function wrappers for packing and unpacking data as well as typing fixes for operating on types directly.
+- New :data:`~caterpillar.context.parentctx` context path
+- Dynamic endian is now used by default (i. e. can be changed via order parameter in pack() or unpack())
+- New generic :class:`~caterpillar.fields.Timestamp` class
+- A number of new test cases
+- New module: ``caterpillar.types``: defines default types that can be used as annotations within struct definitions
+- :data:`~caterpillar.byteorder.O_DEFAULT_ENDIAN` as a global option to set a global default byteorder
+- :data:`~caterpillar.byteorder.O_DEFAULT_ARCH` same concept for arch-like objects
+- new 'strict' option to :class:`~caterpillar.fields.Enum` struct
+- :data:`~caterpillar.shared.ATTR_PACK` and :data:`~caterpillar.shared.ATTR_UNPACK` to *caterpillar.shared*
+- `'order'` and `'arch'` options to pack() and unpack(), which temporarily change the global endianess or arch (compatible with Dynamic byteorder)
+
+Changes
+-------
+
+- Merge all stub files with their corresponding Python files. All typing is now inline.
+- :class:`~caterpillar.fields.Padding` struct now has its own dedicated class
+- sizeof() now always returns an integer
+- ContextPath: dropped support for the call action
+- Field.get_name() now always returns a string
+- Rename `ssize` and `size` to :data:`~caterpillar.fields.pssize` and :data:`~caterpillar.fields.psize` in caterpillar.fields exports
+- options.get_flags() always returns a list
+
+Fixes
+-----
+
+- IntFlag support for :class:`~caterpillar.fields.Enum` structs
+- Fix incorrect Sha1 Digest length
+- Add missing global exports in *caterpillar.py*
+- Bitfield, Struct and Sequence now respect fields with already configured byteorder
+- Prefixed struct now does not require ``as_field=True`` when calling pack() or unpack()
+
+
 .. _changelog_2.7.0:
 
 [2.7.0] - Dynamic Byteorder

docs/sphinx/source/index.rst

@@ -16,16 +16,33 @@ efficient.
 structs that adjust their size based on the current context. This framework enables you
 to write complex structures in a compact and readable manner.
 
+.. tab-set::
 
-.. code-block::
-   :caption: Simple example of a custom struct
+   .. tab-item:: Default Syntax
+
+      .. code-block::
+         :caption: Simple example of a custom struct
+
+         @struct
+         class Format:
+            magic  : b"Foo"                     # constant values
+            name   : CString(...)               # \x00-terminated String without a fixed length
+            value  : le + uint16                # little endian encoding
+            entries: be + CString[uint32::]     # arrays with big-endian prefixed length
+
+
+   .. tab-item:: Extended Syntax (>=2.8.0)
+
+      .. code-block::
+         :caption: Simple example of a custom struct with type annotations in-place
+
+         @struct(kw_only=True)                            # keyword-only constructor arguments
+         class Format:
+            magic  : f[bytes, b"Foo"] = Invisible()       # constant values
+            name   : cstr_t                               # \x00-terminated String without a fixed length
+            value  : f[int, le + uint16]                  # little endian encoding
+            entries: f[list[str], be + CString[uint32::]] # arrays with big-endian prefixed length
 
-   @struct
-   class Format:
-      magic: b"Foo"                       # constant values
-      name: CString(...)                  # \x00-terminated String without a fixed length
-      value: le + uint16                  # little endian encoding
-      entries: be + CString[uint32::]     # arrays with big-endian prefixed length
 
 
 .. admonition:: Hold up, wait a minute!
@@ -58,6 +75,9 @@ Format(magic=b'Foo', name='Hello, World!', value=10, entries=['Bar', 'Baz'])
   to 4 times. More information are provided when discussing available configuration
   :ref:`options`.
 
+- And *YES*, this library tries to enforce strong typing wherever appliacable and supported.
+  More on that topic later in the tutorial: :ref:`first_steps-extended-syntax`
+
 
 Where to start?
 ---------------

docs/sphinx/source/library/byteorder.rst

@@ -15,24 +15,27 @@ Byteorder
 .. autoclass:: DynByteOrder
     :members:
 
-.. autofunction:: byteorder(obj, default: Optional[ByteOrder] = None) -> ByteOrder
+.. autofunction:: byteorder
 .. autofunction:: byteorder_is_little
 
 Standard Byteorder Instances
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-.. autoattribute:: caterpillar.byteorder.Native
+.. autodata:: caterpillar.byteorder.Native
 
-.. autoattribute:: caterpillar.byteorder.BigEndian
+.. autodata:: caterpillar.byteorder.BigEndian
 
-.. autoattribute:: caterpillar.byteorder.LittleEndian
+.. autodata:: caterpillar.byteorder.LittleEndian
 
-.. autoattribute:: caterpillar.byteorder.NetEndian
+.. autodata:: caterpillar.byteorder.NetEndian
 
-.. autoattribute:: caterpillar.byteorder.SysNative
+.. autodata:: caterpillar.byteorder.SysNative
 
-.. autoattribute:: caterpillar.byteorder.Dynamic
+.. autodata:: caterpillar.byteorder.Dynamic
 
+.. autodata:: caterpillar.byteorder.LITTLE_ENDIAN_FMT
+
+.. autodata:: caterpillar.byteorder.O_DEFAULT_ENDIAN
 
 
 Architecture

docs/sphinx/source/library/context.rst

@@ -17,7 +17,7 @@ Context classes
 
 
 .. autoclass:: caterpillar.context.ContextPath
-    :members:
+    :members: __call__, __getattribute__, __repr__, parent, parentctx
 
 .. autoclass:: caterpillar.context.ContextLength
     :members:
@@ -29,6 +29,10 @@ Context classes
 .. autoclass:: caterpillar.context.ConditionContext
 
 
+.. autoclass:: caterpillar.context.SetContextVar
+    :members: __action_pack__, __action_unpack__
+
+
 Extra options
 -------------
 
@@ -52,17 +56,15 @@ Extra options
 Special paths
 -------------
 
-.. autoattribute:: caterpillar.context.this
+.. autodata:: caterpillar.context.this
 
-.. autoattribute:: caterpillar.context.ctx
+.. autodata:: caterpillar.context.ctx
 
-.. autoattribute:: caterpillar.context.parent
+.. autodata:: caterpillar.context.parent
 
-.. autoattribute:: caterpillar.context.root
+.. autodata:: caterpillar.context.parentctx
 
-    Provides access to the root-level context object.
-
-    .. versionadded:: 2.6.0
+.. autodata:: caterpillar.context.root
 
 
 Special Attributes
@@ -112,6 +114,18 @@ Special Attributes
 
     Points to the root of the entire context hierarchy.
 
+.. autoattribute:: caterpillar.context.CTX_ORDER
+
+    Stores the currently used endianess (only in root context).
+
+    .. versionadded:: 2.7.0
+
+.. autoattribute:: caterpillar.context.CTX_ARCH
+
+    Stores the currently used architecture (only in root context).
+
+    .. versionadded:: 2.7.0
+
 
 Expressions
 -----------

docs/sphinx/source/library/ext_types.rst

@@ -0,0 +1,7 @@
+.. _library-extended-types:
+
+Types (Extended Syntax)
+=======================
+
+.. automodule:: caterpillar.types
+    :members: