feat!: add new feature "inline-syntax"

- New *inline* syntax using the `<<` operator to automatically unpack structs from data
- All default struct types now implement `from_bytes`, `from_file` and `to_bytes` for direct packing or unpacking
- Add `PackMixin` and `UnpackMixin` to automatically implement the methods mentioned above for struct types

- `pack`, `unpack` and `sizeof` were moved into their own module (no changed visibe when importing from `caterpillar.models`)

- Drop support for `'x'` format character in `PyStructFormattedField`

- Pre-computed conditions in the `Field` class incorrectly evaluated to true or false unconditionally (in case of a context-lambda). This issue has now been fixed in all places.
- Fix some missing or wrong typing annotations

Merge pull request #59 from MatrixEditor/feat/inline-syntax

Author: MatrixEditor

docs/sphinx/source/library/fields/common.rst

@@ -13,6 +13,9 @@ Numeric Structs
     .. versionchanged:: 2.4.0
         :code:`FormatField` renamed to :code:`PyStructFormattedField`
 
+    .. versionchanged:: 2.8.1
+        Dropped support for format character ``'x'``. Use the :class:`Padding` type instead.
+
 .. data:: caterpillar.fields.uint8
 
     Unsigned 8-bit integer field. Range: ``0`` to ``255``.

docs/sphinx/source/tutorial/first_steps/extended_syntax.rst

@@ -3,6 +3,8 @@
 Extended Syntax
 ===============
 
+.. versionadded:: 2.8.0
+
 The extended syntax introduces a declarative and type-checker-friendly way to
 define structs using ordinary Python classes. Instead of constructing structures
 through procedural builder calls, you describe the format directly through type

docs/sphinx/source/tutorial/first_steps/index.rst

@@ -46,6 +46,7 @@ imported the necessary components from *Caterpillar*:
     structdef
     extended_syntax
     parsing
+    inline_syntax
     configuration
     documentation
 

docs/sphinx/source/tutorial/first_steps/inline_syntax.rst

@@ -0,0 +1,59 @@
+.. _first_steps-inline-syntax:
+
+Inline Syntax
+=============
+
+In addition to the standard methods for packing and unpacking data, *Caterpillar* also
+supports an **inline syntax** feature that allows for more concise and expressive data
+parsing. This syntax uses Python's overloaded left-shift operator (``<<``) to easily
+unpack data from a bytes object or a stream, making the code both compact and readable.
+
+Unpacking Data Inline
+^^^^^^^^^^^^^^^^^^^^^^
+
+With the inline syntax, you can unpack binary data into a struct directly using the
+left-shift operator. This operator is a shorthand for calling the ``from_bytes`` method,
+which means you don't need to explicitly invoke a function to parse your data.
+
+.. code-block:: python
+
+    from io import BytesIO
+    from caterpillar.py import uint8
+
+    # data can be unpacked INLINE using a special operator
+    data = b"\x00\x00\x01\xff"
+
+    # If using on a static bytes object, the struct will always begin at offset zero
+    value1 = uint8 << data
+    value2 = uint8 << data
+    assert value1 == value2
+
+
+In the example above, the binary data is unpacked into the :data:`~caterpillar.fields.uint8` struct.
+The left-shift operator (``<<``) reads the data starting at the beginning of the byte stream and
+automatically handles the parsing for you.
+
+When using a stream, the operator will start unpacking from the current position in the stream. This
+is useful for processing data incrementally or when you want to track your stream's current position
+manually.
+
+>>> stream = BytesIO(data)
+>>> stream.seek(2)  # Move the stream position to byte index 2
+>>> uint8 << stream
+1
+
+Wrapper methods
+^^^^^^^^^^^^^^^
+
+Instead of using the special operator, all struct classes in *Caterpillar* also provide the typical
+wrapper functions for packing and unpacking:
+
+>>> # Instead of using the special operator, all default struct classes provide
+>>> # wrapper functions for packing and unpacking:
+>>> uint8.from_bytes(data)
+0
+>>> uint8.to_bytes(0xFF)
+b"\0xFF"
+
+This inline syntax feature is great for simplifying your code when dealing with binary data, making the
+code both more readable and intuitive.

docs/sphinx/source/tutorial/first_steps/parsing.rst

@@ -36,4 +36,34 @@ RGB(r=1, g=2, b=3)
 
 Now that you've seen how to define, pack, and unpack data with structs in *Caterpillar*, you're
 almost ready to start working with more complex data structures. And remember,
-we've just scratched the surface—there's a lot more to explore!
+we've just scratched the surface—there's a lot more to explore!
+
+.. tip::
+
+    .. versionadded:: 2.8.1
+
+    You can use a special *mixin* class to add wrapper methods to your struct
+    class:
+
+    .. code-block:: python
+
+        @struct
+        class RGB(struct_factory.mixin):
+            ...
+
+    It will then be possible to use the type directly instead of importing
+    :func:`~caterpillar.model.pack` or :func:`~caterpillar.model.unpack`
+    every time.
+
+    >>> obj = RGB.from_bytes(b"\x01\x02\x03")
+    RGB(r=1, g=2, b=3)
+    >>> obj.to_bytes()
+    b"\x01\x02\x03"
+
+    The same applies to all common structs mentioned in the next chapter
+    and :class:`~caterpillar.fields.Field` objects.
+
+    >>> uint8[3].from_bytes(b"\x01\x02\x03")
+    [1,2,3]
+    >>> uint8[3].to_bytes([1,2,3])
+    b"\x01\x02\x03"

examples/formats/caf.py

@@ -1,4 +1,4 @@
-# Copyright (C) MatrixEditor 2023-2025
+# Copyright (C) MatrixEditor 2023-2026
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

examples/formats/itdb.py

@@ -1,4 +1,4 @@
-# Copyright (C) MatrixEditor 2023-2025
+# Copyright (C) MatrixEditor 2023-2026
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

examples/formats/nibarchive.py

@@ -1,4 +1,4 @@
-# Copyright (C) MatrixEditor 2023-2025
+# Copyright (C) MatrixEditor 2023-2026
 #
 # This program is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by

examples/inline_syntax.py

@@ -0,0 +1,22 @@
+from io import BytesIO
+from caterpillar.py import uint8
+
+# data can be unpacked INLINE using a special operator
+data = b"\x00\x00\x01\xff"
+
+# If using on a static bytes object, the struct will always begin at offset zero
+value = uint8 << data
+other_value = uint8 << data
+assert value == other_value
+
+# When using a stream, the current stream position will be modified
+stream = BytesIO(data)
+_ = stream.seek(2)
+value = uint8 << stream
+other_value = uint8 << stream
+assert value != other_value
+
+# Instead of using the special operator, all default struct classes provide
+# wrapper functions for packing and unpacking:
+value = uint8.from_bytes(data)
+assert uint8.to_bytes(value) == data[:1]

pyproject.toml

@@ -8,7 +8,7 @@ cmake.source-dir = "."
 
 [project]
 name = "caterpillar"
-version = "2.8.0"
+version = "2.8.1rc"
 requires-python = ">=3.10"
 description = "Library to pack and unpack structurized binary data."
 authors = [{ name = "MatrixEditor" }]