Skip to content

Commit 4f3ba70

Browse files
Add serialization (#120)
This PR adds a single simple utility module `_serdes.py` that implements (de)serialization. This is useful when one needs to build/parse serialized representations without having to generate code beforehand. I considered extracting this into a separate package but upon looking at how small the module is I decided to embed it into PyDSDL. It integrates very cleanly without affecting the rest of the codebase. This changeset also renames `FrontendError` into just `Error` so that we could also use it for serialization. A compatibility redirect is left behind to avoid API breakage. This is the only change to the rest of the codebase here. A simple demo is added and docs slightly improved. Later I want to add a simple command-line shim that takes serialized data line-by-line at the input and provides JSON at the output, and the other way around. This is intentionally not included here yet.
1 parent 2229ca5 commit 4f3ba70

21 files changed

Lines changed: 4849 additions & 91 deletions

.github/workflows/test-and-release.yml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,12 +46,12 @@ jobs:
4646
- name: Run build and test
4747
run: |
4848
if [ "$RUNNER_OS" == "Linux" ]; then
49-
nox --non-interactive --error-on-missing-interpreters --session test pristine lint --python ${{ matrix.python }}
49+
nox --non-interactive --error-on-missing-interpreters --session test pristine lint demo --python ${{ matrix.python }}
5050
nox --non-interactive --session docs
5151
elif [ "$RUNNER_OS" == "Windows" ]; then
5252
nox --forcecolor --non-interactive --session test pristine lint
5353
elif [ "$RUNNER_OS" == "macOS" ]; then
54-
nox --non-interactive --error-on-missing-interpreters --session test pristine lint --python ${{ matrix.python }}
54+
nox --non-interactive --error-on-missing-interpreters --session test pristine lint demo --python ${{ matrix.python }}
5555
else
5656
echo "${{ runner.os }} not supported"
5757
exit 1

.gitignore

Lines changed: 1 addition & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,7 @@
1-
# Byte-compiled / optimized / DLL files
21
__pycache__/
32
*.py[cod]
43
*$py.class
5-
6-
# C extensions
74
*.so
8-
9-
# Distribution / packaging
105
.Python
116
build/
127
develop-eggs/
@@ -24,18 +19,10 @@ wheels/
2419
.installed.cfg
2520
*.egg
2621
MANIFEST
27-
28-
# PyInstaller
29-
# Usually these files are written by a python script from a template
30-
# before PyInstaller builds the exe, so as to inject date/other infos into it.
3122
*.manifest
3223
*.spec
33-
34-
# Installer logs
3524
pip-log.txt
3625
pip-delete-this-directory.txt
37-
38-
# Unit test / coverage reports
3926
htmlcov/
4027
.tox/
4128
.coverage
@@ -46,42 +33,20 @@ coverage.xml
4633
*.cover
4734
.hypothesis/
4835
.pytest_cache/
49-
50-
# Translations
5136
*.mo
5237
*.pot
53-
54-
# Django stuff:
5538
*.log
5639
local_settings.py
5740
db.sqlite3
58-
59-
# Flask stuff:
6041
instance/
6142
.webassets-cache
62-
63-
# Scrapy stuff:
6443
.scrapy
65-
66-
# Sphinx documentation
6744
docs/_build/
68-
69-
# PyBuilder
7045
target/
71-
72-
# Jupyter Notebook
7346
.ipynb_checkpoints
74-
75-
# pyenv
7647
.python-version
77-
78-
# celery beat schedule file
7948
celerybeat-schedule
80-
81-
# SageMath parsed files
8249
*.sage.py
83-
84-
# Environments
8550
.env
8651
.venv
8752
.pyenv
@@ -90,33 +55,19 @@ venv/
9055
ENV/
9156
env.bak/
9257
venv.bak/
93-
94-
# Spyder project settings
9558
.spyderproject
9659
.spyproject
97-
98-
# Rope project settings
9960
.ropeproject
100-
101-
# mkdocs documentation
61+
.sisyphus/
10262
/site
103-
104-
# mypy
10563
.mypy_cache/
106-
10764
# IDE
10865
**/.idea/*
10966
!**/.idea/dictionaries
11067
!**/.idea/dictionaries/*
11168
.xml
11269
.vscode
113-
114-
# Project-specific testing environment
11570
.dsdl-test/
116-
117-
# OS crap
11871
.DS_Store
119-
120-
# cProfile
12172
/prof
12273

AGENTS.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
# Instructions for clankers
2+
3+
When reading source files, ingest them in their entirety instead of relying on search/grep tools.
4+
5+
Read `README.md` and `CONTRIBUTING.rst` first.
6+
7+
When implementing functional code changes, be sure to read the DSDL specification in <https://github.com/OpenCyphal/specification>.
8+
9+
## Project Structure & Module Organization
10+
11+
- Core library code lives in `pydsdl/`.
12+
- Key internal subpackages are `pydsdl/_expression/`, `pydsdl/_serializable/`.
13+
- Vendored dependencies are under `pydsdl/third_party/` (treat as external code; modify only when intentionally syncing).
14+
- Tests are mostly co-located with implementation. Larger suites are in `pydsdl/_test*.py`, they are never imported.
15+
- Documentation sources are in `docs/`.
16+
- Test and release automation is in `noxfile.py` and `.github/`.
17+
18+
## Commit & Pull Request Guidelines
19+
20+
Provide detailed commit messages explaining the rationale behind the changes. Aim for a brief title with a following expanded description explaining what has been done and why was it necessary.

CLAUDE.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
AGENTS.md

CONTRIBUTING.rst

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,12 @@
33
Development guide
44
=================
55

6-
This document is intended for library developers only.
6+
This document is intended for library developers and AI agents only.
77
If you just want to use the library, you don't need to read it.
88

9-
Development automation is managed by Nox; please see ``noxfile.py``.
9+
Development automation is managed by Nox; please read ``noxfile.py``.
10+
11+
The coding style is PEP8 with max line length 120 characters.
1012

1113

1214
Writing tests
@@ -26,18 +28,16 @@ outside of test-enabled environments.
2628
import pytest # OK to import inside test functions only (rarely useful)
2729
assert get_the_answer() == 42
2830
31+
For targeted test runs: ``pytest pydsdl -k _unittest_whatever -v``.
32+
2933

3034
Supporting newer versions of Python
3135
+++++++++++++++++++++++++++++++++++
3236

33-
Normally, this should be done a few months after a new version of CPython is released:
34-
3537
1. Update the CI/CD pipelines to enable the new Python version.
3638
2. Update the CD configuration to make sure that the library is released using the newest version of Python.
3739
3. Bump the version number using the ``.dev`` suffix to indicate that it is not release-ready until tested.
3840

39-
When the CI/CD pipelines pass, you are all set.
40-
4141

4242
Releasing
4343
+++++++++

demo/DemoMessage.1.0.dsdl

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# A simple demo type.
2+
3+
bool flag
4+
uint32 counter
5+
float64 temperature
6+
float32[4] numeric_data
7+
utf8[<=256] text_data # UTF-8 encoded text data
8+
byte[<=64] binary_data # Raw binary data
9+
10+
@extent 1024 * 8

demo/demo_serdes.py

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
#!/usr/bin/env python3
2+
"""
3+
Self-contained demo of PyDSDL serialization.
4+
"""
5+
6+
import pydsdl
7+
from pathlib import Path
8+
9+
SCRIPT_DIR = Path(__file__).parent
10+
DSDL_FILE = SCRIPT_DIR / "DemoMessage.1.0.dsdl"
11+
12+
13+
def main() -> None:
14+
print("Loading DSDL type from:", DSDL_FILE.name)
15+
types, _ = pydsdl.read_files(
16+
dsdl_files=DSDL_FILE,
17+
root_namespace_directories_or_names=SCRIPT_DIR,
18+
lookup_directories=[],
19+
)
20+
schema = types[0]
21+
print(f"✓ Loaded type: {schema.full_name} v{schema.version.major}.{schema.version.minor}")
22+
print(" Fields:", [f"{f.data_type} {f.name}" for f in schema.fields_except_padding])
23+
24+
print("Creating example object:")
25+
obj = {
26+
"flag": True,
27+
"counter": 42,
28+
"temperature": 23.5,
29+
"numeric_data": [1.0, 2.0, 3, 4],
30+
"text_data": "Hello, SerDes!",
31+
"binary_data": b"\x00\x01\x02\x03",
32+
}
33+
for key, value in obj.items():
34+
value_repr = repr(value)
35+
if len(value_repr) > 50:
36+
value_repr = value_repr[:47] + "..."
37+
print(f" {key:15} = {value_repr} ({type(value).__name__})")
38+
39+
print("Serializing object to bytes")
40+
serialized_data = pydsdl.serialize(schema, obj)
41+
print(f"✓ Serialized to {len(serialized_data)} bytes:")
42+
print(f" {serialized_data.hex()}")
43+
44+
print("Deserializing bytes back to object")
45+
deserialized = pydsdl.deserialize(schema, serialized_data)
46+
print("✓ Deserialized successfully:")
47+
for key, value in deserialized.items():
48+
value_repr = repr(value)
49+
if len(value_repr) > 50:
50+
value_repr = value_repr[:47] + "..."
51+
print(f" {key:15} = {value_repr} ({type(value).__name__})")
52+
53+
print("Verifying roundtrip equality")
54+
assert obj == deserialized, "Roundtrip failed! Objects don't match."
55+
print("✓ Roundtrip verification passed: Original == Deserialized")
56+
57+
58+
if __name__ == "__main__":
59+
main()

docs/pages/pydsdl.rst

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ The main functions
1717

1818
.. autofunction:: pydsdl.read_namespace
1919
.. autofunction:: pydsdl.read_files
20+
.. autofunction:: pydsdl.serialize
21+
.. autofunction:: pydsdl.deserialize
2022

2123

2224
Type model
@@ -36,20 +38,29 @@ Exceptions
3638

3739
.. computron-injection::
3840
:filename: ../descendant_diagram.py
39-
:argv: FrontendError
41+
:argv: Error
4042

41-
.. autoexception:: pydsdl.FrontendError
43+
.. autoexception:: pydsdl.Error
4244
:undoc-members:
4345
:no-inherited-members:
4446
:show-inheritance:
4547
:special-members:
4648

49+
.. note::
50+
``FrontendError`` is retained as a backward-compatibility alias for ``Error``.
51+
4752
.. autoexception:: pydsdl.InvalidDefinitionError
4853
:undoc-members:
4954
:no-inherited-members:
5055
:show-inheritance:
5156
:special-members:
5257

58+
.. autoexception:: pydsdl.SerDesError
59+
:undoc-members:
60+
:no-inherited-members:
61+
:show-inheritance:
62+
:special-members:
63+
5364
.. autoexception:: pydsdl.InternalError
5465
:undoc-members:
5566
:no-inherited-members:

noxfile.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,13 @@ def pristine(session):
7272
exe("import pydsdl")
7373

7474

75+
@nox.session(python=PYTHONS[-1:])
76+
def demo(session):
77+
"""Run the serialization/deserialization demo script."""
78+
session.install("-e", ".")
79+
session.run("python", "demo/demo_serdes.py")
80+
81+
7582
@nox.session(python=PYTHONS, reuse_venv=True)
7683
def lint(session):
7784
session.log("Using the newest supported Python: %s", is_latest_python(session))

pydsdl/__init__.py

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
import sys as _sys
88
from pathlib import Path as _Path
99

10-
__version__ = "1.24.3"
10+
__version__ = "1.25.0.rc0"
1111
__version_info__ = tuple(map(int, __version__.split(".")[:3]))
1212
__license__ = "MIT"
1313
__author__ = "OpenCyphal"
@@ -30,10 +30,13 @@
3030
from ._namespace import read_files as read_files
3131

3232
# Error model.
33-
from ._error import FrontendError as FrontendError
33+
from ._error import Error as Error
3434
from ._error import InvalidDefinitionError as InvalidDefinitionError
3535
from ._error import InternalError as InternalError
3636

37+
# Deprecated compatibility alias, to be removed.
38+
FrontendError = Error
39+
3740
# Data type model - meta types.
3841
from ._serializable import SerializableType as SerializableType
3942
from ._serializable import PrimitiveType as PrimitiveType
@@ -75,4 +78,9 @@
7578
from ._serializable import Version as Version
7679
from ._bit_length_set import BitLengthSet as BitLengthSet
7780

81+
# Serialization/deserialization.
82+
from ._serdes import serialize as serialize
83+
from ._serdes import deserialize as deserialize
84+
from ._serdes import SerDesError as SerDesError
85+
7886
_sys.path = _original_sys_path

0 commit comments

Comments
 (0)