Skip to content

Commit b341385

Browse files
docs: Add mYm references and intrinsic versioning explanation
Add references to mYm format documentation: - MATLAB FileExchange: https://www.mathworks.com/matlabcentral/fileexchange/81208-mym - GitHub repository: https://github.com/datajoint/mym Add intrinsic versioning explanation to plugin codecs guide: - How built-in codecs embed version in data format - Protocol headers in <blob> (mYm\0, dj0\0) - NumPy format version in <npy@> headers - Self-describing structure in <object@> - Why built-in codecs don't need explicit codec_version field Clarifies the distinction between built-in codecs (intrinsic versioning) and plugin codecs (explicit codec_version field).
1 parent 78047a9 commit b341385

3 files changed

Lines changed: 15 additions & 2 deletions

File tree

src/explanation/type-system.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,9 @@ Codecs provide `encode()`/`decode()` semantics for complex Python objects.
114114
Stores NumPy arrays, dicts, lists, and other Python objects using DataJoint's custom binary serialization format.
115115

116116
**Serialization format:**
117-
- **Protocol headers**: `mYm` (MATLAB-compatible) or `dj0` (Python-extended)
117+
- **Protocol headers**:
118+
- `mYm` — MATLAB-compatible format (see [mYm on MATLAB FileExchange](https://www.mathworks.com/matlabcentral/fileexchange/81208-mym) and [mym on GitHub](https://github.com/datajoint/mym))
119+
- `dj0` — Python-extended format supporting additional types
118120
- **Optional compression**: zlib compression for data > 1KB
119121
- **Type-specific encoding**: Each Python type has a specific serialization code
120122
- **Version detection**: Protocol header embedded in blob enables format detection

src/how-to/use-plugin-codecs.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -262,6 +262,17 @@ Plugin codecs evolve over time. Following versioning best practices ensures your
262262

263263
Plugin codecs evolve independently and need to handle data encoded by different plugin versions. Built-in codecs are part of DataJoint's API surface and evolve with the framework itself. When you upgrade DataJoint 2.0 → 3.0, you expect potential breaking changes. When you upgrade a plugin 1.0 → 2.0 while keeping DataJoint 2.0, backward compatibility is critical.
264264

265+
**How built-in codecs handle versioning:**
266+
267+
Built-in formats have **intrinsic versioning** - the format version is embedded in the data itself:
268+
269+
- `<blob>` — Protocol header (`mYm\0` or `dj0\0`) at start of blob
270+
- `<npy@>` — NumPy format version in `.npy` file header
271+
- `<object@>` — Self-describing directory structure
272+
- `<attach>` — Filename + content (format-agnostic)
273+
274+
When DataJoint needs to change a built-in codec's format, it can detect the old format from the embedded version information and handle migration transparently. This is why built-in codecs don't need an explicit `codec_version` field in database metadata.
275+
265276
### Version Strategy
266277

267278
**Two version numbers matter for plugin codecs:**

src/reference/specs/type-system.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -503,7 +503,7 @@ Serializes Python objects using DataJoint's custom binary serialization format.
503503
**Serialization format:**
504504

505505
- **Protocol headers**:
506-
- `mYm` — Original MATLAB-compatible format for numeric arrays, structs, cells
506+
- `mYm` — Original MATLAB-compatible format for numeric arrays, structs, cells (see [mYm on MATLAB FileExchange](https://www.mathworks.com/matlabcentral/fileexchange/81208-mym) and [mym on GitHub](https://github.com/datajoint/mym))
507507
- `dj0` — Extended format supporting Python-specific types (UUID, Decimal, datetime, etc.)
508508
- **Compression**: Automatic zlib compression for data > 1KB
509509
- **Type codes**: Each Python type has a specific serialization code:

0 commit comments

Comments
 (0)