Skip to content

Commit 6fe676b

Browse files
author
brett-bonner_infodesk
committed
docs(codec): clarify scientific envelope behavior
1 parent a705935 commit 6fe676b

1 file changed

Lines changed: 42 additions & 7 deletions

File tree

docs/codec-envelopes.md

Lines changed: 42 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,23 @@ Every scientific envelope has these fields:
4040
- `codecVersion` is `1`. Decoders treat a missing version as legacy version `0`.
4141
- `encoding` identifies the payload representation.
4242

43+
Scientific values can appear at the root of a result or inside nested Python
44+
dictionaries, lists, and tuples. The Python bridge replaces each recognized
45+
value with its envelope. The JavaScript codec then walks plain objects and
46+
arrays and decodes recognized envelopes within the traversal bounds. A decoded
47+
envelope is a terminal value, so fields inside its decoded output are not
48+
scanned again. A `torch.tensor` envelope is the exception because its `value`
49+
field is a required nested ndarray envelope.
50+
51+
Serialization accepts a maximum nesting depth of 900 and at most 1,000,000
52+
visited containers. Decoding accepts a maximum depth of 2048 and the same
53+
1,000,000-container limit. Scientific envelopes count as containers, including
54+
the ndarray nested inside a tensor. Primitive leaves and the elements of a
55+
decoded envelope payload do not consume the container limit. The bridge reports
56+
the result path when a serialization bound is exceeded, and the JavaScript
57+
codec does the same for decoding bounds. The bridge also rejects circular
58+
Python containers.
59+
4360
### SciPy sparse
4461

4562
Supported targets are `scipy.sparse.csr_matrix`, `csc_matrix`, and
@@ -90,7 +107,8 @@ The target is `torch.Tensor`. CPU tensors work by default. Schema sketch:
90107
"encoding": "arrow" | "json",
91108
"b64": "...", // when encoding="arrow"
92109
"data": [ ... ], // when encoding="json"
93-
"shape": [ ... ]
110+
"shape": [ ... ],
111+
"dtype": "float32"
94112
}
95113
}
96114
```
@@ -128,8 +146,24 @@ pyarrow is not available in WASM.
128146

129147
If an Arrow payload arrives without the dependency, decoding fails with an
130148
installation hint. Set `TYWRAP_CODEC_FALLBACK=json` on the Python side when a
131-
JSON fallback is acceptable. JSON fallback is lossy for dtype and missing-value
132-
fidelity.
149+
JSON representation is acceptable.
150+
151+
An ndarray JSON envelope includes `shape`, `dtype`, and plain JSON `data`. The
152+
decoder checks that the data matches the declared shape and supported dtype
153+
domain. It returns plain JavaScript arrays, or a scalar for a zero-dimensional
154+
array. The `dtype` field records provenance for validation; it does not create a
155+
typed JavaScript array. JSON encoding rejects ndarray dtypes and values that it
156+
cannot represent safely, including complex, structured, object, byte-string,
157+
unicode, datetime, timedelta, big-endian, and unsafe integer cases.
158+
159+
DataFrame and Series JSON envelopes are values-only representations. Before
160+
encoding, the bridge requires an unnamed `RangeIndex` starting at 0 with step 1.
161+
It rejects a `MultiIndex`, categorical dtype, unsupported cell values, unsafe
162+
integers, non-finite floats, duplicate DataFrame column labels, and labels that
163+
collide after JSON object-key coercion. `None`, `pd.NA`, and `pd.NaT` become
164+
JSON `null`. Supported booleans, numbers, strings, empty values, and nulls keep
165+
their value semantics, but pandas dtype and index metadata are not retained as
166+
they are with Arrow.
133167

134168
## Size limits and transport framing
135169

@@ -154,7 +188,8 @@ their promises resolve. A result that does not match its declared return type
154188
throws `BridgeValidationError` with the declared type, received shape, and call
155189
site.
156190

157-
Columnar values retain their decoded provenance. Validation checks the envelope
158-
marker, dimensions, and dtype for DataFrame, Series, and ndarray returns
159-
without walking Arrow data. Returns declared as `unknown`, `void`, or `Any` do
160-
not receive a runtime check.
191+
Decoded scientific values retain their envelope provenance. Return validation
192+
recognizes all six markers: `ndarray`, `dataframe`, `series`, `scipy.sparse`,
193+
`torch.tensor`, and `sklearn.estimator`. It checks the marker and, when present
194+
in the return contract, dimensions and dtype without walking Arrow data. Returns
195+
declared as `unknown`, `void`, or `Any` do not receive a runtime check.

0 commit comments

Comments
 (0)