You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The foundation half of the scientific data plane. It lands the measurement, capability, and Arrow-ergonomics groundwork the large-payload transport work (0.8.0) builds on, and captures Python class members the IR used to drop. The wire protocol is unchanged, so a 0.6.x bridge and a 0.7.0 client still talk.
6
+
7
+
### Breaking changes
8
+
9
+
**The IR schema is now `0.3.0` and captures more class members.**`tywrap-ir` reads `@classmethod`, `@property`, and `functools.cached_property` (via `inspect.classify_class_attrs`) and labels `@staticmethod` correctly. Generated wrappers gain `static` members for class/static methods and `readonly` getters for properties, so **generated output changes for any class that uses those decorators** — classes without them are byte-identical. Regenerate your wrappers after upgrading. The TS↔Python IR-version check enforces the bump.
10
+
11
+
### Features
12
+
13
+
- Arrow decoding is frictionless: the runtime auto-registers an `apache-arrow` decoder when the package is present, with no manual wiring. JSON fallback stays opt-in (`TYWRAP_CODEC_FALLBACK=json`), and a missing `apache-arrow` fails with a clear, actionable error instead of silent lossy output. ([#232](https://github.com/bbopen/tywrap/issues/232))
14
+
- Each backend reports a `TransportCapabilities` descriptor (`backend`, `supportsArrow`, `supportsBinary`, `supportsChunking`, `supportsStreaming`, `maxFrameBytes`), surfaced on the bridge via `capabilities()`, with a documented capability matrix. ([#235](https://github.com/bbopen/tywrap/issues/235))
15
+
-`tywrap/dev` gains a watch/reload end-to-end smoke and documented reload failure/recovery behavior. ([#228](https://github.com/bbopen/tywrap/issues/228))
16
+
17
+
### Internal
18
+
19
+
- Measure-first data-plane benchmarks (Arrow round-trip, 100k-row decode, size-guard overhead, pool throughput) land as baselines for 0.8.0's perf gates — no gating yet.
20
+
-`generate()` and `fetchPythonIr()` are decomposed (cognitive complexity 91→16), and a shared `BasePythonBridge` removes the duplicated RPC delegation across the three bridges.
21
+
- The bridge dispatches `@classmethod`/`@staticmethod` through a dotted `call('Class.method')` and reads `@property`/`cached_property` through `call_method`, with the private-attribute guard re-applied per dotted segment.
Copy file name to clipboardExpand all lines: docs/codec-roadmap.md
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,7 +16,9 @@ This is a forward-looking plan for adding codecs beyond numpy/pandas. The focus
16
16
17
17
## DX Defaults (Decisions)
18
18
19
-
- Arrow is the default for ndarray/dataframe/series. The JS runtime should auto-register an Arrow decoder when `apache-arrow` is installed so users do not have to wire it manually.
19
+
- Arrow is the default for ndarray/dataframe/series. The JS runtime auto-registers an Arrow decoder when `apache-arrow` is importable, so users do not have to wire it manually. Node/Bun/Deno and HTTP bridges register eagerly during `init()`; in addition, the codec lazily imports `apache-arrow` on the first Arrow-encoded response (cached for the process) so callers that bypass `init()` still work.
20
+
-`apache-arrow` is an **optional** dependency: it is not in tywrap's runtime `dependencies` (declared as an optional peer). Installing tywrap never pulls it in or fails without it.
21
+
- No silent lossy fallback: if an Arrow-encoded payload arrives and `apache-arrow` is unavailable, decoding fails with an actionable error telling you to `npm install apache-arrow` or set `TYWRAP_CODEC_FALLBACK=json` on the Python side. tywrap never quietly downgrades Arrow data to JSON.
20
22
- JSON fallback is opt-in only (via `TYWRAP_CODEC_FALLBACK=json`) and remains explicitly lossy for dtype/NA fidelity.
21
23
- GPU handling stays explicit: no implicit `.cpu()` or contiguous copies. Opt-in copy/transfer remains available, and GPU-native transport is a follow-up track (DLPack/Arrow CUDA).
22
24
- Large payloads should not be forced through single-line JSONL forever; add an artifact/chunked transport to keep responses reliable without silent truncation.
`tywrap/dev` provides development-time wrapper regeneration plus runtime bridge
4
+
replacement. It does **not** provide application-level hot module reloading — it
5
+
keeps your generated wrappers and the active Python bridge in sync with your
6
+
Python sources while your process stays up.
7
+
8
+
This page documents the reload **lifecycle** and, in particular, the
9
+
**failure / recovery contract**: what happens when a reload fails, and what stays
10
+
live so your app keeps working.
11
+
12
+
> Reload is configured in code through `tywrap/dev`, never in `tywrap.config.*`.
13
+
> The legacy `development` block and `pythonModules.<module>.watch` fields were
14
+
> removed in 0.4.0; using them now raises an explicit migration error.
15
+
16
+
## The two entry points
17
+
18
+
| Helper | Use it for |
19
+
| --- | --- |
20
+
|`startNodeWatchSession(...)`| Node-only. Watches your config file and local Python sources, regenerates wrappers, and swaps the active bridge in place. |
21
+
|`createBridgeReloader(...)`| Cross-runtime manual primitive (e.g. Pyodide). You call `reload()` yourself; there is no filesystem watcher. |
// Force a rebuild now (resolves to `true` on success, `false` on failure):
37
+
const ok =awaitsession.reloadNow();
38
+
39
+
// Stop watching and dispose the active bridge:
40
+
awaitsession.close();
41
+
```
42
+
43
+
`createBridge` receives the **freshly resolved config for that reload cycle**, so
44
+
config edits (e.g. a changed `timeout`) are picked up on the next reload. By
45
+
default the newly created bridge is published to the global runtime registry, so
46
+
existing generated wrappers transparently route through the swapped bridge — no
47
+
imports change and the process never restarts.
48
+
49
+
## Reload lifecycle events
50
+
51
+
Pass an `onEvent` callback to observe the session. Events are emitted in this
52
+
order for a successful cycle:
53
+
54
+
| Event | Meaning |
55
+
| --- | --- |
56
+
|`watchPaths`| The set of paths the session is currently watching (config file, resolved Python package trees, and `extraWatchPaths`). Re-emitted whenever the watched set changes. |
57
+
|`change`| A watched path changed (`manual: false`) — a reload has been scheduled. |
58
+
|`reload-start`| A reload cycle began (`manual: true` for `reloadNow()` / initial startup). |
59
+
|`reload-success`| Regeneration and bridge swap completed. Carries `written` (relative paths of the generated files now on disk) and `warnings`. |
60
+
|`reload-error`| The reload failed. Carries the underlying `error: Error`. |
61
+
62
+
A reload performs these steps atomically with respect to the live state:
63
+
64
+
1.**Generate** the next wrapper set into a temporary staging directory.
65
+
2.**Promote** the staged files into the real output directory (writing new
66
+
files, removing managed files that no longer exist).
67
+
3.**Warm and activate** the next bridge, then dispose the previous one.
68
+
4.**Commit** the refreshed watcher set and emit `reload-success`.
69
+
70
+
`reloadNow()` and filesystem changes are serialized: an in-flight reload
71
+
finishes before the next one starts, and rapid edits are debounced
72
+
(`debounceMs`, default `100`).
73
+
74
+
## Failure / recovery contract
75
+
76
+
If **any** step of a reload throws, the session does **not** tear down. The
77
+
last-known-good state stays live:
78
+
79
+
- The **previously generated wrappers remain on disk unchanged.** Staging happens
80
+
in a temp directory; the real output directory is only touched once generation
81
+
succeeds. If promotion itself fails partway, the previous file contents are
82
+
restored.
83
+
- The **previously active bridge stays active** in the runtime registry and is
84
+
**not disposed.** A bridge that was warmed for the failed reload is disposed
85
+
instead, so you never leak the half-prepared one.
86
+
- A structured `reload-error` event is emitted with the underlying `Error`, and
87
+
the failing call (`reloadNow()` or the debounced auto-reload) resolves to
88
+
`false`. The session keeps watching and will attempt the next reload normally.
89
+
90
+
Two distinct failure sources surface the same way:
91
+
92
+
| Source | What the `error` looks like |
93
+
| --- | --- |
94
+
|**Generation failure** (a watched module's IR can't be produced — e.g. a syntax error). This is the `GenerateFailure` path. |`error.message` begins with `Generation failed for N module(s):` followed by per-module detail. |
95
+
|**Bridge construction failure** (your `createBridge` throws). | The error your factory threw, propagated verbatim. |
96
+
97
+
```typescript
98
+
const session =awaitstartNodeWatchSession({
99
+
configFile: './tywrap.config.ts',
100
+
createBridge,
101
+
onEvent: event=> {
102
+
if (event.type==='reload-error') {
103
+
// Last-good wrappers + bridge are still live here.
0 commit comments