|
| 1 | +# ipc-codegen |
| 2 | + |
| 3 | +Schema-driven IPC code generator for **C++**, **TypeScript**, **Rust**, and **Zig**. |
| 4 | + |
| 5 | +Given a JSON schema describing a service's commands and responses, emits matching |
| 6 | +wire-type definitions plus a typed client and/or server-side dispatcher in the |
| 7 | +target language. Wire format is msgpack; the actual byte transport |
| 8 | +(Unix-domain socket or MPSC shared memory) is provided by |
| 9 | +[`/ipc-runtime`](../ipc-runtime) — clients and servers in different languages |
| 10 | +talk byte-compatibly because they all pack the same wire types. |
| 11 | + |
| 12 | +## Quick start |
| 13 | + |
| 14 | +```sh |
| 15 | +cd ipc-codegen |
| 16 | +./bootstrap.sh build # generate echo example bindings, compile all 4 languages |
| 17 | +./bootstrap.sh test # run the cross-language wire-compat matrix |
| 18 | +``` |
| 19 | + |
| 20 | +## How it fits together |
| 21 | + |
| 22 | +``` |
| 23 | + ┌──────────────────┐ |
| 24 | + │ *_schema.json │ (committed next to the C++ server |
| 25 | + └────────┬─────────┘ that owns the wire format) |
| 26 | + │ |
| 27 | + ▼ |
| 28 | + ┌──────────────────┐ |
| 29 | + │ ipc-codegen │ (this package) |
| 30 | + └────────┬─────────┘ |
| 31 | + │ |
| 32 | + ┌──────────┬───────┴───────┬──────────┐ |
| 33 | + ▼ ▼ ▼ ▼ |
| 34 | + wire types, wire types, wire types, wire types, |
| 35 | + typed typed typed typed |
| 36 | + client + client + client + client + |
| 37 | + server server server server |
| 38 | + (C++) (TS) (Rust) (Zig) |
| 39 | + │ │ │ │ |
| 40 | + └──────────┴────────┬──────┴──────────┘ |
| 41 | + │ |
| 42 | + ▼ |
| 43 | + ┌──────────────────┐ |
| 44 | + │ ipc-runtime │ (transport: UDS / MPSC-SHM, |
| 45 | + └──────────────────┘ same path-suffix dispatch in |
| 46 | + every language) |
| 47 | +``` |
| 48 | + |
| 49 | +ipc-codegen knows nothing about sockets, shared memory, or processes — it just |
| 50 | +serialises typed commands to msgpack bytes and back. ipc-runtime knows nothing |
| 51 | +about your service's commands — it just moves bytes. Consumers wire the two |
| 52 | +together (codegen-emitted dispatcher on top of an ipc-runtime server, or |
| 53 | +codegen-emitted typed client on top of an ipc-runtime client). |
| 54 | + |
| 55 | +## Layout |
| 56 | + |
| 57 | +``` |
| 58 | +ipc-codegen/ |
| 59 | + bootstrap.sh # build / test / update_goldens / hash |
| 60 | + src/ # generator (TypeScript, runs under Node 22+) |
| 61 | + generate.ts # CLI entry point |
| 62 | + schema_visitor.ts # JSON schema -> CompiledSchema IR |
| 63 | + cpp_codegen.ts # IR -> C++ output |
| 64 | + typescript_codegen.ts # IR -> TypeScript output |
| 65 | + rust_codegen.ts # IR -> Rust output |
| 66 | + zig_codegen.ts # IR -> Zig output |
| 67 | + naming.ts # snake_case / PascalCase helpers |
| 68 | + templates/ # static templates copied alongside generated code |
| 69 | + cpp/ipc_codegen/*.hpp # C++ support headers copied into generated output |
| 70 | + rust/{backend,error,ffi_backend}.rs |
| 71 | + zig/{backend,ffi_backend}.zig |
| 72 | + echo_example/ # 4-language echo service (cross-lang test harness) |
| 73 | + SCHEMA_SPEC.md # wire protocol and schema-format reference |
| 74 | +``` |
| 75 | + |
| 76 | +The package contains no service schemas of its own. Each consumer owns and |
| 77 | +commits its schema next to the C++ server that defines the wire format, and |
| 78 | +invokes `generate.ts` with that local path. |
| 79 | + |
| 80 | +## CLI: `src/generate.ts` |
| 81 | + |
| 82 | +Invoked once per (schema, language) pair. Run directly with |
| 83 | +`node --experimental-strip-types`, or via `bootstrap.sh`. |
| 84 | + |
| 85 | +``` |
| 86 | +node --experimental-strip-types --experimental-transform-types --no-warnings \ |
| 87 | + src/generate.ts --schema <file> --lang <ts|cpp|rust|zig> --out <dir> [flags] |
| 88 | +``` |
| 89 | + |
| 90 | +### Required flags |
| 91 | + |
| 92 | +| Flag | Purpose | |
| 93 | +|---|---| |
| 94 | +| `--schema <file>` | Path to the JSON schema. | |
| 95 | +| `--lang <ts\|cpp\|rust\|zig>` | Target language. | |
| 96 | +| `--out <dir>` | Output directory. Generated files are (re)written every run; static templates are copied alongside and re-copied only if missing (so handwritten edits to templated scaffolding are preserved). | |
| 97 | + |
| 98 | +### Role flags |
| 99 | + |
| 100 | +| Flag | Purpose | |
| 101 | +|---|---| |
| 102 | +| `--server` | Emit server dispatch (matches request name to handler, deserialises, calls handler, serialises response). Pair it with an `ipc::IpcServer` from ipc-runtime. | |
| 103 | +| `--client` | Emit a typed client class/struct with one method per command. Pair it with an `ipc::IpcClient` (C++) or the equivalent Rust/Zig/TS binding. | |
| 104 | +| `--package <dir>` | TS only. Emit a complete package wrapper around the generated async client. The wrapper launches a native service binary, connects over UDS or SHM, and resolves the binary from an override path, environment variable, installed arch package, or local `build/<platform>/` directory. | |
| 105 | +| `--uds` | Rust/Zig only. Copies the `Backend` trait template (and `error.rs` for Rust) into `<out>` so consumers can plug ipc-runtime — or any custom transport — behind the generated client. The flag name is historical: the trait is transport-agnostic. | |
| 106 | +| `--ffi` | Rust/Zig only. Adds the `ffi_backend` template (a thin wrapper exposing the generated client over a C ABI for embedding in other languages). | |
| 107 | + |
| 108 | +### Naming flags |
| 109 | + |
| 110 | +| Flag | Purpose | |
| 111 | +|---|---| |
| 112 | +| `--prefix <Str>` | Type prefix applied to generated type names (`<Prefix>CircuitProve`, etc.). Auto-detected from the schema if omitted. | |
| 113 | +| `--strip-method-prefix` | TS only. Drops the prefix from client *method* names: `bbCircuitProve()` → `circuitProve()`. Types keep the prefix. | |
| 114 | + |
| 115 | +### C++-specific flags |
| 116 | + |
| 117 | +| Flag | Purpose | |
| 118 | +|---|---| |
| 119 | +| `--cpp-namespace <ns>` | C++ namespace, e.g. `my::service`. Default: lowercased prefix. | |
| 120 | +| `--cpp-wire-namespace <ns>` | Inner namespace for wire types, default `wire`. | |
| 121 | +| `--cpp-include-dir <path>` | Include-path prefix for cross-includes between generated files, e.g. `myservice/generated`. Leave unset when generated files are in the same directory as their consumer. | |
| 122 | + |
| 123 | +### Other |
| 124 | + |
| 125 | +| Flag | Purpose | |
| 126 | +|---|---| |
| 127 | +| `--curve-constants` | TS only. Also emit `curve_constants.ts` with bn254/grumpkin/secp moduli & generators for schemas that need curve constants. | |
| 128 | +| `--skeleton <dir>` | One-shot scaffolding: writes a `<service>_handlers.{ts,rs,zig,cpp}` stub, `main`, and a build file into `<dir>` if they don't already exist. Skipped on subsequent runs. | |
| 129 | +| `--package-name <name>` | TS package mode only. Package name to write into the generated wrapper `package.json`. | |
| 130 | +| `--binary-name <name>` | TS package mode only. Native service binary name to launch. | |
| 131 | +| `--binary-env-var <name>` | TS package mode only. Environment variable that can override the binary path. Defaults to `<BINARY_NAME>_PATH`. | |
| 132 | +| `--package-transports <uds,shm>` | TS package mode only. Comma-separated transports supported by the generated wrapper. | |
| 133 | +| `--ipc-runtime-dependency <spec>` | TS package mode only. Dependency spec for `@aztec/ipc-runtime`, e.g. a release version or local `file:` dependency in examples. | |
| 134 | + |
| 135 | +## Worked examples |
| 136 | + |
| 137 | +Paths below are illustrative — consumers commit their own schema next to the |
| 138 | +C++ server that owns the wire format and supply absolute or relative paths on |
| 139 | +the command line. |
| 140 | + |
| 141 | +### TypeScript client, with curve constants |
| 142 | + |
| 143 | +```sh |
| 144 | +src/generate.ts \ |
| 145 | + --schema /path/to/myservice_schema.json \ |
| 146 | + --lang ts \ |
| 147 | + --out /path/to/output/generated \ |
| 148 | + --client \ |
| 149 | + --prefix MyService --strip-method-prefix --curve-constants |
| 150 | +``` |
| 151 | + |
| 152 | +Produces `api_types.ts`, `async.ts`, `sync.ts`, `curve_constants.ts`. The TS |
| 153 | +client uses `@aztec/ipc-runtime`'s `UdsIpcClient` or `NapiShmSyncClient` for |
| 154 | +transport — no template copy. |
| 155 | + |
| 156 | +### TypeScript spawned-service package |
| 157 | + |
| 158 | +```sh |
| 159 | +src/generate.ts \ |
| 160 | + --schema /path/to/myservice_schema.json \ |
| 161 | + --lang ts \ |
| 162 | + --out /path/to/myservice/src/generated \ |
| 163 | + --client --strip-method-prefix \ |
| 164 | + --prefix MyService \ |
| 165 | + --package /path/to/myservice \ |
| 166 | + --package-name @aztec/myservice \ |
| 167 | + --binary-name myservice \ |
| 168 | + --package-transports uds,shm |
| 169 | +``` |
| 170 | + |
| 171 | +Produces the generated TS client under `src/generated/` plus a package shell |
| 172 | +(`package.json`, `tsconfig.json`, `src/index.ts`, `src/platform.ts`, and |
| 173 | +`scripts/prepare_arch_packages.sh`). The package exports a |
| 174 | +`MyServiceService.spawn(...)` helper that launches the native binary and wraps |
| 175 | +the generated async client. `scripts/prepare_arch_packages.sh` turns |
| 176 | +`build/<platform>/<binary>` directories into per-architecture npm packages |
| 177 | +matching the binary resolution path. |
| 178 | + |
| 179 | +### C++ server + client, under a project sub-include path |
| 180 | + |
| 181 | +```sh |
| 182 | +src/generate.ts \ |
| 183 | + --schema /path/to/myservice_schema.json \ |
| 184 | + --lang cpp \ |
| 185 | + --out /path/to/myservice/generated \ |
| 186 | + --server --client \ |
| 187 | + --cpp-namespace my::ns --prefix MyService \ |
| 188 | + --cpp-include-dir myservice/generated |
| 189 | +``` |
| 190 | + |
| 191 | +Produces `myservice_types.hpp`, `myservice_ipc_client.{hpp,cpp}`, and |
| 192 | +`myservice_ipc_server.hpp`. Cross-includes use the supplied `--cpp-include-dir` prefix |
| 193 | +(`#include "myservice/generated/myservice_types.hpp"`). Wire to an |
| 194 | +`ipc::IpcServer` (from ipc-runtime) plus a hand-written |
| 195 | +`<service>_handlers.cpp` that supplies one `handle_<method>(...)` per command. |
| 196 | +Generated C++ includes support headers as `ipc_codegen/...`; the generator |
| 197 | +copies those headers from `templates/cpp/ipc_codegen/` into the output |
| 198 | +directory. |
| 199 | + |
| 200 | +### Rust client + FFI backend |
| 201 | + |
| 202 | +```sh |
| 203 | +src/generate.ts \ |
| 204 | + --schema /path/to/myservice_schema.json \ |
| 205 | + --lang rust \ |
| 206 | + --out /path/to/crate/src/generated \ |
| 207 | + --client --uds --ffi \ |
| 208 | + --prefix MyService \ |
| 209 | + --skeleton /path/to/crate/src |
| 210 | +``` |
| 211 | + |
| 212 | +Produces `myservice_types.rs`, `myservice_client.rs`, plus `backend.rs`, |
| 213 | +`error.rs`, `ffi_backend.rs`. UDS/SHM transport is provided by the |
| 214 | +`ipc-runtime` Rust crate; the consumer chooses which to use via the path |
| 215 | +suffix passed at runtime. The skeleton flag also writes a one-time |
| 216 | +`myservice_handlers.rs`, `main.rs`, `Cargo.toml`, and `generate.sh` into the |
| 217 | +skeleton dir so a new service crate is buildable on first run. |
| 218 | + |
| 219 | +### Zig client + server |
| 220 | + |
| 221 | +```sh |
| 222 | +src/generate.ts \ |
| 223 | + --schema /path/to/myservice_schema.json \ |
| 224 | + --lang zig \ |
| 225 | + --out /path/to/output/generated \ |
| 226 | + --server --client --uds --ffi \ |
| 227 | + --prefix MyService |
| 228 | +``` |
| 229 | + |
| 230 | +Produces `myservice_types.zig`, `myservice_client.zig`, |
| 231 | +`myservice_server.zig`, plus `backend.zig` and `ffi_backend.zig`. Consumers |
| 232 | +`@import("ipc_runtime")` for transport. |
| 233 | + |
| 234 | +## Adding a new service |
| 235 | + |
| 236 | +1. **Define the C++ command structs** in your service's `.hpp`, each with |
| 237 | + `MSGPACK_SCHEMA_NAME` and `SERIALIZATION_FIELDS(...)`. Group them into a |
| 238 | + single `Command` and `Response` `NamedUnion`. |
| 239 | +2. **Snapshot the schema.** Build the service binary and run |
| 240 | + `<binary> msgpack schema` to dump the JSON. Commit it next to the C++ |
| 241 | + source that defines it (e.g. alongside the `Command` / `Response` |
| 242 | + headers). This is the wire-format source of truth. |
| 243 | +3. **Wire your consumer's build to invoke `src/generate.ts`** with the flags |
| 244 | + above, passing the absolute path to the committed schema and the desired |
| 245 | + output directory. Generated files go under a `generated/` directory which |
| 246 | + is gitignored by convention. |
| 247 | +4. **Wire transport.** On the C++ server side, instantiate an |
| 248 | + `ipc::IpcServer` via `ipc::make_server(path)` (from ipc-runtime) and feed |
| 249 | + it the codegen-emitted `make_<prefix>_handler(...)`. On the client side |
| 250 | + (any language), point an `ipc::IpcClient` / equivalent at the same path |
| 251 | + and wrap it with the codegen-emitted client. |
| 252 | +5. **Run `./bootstrap.sh test`** in `ipc-codegen/` to confirm the codegen and |
| 253 | + cross-language wire-compat tests still pass. |
| 254 | + |
| 255 | +## Schemas are the source of truth |
| 256 | + |
| 257 | +The JSON schema is the wire contract between client and server. Consumers |
| 258 | +commit it next to the C++ server that defines the underlying |
| 259 | +`SERIALIZATION_FIELDS`, so the file lives close to what it describes and |
| 260 | +tracks with that code. Whenever a server-side command changes, refresh the |
| 261 | +JSON snapshot by running `<binary> msgpack schema` against the rebuilt |
| 262 | +binary and committing the diff. Diverged schemas are a CI failure (each |
| 263 | +consumer is responsible for guarding its own snapshot). |
| 264 | + |
| 265 | +Each generated file embeds a `SCHEMA_HASH` so callers can detect at |
| 266 | +connection time that their bindings predate the server. |
| 267 | + |
| 268 | +## Wire-format contract |
| 269 | + |
| 270 | +`echo_example/schema/golden/*.msgpack` is a frozen set of byte-level |
| 271 | +fixtures covering every relevant msgpack encoding boundary (variable-width |
| 272 | +ints, fixstr/str8/str16, bin8/bin16, optional `Some`/`None`, empty |
| 273 | +containers, multi-byte UTF-8). The per-language golden tests |
| 274 | +(`echo_example/{rust,ts}/...`) both decode the fixtures and re-encode |
| 275 | +round-trip — pinning down canonical msgpack output across implementations. |
| 276 | + |
| 277 | +If you intentionally change the wire format, run |
| 278 | +`./bootstrap.sh update_goldens` and review the diff. Any byte-level change |
| 279 | +is a breaking change for external implementations of the schema. |
| 280 | + |
| 281 | +See `SCHEMA_SPEC.md` for the wire protocol details. |
0 commit comments