Skip to content

Commit 275fbe4

Browse files
authored
feat(linkable): add allocation-free postcard encode_into (#180)
* feat(linkable): add allocation-free postcard `encode_into` Add a caller-provided buffer API to `Linkable` while retaining the existing from_bytes and to_bytes signatures for compatibility. Introduce `LinkCodecError` and an optional into-slice serializer path in core. Each outbound route owns one bounded reusable scratch buffer, validates the returned length, and falls back to the existing `Vec` serializer when the buffer is too small. Legacy and custom codecs keep their previous behavior. Generate a true postcard override using `postcard::to_slice` with a 256-byte preferred capacity. Extend codegen drift coverage for generated connector wiring, exact and undersized buffers, Cortex-M compilation and the JSON-free dependency graph. Add an allocation-counting benchmark covering 10000 `encode_into` calls with zero allocation calls and zero allocated bytes. This removes the codec allocation only, connector-internal payload copies remain outside the claim. * refactor(linkable): reuse `SerializeError` for `encode_into` Remove the newly introduced `LinkCodecError` and reuse the existing `aimdb_core::connector::SerializeError` across the into-slice API, postcard codegen, tests, benchmarks and documentation. Keep the legacy Linkable `from_bytes`/`to_bytes` String errors unchanged under the compatibility-first approach agreed in #177.
1 parent 9bed3e2 commit 275fbe4

20 files changed

Lines changed: 896 additions & 37 deletions

File tree

CHANGELOG.md

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

88
> **Note**: This is the global changelog for the AimDB project. For detailed changes to individual crates, see their respective CHANGELOG.md files:
9+
>
910
> - [aimdb-core/CHANGELOG.md](aimdb-core/CHANGELOG.md)
1011
> - [aimdb-codegen/CHANGELOG.md](aimdb-codegen/CHANGELOG.md)
1112
> - [aimdb-data-contracts/CHANGELOG.md](aimdb-data-contracts/CHANGELOG.md)
@@ -105,6 +106,10 @@ multi-step migration round-trip suite and a trybuild compile-fail harness.
105106

106107
### Added
107108

109+
- **Zero-allocation generated Postcard encoding (Issue #177).** `Linkable` gains a source-compatible `encode_into(&mut [u8])` fast path and generated Postcard records implement it with `postcard::to_slice`. Core reuses onebounded 256-byte scratch allocation per generated Postcard outbound route (raw builders choose their capacity) and falls back to the existing owned serializer when a value does not fit. The codec seam is gated by a 10,000-iteration allocation-counting bench (0 allocations, 0 bytes) plus
110+
host, `no_std` Cortex-M, and codegen-drift tests. The claim intentionally excludes connector-internal payload copies and the existing boxed connector reader future. ([aimdb-core](aimdb-core/CHANGELOG.md),
111+
[aimdb-data-contracts](aimdb-data-contracts/CHANGELOG.md),
112+
[aimdb-codegen](aimdb-codegen/CHANGELOG.md))
108113
- **Embassy MQTT client TLS — `mqtts://` for embedded, WP7 ([design 044](docs/design/044-embassy-mqtt-tls.md)).** The Embassy MQTT connector can now dial a broker over TLS 1.3 (`embedded-tls`, pure-Rust `rustpki` certificate verification with `rsa`/`p384` so public CA chains verify out of the box), with SNI/hostname verification, DNS resolution, and a connector-internal SNTP time source that gates the first handshake so certificate validity is always checked against real time. Behind the new `embassy-tls` feature; `MqttConnectorBuilder::new` picks the transport from the URL scheme (`mqtt://` plain, `mqtts://` TLS) and gains `with_tls(TlsOptions)` (entropy, record buffers, SNTP server) and `with_credentials(username, password)` (MQTT CONNECT auth, both transports). The plain `mqtt://` path is untouched. The `embassy-mqtt-connector-demo` example gains a `tls` feature demonstrating the full flow against the repo's `dev/mosquitto` bench broker. ([aimdb-mqtt-connector](aimdb-mqtt-connector/CHANGELOG.md))
109114
- **Design 034 Phase 3 — sans-io KNX/IP tunneling engine shared by both transports (Issue #135, [review doc §3.7](docs/design/034-technical-debt-review.md)).** The entire tunneling lifecycle — CONNECT_REQUEST/RESPONSE handshake, TUNNELING_REQUEST/ACK sequence + pending-ACK bookkeeping, keepalive (CONNECTIONSTATE_REQUEST) scheduling, ACK-timeout sweeps, and reconnect-with-backoff — now lives **once**, in the new runtime-neutral `aimdb_knx_connector::tunnel` module (`no_std + alloc`, no tokio/embassy imports), driven as a poll-based state machine (events in, `Action`s out, `next_deadline()` for timer arming). `tokio_client.rs` (988 → ~530 lines incl. a new fake-gateway integration test) and `embassy_client.rs` (1,055 → ~450 lines) are reduced to socket shims; the previously untestable handshake/ACK/keepalive/reconnect paths now have 15 host-run unit tests plus a scripted localhost-UDP roundtrip test. Behavioral unifications: Embassy gains the 5 s CONNECT_RESPONSE timeout (previously waited forever), both shims reconnect on fatal socket errors, and the dead tokio-only per-publish ACK oneshot is dropped (it was always `None`); the CONNECT_REQUEST HPAI stays per-transport (`LocalEndpoint`: tokio = real bound address, Embassy = NAT mode). ([aimdb-knx-connector](aimdb-knx-connector/CHANGELOG.md))
110115

@@ -418,6 +423,7 @@ builder.configure::<Temperature>("sensor.temperature", |reg| {
418423
```
419424

420425
**Key naming conventions:**
426+
421427
- Use dot-separated hierarchical names: `"sensors.indoor"`, `"config.app"`
422428
- Keys must be unique across all records (duplicate keys panic at registration)
423429
- For single-instance records, any descriptive string works (e.g., `"sensor.temperature"`, `"app.config"`)
@@ -513,6 +519,7 @@ This release introduces **bidirectional connector support**, enabling true two-w
513519
### Modified Crates
514520

515521
See individual changelogs for detailed changes:
522+
516523
- **[aimdb-core](aimdb-core/CHANGELOG.md)**: Core connector architecture, router system, bidirectional APIs
517524
- **[aimdb-tokio-adapter](aimdb-tokio-adapter/CHANGELOG.md)**: Connector builder integration
518525
- **[aimdb-embassy-adapter](aimdb-embassy-adapter/CHANGELOG.md)**: Network stack access, connector support
@@ -522,6 +529,7 @@ See individual changelogs for detailed changes:
522529
### Migration Guide
523530

524531
**1. Update connector registration:**
532+
525533
```rust
526534
// Old (v0.1.0)
527535
let mqtt = MqttConnector::new("mqtt://broker:1883").await?;
@@ -532,6 +540,7 @@ builder.with_connector(MqttConnectorBuilder::new("mqtt://broker:1883"))
532540
```
533541

534542
**2. Make build() async:**
543+
535544
```rust
536545
// Old (v0.1.0)
537546
let db = builder.build()?;
@@ -541,6 +550,7 @@ let db = builder.build().await?;
541550
```
542551

543552
**3. Use directional link methods:**
553+
544554
```rust
545555
// Old (v0.1.0)
546556
.link("mqtt://sensors/temp")
@@ -551,6 +561,7 @@ let db = builder.build().await?;
551561
```
552562

553563
**4. Remove manual MQTT task spawning (Embassy):**
564+
554565
```rust
555566
// Old (v0.1.0) - Manual task spawning required
556567
let mqtt_result = MqttConnector::create(...).await?;
@@ -627,12 +638,14 @@ impl MyConnector {
627638
```
628639

629640
**Why this change?** The new trait-based architecture provides:
641+
630642
- ✅ Symmetry with inbound routing (`ProducerTrait``ConsumerTrait`)
631643
- ✅ Testability (can mock `ConsumerTrait` without real records)
632644
- ✅ Type safety via factory pattern (type capture at configuration time)
633645
- ✅ Maintainability (connector logic stays in connector crate)
634646

635647
**Migration checklist for custom connectors:**
648+
636649
- [ ] Add `spawn_outbound_publishers()` method to connector implementation
637650
- [ ] Call `db.collect_outbound_routes(protocol_name)` in `ConnectorBuilder::build()`
638651
- [ ] Call `connector.spawn_outbound_publishers(db, routes)?` before returning
@@ -653,6 +666,7 @@ See [Connector Development Guide](docs/design/012-M5-connector-development-guide
653666
### Added
654667

655668
#### Core Database (`aimdb-core`)
669+
656670
- Initial release of AimDB async in-memory database engine
657671
- Type-safe record system using `TypeId`-based routing
658672
- Three buffer types for different data flow patterns:
@@ -666,6 +680,7 @@ See [Connector Development Guide](docs/design/012-M5-connector-development-guide
666680
- Remote access protocol (AimX v1) for cross-process introspection
667681

668682
#### Runtime Adapters
683+
669684
- **Tokio Adapter** (`aimdb-tokio-adapter`): Full-featured std runtime support
670685
- Lock-free buffer implementations
671686
- Configurable buffer capacities
@@ -676,6 +691,7 @@ See [Connector Development Guide](docs/design/012-M5-connector-development-guide
676691
- Compatible with ARM Cortex-M targets
677692

678693
#### MQTT Connector (`aimdb-mqtt-connector`)
694+
679695
- Dual runtime support for both Tokio and Embassy
680696
- Automatic consumer registration via builder pattern
681697
- Topic mapping with QoS and retain configuration
@@ -685,6 +701,7 @@ See [Connector Development Guide](docs/design/012-M5-connector-development-guide
685701
- Uses `mountain-mqtt` for embedded environments
686702

687703
#### Developer Tools
704+
688705
- **MCP Server** (`aimdb-mcp`): LLM-powered introspection and debugging
689706
- Discover running AimDB instances
690707
- Query record values and schemas
@@ -702,12 +719,14 @@ See [Connector Development Guide](docs/design/012-M5-connector-development-guide
702719
- Clean error handling
703720

704721
#### Synchronous API (`aimdb-sync`)
722+
705723
- Blocking wrapper around async AimDB core
706724
- Thread-safe synchronous record access
707725
- Automatic Tokio runtime management
708726
- Ideal for gradual migration from sync to async
709727

710728
#### Documentation & Examples
729+
711730
- Comprehensive README with architecture overview
712731
- Individual crate documentation with examples
713732
- 12 detailed design documents in `/docs/design`
@@ -718,6 +737,7 @@ See [Connector Development Guide](docs/design/012-M5-connector-development-guide
718737
- `remote-access-demo`: Cross-process introspection server
719738

720739
#### Build & CI Infrastructure
740+
721741
- Comprehensive Makefile with color-coded output
722742
- GitHub Actions workflows:
723743
- Continuous integration (format, lint, test)
@@ -729,28 +749,33 @@ See [Connector Development Guide](docs/design/012-M5-connector-development-guide
729749
- Dev container setup for consistent development environment
730750

731751
### Design Goals Achieved
752+
732753
- ✅ Sub-50ms latency for data synchronization
733754
- ✅ Lock-free buffer operations
734755
- ✅ Cross-platform support (MCU → edge → cloud)
735756
- ✅ Type safety with zero-cost abstractions
736757
- ✅ Protocol-agnostic connector architecture
737758

738759
### Known Limitations
760+
739761
- Kafka and DDS connectors planned for future releases
740762
- CLI tool is currently skeleton implementation
741763
- Performance benchmarks not yet included
742764
- Limited to Unix domain sockets for remote access (no TCP yet)
743765

744766
### Dependencies
767+
745768
- Rust 1.75+ required
746769
- Tokio 1.47+ for std environments
747770
- Embassy 0.9+ for embedded environments
748771
- See `deny.toml` for approved dependency licenses
749772

750773
### Breaking Changes
774+
751775
None (initial release)
752776

753777
### Migration Guide
778+
754779
Not applicable (initial release)
755780

756781
---
@@ -762,6 +787,7 @@ Not applicable (initial release)
762787
AimDB v0.1.0 establishes the foundational architecture for async, in-memory data synchronization across MCU → edge → cloud environments. This release focuses on core functionality, dual runtime support, and developer tooling.
763788

764789
**Highlights:**
790+
765791
- 🚀 Dual runtime support: Works on both standard library (Tokio) and embedded (Embassy)
766792
- 🔒 Type-safe record system eliminates runtime string lookups
767793
- 📦 Three buffer types cover most data patterns
@@ -770,6 +796,7 @@ AimDB v0.1.0 establishes the foundational architecture for async, in-memory data
770796
- ✅ 27+ core tests, comprehensive CI/CD, security auditing
771797

772798
**Get Started:**
799+
773800
```bash
774801
cargo add aimdb-core aimdb-tokio-adapter
775802
```
@@ -778,7 +805,7 @@ See [README.md](README.md) for quickstart guide and examples.
778805

779806
**Feedback Welcome:**
780807
This is an early release. Please report issues, suggest features, or contribute at:
781-
https://github.com/aimdb-dev/aimdb
808+
<https://github.com/aimdb-dev/aimdb>
782809

783810
---
784811

Cargo.lock

Lines changed: 29 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

aimdb-bench/Cargo.toml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,10 @@ harness = false
4040
name = "b0_alloc_migration"
4141
harness = false
4242

43+
[[bench]]
44+
name = "b0_alloc_linkable"
45+
harness = false
46+
4347
[features]
4448
default = ["std"]
4549
# Gates `profiles`/`reports`/`harness` and their criterion/serde_json/
@@ -89,6 +93,7 @@ serde_json = { workspace = true, optional = true }
8993
# criterion/serde_json above, since that's the only bench that needs it.
9094
aimdb-data-contracts = { path = "../aimdb-data-contracts", default-features = false, features = [
9195
"alloc",
96+
"linkable",
9297
"migratable",
9398
], optional = true }
9499

@@ -124,3 +129,4 @@ critical-section = { version = "1.1", features = ["std"] }
124129
# defmt logger / panic-handler + embassy-time driver stubs for the host bench
125130
defmt = { workspace = true }
126131
embassy-time-driver = { path = "../_external/embassy/embassy-time-driver" }
132+
postcard = { version = "1.0", default-features = false, features = ["alloc"] }

aimdb-bench/README.md

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ Plus two informational benches that exercise the full runner-driven pipeline.
2020
**Adapters covered:**
2121

2222
- **Tokio**`b0_alloc_tokio`, `b1_b2_tokio` (host).
23+
- **postcard `Linkable` codec**`b0_alloc_linkable` (host). This isolates the
24+
issue #177 `encode_into` seam; it is not a whole-connector allocation claim.
2325
- **Embassy**`b0_alloc_embassy`, `b1_b2_embassy`
2426
(host). These drive the real [`EmbassyBuffer`] backend via
2527
`futures::executor::block_on` over embassy-sync's poll methods — no
@@ -53,6 +55,9 @@ Always run from the workspace root (`/aimdb_ws/aimdb`).
5355
# B0 — allocation gate (buffer layer)
5456
cargo bench -p aimdb-bench --bench b0_alloc_tokio
5557

58+
# B0 — allocation gate (Postcard Linkable::encode_into codec seam)
59+
cargo bench -p aimdb-bench --bench b0_alloc_linkable
60+
5661
# B1 + B2 — latency (time/iter) and throughput (msgs/sec), one Criterion suite
5762
cargo bench -p aimdb-bench --bench b1_b2_tokio
5863

@@ -91,6 +96,7 @@ Criterion writes HTML reports to `target/criterion/`.
9196
`b0_alloc_tokio` does not use Criterion. It runs a fixed warmup + batch cycle and writes JSON results to `aimdb-bench/target/bench-results/b0_alloc_tokio.json` (the path is anchored to the crate dir, so it is the same regardless of the directory you run from).
9297

9398
**Measurement model:**
99+
94100
1. Create buffer + reader.
95101
2. Warmup ≥ 200 push → recv cycles (excluded from counters).
96102
3. Reset allocation counters.
@@ -103,6 +109,9 @@ The committed baseline lives in `data/baselines/b0_alloc_tokio.json`. When a cha
103109
104110
`b0_alloc_embassy` mirrors this against the Embassy buffer backend and writes `data/baselines/b0_alloc_embassy.json` — also **0 allocs/msg** across all three profiles, confirming the Embassy `poll_recv` path is allocation-free on the host. The on-target B3 bench (`examples/embassy-bench-stm32h5`) re-checks the same 0-alloc claim against the real embedded allocator.
105111

112+
`b0_alloc_linkable` warms up for 1,000 iterations, then measures 10,000 generated-shape postcard `Linkable::encode_into` calls into one stack buffer.
113+
The required result is **0 allocation calls and 0 allocated bytes**. It isolates the codec seam: `SerializedReader` still returns a boxed future, dynamic topics may allocate and connector implementations may copy payload ownership after the core pump lends them the scratch slice.
114+
106115
> **Embassy eager registration (design 039 F8/F9).** An Embassy `SpmcRing` reader registers its embassy `Subscriber` eagerly, at `subscribe()` time — matching Tokio's `broadcast` — so no separate priming step is needed before the first `push`.
107116
108117
**Noise reduction:** a `new_current_thread()` Tokio executor is used so there are no work-stealing threads and Tokio's scheduler does not allocate per-poll in the hot path.
@@ -127,5 +136,3 @@ Use them as a comparison point, not a regression gate. If they regress, `b0_allo
127136
- Criterion p99 can vary ±5–10% on noisy CI runners. Use p50 medians for trend comparisons.
128137
- Always specify `--release` or debug build consistently when comparing runs; optimizations differ by 5–50×.
129138
- `b_alloc_pipeline` uses a paced source: per-message pace tokens and notification channels. The coordination overhead is included in the measured window.
130-
131-

0 commit comments

Comments
 (0)