feat(ffi): per-object metadata accessors so C clients don't reimplement CBOR by tlmquintino · Pull Request #156 · ecmwf/tensogram

tlmquintino · 2026-07-19T13:36:28Z

Problem

The C FFI exposed metadata only through first-match getters —
tgm_metadata_get_string/int/float(meta, key) internally scan every base[i]
and return the first hit. For a multi-object message (e.g. a merged GRIB
re-encoded as one tensogram message), base[i] for i>0 is unreachable, so a
C client that needs per-object metadata (short name, level, geometry, …) had to
reimplement CBOR decoding to walk the metadata frame by hand. That's exactly
what the ushow atmospheric-data viewer had to do, coupling it to tensogram's
wire format.

Design

Grounded in the canonical Rust model — GlobalMetadata { base: Vec<Map>, … },
where per-object metadata is base[i] — and Python's meta.base[i] full-dict
exposure. Two cohesive, backward-compatible additions (existing functions
untouched):

1. Per-object typed getters — the faithful C translation of
meta.base[obj_index].get(key), scoped to one object, dot-path nested
navigation, _reserved_ skipped (as the message-level getters do):

tgm_metadata_get_string_at(meta, obj_index, key)
tgm_metadata_get_int_at(meta, obj_index, key, default)
tgm_metadata_get_float_at(meta, obj_index, key, default)

2. JSON export — for callers that want to enumerate/display an object's
whole metadata without knowing the keys (mirrors Python's meta.base[i]
dict; no client parser needed for display):

tgm_metadata_object_to_json(meta, obj_index) → full base[i] as JSON
tgm_metadata_to_json(meta) → {base, _reserved_, _extra_}

Both JSON exporters and the string getter return handle-cached pointers (no
separate free), consistent with the existing tgm_metadata_get_string.

The C++ RAII wrapper gains matching metadata::get_string_at/get_int_at/ get_float_at/object_to_json/to_json.

What this replaces for ushow

// before: hand-rolled cbor_head/cbor_skip/cbor_map_find over the raw frames…
// after:
tgm_metadata_t *m; tgm_decode_metadata(buf, len, &m);
for (size_t i = 0; i < tgm_metadata_num_objects(m); i++) {
    const char *name = tgm_metadata_get_string_at(m, i, "shortName");
    int64_t level    = tgm_metadata_get_int_at(m, i, "level", -1);
    const char *grid = tgm_metadata_get_string_at(m, i, "geometry.gridType");
    // or: const char *json = tgm_metadata_object_to_json(m, i);  // display all
}

Tests

Rust FFI: 10 new unit + extern-ABI tests (per-object scoping, nested
dot-paths, _reserved_ skip, out-of-range/null, JSON round-trip, CBOR→JSON
edge cases). Full FFI suite: 179 lib + 32 + 100 doctests green.
C++: 4 new MetadataTest cases (per-object scoping vs first-match, int/
float/nested, out-of-range, JSON enumeration). Full ctest: 186/186 green.
Pure C: extended the cargo-c smoke test with a 2-object metadata walk —
the ushow scenario, no CBOR reader. Validated locally by compiling the C
program against the built lib:
n=2 base0.shortName=2t base1.shortName=msl base1.level=500 base1.grid=regular_ll.
cargo fmt + clippy -D warnings clean; cbindgen header regenerated in place
(no manual drift).

Docs Preview
https://sites.ecmwf.int/docs/tensogram/pull-requests/PR-156

The C API exposed metadata only through first-match getters (tgm_metadata_get_string/int/float, which scan every base[i] and return the first hit), so per-object metadata in a multi-object message was unreachable — forcing C clients (e.g. the ushow viewer) to reimplement CBOR parsing to walk the metadata frame themselves. Add per-object accessors, the C equivalent of Rust's meta.base[obj_index]: - tgm_metadata_get_string_at / _int_at / _float_at(meta, obj_index, key, ...) scope the lookup to one object (dot-paths navigate nested maps, _reserved_ is skipped, as with the message-level getters). - tgm_metadata_object_to_json(meta, obj_index) / tgm_metadata_to_json(meta) export the full metadata as JSON so a viewer can enumerate/display an object without knowing the key set (mirrors Python's meta.base[i] dict). Both JSON exporters and the string getter return handle-cached pointers (no separate free), consistent with the existing tgm_metadata_get_string. Existing functions are unchanged. The cbindgen header is regenerated in place. Extend the pure-C cargo-c smoke test to build a 2-object message and walk each object's metadata via the new accessors — the ushow scenario, no CBOR reader.

Add metadata::get_string_at / get_int_at / get_float_at(obj_index, key, ...) and object_to_json / to_json to the RAII metadata wrapper, forwarding to the new C FFI functions. Tests build a 2-object message via the top-level "base" array and assert per-object scoping, nested dot-paths, out-of-range defaults, and the JSON enumeration path.

Review pass (1-2). Two helpers remove the duplication the per-object accessors introduced: - cbor_map_to_json() replaces the CBOR-map -> JSON-object idiom that was written out four times (once in object_to_json, three times in to_json). - cache_cstr() replaces the cache-entry-and-return-pointer boilerplate inlined in all four string-returning accessors (get_string, get_string_at, and both JSON exporters). No API/ABI change (the generated header is byte-identical, guarded by the cargo-c header-drift check); behaviour and all tests unchanged.

Pass 3-4 hardening. Add tests pinning the edge behaviours: - get_string_at on a container value (a map) returns NULL, not a stringified map - obj_index == num_objects (the off-by-one boundary) misses cleanly, no panic - null handle on the JSON exporters and _at getters returns NULL / default - to_json on empty metadata serialises to "{}" - cache_cstr degrades an interior-NUL value to "" rather than panicking (C strings cannot hold NUL) No library-code change; all accessors were already panic-free.

Pass 5 polish. cache_cstr now takes a FnOnce closure instead of an eagerly-built String, so the value — for the JSON exporters, the whole serialisation — is produced only on a cache miss; a repeat object_to_json/ to_json call returns the cached pointer without re-serialising. Document the cache-key namespacing scheme (\0-separated, disjoint from NUL-free C keys). Add zero-tolerance edge/error-path tests: no cross-object first-match bleed at the FFI boundary, a NUL-bearing value degrading to "" end-to-end via get_string_at, get_float_at wrong-type -> default, array serialisation in object_to_json, and cache-pointer stability across repeat calls (proving the lazy build runs once). No API/ABI change (header byte-identical).

Copilot

Pull request overview

This PR extends the tensogram C FFI to expose per-object (base[i]) metadata and JSON exports, eliminating the need for C clients to manually decode CBOR when working with multi-object messages. The C++ RAII wrapper is updated to mirror the new FFI surface, and tests are added/extended to validate the new behavior end-to-end.

Changes:

Added per-object metadata typed getters (*_get_{string,int,float}_at) plus JSON exporters (*_object_to_json, *_to_json) to the C FFI.
Refactored/extended Rust FFI internals to support object-scoped dot-path lookups and CBOR→JSON conversion with handle-cached string storage.
Updated the C++ wrapper API and expanded C++ + cargo-c smoke tests; documented the feature in CHANGELOG.md.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
rust/tensogram-ffi/tensogram.h	Declares new per-object metadata getters and JSON export functions in the C header.
rust/tensogram-ffi/src/lib.rs	Implements per-object accessors, JSON export, cache helper, and adds Rust tests covering the new FFI behavior.
cpp/include/tensogram.hpp	Adds C++ `metadata` wrapper methods matching the new C FFI entry points.
cpp/tests/test_metadata.cpp	Adds C++ tests validating per-object scoping and JSON export behavior on multi-object messages.
cpp/tests/test_cargo_c_smoke.sh	Extends the cargo-c smoke test with a pure-C multi-object per-object metadata walk using the new accessors.
CHANGELOG.md	Documents the newly added per-object metadata accessors and JSON export APIs.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

A metadata string value containing an interior NUL byte cannot be a C string. cache_cstr previously fell back to unwrap_or_default(), returning a non-NULL pointer to a silently-truncated empty string — which conflates with a genuinely empty value and contradicted the getters' docstrings ("returns NULL if ..."). Return NULL instead: a value with an interior NUL is 'no usable value', matching the existing 'not a string' NULL contract and unambiguous for callers. Update the get_string / get_string_at docs (and the regenerated header) to state it; the JSON exporters are unaffected (serde escapes NUL as \u0000, so serialised JSON never contains a literal NUL). Tests updated to assert NULL.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

get_string's doc claimed it returns NULL when the value 'is not a string', but cbor_as_string coerces integer/float/bool leaves to their string form (only map/array/null/bytes yield NULL). Correct the doc to say so, and while aligning the getter family: spell out get_string_at's precise NULL cases (container / null / bytes) and note that the message-level get_float widens integer leaves and returns default_val on a non-numeric value. Regenerated header included.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

The C++ get_string_at doc said '_reserved_ is skipped', implying unconditional; the underlying lookup only blocks it at the first path segment (a nested 'foo._reserved_.bar' would be reachable). Align the wording with the C FFI and Rust docs ('at the first level').

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

The bytes->hex conversion allocated a temporary String per byte via format!("{x:02x}"). Write into a pre-sized String buffer instead — one allocation regardless of length — keeping object_to_json/to_json fast on byte-valued metadata (e.g. a local-section blob).

tgm_metadata_get_string / get_int special-case the key "version" (returning the wire-format version from the preamble, via lookup_*_key), but the docs only described CBOR-backed lookups. Document the pseudo-key so consumers don't assume a CBOR version field is read. Regenerated header included.

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

make mutants-diff hard-codes --features async,remote, which cargo-mutants applies to whatever package the diff touches; only the core tensogram crate has a remote feature, so a diff confined to tensogram-ffi (remote surface is async-remote) fails to build before any mutant runs. Gate the job behind the RUN_MUTANTS repository variable (unset -> skipped) so it stops running on PRs and main; re-enable by setting RUN_MUTANTS=true once feature selection is made per-package.

tlmquintino added 2 commits July 19, 2026 14:35

tlmquintino requested review from HCookie, sametd and tmi as code owners July 19, 2026 13:36

tlmquintino added 3 commits July 19, 2026 14:55

tlmquintino requested a review from Copilot July 20, 2026 06:42

Copilot started reviewing on behalf of tlmquintino July 20, 2026 06:43 View session