Skip to content

metadata-lifecycle.mdx

Latest commit

 

History

History
182 lines (134 loc) · 13.4 KB

metadata-lifecycle.mdx

File metadata and controls

182 lines (134 loc) · 13.4 KB
Raw
title Metadata Lifecycle & HMR
description How metadata flows through Repository → Change Log → Cache → Registry — the canonical event stream that powers Studio HMR, REST writes, and future cloud editing.

Metadata Lifecycle & HMR

This page documents the metadata data path introduced by ADR-0008 and refined by ADR-0005. It is the canonical event stream that powers Studio Hot Module Replacement (HMR), REST writes, and future cloud editing.

**TL;DR** — Every write goes through a single `MetadataRepository.put()` call. The repository appends to a change log, emits a watch event with a monotonic `seq`, and broadcasts over SSE to the Studio UI. The Studio's `useMetadataHmr` hook displays the latest `seq` in the status badge and re-fetches affected views.

Architecture (one diagram)

   ┌─────────────────────────────────────────────┐
   │                  Studio UI                  │
   │      (useMetadataHmr / HmrStatusBadge)      │
   └────────────────────┬────────────────────────┘
                        │ SSE  /api/v1/dev/metadata-events
                        ▼
   ┌─────────────────────────────────────────────┐
   │           MetadataManager (bridge)          │
   │   forwards Repository events → SSE channel  │
   └────────────────────┬────────────────────────┘
                        │ ChangeEvent{kind, metadataType, name, seq?}
                        ▼
   ┌─────────────────────────────────────────────┐
   │            LayeredRepository                │
   │   ┌─────────────────────────────────────┐   │
   │   │  Top: SysMetadataRepository (org)   │   │ ← writable overlay
   │   │  Bottom: FS / InMemory (artifact)   │   │ ← read-only baseline
   │   └─────────────────────────────────────┘   │
   └────────────────────┬────────────────────────┘
                        │ append(change log) + emit watch event
                        ▼
                   sys_metadata table
                  (or filesystem artifact)

Reads walk top-to-bottom: the first non-null layer wins. Writes always route to the topmost writable layer (the overlay). Deletes from the layered API only remove the overlay row — the artifact baseline survives.

The four primitives

Primitive Package Purpose
Repository @objectstack/metadata-core CRUD + watch interface over a single metadata source. InMemoryRepository and LayeredRepository ship from @objectstack/metadata-core; FileSystemRepository from @objectstack/metadata-fs; SysMetadataRepository from @objectstack/objectql.
Change Log @objectstack/metadata-core Append-only log of every mutation, tagged with a monotonic seq. Watchers can replay from any since.
Cache @objectstack/metadata In-memory snapshot of the registry, keyed by MetaRef. Invalidated by change-log events.
Registry @objectstack/metadata Typed registry the Kernel and plugins query. Built from the cache.

MetaRef = (type, name, org). As of ADR-0008 §0 amendment (2026-04-13), project and branch are removed from the runtime tuple. Project survives only as an artifact-packaging concept on the objectstack.json envelope; branching is left to Git.

Write path: Studio → SSE → repository

When a user edits a view in Studio:

  1. Studio calls PUT /api/v1/metadata/views/case_grid (REST).
  2. protocol.ts:saveMetaItem() validates against MetadataTypeRegistry.allowOrgOverride (see overlay whitelist).
  3. If allowed, the call lands on MetadataRepository.put(ref, body, { parentVersion, actor }).
  4. The repository:
    • Verifies parentVersion matches the current head (ConflictError on mismatch).
    • Writes the new row + appends a change-log entry with the next seq.
    • Emits a MetadataEvent { op: 'create' | 'update' | 'delete', ref, seq, source }.
  5. The MetadataManager bridge forwards the event over the SSE channel /api/v1/dev/metadata-events. The wire payload is not the internal MetadataEvent — it is a ChangeEvent { kind: 'metadata-change', type, metadataType, name, path?, timestamp, seq? } (emitted as event: metadata-change). seq is the canonical repository sequence and is absent for FS-watcher (chokidar) dev events.
  6. The Studio's useMetadataHmr hook receives the event, updates lastSeq, and triggers a refetch of the affected view.

The seq is the single source of truth for ordering. The Studio status badge (HmrStatusBadge) shows Repo seq: #N in its tooltip.

Package-first authoring (ADR-0070)

Every runtime-authored item lives inside a writable package — there are no orphans. Code-defined and installed packages are read-only at runtime, so the first step of any Studio/API authoring action is to target a writable package (a "base"): a create/update aimed at a read-only package is rejected, and the author is asked to pick or create a writable base first. New objects, fields, views, and flows are namespaced into that package — which is exactly what os package publish later ships. See ADR-0070.

Note: "package" here is the runtime authoring base (a package_id-bearing container of metadata), distinct from the npm @objectstack/* packages listed in the repo README.

Overlay whitelist (shared-DB tenancy invariant)

In shared-database multi-tenancy, most metadata types must not be per-org customizable — overriding them would break the physical schema. The whitelist lives in one place: MetadataTypeRegistryEntry.allowOrgOverride in packages/spec/src/kernel/metadata-plugin.zod.ts.

Type allowOrgOverride Rationale
view, dashboard, report, email_template ✅ Pure rendering. Per-org customization is safe.
flow, agent ✅ Per-org overlays are allowed for automation and agent definitions.
permission, role, profile ✅ Per-org overlays are allowed; tenant-level controls layer on top.
object, field ❌ Defines the table schema. Overriding would break existing data.
datasource ❌ Connection strings; multi-tenant isolation is enforced at a higher layer.

There is no workflow metadata type (per ADR-0020, record state machines are a state_machine validation). The runtime gate is implemented in OVERLAY_ALLOWED_TYPES (derived from the registry) and enforced by SysMetadataRepository.put(). Denied types return 403 not_overridable.

See ADR-0005 for the full design and amendments.

Conflict handling

Every put() requires a parentVersion (or null for fresh creates). If the head has moved since the caller read it, ConflictError(ref, expectedParent, actualHead) is thrown. Studio surfaces this as a "metadata moved" toast and offers reload-and-retry.

The hash is sha256: + 64-hex of a canonical (sorted-keys, no-undefined) JSON serialization of the body. Identical bodies produce identical hashes — put() short-circuits on no-op writes.

HMR end-to-end (latency & ordering)

  • Latency. A typical write-to-render round-trip is < 100ms on localhost (REST PUT → DB write → SSE flush → Studio refetch → React re-render).
  • Ordering. The seq is monotonic per repository. The repository-level watch() API supports replay from a since cursor, but the dev HMR SSE endpoint does not replay on reconnect — it registers a fresh listener, emits a ready event, and then streams only live events. A tab that disconnects and reconnects will miss any events that occurred while it was offline; it should refetch the affected metadata on reconnect.
  • Multi-tab. Each tab has its own seq counter. Out-of-order delivery between tabs is impossible because they share the server change log.

Cross-references

Status (as of 2026-04)

Component State
InMemoryRepository, FileSystemRepository, LayeredRepository ✅ Shipped (@objectstack/metadata-core)
Change log + seq (per-org, monotonic) ✅ Shipped
SSE bridge (/api/v1/dev/metadata-events, event: metadata-change) ✅ Shipped
Studio useMetadataHmr + HmrStatusBadge ✅ Shipped
Console dev-mode HMR reloader (MetadataHmrReloader) ✅ Shipped
SysMetadataRepository (overlay over sys_metadata) ✅ Shipped (@objectstack/objectql)
LayeredRepository(SysMeta + artifact) composition ✅ Shipped
protocol.ts:saveMetaItem routed through SysMetadataRepository.put ✅ Shipped (PR-10d.6, flag removed)
sys_metadata_history table (durable, org-keyed change log) ✅ Shipped (@objectstack/objectql)
Cache + Registry refactor against MetadataRepository ⏳ Post-M0

Cross-replica overlay sync is out of scope. Single-instance deployments are the supported topology for the metadata overlay. If you run multiple replicas and need them to converge on the same overlay state, point them at the same database and rely on each replica re-reading on demand — there is no LISTEN/NOTIFY (or equivalent) push mechanism on the roadmap.

FAQ — When does code-defined metadata enter the database?

Short answer: never. This is a deliberate design choice; the table below is the authoritative reference and supersedes any other description you may find.

Where the metadata came from Lands in sys_metadata? Lands in history?
defineView(...) / defineFlow(...) / any source file → compiled into dist/objectstack.json ❌ Never. Loaded into the in-memory registry on boot; refreshed via HMR in dev. ❌ The artifact's own version history is Git. The metadata layer does not duplicate it.
Editing a .json under <root>/<type>/<name>.json (FS overlay, e.g. <root>/view/case_grid.json) ❌ FS layer is independent of DB. ✅ Appended to the change log at <root>/.objectstack/.log/main.jsonl by FileSystemRepository.
Studio inline edit, or PUT /api/v1/metadata/... (REST) on an allowOrgOverride: true type ✅ Written by SysMetadataRepository.put() as an overlay row scoped to organization_id. ✅ Appended to sys_metadata_history (per-org event_seq) in the same transaction as the sys_metadata write (single-instance scope; no cross-replica push).
Deploying a new build (new dist/objectstack.json) ❌ The artifact is loaded into memory, not synced into sys_metadata. ❌ Use Git tags / your deployment platform's release log; that's where artifact "version history" lives.

Why artifact never enters the database

  1. Single source of truth. Code → artifact → memory. If artifact also wrote to sys_metadata, every deploy would need a reconciliation step ("the row says v3.0.0 but the code says v3.0.1") that is impossible to get right under concurrent edits.
  2. Overlay stays small and auditable. Per-org overlay = "what this organization deliberately changed at runtime". A clean table that's safe to dump, diff, or reset to factory.
  3. Immutable infrastructure. Rolling back a deploy = switching image tags. No DB migration, no reverse put, no history replay required.
  4. Per-org multi-tenant economics. If artifacts were materialised into sys_metadata, every org would carry tens of thousands of baseline rows. Overlay-only keeps each org's row count proportional to its actual customisation.

What if I want auditable history of deployed artifact changes?

That's a deployment log, not a metadata change log. Track the envelope hash of each deployed objectstack.json against the org/environment in a deployment-history table. The internal items inside the artifact need no per-item history because their canonical history is the Git repo that produced them.