Skip to content

PHASE_1_IMPLEMENTATION.md

Latest commit

 

History

History
139 lines (103 loc) · 4.61 KB

PHASE_1_IMPLEMENTATION.md

File metadata and controls

139 lines (103 loc) · 4.61 KB

Phase 1 Implementation Summary: Dual-Table Metadata Projection

Overview

Successfully implemented the dual-table projection logic for the metadata service. This establishes the architectural foundation for treating metadata as queryable data following industry best practices from Salesforce, ServiceNow, and Kubernetes.

Architecture

The dual-table pattern maintains two layers:

  1. sys_metadata (Source of Truth)

    • Package management
    • Version control
    • Deployment tracking
    • History via sys_metadata_history

  2. Type-Specific Tables (Queryable Projections)

    • sys_object — Object definitions
    • sys_view — View configurations
    • sys_agent — AI agent metadata
    • sys_tool — AI tool registry
    • sys_flow — Automation flows

Implementation Details

1. MetadataProjector Service

Created packages/metadata/src/projection/metadata-projector.ts with:

  • Project transformation functions for each metadata type
  • Denormalization logic to flatten complex structures for querying
  • Support for both IDataDriver and IDataEngine (ObjectQL)
  • Automatic CRUD operations on projection tables

Key methods:

  • project(type, name, data) — Create/update projection
  • deleteProjection(type, name) — Remove projection
  • transformToProjection(type, name, data) — Type-specific transformation

2. DatabaseLoader Integration

Updated packages/metadata/src/loaders/database-loader.ts:

  • Added enableProjection configuration option (default: true)
  • Integrated MetadataProjector into save() flow
  • Added projection cleanup in delete() flow
  • Projection occurs AFTER sys_metadata save (async safety)

3. System Object Registration

Updated packages/metadata/src/plugin.ts:

  • Added dependency on @objectstack/objectos
  • Registered all system objects from SystemObjects registry
  • Objects registered via manifest service during plugin init()
  • Includes: sys_object, sys_view, sys_agent, sys_tool, sys_flow

Projection Mapping

Each metadata type is projected with denormalized fields for efficient querying:

Object Projection (object → sys_object)

  • Complex structures (fields, indexes, validations) → JSON columns
  • Capabilities → individual boolean columns for filtering
  • Denormalized field_count for sorting/filtering

View Projection (view → sys_view)

  • Columns, filters, sort, config → JSON columns
  • Display options → individual columns
  • Object reference for joins

Agent Projection (agent → sys_agent)

  • Model configuration → individual columns
  • Tools, skills → JSON columns
  • Memory settings → individual columns

Tool Projection (tool → sys_tool)

  • Parameters schema → JSON column
  • Handler code → text column

Flow Projection (flow → sys_flow)

  • Nodes, edges, variables → JSON columns
  • Trigger configuration → individual columns
  • Active status for filtering

Benefits Achieved

  1. Unified Query Protocol: Metadata can be queried using Object Protocol API
  2. Studio Auto-Generation: UI can use /api/v1/data/sys_* endpoints
  3. Efficient Filtering: Denormalized fields enable fast queries
  4. Preserved History: sys_metadata maintains full version control
  5. Package Tracking: All projections include package_id and managed_by

Files Changed

New Files

  • packages/metadata/src/projection/metadata-projector.ts — Projection service
  • packages/metadata/src/projection/index.ts — Module exports

Modified Files

  • packages/metadata/src/loaders/database-loader.ts — Added projection integration
  • packages/metadata/src/plugin.ts — Registered system objects
  • packages/metadata/src/index.ts — Exported projection module
  • packages/metadata/package.json — Added @objectstack/objectos dependency
  • OBJECTOS_IMPLEMENTATION.md — Updated Phase 1 status

Usage Example

// Projection happens automatically when saving metadata
await metadataService.register('object', 'account', {
  name: 'account',
  label: 'Account',
  fields: { /* ... */ },
  // ... object definition
});

// Results in TWO database writes:
// 1. sys_metadata: Full envelope with JSON payload
// 2. sys_object: Denormalized projection with queryable columns

// Studio can now query via Object Protocol
const objects = await client.data.find('sys_object', {
  filter: { namespace: 'crm' },
  sort: 'name',
});

Next Steps

Phase 2 will focus on Studio integration:

  • Update Studio to use type-specific table queries
  • Auto-generate metadata management UI
  • Leverage existing grid/form/kanban components

Phase 3 will add:

  • Comprehensive test coverage
  • Documentation updates
  • Migration guides for existing deployments