feat(metadata): implement dual-table projection logic for Phase 1

Claude · hotlong · web-flow · commit 8901d29efe0a · 2026-04-20T11:48:21.000Z
- Create MetadataProjector service for sys_metadata → type-specific table projections - Add projection functions for object, view, agent, tool, flow metadata types - Integrate projection into DatabaseLoader save() and delete() operations - Register system objects from @objectstack/objectos in MetadataPlugin - Add enableProjection config option (default: true) - Projection enables Studio to query metadata via Object Protocol API Phase 1 implementation complete. Next: Studio integration (Phase 2) Agent-Logs-Url: https://github.com/objectstack-ai/framework/sessions/e95ef127-f74b-4ddd-8a6e-fe86e33fe1b0 Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>
diff --git a/OBJECTOS_IMPLEMENTATION.md b/OBJECTOS_IMPLEMENTATION.md
@@ -106,10 +106,17 @@ packages/objectos/
 
 ## Next Steps
 
-### Phase 1: Integration (Immediate)
-- [ ] Update `packages/metadata` service to support projection
-- [ ] Implement dual-table sync logic
-- [ ] Register system objects in runtime bootstrap
+### Phase 1: Integration (Immediate) ✅ COMPLETED
+- [x] Update `packages/metadata` service to support projection
+- [x] Implement dual-table sync logic
+- [x] Register system objects in runtime bootstrap
+
+**Implementation Details:**
+- Created `MetadataProjector` service in `packages/metadata/src/projection/`
+- Integrated projection into `DatabaseLoader.save()` and `DatabaseLoader.delete()`
+- Added projection functions for each metadata type: object, view, agent, tool, flow
+- Updated `MetadataPlugin` to register all system objects from `@objectstack/objectos`
+- Projection is enabled by default, can be disabled via `enableProjection: false` option
 
 ### Phase 2: Studio Integration (Next)
 - [ ] Update Studio to query type-specific tables
diff --git a/PHASE_1_IMPLEMENTATION.md b/PHASE_1_IMPLEMENTATION.md
@@ -0,0 +1,139 @@
+# Phase 1 Implementation Summary: Dual-Table Metadata Projection
+
+## Overview
+
+Successfully implemented the dual-table projection logic for the metadata service. This establishes the architectural foundation for treating metadata as queryable data following industry best practices from Salesforce, ServiceNow, and Kubernetes.
+
+## Architecture
+
+The dual-table pattern maintains two layers:
+
+1. **sys_metadata** (Source of Truth)
+   - Package management
+   - Version control
+   - Deployment tracking
+   - History via sys_metadata_history
+
+2. **Type-Specific Tables** (Queryable Projections)
+   - sys_object — Object definitions
+   - sys_view — View configurations
+   - sys_agent — AI agent metadata
+   - sys_tool — AI tool registry
+   - sys_flow — Automation flows
+
+## Implementation Details
+
+### 1. MetadataProjector Service
+
+Created `packages/metadata/src/projection/metadata-projector.ts` with:
+
+- **Project transformation functions** for each metadata type
+- **Denormalization logic** to flatten complex structures for querying
+- **Support for both IDataDriver and IDataEngine** (ObjectQL)
+- **Automatic CRUD operations** on projection tables
+
+Key methods:
+- `project(type, name, data)` — Create/update projection
+- `deleteProjection(type, name)` — Remove projection
+- `transformToProjection(type, name, data)` — Type-specific transformation
+
+### 2. DatabaseLoader Integration
+
+Updated `packages/metadata/src/loaders/database-loader.ts`:
+
+- Added `enableProjection` configuration option (default: true)
+- Integrated `MetadataProjector` into save() flow
+- Added projection cleanup in delete() flow
+- Projection occurs AFTER sys_metadata save (async safety)
+
+### 3. System Object Registration
+
+Updated `packages/metadata/src/plugin.ts`:
+
+- Added dependency on `@objectstack/objectos`
+- Registered all system objects from SystemObjects registry
+- Objects registered via manifest service during plugin init()
+- Includes: sys_object, sys_view, sys_agent, sys_tool, sys_flow
+
+## Projection Mapping
+
+Each metadata type is projected with denormalized fields for efficient querying:
+
+### Object Projection (object → sys_object)
+- Complex structures (fields, indexes, validations) → JSON columns
+- Capabilities → individual boolean columns for filtering
+- Denormalized field_count for sorting/filtering
+
+### View Projection (view → sys_view)
+- Columns, filters, sort, config → JSON columns
+- Display options → individual columns
+- Object reference for joins
+
+### Agent Projection (agent → sys_agent)
+- Model configuration → individual columns
+- Tools, skills → JSON columns
+- Memory settings → individual columns
+
+### Tool Projection (tool → sys_tool)
+- Parameters schema → JSON column
+- Handler code → text column
+
+### Flow Projection (flow → sys_flow)
+- Nodes, edges, variables → JSON columns
+- Trigger configuration → individual columns
+- Active status for filtering
+
+## Benefits Achieved
+
+1. **Unified Query Protocol**: Metadata can be queried using Object Protocol API
+2. **Studio Auto-Generation**: UI can use `/api/v1/data/sys_*` endpoints
+3. **Efficient Filtering**: Denormalized fields enable fast queries
+4. **Preserved History**: sys_metadata maintains full version control
+5. **Package Tracking**: All projections include package_id and managed_by
+
+## Files Changed
+
+### New Files
+- `packages/metadata/src/projection/metadata-projector.ts` — Projection service
+- `packages/metadata/src/projection/index.ts` — Module exports
+
+### Modified Files
+- `packages/metadata/src/loaders/database-loader.ts` — Added projection integration
+- `packages/metadata/src/plugin.ts` — Registered system objects
+- `packages/metadata/src/index.ts` — Exported projection module
+- `packages/metadata/package.json` — Added @objectstack/objectos dependency
+- `OBJECTOS_IMPLEMENTATION.md` — Updated Phase 1 status
+
+## Usage Example
+
+```typescript
+// Projection happens automatically when saving metadata
+await metadataService.register('object', 'account', {
+  name: 'account',
+  label: 'Account',
+  fields: { /* ... */ },
+  // ... object definition
+});
+
+// Results in TWO database writes:
+// 1. sys_metadata: Full envelope with JSON payload
+// 2. sys_object: Denormalized projection with queryable columns
+
+// Studio can now query via Object Protocol
+const objects = await client.data.find('sys_object', {
+  filter: { namespace: 'crm' },
+  sort: 'name',
+});
+```
+
+## Next Steps
+
+Phase 2 will focus on Studio integration:
+- Update Studio to use type-specific table queries
+- Auto-generate metadata management UI
+- Leverage existing grid/form/kanban components
+
+Phase 3 will add:
+- Comprehensive test coverage
+- Documentation updates
+- Migration guides for existing deployments
diff --git a/packages/metadata/package.json b/packages/metadata/package.json
@@ -39,6 +39,7 @@
   ],
   "dependencies": {
     "@objectstack/core": "workspace:*",
+    "@objectstack/objectos": "workspace:*",
     "@objectstack/spec": "workspace:*",
     "@objectstack/types": "workspace:*",
     "chokidar": "^5.0.0",
diff --git a/packages/metadata/src/index.ts b/packages/metadata/src/index.ts
@@ -13,6 +13,9 @@ export { MetadataManager, type WatchCallback, type MetadataManagerOptions } from
 // Plugin
 export { MetadataPlugin } from './plugin.js';
 
+// Projection
+export { MetadataProjector, type MetadataProjectorOptions } from './projection/index.js';
+
 // Loaders
 export { type MetadataLoader } from './loaders/loader-interface.js';
 export { MemoryLoader } from './loaders/memory-loader.js';
diff --git a/packages/metadata/src/loaders/database-loader.ts b/packages/metadata/src/loaders/database-loader.ts
@@ -24,6 +24,7 @@ import { SysMetadataHistoryObject } from '../objects/sys-metadata-history.object
 import type { IDataDriver, IDataEngine } from '@objectstack/spec/contracts';
 import type { MetadataLoader } from './loader-interface.js';
 import { calculateChecksum } from '../utils/metadata-history-utils.js';
+import { MetadataProjector } from '../projection/metadata-projector.js';
 
 /**
  * Configuration for the DatabaseLoader.
@@ -54,6 +55,9 @@ export interface DatabaseLoaderOptions {
 
   /** Enable history tracking (default: true) */
   trackHistory?: boolean;
+
+  /** Enable metadata projection to type-specific tables (default: true) */
+  enableProjection?: boolean;
 }
 
 /**
@@ -84,6 +88,8 @@ export class DatabaseLoader implements MetadataLoader {
   private trackHistory: boolean;
   private schemaReady = false;
   private historySchemaReady = false;
+  private enableProjection: boolean;
+  private projector?: MetadataProjector;
 
   constructor(options: DatabaseLoaderOptions) {
     if (!options.driver && !options.engine) {
@@ -96,6 +102,17 @@ export class DatabaseLoader implements MetadataLoader {
     this.organizationId = options.organizationId;
     this.environmentId = options.environmentId;
     this.trackHistory = options.trackHistory !== false; // Default to true
+    this.enableProjection = options.enableProjection !== false; // Default to true
+
+    // Initialize projector if projection is enabled
+    if (this.enableProjection) {
+      this.projector = new MetadataProjector({
+        driver: this.driver,
+        engine: this.engine,
+        organizationId: this.organizationId,
+        environmentId: this.environmentId,
+      });
+    }
   }
 
   // ==========================================
@@ -709,6 +726,11 @@ export class DatabaseLoader implements MetadataLoader {
           previousChecksum
         );
 
+        // Project to type-specific table
+        if (this.projector) {
+          await this.projector.project(type, name, data);
+        }
+
         return {
           success: true,
           path: `datasource://${this.tableName}/${type}/${name}`,
@@ -746,6 +768,11 @@ export class DatabaseLoader implements MetadataLoader {
           'create'
         );
 
+        // Project to type-specific table
+        if (this.projector) {
+          await this.projector.project(type, name, data);
+        }
+
         return {
           success: true,
           path: `datasource://${this.tableName}/${type}/${name}`,
@@ -780,6 +807,11 @@ export class DatabaseLoader implements MetadataLoader {
 
     // Delete from the main metadata table using the record's ID
     await this._delete(this.tableName, existing.id as string);
+
+    // Delete projection from type-specific table
+    if (this.projector) {
+      await this.projector.deleteProjection(type, name);
+    }
   }
 }
 
diff --git a/packages/metadata/src/plugin.ts b/packages/metadata/src/plugin.ts
@@ -6,6 +6,7 @@ import { DEFAULT_METADATA_TYPE_REGISTRY } from '@objectstack/spec/kernel';
 import type { MetadataPluginConfig } from '@objectstack/spec/kernel';
 import { SysMetadataObject } from './objects/sys-metadata.object.js';
 import { SysMetadataHistoryObject } from './objects/sys-metadata-history.object.js';
+import { SystemObjects } from '@objectstack/objectos';
 
 export interface MetadataPluginOptions {
     rootDir?: string;
@@ -56,7 +57,10 @@ export class MetadataPlugin implements Plugin {
         // Register metadata system objects via the manifest service (if available).
         // MetadataPlugin may init before ObjectQLPlugin, so wrap in try/catch.
         try {
-            ctx.getService<{ register(m: any): void }>('manifest').register({
+            const manifestService = ctx.getService<{ register(m: any): void }>('manifest');
+
+            // Register the metadata envelope tables
+            manifestService.register({
                 id: 'com.objectstack.metadata',
                 name: 'Metadata',
                 version: '1.0.0',
@@ -65,6 +69,22 @@ export class MetadataPlugin implements Plugin {
                 namespace: 'sys',
                 objects: [SysMetadataObject, SysMetadataHistoryObject],
             });
+
+            // Register the queryable system objects from @objectstack/objectos
+            manifestService.register({
+                id: 'com.objectstack.objectos',
+                name: 'ObjectOS System Objects',
+                version: '1.0.0',
+                type: 'plugin',
+                scope: 'platform',
+                namespace: 'sys',
+                objects: Object.values(SystemObjects),
+            });
+
+            ctx.logger.info('Registered system metadata objects', {
+                metadata: ['sys_metadata', 'sys_metadata_history'],
+                objectos: Object.keys(SystemObjects),
+            });
         } catch {
             // ObjectQL not loaded yet — objects will be discovered via legacy fallback
         }
diff --git a/packages/metadata/src/projection/index.ts b/packages/metadata/src/projection/index.ts
@@ -0,0 +1,9 @@
+// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * Metadata Projection
+ *
+ * Dual-table architecture: sys_metadata → type-specific tables
+ */
+
+export { MetadataProjector, type MetadataProjectorOptions } from './metadata-projector.js';
diff --git a/packages/metadata/src/projection/metadata-projector.ts b/packages/metadata/src/projection/metadata-projector.ts
