Skip to content

Commit 0a24a37

Browse files
hotlonghotlong
authored
Merge pull request #1194 from objectstack-ai/claude/update-metadata-service-dual-table-projection
feat(metadata): implement dual-table projection for queryable metadata
2 parents 8b57fc3 + 40332f5 commit 0a24a37

13 files changed

Lines changed: 826 additions & 25 deletions

File tree

‎OBJECTOS_IMPLEMENTATION.md‎

Lines changed: 185 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,185 @@
1+
# ObjectOS Implementation Summary
2+
3+
## Overview
4+
5+
Successfully implemented the ObjectOS layer - a new `@objectstack/objectos` package containing system runtime object definitions that represent metadata as queryable data.
6+
7+
## Architecture Decision
8+
9+
Based on architectural discussions, we established:
10+
11+
1. **Location**: `packages/objectos` (NOT `packages/plugins/plugin-system`)
12+
- Rationale: System objects are core infrastructure, not optional plugins
13+
- ObjectOS represents the OS-level primitives of the platform
14+
15+
2. **Dual-Table Pattern**: Keep BOTH systems (do NOT deprecate `sys_metadata`)
16+
- `sys_metadata`: Source of truth for package management, version control, deployment
17+
- Type-specific tables (`sys_object`, `sys_view`, etc.): Queryable data for UI/reporting
18+
19+
## Package Structure
20+
21+
```
22+
packages/objectos/
23+
├── src/
24+
│ ├── objects/
25+
│ │ ├── sys-metadata.object.ts # Generic metadata envelope
26+
│ │ ├── sys-object.object.ts # Object definitions
27+
│ │ ├── sys-view.object.ts # View definitions
28+
│ │ ├── sys-agent.object.ts # AI Agent definitions
29+
│ │ ├── sys-tool.object.ts # AI Tool definitions
30+
│ │ ├── sys-flow.object.ts # Flow definitions
31+
│ │ └── index.ts
32+
│ ├── index.ts # Package entry point
33+
│ └── registry.ts # System object registry
34+
├── package.json
35+
├── tsconfig.json
36+
├── tsup.config.ts
37+
└── README.md
38+
```
39+
40+
## System Objects Implemented
41+
42+
### 1. sys_metadata (Generic Envelope)
43+
- **Purpose**: Source of truth for package management
44+
- **Features**: Version control, checksums, package ownership, deployment tracking
45+
- **Fields**: 20+ fields including version, checksum, package_id, managed_by, scope
46+
47+
### 2. sys_object (Queryable Object Definitions)
48+
- **Purpose**: Browse/filter/search object definitions in Studio
49+
- **Features**: Denormalized data, complex fields as JSON
50+
- **Fields**: 30+ fields including fields_json, capabilities_json, field_count
51+
52+
### 3. sys_view (Queryable View Definitions)
53+
- **Purpose**: Manage view metadata through Object Protocol
54+
- **Features**: View type filtering, object references
55+
- **Fields**: columns_json, filters_json, sort_json, config_json
56+
57+
### 4. sys_agent (AI Agent Definitions)
58+
- **Purpose**: AI agent configuration as data
59+
- **Features**: Model config, tools/skills management
60+
- **Fields**: model, temperature, system_prompt, tools_json, skills_json
61+
62+
### 5. sys_tool (AI Tool Definitions)
63+
- **Purpose**: AI tool registry as queryable data
64+
- **Features**: Parameter schemas, handler code
65+
- **Fields**: parameters_json, handler_code
66+
67+
### 6. sys_flow (Automation Flow Definitions)
68+
- **Purpose**: Flow metadata management
69+
- **Features**: Flow types, trigger configuration
70+
- **Fields**: flow_type, nodes_json, edges_json, trigger_type
71+
72+
## Key Features
73+
74+
1. **Metadata as Data**
75+
- All metadata types are queryable using Object Protocol
76+
- Same CRUD operations as business data
77+
- Consistent API: `/api/v1/data/sys_object`, `/api/v1/data/sys_view`
78+
79+
2. **Dual-Table Architecture**
80+
```
81+
Package Loader
82+
↓
83+
sys_metadata (source of truth)
84+
↓ (projection)
85+
sys_object, sys_view, etc. (queryable)
86+
↓
87+
Studio UI (auto-generated)
88+
```
89+
90+
3. **Version Management**
91+
- `sys_metadata` tracks all versions
92+
- `sys_metadata_history` table for history
93+
- Checksum-based change detection
94+
- Package upgrade/downgrade support
95+
96+
4. **Auto-Generated UI**
97+
- Studio uses Object Protocol
98+
- No custom UI code per metadata type
99+
- Leverage grid/form/kanban views
100+
101+
## Industry Alignment
102+
103+
- **Salesforce**: CustomObject, CustomField (queryable metadata)
104+
- **ServiceNow**: sys_db_object, sys_dictionary (table-based metadata)
105+
- **Kubernetes**: CRDs as structured resources
106+
107+
## Next Steps
108+
109+
### Phase 1: Integration (Immediate) ✅ COMPLETED
110+
- [x] Update `packages/metadata` service to support projection
111+
- [x] Implement dual-table sync logic
112+
- [x] Register system objects in runtime bootstrap
113+
114+
**Implementation Details:**
115+
- Created `MetadataProjector` service in `packages/metadata/src/projection/`
116+
- Integrated projection into `DatabaseLoader.save()` and `DatabaseLoader.delete()`
117+
- Added projection functions for each metadata type: object, view, agent, tool, flow
118+
- Updated `MetadataPlugin` to register all system objects from `@objectstack/objectos`
119+
- Projection is enabled by default, can be disabled via `enableProjection: false` option
120+
121+
### Phase 2: Studio Integration (Next)
122+
- [ ] Update Studio to query type-specific tables
123+
- [ ] Use `/api/v1/data/sys_object` for browsing
124+
- [ ] Auto-generate metadata forms
125+
126+
### Phase 3: Testing & Documentation (Later)
127+
- [ ] Add comprehensive test coverage
128+
- [ ] Update documentation
129+
- [ ] Create migration guides
130+
131+
## Usage Example
132+
133+
```typescript
134+
import { SystemObjects } from '@objectstack/objectos';
135+
136+
// Register all system objects during bootstrap
137+
for (const [name, definition] of Object.entries(SystemObjects)) {
138+
await kernel.metadata.register('object', name, definition, {
139+
scope: 'system',
140+
isSystem: true,
141+
managedBy: 'platform',
142+
});
143+
}
144+
145+
// Query metadata using Object Protocol
146+
const objects = await client.data.find('sys_object', {
147+
filter: { namespace: 'crm' },
148+
sort: 'name',
149+
});
150+
151+
// Studio auto-generates UI
152+
<GridView object="sys_object" />
153+
<FormView object="sys_object" recordId="account" />
154+
```
155+
156+
## Benefits
157+
158+
1. ✅ **Unified Protocol**: One protocol for both data and metadata
159+
2. ✅ **Auto-Generated UI**: Studio reuses existing components
160+
3. ✅ **Better DX**: Consistent API for all entity types
161+
4. ✅ **Version Control**: Full history via sys_metadata_history
162+
5. ✅ **Package Management**: Track ownership and deployments
163+
6. ✅ **Industry Standard**: Follows Salesforce/ServiceNow patterns
164+
165+
## Files Created
166+
167+
- `/home/runner/work/framework/framework/packages/objectos/package.json`
168+
- `/home/runner/work/framework/framework/packages/objectos/tsconfig.json`
169+
- `/home/runner/work/framework/framework/packages/objectos/tsup.config.ts`
170+
- `/home/runner/work/framework/framework/packages/objectos/README.md`
171+
- `/home/runner/work/framework/framework/packages/objectos/src/index.ts`
172+
- `/home/runner/work/framework/framework/packages/objectos/src/registry.ts`
173+
- `/home/runner/work/framework/framework/packages/objectos/src/objects/index.ts`
174+
- `/home/runner/work/framework/framework/packages/objectos/src/objects/sys-metadata.object.ts`
175+
- `/home/runner/work/framework/framework/packages/objectos/src/objects/sys-object.object.ts`
176+
- `/home/runner/work/framework/framework/packages/objectos/src/objects/sys-view.object.ts`
177+
- `/home/runner/work/framework/framework/packages/objectos/src/objects/sys-agent.object.ts`
178+
- `/home/runner/work/framework/framework/packages/objectos/src/objects/sys-tool.object.ts`
179+
- `/home/runner/work/framework/framework/packages/objectos/src/objects/sys-flow.object.ts`
180+
181+
## Conclusion
182+
183+
The ObjectOS package establishes a clean architectural foundation for treating metadata as queryable data. This enables auto-generated Studio UI, unified APIs, and follows industry best practices from Salesforce, ServiceNow, and Kubernetes.
184+
185+
The dual-table architecture preserves the benefits of `sys_metadata` for package management while adding queryability through type-specific tables.

‎PHASE_1_IMPLEMENTATION.md‎

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# Phase 1 Implementation Summary: Dual-Table Metadata Projection
2+
3+
## Overview
4+
5+
Successfully implemented the dual-table projection logic for the metadata service. This establishes the architectural foundation for treating metadata as queryable data following industry best practices from Salesforce, ServiceNow, and Kubernetes.
6+
7+
## Architecture
8+
9+
The dual-table pattern maintains two layers:
10+
11+
1. **sys_metadata** (Source of Truth)
12+
- Package management
13+
- Version control
14+
- Deployment tracking
15+
- History via sys_metadata_history
16+
17+
2. **Type-Specific Tables** (Queryable Projections)
18+
- sys_object — Object definitions
19+
- sys_view — View configurations
20+
- sys_agent — AI agent metadata
21+
- sys_tool — AI tool registry
22+
- sys_flow — Automation flows
23+
24+
## Implementation Details
25+
26+
### 1. MetadataProjector Service
27+
28+
Created `packages/metadata/src/projection/metadata-projector.ts` with:
29+
30+
- **Project transformation functions** for each metadata type
31+
- **Denormalization logic** to flatten complex structures for querying
32+
- **Support for both IDataDriver and IDataEngine** (ObjectQL)
33+
- **Automatic CRUD operations** on projection tables
34+
35+
Key methods:
36+
- `project(type, name, data)` — Create/update projection
37+
- `deleteProjection(type, name)` — Remove projection
38+
- `transformToProjection(type, name, data)` — Type-specific transformation
39+
40+
### 2. DatabaseLoader Integration
41+
42+
Updated `packages/metadata/src/loaders/database-loader.ts`:
43+
44+
- Added `enableProjection` configuration option (default: true)
45+
- Integrated `MetadataProjector` into save() flow
46+
- Added projection cleanup in delete() flow
47+
- Projection occurs AFTER sys_metadata save (async safety)
48+
49+
### 3. System Object Registration
50+
51+
Updated `packages/metadata/src/plugin.ts`:
52+
53+
- Added dependency on `@objectstack/objectos`
54+
- Registered all system objects from SystemObjects registry
55+
- Objects registered via manifest service during plugin init()
56+
- Includes: sys_object, sys_view, sys_agent, sys_tool, sys_flow
57+
58+
## Projection Mapping
59+
60+
Each metadata type is projected with denormalized fields for efficient querying:
61+
62+
### Object Projection (object → sys_object)
63+
- Complex structures (fields, indexes, validations) → JSON columns
64+
- Capabilities → individual boolean columns for filtering
65+
- Denormalized field_count for sorting/filtering
66+
67+
### View Projection (view → sys_view)
68+
- Columns, filters, sort, config → JSON columns
69+
- Display options → individual columns
70+
- Object reference for joins
71+
72+
### Agent Projection (agent → sys_agent)
73+
- Model configuration → individual columns
74+
- Tools, skills → JSON columns
75+
- Memory settings → individual columns
76+
77+
### Tool Projection (tool → sys_tool)
78+
- Parameters schema → JSON column
79+
- Handler code → text column
80+
81+
### Flow Projection (flow → sys_flow)
82+
- Nodes, edges, variables → JSON columns
83+
- Trigger configuration → individual columns
84+
- Active status for filtering
85+
86+
## Benefits Achieved
87+
88+
1. **Unified Query Protocol**: Metadata can be queried using Object Protocol API
89+
2. **Studio Auto-Generation**: UI can use `/api/v1/data/sys_*` endpoints
90+
3. **Efficient Filtering**: Denormalized fields enable fast queries
91+
4. **Preserved History**: sys_metadata maintains full version control
92+
5. **Package Tracking**: All projections include package_id and managed_by
93+
94+
## Files Changed
95+
96+
### New Files
97+
- `packages/metadata/src/projection/metadata-projector.ts` — Projection service
98+
- `packages/metadata/src/projection/index.ts` — Module exports
99+
100+
### Modified Files
101+
- `packages/metadata/src/loaders/database-loader.ts` — Added projection integration
102+
- `packages/metadata/src/plugin.ts` — Registered system objects
103+
- `packages/metadata/src/index.ts` — Exported projection module
104+
- `packages/metadata/package.json` — Added @objectstack/objectos dependency
105+
- `OBJECTOS_IMPLEMENTATION.md` — Updated Phase 1 status
106+
107+
## Usage Example
108+
109+
```typescript
110+
// Projection happens automatically when saving metadata
111+
await metadataService.register('object', 'account', {
112+
name: 'account',
113+
label: 'Account',
114+
fields: { /* ... */ },
115+
// ... object definition
116+
});
117+
118+
// Results in TWO database writes:
119+
// 1. sys_metadata: Full envelope with JSON payload
120+
// 2. sys_object: Denormalized projection with queryable columns
121+
122+
// Studio can now query via Object Protocol
123+
const objects = await client.data.find('sys_object', {
124+
filter: { namespace: 'crm' },
125+
sort: 'name',
126+
});
127+
```
128+
129+
## Next Steps
130+
131+
Phase 2 will focus on Studio integration:
132+
- Update Studio to use type-specific table queries
133+
- Auto-generate metadata management UI
134+
- Leverage existing grid/form/kanban components
135+
136+
Phase 3 will add:
137+
- Comprehensive test coverage
138+
- Documentation updates
139+
- Migration guides for existing deployments

‎apps/studio/test/ai-chat-panel.test.tsx‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -97,10 +97,10 @@ describe('use-ai-chat-panel', () => {
9797
});
9898

9999
it('does not throw when localStorage is unavailable', () => {
100-
const originalSetItem = Storage.prototype.setItem;
101-
Storage.prototype.setItem = () => { throw new Error('QuotaExceeded'); };
100+
const originalSetItem = localStorage.setItem.bind(localStorage);
101+
localStorage.setItem = () => { throw new Error('QuotaExceeded'); };
102102
expect(() => saveMessages([makeMsg({ id: '1', role: 'user', content: 'A' })])).not.toThrow();
103-
Storage.prototype.setItem = originalSetItem;
103+
localStorage.setItem = originalSetItem;
104104
});
105105
});
106106
});
@@ -260,10 +260,10 @@ describe('Agent Selector', () => {
260260
});
261261

262262
it('should not throw when localStorage is unavailable', () => {
263-
const originalSetItem = Storage.prototype.setItem;
264-
Storage.prototype.setItem = () => { throw new Error('QuotaExceeded'); };
263+
const originalSetItem = localStorage.setItem.bind(localStorage);
264+
localStorage.setItem = () => { throw new Error('QuotaExceeded'); };
265265
expect(() => saveSelectedAgent('metadata_assistant')).not.toThrow();
266-
Storage.prototype.setItem = originalSetItem;
266+
localStorage.setItem = originalSetItem;
267267
});
268268
});
269269

0 commit comments

Comments
 (0)