|
| 1 | +# Metadata Service: Architecture and Flow |
| 2 | + |
| 3 | +This document explains how metadata flows through ObjectStack from definition to runtime. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +ObjectStack supports **two modes** for metadata provision: |
| 8 | + |
| 9 | +1. **Simple Mode (ObjectQL-only)**: Metadata defined in code, stored in memory |
| 10 | +2. **Advanced Mode (with MetadataPlugin)**: Metadata loaded from files, with watch/export/persistence features |
| 11 | + |
| 12 | +Both modes are compatible and use the same service interface. |
| 13 | + |
| 14 | +## Architecture Decision |
| 15 | + |
| 16 | +See [ADR-0001: Metadata Service Architecture](./adr/0001-metadata-service-architecture.md) for the full decision rationale. |
| 17 | + |
| 18 | +## Service Providers |
| 19 | + |
| 20 | +### ObjectQL as Metadata Provider |
| 21 | + |
| 22 | +**Package**: `@objectstack/objectql` |
| 23 | +**Service Registered**: `metadata`, `objectql`, `data` |
| 24 | +**When Used**: Default fallback when MetadataPlugin is not loaded |
| 25 | + |
| 26 | +**Characteristics**: |
| 27 | +- ✅ Simple setup - no additional configuration |
| 28 | +- ✅ Fast in-memory access |
| 29 | +- ✅ Good for testing and simple applications |
| 30 | +- ⚠️ No file persistence |
| 31 | +- ⚠️ No file watching |
| 32 | +- ⚠️ Limited to programmatically registered metadata |
| 33 | + |
| 34 | +**Flow**: |
| 35 | +``` |
| 36 | +objectstack.config.ts |
| 37 | + ↓ |
| 38 | +defineStack({ plugins: [new AppPlugin(manifest)] }) |
| 39 | + ↓ |
| 40 | +ObjectQLPlugin.start() discovers apps via kernel.getServices() |
| 41 | + ↓ |
| 42 | +ObjectQL.registerApp(manifest) → Internal Registry |
| 43 | + ↓ |
| 44 | +metadata service queries → Registry lookup → Response |
| 45 | +``` |
| 46 | + |
| 47 | +### MetadataPlugin as Provider |
| 48 | + |
| 49 | +**Package**: `@objectstack/metadata` |
| 50 | +**Service Registered**: `metadata` |
| 51 | +**When Used**: Explicitly loaded in plugin configuration |
| 52 | + |
| 53 | +**Characteristics**: |
| 54 | +- ✅ File system persistence |
| 55 | +- ✅ File watching for hot reload |
| 56 | +- ✅ Multi-format support (YAML, JSON, TypeScript) |
| 57 | +- ✅ Multi-source (filesystem, HTTP, database) |
| 58 | +- ✅ Export/import capabilities |
| 59 | +- ⚠️ Requires explicit setup |
| 60 | +- ⚠️ Slightly more complex |
| 61 | + |
| 62 | +**Flow**: |
| 63 | +``` |
| 64 | +File System (objects/, views/, apps/, etc.) |
| 65 | + ↓ |
| 66 | +MetadataPlugin.start() loads all metadata types |
| 67 | + ↓ |
| 68 | +NodeMetadataManager (uses FilesystemLoader) |
| 69 | + ↓ |
| 70 | +Serializers (YAML/JSON/TS) parse files |
| 71 | + ↓ |
| 72 | +metadata service queries → MetadataManager.load() → File lookup → Response |
| 73 | +``` |
| 74 | + |
| 75 | +## Integration Flow |
| 76 | + |
| 77 | +When **both** ObjectQL and MetadataPlugin are loaded: |
| 78 | + |
| 79 | +``` |
| 80 | +1. MetadataPlugin.init() |
| 81 | + → Registers 'metadata' service FIRST |
| 82 | + |
| 83 | +2. ObjectQLPlugin.init() |
| 84 | + → Checks for existing 'metadata' service |
| 85 | + → Finds MetadataPlugin |
| 86 | + → Does NOT register 'metadata' (already exists) |
| 87 | + → Registers 'objectql' and 'data' only |
| 88 | + |
| 89 | +3. MetadataPlugin.start() |
| 90 | + → Loads metadata from file system |
| 91 | + → Service now provides file-based metadata |
| 92 | + |
| 93 | +4. ObjectQLPlugin.start() |
| 94 | + → Detects external metadata service |
| 95 | + → Loads definitions from it |
| 96 | + → Populates internal registry for fast queries |
| 97 | + → Discovers apps and drivers from kernel |
| 98 | + |
| 99 | +5. Runtime Queries |
| 100 | + → API calls ctx.getService('metadata') |
| 101 | + → Gets MetadataPlugin instance |
| 102 | + → Returns file-based metadata |
| 103 | + |
| 104 | + → ObjectQL queries use registry (pre-loaded from MetadataPlugin) |
| 105 | + → Fast in-memory access for data operations |
| 106 | +``` |
| 107 | + |
| 108 | +## Example Configurations |
| 109 | + |
| 110 | +### Simple Mode (ObjectQL Only) |
| 111 | + |
| 112 | +```typescript |
| 113 | +// objectstack.config.ts |
| 114 | +import { defineStack } from '@objectstack/spec'; |
| 115 | +import { ObjectQLPlugin } from '@objectstack/objectql'; |
| 116 | +import { AppPlugin } from '@objectstack/runtime'; |
| 117 | +import myApp from './myapp.config'; |
| 118 | + |
| 119 | +export default defineStack({ |
| 120 | + manifest: { |
| 121 | + id: 'my-app', |
| 122 | + name: 'my_app', |
| 123 | + version: '1.0.0', |
| 124 | + type: 'app' |
| 125 | + }, |
| 126 | + plugins: [ |
| 127 | + new ObjectQLPlugin(), |
| 128 | + new AppPlugin(myApp), // Metadata defined in code |
| 129 | + ] |
| 130 | +}); |
| 131 | +``` |
| 132 | + |
| 133 | +**Result**: ObjectQL provides metadata service, serves in-memory definitions. |
| 134 | + |
| 135 | +### Advanced Mode (With MetadataPlugin) |
| 136 | + |
| 137 | +```typescript |
| 138 | +// objectstack.config.ts |
| 139 | +import { defineStack } from '@objectstack/spec'; |
| 140 | +import { ObjectQLPlugin } from '@objectstack/objectql'; |
| 141 | +import { MetadataPlugin } from '@objectstack/metadata'; |
| 142 | + |
| 143 | +export default defineStack({ |
| 144 | + manifest: { |
| 145 | + id: 'my-app', |
| 146 | + name: 'my_app', |
| 147 | + version: '1.0.0', |
| 148 | + type: 'app' |
| 149 | + }, |
| 150 | + plugins: [ |
| 151 | + new MetadataPlugin({ |
| 152 | + rootDir: process.cwd(), |
| 153 | + watch: true |
| 154 | + }), |
| 155 | + new ObjectQLPlugin(), |
| 156 | + ] |
| 157 | +}); |
| 158 | +``` |
| 159 | + |
| 160 | +**Result**: MetadataPlugin provides metadata service, ObjectQL reads from it. |
| 161 | + |
| 162 | +**File Structure**: |
| 163 | +``` |
| 164 | +project/ |
| 165 | +├── objectstack.config.ts |
| 166 | +├── objects/ |
| 167 | +│ ├── account.object.ts |
| 168 | +│ └── contact.object.ts |
| 169 | +├── views/ |
| 170 | +│ ├── account-list.view.yaml |
| 171 | +│ └── contact-form.view.json |
| 172 | +└── apps/ |
| 173 | + └── crm.app.ts |
| 174 | +``` |
| 175 | + |
| 176 | +## API Metadata Endpoints |
| 177 | + |
| 178 | +All API metadata endpoints use the kernel's `metadata` service: |
| 179 | + |
| 180 | +```typescript |
| 181 | +// In API handler |
| 182 | +const metadataService = ctx.getService('metadata'); |
| 183 | + |
| 184 | +// Single object |
| 185 | +const object = await metadataService.load('object', 'account'); |
| 186 | +return { data: object }; |
| 187 | + |
| 188 | +// List all objects |
| 189 | +const objects = await metadataService.loadMany('object'); |
| 190 | +return { data: objects }; |
| 191 | +``` |
| 192 | + |
| 193 | +The API **doesn't know or care** whether metadata comes from ObjectQL or MetadataPlugin. |
| 194 | + |
| 195 | +## When to Use Each Mode |
| 196 | + |
| 197 | +### Use ObjectQL-only when: |
| 198 | +- Building prototypes or POCs |
| 199 | +- Writing tests |
| 200 | +- Creating simple single-file applications |
| 201 | +- All metadata is programmatically generated |
| 202 | + |
| 203 | +### Use MetadataPlugin when: |
| 204 | +- Building production applications |
| 205 | +- Need file-based metadata (version control friendly) |
| 206 | +- Want hot reload during development |
| 207 | +- Need export/import capabilities |
| 208 | +- Multiple developers editing metadata |
| 209 | +- Metadata stored in external systems |
| 210 | + |
| 211 | +## Metadata Service Interface |
| 212 | + |
| 213 | +Both providers implement this interface: |
| 214 | + |
| 215 | +```typescript |
| 216 | +interface IMetadataService { |
| 217 | + /** |
| 218 | + * Load a single metadata item |
| 219 | + */ |
| 220 | + load<T>(type: string, name: string, options?: MetadataLoadOptions): Promise<T | null>; |
| 221 | + |
| 222 | + /** |
| 223 | + * Load multiple metadata items of a type |
| 224 | + */ |
| 225 | + loadMany<T>(type: string, options?: MetadataLoadOptions): Promise<T[]>; |
| 226 | + |
| 227 | + /** |
| 228 | + * Save a metadata item |
| 229 | + */ |
| 230 | + save<T>(type: string, name: string, data: T, options?: MetadataSaveOptions): Promise<MetadataSaveResult>; |
| 231 | + |
| 232 | + /** |
| 233 | + * Check if metadata item exists |
| 234 | + */ |
| 235 | + exists(type: string, name: string): Promise<boolean>; |
| 236 | + |
| 237 | + /** |
| 238 | + * List all items of a type |
| 239 | + */ |
| 240 | + list(type: string): Promise<string[]>; |
| 241 | +} |
| 242 | +``` |
| 243 | + |
| 244 | +## Troubleshooting |
| 245 | + |
| 246 | +### "Service 'metadata' already registered" Error |
| 247 | + |
| 248 | +**Cause**: Both ObjectQL and MetadataPlugin trying to register the service. |
| 249 | + |
| 250 | +**Solution**: Ensure MetadataPlugin is loaded BEFORE ObjectQLPlugin in the plugins array. ObjectQL will detect it and not register. |
| 251 | + |
| 252 | +```typescript |
| 253 | +// ❌ Wrong order |
| 254 | +plugins: [ |
| 255 | + new ObjectQLPlugin(), |
| 256 | + new MetadataPlugin(), // Too late, ObjectQL already registered metadata |
| 257 | +] |
| 258 | + |
| 259 | +// ✅ Correct order |
| 260 | +plugins: [ |
| 261 | + new MetadataPlugin(), // Registers first |
| 262 | + new ObjectQLPlugin(), // Detects metadata service, doesn't register |
| 263 | +] |
| 264 | +``` |
| 265 | + |
| 266 | +### Metadata Changes Not Reflected |
| 267 | + |
| 268 | +**Cause**: Using ObjectQL-only mode, which doesn't watch files. |
| 269 | + |
| 270 | +**Solution**: Add MetadataPlugin with `watch: true`: |
| 271 | + |
| 272 | +```typescript |
| 273 | +plugins: [ |
| 274 | + new MetadataPlugin({ watch: true }), |
| 275 | + new ObjectQLPlugin(), |
| 276 | +] |
| 277 | +``` |
| 278 | + |
| 279 | +### "Cannot find metadata" in API |
| 280 | + |
| 281 | +**Cause**: Metadata not loaded into the service. |
| 282 | + |
| 283 | +**Debug**: |
| 284 | +1. Check logs for "Loaded X objects" messages |
| 285 | +2. Verify file paths are correct |
| 286 | +3. Check that files have correct naming (e.g., `*.object.ts`, `*.view.yaml`) |
| 287 | +4. Ensure MetadataPlugin `rootDir` points to correct location |
| 288 | + |
| 289 | +## References |
| 290 | + |
| 291 | +- [ADR-0001: Metadata Service Architecture](./adr/0001-metadata-service-architecture.md) |
| 292 | +- [ObjectQL Package](../packages/objectql/README.md) |
| 293 | +- [Metadata Package](../packages/metadata/README.md) |
| 294 | +- [Metadata Spec](../packages/spec/src/api/metadata.zod.ts) |
| 295 | +- [Metadata Loader Protocol](../packages/spec/src/kernel/metadata-loader.zod.ts) |
0 commit comments