Skip to content

data-flow.mdx

Latest commit

 

History

History
322 lines (245 loc) · 11.2 KB

data-flow.mdx

File metadata and controls

322 lines (245 loc) · 11.2 KB
Raw
title Data Flow Diagrams
description How data moves through the ObjectStack system — from defineStack() configuration to database queries, realtime events, and metadata resolution

Data Flow Diagrams

This guide provides detailed diagrams showing how data moves through every layer of the ObjectStack system — from developer configuration to database operations and back to the client.

**Architecture Overview:** For a high-level view of how protocol layers connect, see the [Protocol Relationship Diagram](/docs/protocol/diagram).

1. Configuration Flow: defineStack() → Database

How your objectstack.config.ts becomes a running application.

sequenceDiagram
    participant Dev as Developer
    participant Cfg as objectstack.config.ts
    participant CLI as CLI / Build
    participant Man as Manifest
    participant Ker as Kernel
    participant Plu as Plugins
    participant Drv as Driver
    participant DB as Database

    Dev->>Cfg: defineStack({ objects, views, flows })
    Cfg->>CLI: Export configuration
    CLI->>Man: Compile & validate (Zod)

    alt Validation Failed
        Man-->>CLI: Schema errors
        CLI-->>Dev: Error output with suggestions
    end

    CLI->>Ker: Boot Kernel with Manifest
    Ker->>Plu: Initialize plugins
    Plu-->>Ker: Contracts registered

    Ker->>Drv: Connect to datasource
    Drv->>DB: Run migrations (if needed)
    DB-->>Drv: Schema ready

    Ker-->>CLI: Server ready
    CLI-->>Dev: Listening on http://localhost:3000
Loading
Stage What Happens
defineStack() Developer declares objects, fields, views, flows in TypeScript (array or map format)
Compile CLI validates all definitions against Zod schemas
Manifest Compiled metadata package containing all configuration
Kernel Boot Kernel loads the manifest and initializes the plugin graph
Plugin Init Each plugin registers its contracts (DataEngine, AuthService, etc.)
Driver Connect Database driver connects and runs any pending migrations

2. Read Flow: Client → API → Database → Response

How a GET request for a single record flows through the system.

sequenceDiagram
    participant C as Client
    participant API as REST API
    participant Auth as Auth Middleware
    participant K as Kernel
    participant Sec as Security
    participant QE as Query Engine
    participant H as Hooks
    participant D as Driver
    participant DB as Database

    C->>API: GET /api/v1/data/task/tsk_123
    API->>Auth: Verify JWT token
    Auth-->>API: AuthUser resolved

    API->>K: read('task', 'tsk_123', context)

    K->>Sec: checkPermission('task', 'read', user)
    Sec-->>K: ✓ Allowed

    K->>QE: buildQuery('task', { id: 'tsk_123' })
    QE->>Sec: applyRLS(query, user)
    Sec-->>QE: Filtered query

    QE->>D: execute(filteredQuery)
    D->>DB: SELECT * FROM tasks WHERE id = 'tsk_123' AND ...
    DB-->>D: Row data
    D-->>QE: Typed record

    QE-->>K: DataRecord

    K->>H: afterFind hooks (fire for findOne too)
    H-->>K: Transformed record

    K->>Sec: applyFieldSecurity(record, user)
    Sec-->>K: Fields filtered by profile

    K-->>API: Final record
    API-->>C: 200 OK { object: 'task', id: 'tsk_123', record: { ... } }
Loading
**Row-Level Security (RLS):** Security filters are applied at the query level, not after fetching. This means the database only returns rows the user is allowed to see — efficient even on large tables.

3. Write Flow: Client → Validation → Hooks → Database → Response

How a POST (create) request is processed with full validation and hook execution.

sequenceDiagram
    participant C as Client
    participant API as REST API
    participant K as Kernel
    participant V as Validator
    participant BH as Before Hooks
    participant D as Driver
    participant DB as Database
    participant AH as After Hooks
    participant EB as Event Bus
    participant WS as WebSocket

    C->>API: POST /api/v1/data/task { title: '...', ... }
    API->>K: create('task', data, context)

    K->>V: validate(schema, data)

    alt Validation Failed
        V-->>K: ValidationError { fields: [...] }
        K-->>API: 400 VALIDATION_FAILED
        API-->>C: { error, code: 'VALIDATION_FAILED', fields: [...] }
    end

    V-->>K: ✓ Valid data

    K->>BH: beforeInsert hooks
    Note over BH: Transform data, enforce rules,<br/>set defaults, check permissions

    alt Hook Throws Error
        BH-->>K: BusinessError
        K-->>API: 4xx/5xx Error
        API-->>C: Error Response
    end

    BH-->>K: Transformed data

    K->>D: insert('task', transformedData)
    D->>DB: INSERT INTO tasks (...) VALUES (...)
    DB-->>D: Inserted row with ID
    D-->>K: DataRecord { id: 'tsk_456', ... }

    K->>AH: afterInsert hooks
    Note over AH: Send notifications,<br/>update counters,<br/>trigger workflows

    K->>EB: emit('task.created', record)
    EB->>WS: Broadcast to subscribers

    K-->>API: Created record
    API-->>C: 201 Created { object: 'task', id: 'tsk_456', record: { ... } }
Loading

4. Metadata Flow: defineStack() → IDE Autocomplete

How object definitions flow from configuration to runtime API and IDE tooling.

flowchart LR
    subgraph "Development"
        A["defineStack()"] -->|"TypeScript"| B[objectstack.config.ts]
        B -->|"compile"| C[Zod Validation]
    end

    subgraph "Build Time"
        C -->|"generate"| D[JSON Schema]
        C -->|"generate"| E[TypeScript Types]
        D -->|"feed"| F[IDE Autocomplete]
        E -->|"feed"| F
    end

    subgraph "Runtime"
        C -->|"bundle"| G[Manifest]
        G -->|"load"| H[Schema Registry]
        H -->|"serve"| I["GET /api/v1/meta/object"]
        I -->|"discover"| J[Client SDK]
        J -->|"generate"| K[Typed Client]
    end
Loading
Output Used By Purpose
JSON Schema VS Code, IntelliJ Autocomplete and validation for objectstack.config.ts (via os generate schema)
TypeScript Types Plugin developers Type-safe access to object definitions
Manifest Kernel Runtime metadata for query validation and execution
Metadata API Client SDK Dynamic object/field discovery
Typed Client Frontend apps client.data.create('task', { title: '...' })

5. Realtime Flow: Database Change → Client

How data changes propagate to connected clients in real time.

**Implementation status:** This diagram (and the `emit(...)` → `EB->>WS` broadcast step in the Write Flow diagram above) shows the target design. The realtime service that ships today is an **in-memory pub/sub adapter** (`InMemoryRealtimeAdapter` in `@objectstack/service-realtime`), delivered to JS clients via **long-polling** (`RealtimeAPI` in `@objectstack/client`) — no WebSocket transport is wired up yet. The adapter also only matches subscriptions by `object` and `eventTypes`; per-field `filter` conditions (e.g. `{ project: 'prj_1' }`) are accepted in the request shape but are not yet enforced server-side. See [Real-Time Protocols](/docs/protocol/kernel/realtime-protocol) for the full implementation-status breakdown. 
sequenceDiagram
    participant W as Writer Client
    participant API as API Layer
    participant K as Kernel
    participant DB as Database
    participant EB as Event Bus
    participant Sub as Subscription Manager
    participant R1 as Reader Client 1
    participant R2 as Reader Client 2

    Note over R1,R2: Clients subscribe via WebSocket

    R1->>Sub: subscribe('task', { filter: { project: 'prj_1' } })
    R2->>Sub: subscribe('task', { filter: { assigned_to: 'usr_2' } })
    Sub-->>R1: Subscribed (channelId: ch_1)
    Sub-->>R2: Subscribed (channelId: ch_2)

    W->>API: PATCH /api/v1/data/task/tsk_123 { status: 'done' }
    API->>K: update(...)
    K->>DB: UPDATE tasks SET status = 'done' WHERE id = 'tsk_123'
    DB-->>K: Updated record

    K->>EB: emit('task.updated', { record, changes })

    EB->>Sub: Route event to matching subscriptions
    Sub->>Sub: Check filters

    Note over Sub: tsk_123.project = 'prj_1' ✓ → ch_1
    Note over Sub: tsk_123.assigned_to = 'usr_2' ✓ → ch_2

    Sub->>R1: { event: 'task.updated', data: { id: 'tsk_123', ... } }
    Sub->>R2: { event: 'task.updated', data: { id: 'tsk_123', ... } }

    K-->>API: Updated record
    API-->>W: 200 OK { object: 'task', id: 'tsk_123', record: { ... } }
Loading
**Subscription Filtering:** The Subscription Manager applies each client's filter server-side. Clients only receive events for records that match their subscription criteria — no over-fetching.

6. Query Execution Flow

How a structured query is translated to SQL and executed.

flowchart TD
    A[QueryInput] -->|parse| B[Query Parser]
    B -->|validate| C[Schema Validator]
    C -->|optimize| D[Query Optimizer]
    D -->|secure| E[RLS Filter Injection]
    E -->|translate| F[SQL Generator]
    F -->|execute| G[Database]
    G -->|map| H[Result Mapper]
    H -->|paginate| I[QueryResult]
Loading
Stage Description
Query Parser Parses the JSON QueryInput into an AST
Schema Validator Verifies field names and operators against the object schema
Query Optimizer Reorders filters, selects indexes, applies batch strategies
RLS Filter Injection Appends row-level security conditions based on the user's profile
SQL Generator Translates the AST to the target database dialect (PostgreSQL, MySQL, SQLite)
Database Executes the query and returns raw rows
Result Mapper Converts raw rows to typed DataRecord objects
Pagination Calculates total, hasMore, offset for the response

7. Hook Execution Order

The precise order in which hooks fire during each operation.

flowchart TD
    subgraph "Insert Operation"
        C4[beforeInsert] --> C5["DB INSERT"]
        C5 --> C6[afterInsert]
    end

    subgraph "Update Operation"
        U4[beforeUpdate] --> U5["DB UPDATE"]
        U5 --> U6[afterUpdate]
    end

    subgraph "Delete Operation"
        D1[beforeDelete] --> D2["DB DELETE"]
        D2 --> D3[afterDelete]
    end
Loading

The lifecycle events are defined by the HookEvent enum (8 events). Reads fire beforeFind/afterFind — for both find and findOne, so one subscription covers every read shape. Bulk writes (multi: true) fire the same beforeUpdate/beforeDelete/afterUpdate/afterDelete events as single-id writes, with the row-scoping predicate in ctx.input.ast. There are deliberately no per-method (findOne/count/aggregate) or *Many events: read authorization and row filtering are RLS/permission-rule concerns, and field masking is field-level metadata.

Hook Phase Can Modify? Can Abort?
beforeInsert/Update/Delete Pre-persist ✅ Yes (mutate input) ✅ Yes (throw with onError: 'abort')
afterInsert/Update/Delete Post-persist ✅ Yes (mutate result) ❌ No (record is saved)
**Side Effects:** Place side effects (emails, notifications, external API calls) in `after*` hooks. Set `async: true` so they run fire-and-forget without blocking the operation, and use `onError: 'log'` so a failing side effect doesn't roll back the write.