feat: add constructive-compute-stubs — schema-only DDL for FK resolution#51
Closed
pyramation wants to merge 100 commits into
Closed
feat: add constructive-compute-stubs — schema-only DDL for FK resolution#51pyramation wants to merge 100 commits into
pyramation wants to merge 100 commits into
Conversation
Sliced from constructive-db's monolithic constructive module using: pnpm run slice:constructive -- --renumber-alterations --strip-cross-package-deps Package includes: - constructive_infra_public schema, types, and all tables - constructive_infra_private schema and trigger functions - 254 changes, 764 files Standalone: requires only plpgsql, pgpm-inflection, pgpm-stamps No RLS policies, no grants, no job triggers
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
…mignore) The slicer now generates a complete pgpm module with: - package.json with @pgpm/* dependencies mapped from .control requires - Makefile with PGXS include - .npmignore excluding test files This fixes 'No package.json found' error when running pgpm install.
|
All alerts resolved. Learn more about Socket for GitHub. This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored. |
…ema stripping - Add missing deploy dependencies (type, constraint→column, FK→pkey) - Plan is now topologically sorted for correct deploy order - Remove metaschema partition registrations (INSERT INTO metaschema_public.*) that require the closed-source MetaSchema system - Successfully tested: pgpm deploy --yes --createdb --database testinfra
…tion Separate pgpm module that registers the constructive-infra function tables as a MetaSchema function_module. This allows querying MetaSchema to discover the infra function tables (definitions, invocations, execution_logs, secret_definitions). Dependencies: - constructive-infra (the raw DDL) - @pgpm/metaschema-modules (function_module table) - @pgpm/services (service registration layer) The registration INSERT provides explicit schema/table names for the platform-scoped infra tables, letting the MetaSchema trigger system handle ID resolution and API routing.
Adds a CI workflow that runs 'pgpm test-packages --full-cycle' on every PR that touches pgpm/ files. This validates the full deploy/verify/revert/deploy cycle for constructive-infra against a real postgres-plus:18 instance. constructive-infra-services is excluded since it requires the full MetaSchema system (closed-source triggers) to deploy. Also fixes the function_requirement type revert SQL which was missing the TYPE keyword (DROP → DROP TYPE).
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
- job/compute-worker: platform-aware worker that discovers functions from constructive_infra_public.platform_function_definitions (TTL-cached), tracks invocations in platform_function_invocations, dispatches via HTTP - job/compute-service: orchestrator (callback server + ComputeWorker + Scheduler), mirrors job/service patterns - scripts/setup-platform-db.sh: Tier 1 setup (pgpm deploy + seed) - scripts/dev-compute.ts: dev launcher for compute-service + functions - scripts/seed-functions.sql: registers send-email + send-verification-link - docker-compose.yml: adds platform-setup service (Tier 2) - k8s/overlays/local-simple/compute-service.yaml: K8s manifest (Tier 3) - Makefile: adds setup-platform, dev-compute targets + tier documentation - .agents/skills/dev-tiers: skill for the 3-tier dev model - .agents/skills/compute-worker: skill for the compute-worker system - pgpm/constructive-infra: adds @pgpm/database-jobs dependency
…error) pgpm deploy requires module dependencies to be pre-installed via 'pgpm install' before deployment. Without this, packages like @pgpm/database-jobs fail with 'extension not available'. Adds pgpm install to: - scripts/setup-platform-db.sh (Tier 1) - docker-compose.yml platform-setup service (Tier 2) - .github/workflows/pgpm-test.yaml (CI)
- make status: shows Docker containers, PG connection, databases with infra schema, Node/pnpm/pgpm versions, and build state - make verify-platform: checks DB exists, infra + jobs schemas deployed, tables present, functions seeded — exits non-zero with fix instructions if anything is wrong
Dan committed all extension deps (database-jobs, metaschema-modules, services, etc.) directly into extensions/. pgpm install is no longer needed before deploy. Also restores pgpm volume to :ro in Docker Compose and mounts extensions/ for the platform-setup service.
Built-in function definitions (send-email, send-verification-link) are now deployed as a pgpm fixture inside constructive-infra. The fixture uses the standard deploy/revert/verify pattern following the inflection module's fixtures convention. - deploy: INSERT ON CONFLICT DO NOTHING - revert: DELETE WHERE is_built_in AND scope='platform' - verify: SELECT 1 from seed row Removes manual psql seed step from setup script, docker-compose, and verify-platform fix instructions. pgpm deploy now handles everything.
Moves the built-in function seed data (send-email, send-verification-link) from constructive-infra into its own pgpm package: constructive-infra-seed. This keeps constructive-infra as pure DDL (schemas, tables, triggers) and the seed data as a separate deployable unit. The seed package depends on constructive-infra via the .control file requires. Both packages pass pgpm test-packages --full-cycle.
compute-service discovers functions from the database, so it doesn't need direct workspace deps on send-email-fn or send-verification-link-fn (which are generated packages in generated/ and not in the workspace).
…nfigs - New fixture: seed_built_in_secrets seeds MAILGUN_* and SMTP_* into platform_secret_definitions (with well-known default database_id) - Function definitions now include required_secrets and required_configs arrays (function_requirement[] type) linking functions to their deps - All SQL files now use proper pgpm format: BEGIN/COMMIT wrappers, '-- Deploy:' header style with '-- made with <3 @ constructive.io' - New script: load-platform-env.sh reads .env, cross-references against DB function requirements, reports satisfied vs missing keys - .env.example updated with SMTP/Mailpit + dry-run toggle sections - Makefile: added 'make check-env' target
Procedural lifecycle targets: make up # prereqs → docker → bootstrap → deploy → seed → verify make up DB_NAME=mydb # same with custom DB make down # stop docker compose + pgpm docker stop DROP=1 make down DB_NAME=mydb # also drop the DB make up:email-job # start mailpit + compute-service (SMTP mode) make down:email-job # stop mailpit + compute-service email-job-up verifies platform is up first, starts mailpit, loads .env (with sane SMTP defaults), then launches compute-service. status now shows mailpit container state.
…rrides - Skip python-example and other non-node-graphql functions from startup - Print a clear port/service summary table before launching processes - Respect SEND_EMAIL_DRY_RUN, SEND_VERIFICATION_LINK_DRY_RUN, EMAIL_SEND_USE_SMTP, and SMTP_FROM from environment (was hardcoded)
- www/: Vite + React + Express app with 6 tabs (Functions, Secrets, Jobs, Invocations, Commands, Terminal) - Express backend: REST API for DB queries + WebSocket terminal via child_process - Secrets & Namespaces tab: shows seeded secret definitions and default namespace - Seed: add default platform namespace, link functions to namespace_id - Function API: parse composite-type arrays into proper JSON - Makefile: add make up:www target - scripts/www-up.sh: checks platform, installs deps, starts Vite + Express
…ster management - Install commander and kubernetesjs packages in www/ - Add K8s proxy endpoint (/api/k8s/*) in Express server - Add K8s tab with Pods/Deployments/Services views per namespace - Namespace selector defaults to constructive-functions - Graceful error when kubectl proxy not running
… tab - ansiToHtml() converts ANSI escape sequences into colored <span> elements - Remove max-h-60 so output expands naturally with content - Exit code shown in a separate footer bar - Strip remaining escape sequences that don't match known codes
Step 4b cross-references loaded env vars against platform_function_definitions required_secrets/required_configs. Warns about missing secrets but doesn't block (defaults still work for Mailpit/SMTP mode). Suggests 'make check-env' for details.
- Reads .env file and merges with process.env + dev defaults - Queries platform_function_definitions for required_secrets/required_configs - Reports per-function coverage at startup (✓ all set / ● N missing) - Priority: .env > process.env > hardcoded defaults - Graceful fallback if DB query fails (schema not deployed)
Both tables are PARTITION BY RANGE(created_at) but had no partitions, causing 'no partition of relation found for row' on INSERT. Adds DEFAULT partition for each so rows land somewhere until time-based partitions are created.
Secrets tab now shows: - Function coverage badges (e.g. send-email 5/8) - All required secrets/configs from DB + .env merged - Editable input fields with eye/reveal toggle for sensitive keys - 'Save to .env' button writes grouped .env file to project root - Shows which functions require each secret - File status indicator (.env exists / will be created) Backend adds: - GET /api/env — reads .env and returns parsed vars - POST /api/env — merges values + writes grouped .env file
Each function card now has a 'Trigger' button that opens an inline payload editor with sensible defaults (send-email gets to/subject/html, send-verification-link gets to/type/link). Submitting creates a job in app_jobs.jobs and shows success/error inline — no need to switch tabs.
fix: auto-generate function packages when missing
…efinitions - scripts/dev-compute.ts: fix DEFAULT_DATABASE_ID from 'dbe' to proper nil UUID, update function_definitions query to use constructive_compute_public - scripts/email-job-up.sh: check for constructive_compute_public schema, query function_definitions from constructive_compute_public - scripts/register-functions.ts: INSERT into constructive_compute_public - scripts/seed-functions.sql: INSERT into constructive_compute_public - scripts/status.sh: check constructive_compute_public for function counts - scripts/setup-platform-db.sh: verify function count from correct schema - scripts/secrets-sync.sh: query function requirements from correct schema - scripts/load-platform-env.sh: query function definitions from correct schema platform_namespaces remains in constructive_infra_public (not moved).
fix: update scripts to use constructive_compute_public for function definitions
- module-loader: catch missing function_invocation_module table gracefully - discovery: fall back to constructive_compute_public when function_module config not populated in metaschema - compute-service: fall back to direct schema check in waitForComputePrereqs when metaschema database row doesn't exist
Updates all @pgpm dependency packages including metaschema-modules which now includes the function_prefix column on merkle_store_module.
Replaces constructive-compute-fbp with constructive-platform-function-graph (unprefixed merkle store functions: get_all, get_node_at_path, etc.) Updates constructive-compute with FBP operations now consolidated into constructive_compute_public schema.
…platform-function-graph - up.sh: deploy constructive-platform-function-graph instead of constructive-compute-fbp - verify-platform.sh: check new schema names - infra-seed: update partition tables to constructive_compute_private - copy-slicer-packages.py: update package mapping
Replaced by constructive-platform-function-graph (merkle store functions) and constructive-compute (FBP operations) after schema consolidation.
…-update feat: unprefixed functions + schema consolidation + graceful fallback
The platform_files_file_path and platform_files_rename functions reference columns (filename, id) via composite type parameters. PostgreSQL validates SQL function bodies at creation time, so the columns must exist before the functions are created. Moved procedure entries in pgpm.plan to after all column additions and added explicit column dependencies in the deploy SQL requires headers.
fix: reorder pgpm.plan so storage procedures deploy after columns
… RLS/partman Fixes all pgpm deploy errors for standalone (non-monolith) mode: 1. pgpm.plan ordering: Move procedures/trigger_fns that reference table composite types to AFTER the table definitions they depend on. Affects: constructive-objects, constructive-platform-function-graph, constructive-compute. 2. Remove constructive-compute-stubs: Replace all stubs references with constructive-infra (the actual module). Update .control files and pgpm.plan cross-package deps. Delete the stubs directory entirely. 3. Strip RLS policies: Remove enable_row_level_security and auth_* policy entries from objects/storage/store pgpm.plan files — these reference jwt_public and constructive_memberships schemas not available in standalone mode. 4. Strip jwt_public triggers: Remove force_current_user_actor_id trigger_fns and their triggers from storage (depend on jwt_public). 5. Strip partman entries: Remove metaschema_public.partition INSERTs from compute pgpm.plan (metaschema not available standalone). 6. Strip monolith trigger: Remove platform_function_definitions job trigger that references the constructive monolith package. 7. Fix seed data: Update partition seed to use platform_function_* table names (was app_function_*). Remove stale invocations_table_name column from function_module registration. Fix verify script table name.
fix: pgpm deploy standalone — reorder procedures, remove stubs, strip RLS/partman
…e_compute_public After schema consolidation, platform_function_definitions, platform_function_invocations, and function_requirement type moved from constructive_infra_public to constructive_compute_public. Updates: www/server/index.ts, scripts/register-functions.ts, docs/spec/fbp-integration.md, compute-worker + fbp skill files.
…-schema-refs fix: update schema refs from constructive_infra_public to constructive_compute_public
New pgpm package that seeds the full API routing chain for standalone mode: - database row (well-known 00000000-...0000 sentinel) - 12 schema registrations (7 public + 5 private) - 2 table registrations (definitions + secrets) for module FK refs - compute API linked to all 7 public schemas - compute.localhost domain - app.localhost site + site domain Also updates constructive-infra-services to properly resolve schema_id, private_schema_id, definitions_table_id, and secret_definitions_table_id via JOINs against the seeded metaschema rows. up.sh gains step 5 (platform-seed) before step 6 (infra-seed + services).
feat: add constructive-platform-seed (metaschema routing chain)
Adds step 8 to up.sh: starts @constructive-io/graphql-server as a background process using the sibling constructive repo, with PID file for clean shutdown. - start-graphql-server.sh: resolves ../constructive, starts server with enableServicesApi=true + isPublic=true, writes PID to .graphql-server.pid - up.sh: step 8 starts the server, prints compute.localhost URLs - down.sh: kills the server via PID file before stopping MinIO/PG The server uses the services_public routing chain seeded by constructive-platform-seed to resolve compute.localhost -> 7 schemas. Total steps: 9 -> 10.
feat: integrate GraphQL server into make up/down (port 6464)
- Compute API (compute.localhost): users, infra, objects, storage, store, compute - Graph API (graph.localhost): platform_function_graph, compute Prevents PostGraphile function name collisions between constructive_objects_public and constructive_platform_function_graph_public (both export unprefixed merkle store functions: init_empty_repo, get_all, etc.)
feat: split compute and graph into separate APIs
Restructure seed to match the canonical API-to-schema mapping from constructive-db's introspection.provision_base_modules: - api: users, infra, storage, store (general platform schemas) - compute: compute_public + platform_function_graph_public (matches COMPUTE_SCHEMAS = [compute_public, compute_fbp_public]) - objects: objects_public (content-addressed merkle store) - flow: empty placeholder (populated at runtime in constructive-db) Previously objects was lumped into compute, causing PostGraphile collisions between the objects and graph merkle store functions. Now each API has its own domain: api/compute/objects/flow.localhost.
feat: align API routing with constructive-db provision_base_modules
feat: add make kill (down + pgpm kill)
Adds 4 SDK packages matching constructive-db's structure: - constructive-functions-schema: exports .graphql from live endpoints (api, compute, objects) via introspection - constructive-functions-sdk: generates typed ORM client from schema files - constructive-functions-cli: generates unified CLI from schema files - constructive-functions-hooks: generates React Query hooks for www/ Includes: - Root-level generate scripts (generate:schemas, generate:sdk, etc.) - Makefile targets (make generate:schemas, make generate:sdk-all) - CI workflow (.github/workflows/generate-sdk.yaml) for automated regen - Auto-generated .agents/skills/ for ORM, CLI, and hooks reference - All generated output from current schema (3 APIs, 25 tables total)
feat: SDK generation pipeline (schemas, ORM, CLI, hooks)
|
Superseded by #95 — includes constructive-compute-stubs, then replaced with per-module packages, plus infra-seed, platform-seed, and infra-services. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
database/constructive-compute-stubs/— a pgpm package containing schema-only DDL (tables, columns, constraints, indexes) for 19 tables that the compute layer needs for FK resolution, with zero behavioral code.4 schemas, 19 tables:
services_public(15): apps, apis, domains, sites, api_modules, api_schemas, api_settings, cors_settings, database_settings, pubkey_settings, rls_settings, site_metadata, site_modules, site_themes, webauthn_settingsconstructive_users_public(2): users, role_typesconstructive_infra_public(2): platform_namespaces, platform_namespace_eventsconstructive_infra_private: schema only (no tables)What's excluded (via upstream
excludeCategoriesin constructive-db#1590):'auth''memberships''permissions'stubModeZero: policies, grants, triggers, trigger functions, procedures, views, or fixtures.
Link to Devin session: https://app.devin.ai/sessions/633efc205f0c445dbbb61c40679c2f78
Requested by: @pyramation