feat(pgpm): add execution node_states table + fail_node procedure + SDK regen#96
feat(pgpm): add execution node_states table + fail_node procedure + SDK regen#96pyramation wants to merge 175 commits into
Conversation
Sliced from constructive-db's monolithic constructive module using: pnpm run slice:constructive -- --renumber-alterations --strip-cross-package-deps Package includes: - constructive_infra_public schema, types, and all tables - constructive_infra_private schema and trigger functions - 254 changes, 764 files Standalone: requires only plpgsql, pgpm-inflection, pgpm-stamps No RLS policies, no grants, no job triggers
…mignore) The slicer now generates a complete pgpm module with: - package.json with @pgpm/* dependencies mapped from .control requires - Makefile with PGXS include - .npmignore excluding test files This fixes 'No package.json found' error when running pgpm install.
…ema stripping - Add missing deploy dependencies (type, constraint→column, FK→pkey) - Plan is now topologically sorted for correct deploy order - Remove metaschema partition registrations (INSERT INTO metaschema_public.*) that require the closed-source MetaSchema system - Successfully tested: pgpm deploy --yes --createdb --database testinfra
…tion Separate pgpm module that registers the constructive-infra function tables as a MetaSchema function_module. This allows querying MetaSchema to discover the infra function tables (definitions, invocations, execution_logs, secret_definitions). Dependencies: - constructive-infra (the raw DDL) - @pgpm/metaschema-modules (function_module table) - @pgpm/services (service registration layer) The registration INSERT provides explicit schema/table names for the platform-scoped infra tables, letting the MetaSchema trigger system handle ID resolution and API routing.
Adds a CI workflow that runs 'pgpm test-packages --full-cycle' on every PR that touches pgpm/ files. This validates the full deploy/verify/revert/deploy cycle for constructive-infra against a real postgres-plus:18 instance. constructive-infra-services is excluded since it requires the full MetaSchema system (closed-source triggers) to deploy. Also fixes the function_requirement type revert SQL which was missing the TYPE keyword (DROP → DROP TYPE).
- job/compute-worker: platform-aware worker that discovers functions from constructive_infra_public.platform_function_definitions (TTL-cached), tracks invocations in platform_function_invocations, dispatches via HTTP - job/compute-service: orchestrator (callback server + ComputeWorker + Scheduler), mirrors job/service patterns - scripts/setup-platform-db.sh: Tier 1 setup (pgpm deploy + seed) - scripts/dev-compute.ts: dev launcher for compute-service + functions - scripts/seed-functions.sql: registers send-email + send-verification-link - docker-compose.yml: adds platform-setup service (Tier 2) - k8s/overlays/local-simple/compute-service.yaml: K8s manifest (Tier 3) - Makefile: adds setup-platform, dev-compute targets + tier documentation - .agents/skills/dev-tiers: skill for the 3-tier dev model - .agents/skills/compute-worker: skill for the compute-worker system - pgpm/constructive-infra: adds @pgpm/database-jobs dependency
…error) pgpm deploy requires module dependencies to be pre-installed via 'pgpm install' before deployment. Without this, packages like @pgpm/database-jobs fail with 'extension not available'. Adds pgpm install to: - scripts/setup-platform-db.sh (Tier 1) - docker-compose.yml platform-setup service (Tier 2) - .github/workflows/pgpm-test.yaml (CI)
- make status: shows Docker containers, PG connection, databases with infra schema, Node/pnpm/pgpm versions, and build state - make verify-platform: checks DB exists, infra + jobs schemas deployed, tables present, functions seeded — exits non-zero with fix instructions if anything is wrong
Dan committed all extension deps (database-jobs, metaschema-modules, services, etc.) directly into extensions/. pgpm install is no longer needed before deploy. Also restores pgpm volume to :ro in Docker Compose and mounts extensions/ for the platform-setup service.
Built-in function definitions (send-email, send-verification-link) are now deployed as a pgpm fixture inside constructive-infra. The fixture uses the standard deploy/revert/verify pattern following the inflection module's fixtures convention. - deploy: INSERT ON CONFLICT DO NOTHING - revert: DELETE WHERE is_built_in AND scope='platform' - verify: SELECT 1 from seed row Removes manual psql seed step from setup script, docker-compose, and verify-platform fix instructions. pgpm deploy now handles everything.
Moves the built-in function seed data (send-email, send-verification-link) from constructive-infra into its own pgpm package: constructive-infra-seed. This keeps constructive-infra as pure DDL (schemas, tables, triggers) and the seed data as a separate deployable unit. The seed package depends on constructive-infra via the .control file requires. Both packages pass pgpm test-packages --full-cycle.
compute-service discovers functions from the database, so it doesn't need direct workspace deps on send-email-fn or send-verification-link-fn (which are generated packages in generated/ and not in the workspace).
…nfigs - New fixture: seed_built_in_secrets seeds MAILGUN_* and SMTP_* into platform_secret_definitions (with well-known default database_id) - Function definitions now include required_secrets and required_configs arrays (function_requirement[] type) linking functions to their deps - All SQL files now use proper pgpm format: BEGIN/COMMIT wrappers, '-- Deploy:' header style with '-- made with <3 @ constructive.io' - New script: load-platform-env.sh reads .env, cross-references against DB function requirements, reports satisfied vs missing keys - .env.example updated with SMTP/Mailpit + dry-run toggle sections - Makefile: added 'make check-env' target
Procedural lifecycle targets: make up # prereqs → docker → bootstrap → deploy → seed → verify make up DB_NAME=mydb # same with custom DB make down # stop docker compose + pgpm docker stop DROP=1 make down DB_NAME=mydb # also drop the DB make up:email-job # start mailpit + compute-service (SMTP mode) make down:email-job # stop mailpit + compute-service email-job-up verifies platform is up first, starts mailpit, loads .env (with sane SMTP defaults), then launches compute-service. status now shows mailpit container state.
…rrides - Skip python-example and other non-node-graphql functions from startup - Print a clear port/service summary table before launching processes - Respect SEND_EMAIL_DRY_RUN, SEND_VERIFICATION_LINK_DRY_RUN, EMAIL_SEND_USE_SMTP, and SMTP_FROM from environment (was hardcoded)
- www/: Vite + React + Express app with 6 tabs (Functions, Secrets, Jobs, Invocations, Commands, Terminal) - Express backend: REST API for DB queries + WebSocket terminal via child_process - Secrets & Namespaces tab: shows seeded secret definitions and default namespace - Seed: add default platform namespace, link functions to namespace_id - Function API: parse composite-type arrays into proper JSON - Makefile: add make up:www target - scripts/www-up.sh: checks platform, installs deps, starts Vite + Express
…ster management - Install commander and kubernetesjs packages in www/ - Add K8s proxy endpoint (/api/k8s/*) in Express server - Add K8s tab with Pods/Deployments/Services views per namespace - Namespace selector defaults to constructive-functions - Graceful error when kubectl proxy not running
… tab - ansiToHtml() converts ANSI escape sequences into colored <span> elements - Remove max-h-60 so output expands naturally with content - Exit code shown in a separate footer bar - Strip remaining escape sequences that don't match known codes
Step 4b cross-references loaded env vars against platform_function_definitions required_secrets/required_configs. Warns about missing secrets but doesn't block (defaults still work for Mailpit/SMTP mode). Suggests 'make check-env' for details.
- Reads .env file and merges with process.env + dev defaults - Queries platform_function_definitions for required_secrets/required_configs - Reports per-function coverage at startup (✓ all set / ● N missing) - Priority: .env > process.env > hardcoded defaults - Graceful fallback if DB query fails (schema not deployed)
Both tables are PARTITION BY RANGE(created_at) but had no partitions, causing 'no partition of relation found for row' on INSERT. Adds DEFAULT partition for each so rows land somewhere until time-based partitions are created.
Secrets tab now shows: - Function coverage badges (e.g. send-email 5/8) - All required secrets/configs from DB + .env merged - Editable input fields with eye/reveal toggle for sensitive keys - 'Save to .env' button writes grouped .env file to project root - Shows which functions require each secret - File status indicator (.env exists / will be created) Backend adds: - GET /api/env — reads .env and returns parsed vars - POST /api/env — merges values + writes grouped .env file
Each function card now has a 'Trigger' button that opens an inline payload editor with sensible defaults (send-email gets to/subject/html, send-verification-link gets to/type/link). Submitting creates a job in app_jobs.jobs and shows success/error inline — no need to switch tabs.
Replace declare -A (bash 4+) with newline-separated string + grep. Replace read -ra with portable IFS-based for loops. Replace ((...)) arithmetic with $((...)) POSIX form. macOS ships bash 3.2 — these scripts now work on both.
Jobs created from the UI or without a database_id context now fall back to 00000000-0000-0000-0000-000000000000 (the well-known default used by the seed). Fixes NOT NULL violation on platform_function_invocations.
…ents - Add .agents/skills/fbp/SKILL.md mapping FBP NodeDefinitions to platform functions - Add docs/spec/fbp-integration.md research doc with full mapping spec - Add Flows tab with React Flow: drag-and-drop function nodes, edge creation, localStorage persistence, sidebar palette, and minimap - Enhance FunctionsPanel trigger success message with Invocations tab link - Install @xyflow/react for the flow graph canvas
…tion - Add platform_secret_values table (pgpm migration: deploy/revert/verify) Columns: id, secret_name, configured_value, database_id, created_at, updated_at Unique constraint on (secret_name, database_id) - Backend API (www/server/index.ts): GET /api/secret-values — read configured values from DB POST /api/secret-values — write configured values to DB POST /api/secrets/sync-from-db — DB values -> .env POST /api/secrets/sync-to-db — .env values -> DB POST /api/env now also syncs to DB on save (best-effort) - Frontend (SecretsPanel.tsx): Add 'Sync from DB' and 'Sync to DB' buttons DB status indicator in header - Refactor scripts/dev-compute.ts: Secrets pipeline: .env > DB > hardcoded defaults Per-function env injection (only needed secrets/configs) Fail fast on missing required secrets, warn on optional - Add make secrets:sync (bidirectional sync script) - Add scripts/secrets-sync.sh for CLI-based sync
…low) run in-process - Add 'runtime' column (http|inline) to platform_function_definitions - Split doWork() into doWorkInline() and doWorkHttp() paths - InlineNodeRegistry with 13 implementations: math/add, math/multiply, math/subtract, const/number, const/string, const/boolean, json/select, json/object, json/merge, json/split, string/template, flow/guard, coerce - 23 unit tests for inline node implementations - register-functions.ts includes inline nodes in seed SQL - FlowsPanel MOCK_FUNCTIONS show inline nodes with Zap badge - pgpm runtime column with deploy/revert/verify scripts
…lette - Remove http-request, json-transform, text-template (no running services) - Seed SQL now has 4 HTTP + 13 inline = 17 functions - Palette shows 'inline' (amber) and 'http' (cyan) badges per node - Clear visual demarcation between in-process and HTTP-dispatched functions
- Add unique icon to every function: mail, send, code, terminal - Add unique icon to every inline node: plus, x, minus, hash, type, toggle, filter, git-merge, braces, scissors, quote, shield, repeat - Fix sidebar: use def.icon for all nodes (not just non-functions) - Color icons by type: amber for inline, cyan for HTTP, zinc for built-in - Add missing MOCK_FUNCTIONS: const/number, const/string, const/boolean, json/object (now 17 total: 4 HTTP + 13 inline) - Regenerate seed SQL with updated icons
Updates packages/fbp-graph-editor NodeIcon.tsx with mail, send, code, terminal, minus, globe, git-merge, scissors, shield, repeat, toggle, cpu, box, layers, filter, database icons — both React iconMap and SVG iconPaths for canvas rendering.
…luator Filters out @fbp/evaluator definitions when a platform function with the same name exists (e.g. const/number, json/select, flow/guard). Platform defs carry runtime metadata (inline/http) that evaluator defs lack, so they must win.
- tick_execution SQL now includes 'props' in job payload alongside 'inputs'
- compute-worker converts props array [{name,type,value}] to flat object for impl()
- const/string, const/number, const/boolean now work end-to-end in graph execution
- graph node failures no longer retry (return instead of re-throw after failGraphExecution)
- add Export/Import JSON graph buttons to FlowsPanel toolbar for debugging
- tick_execution: mark nodes as enqueued (null sentinel in node_outputs) after
inserting job, preventing re-enqueue on subsequent ticks. Root cause of 16x
email sends: each inline node completion triggered tick which re-enqueued
all not-yet-completed nodes.
- Node naming: use {type}{N} pattern (string1, string2, send-email1) instead
of random suffixes or _copy_ chains. Shared nextNodeName() utility used by
drag-drop, duplicate, paste, and sidebar add.
- Graph name: sync flowName into graph object on save/execute so exported JSON
has the correct name instead of empty string.
- Export/Import: download/upload buttons in toolbar for JSON graph debugging.
…store by (database_id, name)
Renames scripts/email-job-{up,down}.sh → scripts/compute-{up,down}.sh
and updates all references in Makefile, www UI, status.sh, up.sh.
The command starts the full compute-service (inline + HTTP nodes) plus
mailpit, not just email — the new name reflects that.
When no stores exist, shows an explicit 'Create Store' button instead of just 'No saved flows yet' text. Uses the flow name input value (or 'default') as the store name.
gqlFetch now checks content-type before parsing JSON. If the server returns HTML (e.g. Vite's index.html fallback when port 6464 is down), the error message tells the user to run 'make up' instead of showing 'Unexpected token <'.
…ines, remove const inputs 1. platform_create_function_graph: SELECT existing graph by (store_id, name) before INSERT — prevents duplicate key on re-execute 2. Node name label (e.g. 'string1', 'send-email1') shown above each node on the canvas in both compact and rich rendering modes 3. Optional input ports render as outlines (hollow circle with colored stroke) vs required ports which are solid filled. Subtle visual cue. 4. Const nodes (const/string, const/number, const/boolean) no longer get a spurious 'payload' input — platformFnToDefinition returns empty inputs array when none are defined instead of the fallback.
- send-email handler returns { result: { complete: true } } matching
handler.json output port name 'result'
- send-verification-link handler wraps all returns in { result: {...} }
- failGraphExecution now tracks node name in error message:
'[nodeName] error details'
- failGraphExecution sets error_code = 'NODE_EXECUTION_FAILED' and
completed_at for proper lifecycle tracking
- Race condition handled: if execution already completed/failed, logs
a warning instead of swallowing silently
- Removed try/catch wrapper around failGraphExecution — errors from
the UPDATE propagate to the job queue's handleError
- Updated unit tests for new port-named return values
Slice export from constructive-db PR #1621: - New: platform_function_graph_execution_node_states (partitioned by created_at) - New: platform_fail_node procedure - Updated: platform_complete_node (adds node_states UPDATE on completion) - Updated: platform_tick_execution (adds node_states INSERT on enqueue)
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
Warning Review the following alerts detected in dependencies. According to your organization's Security Policy, it is recommended to resolve "Warn" alerts. Learn more about Socket for GitHub.
|
Procedures (fail_node, complete_node, tick_execution) are now stubs — actual bodies are generated at provision time by graph_module AST builders. Added 36 revert scripts for all node_states table entries (table, columns, alterations, indexes, constraints) to fix pgpm deploy test.
The graph test framework deploys pgpm modules standalone (no metaschema provision), so procedures need working bodies. Stubs break 11 graph tests since start_execution calls tick_execution internally. In the full system, AST body builders replace these at provision time. In constructive-functions standalone, the exported bodies are self-contained and reference only tables present in the pgpm module.
1 participant
Summary
Imports Phase 1 execution debugging schema from constructive-db PR #1621 via slice export into
pgpm/constructive-compute/.New table:
platform_function_graph_execution_node_states(partitioned bycreated_at) — tracks per-node execution status (pending/queued/running/completed/failed), timing (started_at/completed_at), errors (error_code/error_message), and output refs (output_id).New procedure:
platform_fail_node(execution_id, node_name, error_code, error_message)— marks node as failed + marks entire execution as failed with[node_name] error_message.Updated procedures:
platform_tick_execution— INSERTs into node_states when enqueuing a nodeplatform_complete_node— UPDATEs node_states when a node completes (status='completed', output_id set)UPDATE statements use
AS nstable alias to avoid PL/pgSQL column ambiguity (function parameter names shadow column names).Procedures retain full bodies (not stubs) because
functions-testdeploys pgpm standalone without constructive-db's metaschema provision. In the full system, AST body builders replace these at provision time.36 revert scripts for node_states entries.
created_atcolumn revert is a no-op (SELECT 1) since it's the partition key. Partman entry removed (FK to metaschema doesn't resolve standalone).SDK regenerated (schemas, ORM, CLI, hooks).
Ref: constructive-planning #1057
Link to Devin session: https://app.devin.ai/sessions/b2291a8e333e445aa125a2efd1996206
Requested by: @pyramation