feat: compute usage logging (Phase 1 — billing/metering foundation)#91
Closed
pyramation wants to merge 118 commits into
Closed
feat: compute usage logging (Phase 1 — billing/metering foundation)#91pyramation wants to merge 118 commits into
pyramation wants to merge 118 commits into
Conversation
Sliced from constructive-db's monolithic constructive module using: pnpm run slice:constructive -- --renumber-alterations --strip-cross-package-deps Package includes: - constructive_infra_public schema, types, and all tables - constructive_infra_private schema and trigger functions - 254 changes, 764 files Standalone: requires only plpgsql, pgpm-inflection, pgpm-stamps No RLS policies, no grants, no job triggers
…mignore) The slicer now generates a complete pgpm module with: - package.json with @pgpm/* dependencies mapped from .control requires - Makefile with PGXS include - .npmignore excluding test files This fixes 'No package.json found' error when running pgpm install.
…ema stripping - Add missing deploy dependencies (type, constraint→column, FK→pkey) - Plan is now topologically sorted for correct deploy order - Remove metaschema partition registrations (INSERT INTO metaschema_public.*) that require the closed-source MetaSchema system - Successfully tested: pgpm deploy --yes --createdb --database testinfra
…tion Separate pgpm module that registers the constructive-infra function tables as a MetaSchema function_module. This allows querying MetaSchema to discover the infra function tables (definitions, invocations, execution_logs, secret_definitions). Dependencies: - constructive-infra (the raw DDL) - @pgpm/metaschema-modules (function_module table) - @pgpm/services (service registration layer) The registration INSERT provides explicit schema/table names for the platform-scoped infra tables, letting the MetaSchema trigger system handle ID resolution and API routing.
Adds a CI workflow that runs 'pgpm test-packages --full-cycle' on every PR that touches pgpm/ files. This validates the full deploy/verify/revert/deploy cycle for constructive-infra against a real postgres-plus:18 instance. constructive-infra-services is excluded since it requires the full MetaSchema system (closed-source triggers) to deploy. Also fixes the function_requirement type revert SQL which was missing the TYPE keyword (DROP → DROP TYPE).
- job/compute-worker: platform-aware worker that discovers functions from constructive_infra_public.platform_function_definitions (TTL-cached), tracks invocations in platform_function_invocations, dispatches via HTTP - job/compute-service: orchestrator (callback server + ComputeWorker + Scheduler), mirrors job/service patterns - scripts/setup-platform-db.sh: Tier 1 setup (pgpm deploy + seed) - scripts/dev-compute.ts: dev launcher for compute-service + functions - scripts/seed-functions.sql: registers send-email + send-verification-link - docker-compose.yml: adds platform-setup service (Tier 2) - k8s/overlays/local-simple/compute-service.yaml: K8s manifest (Tier 3) - Makefile: adds setup-platform, dev-compute targets + tier documentation - .agents/skills/dev-tiers: skill for the 3-tier dev model - .agents/skills/compute-worker: skill for the compute-worker system - pgpm/constructive-infra: adds @pgpm/database-jobs dependency
…error) pgpm deploy requires module dependencies to be pre-installed via 'pgpm install' before deployment. Without this, packages like @pgpm/database-jobs fail with 'extension not available'. Adds pgpm install to: - scripts/setup-platform-db.sh (Tier 1) - docker-compose.yml platform-setup service (Tier 2) - .github/workflows/pgpm-test.yaml (CI)
- make status: shows Docker containers, PG connection, databases with infra schema, Node/pnpm/pgpm versions, and build state - make verify-platform: checks DB exists, infra + jobs schemas deployed, tables present, functions seeded — exits non-zero with fix instructions if anything is wrong
Dan committed all extension deps (database-jobs, metaschema-modules, services, etc.) directly into extensions/. pgpm install is no longer needed before deploy. Also restores pgpm volume to :ro in Docker Compose and mounts extensions/ for the platform-setup service.
Built-in function definitions (send-email, send-verification-link) are now deployed as a pgpm fixture inside constructive-infra. The fixture uses the standard deploy/revert/verify pattern following the inflection module's fixtures convention. - deploy: INSERT ON CONFLICT DO NOTHING - revert: DELETE WHERE is_built_in AND scope='platform' - verify: SELECT 1 from seed row Removes manual psql seed step from setup script, docker-compose, and verify-platform fix instructions. pgpm deploy now handles everything.
Moves the built-in function seed data (send-email, send-verification-link) from constructive-infra into its own pgpm package: constructive-infra-seed. This keeps constructive-infra as pure DDL (schemas, tables, triggers) and the seed data as a separate deployable unit. The seed package depends on constructive-infra via the .control file requires. Both packages pass pgpm test-packages --full-cycle.
compute-service discovers functions from the database, so it doesn't need direct workspace deps on send-email-fn or send-verification-link-fn (which are generated packages in generated/ and not in the workspace).
…nfigs - New fixture: seed_built_in_secrets seeds MAILGUN_* and SMTP_* into platform_secret_definitions (with well-known default database_id) - Function definitions now include required_secrets and required_configs arrays (function_requirement[] type) linking functions to their deps - All SQL files now use proper pgpm format: BEGIN/COMMIT wrappers, '-- Deploy:' header style with '-- made with <3 @ constructive.io' - New script: load-platform-env.sh reads .env, cross-references against DB function requirements, reports satisfied vs missing keys - .env.example updated with SMTP/Mailpit + dry-run toggle sections - Makefile: added 'make check-env' target
Procedural lifecycle targets: make up # prereqs → docker → bootstrap → deploy → seed → verify make up DB_NAME=mydb # same with custom DB make down # stop docker compose + pgpm docker stop DROP=1 make down DB_NAME=mydb # also drop the DB make up:email-job # start mailpit + compute-service (SMTP mode) make down:email-job # stop mailpit + compute-service email-job-up verifies platform is up first, starts mailpit, loads .env (with sane SMTP defaults), then launches compute-service. status now shows mailpit container state.
…rrides - Skip python-example and other non-node-graphql functions from startup - Print a clear port/service summary table before launching processes - Respect SEND_EMAIL_DRY_RUN, SEND_VERIFICATION_LINK_DRY_RUN, EMAIL_SEND_USE_SMTP, and SMTP_FROM from environment (was hardcoded)
- www/: Vite + React + Express app with 6 tabs (Functions, Secrets, Jobs, Invocations, Commands, Terminal) - Express backend: REST API for DB queries + WebSocket terminal via child_process - Secrets & Namespaces tab: shows seeded secret definitions and default namespace - Seed: add default platform namespace, link functions to namespace_id - Function API: parse composite-type arrays into proper JSON - Makefile: add make up:www target - scripts/www-up.sh: checks platform, installs deps, starts Vite + Express
…ster management - Install commander and kubernetesjs packages in www/ - Add K8s proxy endpoint (/api/k8s/*) in Express server - Add K8s tab with Pods/Deployments/Services views per namespace - Namespace selector defaults to constructive-functions - Graceful error when kubectl proxy not running
… tab - ansiToHtml() converts ANSI escape sequences into colored <span> elements - Remove max-h-60 so output expands naturally with content - Exit code shown in a separate footer bar - Strip remaining escape sequences that don't match known codes
Step 4b cross-references loaded env vars against platform_function_definitions required_secrets/required_configs. Warns about missing secrets but doesn't block (defaults still work for Mailpit/SMTP mode). Suggests 'make check-env' for details.
- Reads .env file and merges with process.env + dev defaults - Queries platform_function_definitions for required_secrets/required_configs - Reports per-function coverage at startup (✓ all set / ● N missing) - Priority: .env > process.env > hardcoded defaults - Graceful fallback if DB query fails (schema not deployed)
Both tables are PARTITION BY RANGE(created_at) but had no partitions, causing 'no partition of relation found for row' on INSERT. Adds DEFAULT partition for each so rows land somewhere until time-based partitions are created.
Secrets tab now shows: - Function coverage badges (e.g. send-email 5/8) - All required secrets/configs from DB + .env merged - Editable input fields with eye/reveal toggle for sensitive keys - 'Save to .env' button writes grouped .env file to project root - Shows which functions require each secret - File status indicator (.env exists / will be created) Backend adds: - GET /api/env — reads .env and returns parsed vars - POST /api/env — merges values + writes grouped .env file
Each function card now has a 'Trigger' button that opens an inline payload editor with sensible defaults (send-email gets to/subject/html, send-verification-link gets to/type/link). Submitting creates a job in app_jobs.jobs and shows success/error inline — no need to switch tabs.
Replace declare -A (bash 4+) with newline-separated string + grep. Replace read -ra with portable IFS-based for loops. Replace ((...)) arithmetic with $((...)) POSIX form. macOS ships bash 3.2 — these scripts now work on both.
Jobs created from the UI or without a database_id context now fall back to 00000000-0000-0000-0000-000000000000 (the well-known default used by the seed). Fixes NOT NULL violation on platform_function_invocations.
…ents - Add .agents/skills/fbp/SKILL.md mapping FBP NodeDefinitions to platform functions - Add docs/spec/fbp-integration.md research doc with full mapping spec - Add Flows tab with React Flow: drag-and-drop function nodes, edge creation, localStorage persistence, sidebar palette, and minimap - Enhance FunctionsPanel trigger success message with Invocations tab link - Install @xyflow/react for the flow graph canvas
…tion - Add platform_secret_values table (pgpm migration: deploy/revert/verify) Columns: id, secret_name, configured_value, database_id, created_at, updated_at Unique constraint on (secret_name, database_id) - Backend API (www/server/index.ts): GET /api/secret-values — read configured values from DB POST /api/secret-values — write configured values to DB POST /api/secrets/sync-from-db — DB values -> .env POST /api/secrets/sync-to-db — .env values -> DB POST /api/env now also syncs to DB on save (best-effort) - Frontend (SecretsPanel.tsx): Add 'Sync from DB' and 'Sync to DB' buttons DB status indicator in header - Refactor scripts/dev-compute.ts: Secrets pipeline: .env > DB > hardcoded defaults Per-function env injection (only needed secrets/configs) Fail fast on missing required secrets, warn on optional - Add make secrets:sync (bidirectional sync script) - Add scripts/secrets-sync.sh for CLI-based sync
feat: align API routing with constructive-db provision_base_modules
feat: add make kill (down + pgpm kill)
Adds 4 SDK packages matching constructive-db's structure: - constructive-functions-schema: exports .graphql from live endpoints (api, compute, objects) via introspection - constructive-functions-sdk: generates typed ORM client from schema files - constructive-functions-cli: generates unified CLI from schema files - constructive-functions-hooks: generates React Query hooks for www/ Includes: - Root-level generate scripts (generate:schemas, generate:sdk, etc.) - Makefile targets (make generate:schemas, make generate:sdk-all) - CI workflow (.github/workflows/generate-sdk.yaml) for automated regen - Auto-generated .agents/skills/ for ORM, CLI, and hooks reference - All generated output from current schema (3 APIs, 25 tables total)
feat: SDK generation pipeline (schemas, ORM, CLI, hooks)
- Add React Query provider + QueryClientProvider to main.tsx
- Add Vite proxy for /graphql/{compute,api,objects} endpoints
- Add graphql.ts client configuration (compute, api, objects)
- Convert FunctionsPanel: compute.usePlatformFunctionDefinitionsQuery
- Convert InvocationsPanel: compute.usePlatformFunctionInvocationsQuery (3s polling)
- Convert SecretsPanel: compute.usePlatformSecretDefinitionsQuery + api.usePlatformNamespacesQuery
- Convert FlowsPanel: compute.usePlatformFunctionDefinitionsQuery for node palette
- Convert JobsPanel NewJobForm: compute.usePlatformFunctionDefinitionsQuery for function dropdown
REST endpoints kept for: StatusBar (aggregate data), job list (app_jobs), env sync, K8s, Terminal
Resolves import failure by adding explicit resolve.alias pointing to sdk/constructive-functions-hooks/src, matching the @fbp/* alias pattern.
- Add --apply mode to register-functions.ts (pipes generated SQL to psql) - Wire into up.sh step 6 so functions are always registered from handler.json - Add make register:apply target - Bump @constructive-io/graphql-codegen to 4.47.7 (FunctionRequirementInput fix) - Regenerate all SDK packages (cli, sdk, hooks) - Remove stale constructive_infra_public seed files (moved to compute_public) - Fix SEED_PATH and SQL header comments to use correct schema path
The platform_function_definitions table doesn't have a payload_schema column. register-functions.ts was generating SQL referencing it, causing pgpm deploy to fail with 'column payload_schema does not exist'. Tested: make kill && make up — all 10 steps pass, 4 functions registered.
… grants, use npm graphql-server - Remove requiredSecrets/requiredConfigs from all panel field selections (composite types not yet supported by ORM codegen) - SecretsPanel: use REST fallback for function requirement data - Add pgpm grants for standalone dev: schema USAGE + table access for authenticated role - Set anon_role=authenticated for dev mode (no JWT auth) - Switch graphql-server from sibling repo path to npm package (@constructive-io/graphql-server) - Add lsof fallback for port check in start-graphql-server.sh
… mismatch - Add seed_invocation_tables migration to register platform_function_invocations, platform_function_execution_logs, and org equivalents in metaschema catalog - Add register_invocation_module migration for both platform and org scopes - Remove function_id column from InvocationTracker INSERT (column doesn't exist) - Remove entity_id/owner_id from CreateInvocationInput (column doesn't exist) - Tested: job dispatch creates invocation record, completes with duration_ms
…sistence - compute-worker: set jwt.claims.* GUCs before invocation dispatch - compute-worker: add X-Organization-Id header to HTTP requests - compute-worker: add BillingTracker (checkQuota + recordUsage) - www/FlowsPanel: persist flows to platform_function_graph_* tables (store → object → commit → ref) instead of localStorage - grants: add graph private schema access for trigger functions - docs: add evolution plan roadmap
- Add platform_compute_log table (partitioned by completed_at) - Add platform_usage_daily rollup table with unique upsert index - Add rollup_compute_daily() function in compute_private schema - Register compute_log_module in metaschema for dynamic discovery - Create ComputeLogTracker class with graceful no-op when not provisioned - Wire into ComputeWorker.doWork() for both success and failure paths - Extend ComputeModuleLoader to query compute_log_module - Add Usage panel to www dashboard (daily rollup + raw log views) - Regenerate SDK with PlatformComputeLog and PlatformUsageDaily hooks Refs: constructive-planning#1051
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
All alerts resolved. Learn more about Socket for GitHub. This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored. |
Schema verification (31 tests): - All expected schemas, tables, functions, indexes - Metaschema module registrations (function, invocation, compute_log) - Partition strategy + default partition - API routing configuration - Function definitions seeded Compute usage pipeline (requires make dev-compute): - Job dispatch → compute-worker pickup - Invocation tracking verification - Compute log entry verification - usage_daily rollup via rollup_compute_daily() - GraphQL exposure of both tables Usage: make test:workflow:schema # schema-only (needs make up) make test:workflow # full pipeline (needs make up + make dev-compute)
…p to constructive-store - eslint: add globals.node to .js/.mjs/.cjs files (fixes 'module is not defined' in jest.config.js) - pgpm: constructive-store.control requires constructive-infra (platform_config_get depends on platform_namespaces)
information_schema.role_table_grants stores privilege_type as uppercase (DELETE, INSERT, UPDATE, SELECT). The verify_table_grant function does a case-sensitive comparison, so lowercase 'delete' fails. Fixed across constructive-store, constructive-objects, constructive-storage.
GRANT UPDATE (col1, col2) does not appear in information_schema.role_table_grants (only full-table grants do). Column-level grants appear in column_privileges. Fixed the verify SQL for platform_files UPDATE grant.
register_compute_log_module fixture depends on metaschema-modules:compute_log_module/table
standalone_schema_access grants reference constructive_users_public
standalone_anon_authenticated updates services_public.apis rows that are seeded by constructive-platform-seed
pgpm test-packages --full-cycle requires revert scripts for all plan entries. Generated reverts for platform_compute_log (23 files), platform_usage_daily (25 files), and rollup_compute_daily function.
Index names in revert must match what CREATE INDEX produced: - idx_task_completed → platform_compute_log_task_completed_idx - idx_db_completed → platform_compute_log_db_completed_idx - idx_actor_completed → platform_compute_log_actor_completed_idx - idx_unique_entity_task_date → platform_usage_daily_entity_task_date_idx - pk_completed_at_id → platform_compute_log_pkey - pk_id → platform_usage_daily_pkey
|
Superseded by #95 (combined into single PR targeting main) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 1 of billing/metering (constructive-planning#1051): every compute-worker dispatch writes to
platform_compute_log;rollup_compute_daily(since)aggregates intoplatform_usage_daily(total_calls, successful, failed, duration stats per entity/task/day). Both tables registered incompute_log_modulemetaschema.Key design decisions:
platform_compute_logis partitioned bycompleted_at(monthly) — same pattern as invocations tableComputeLogTrackerruns alongsideInvocationTrackerindoWork(), writes after dispatch completesINSERT ... ON CONFLICT DO UPDATEto be idempotent — can re-run safelyCI fixes included:
constructive-store→constructive-infra,constructive-platform-seed→metaschema-modules,constructive-infra-seed→constructive-users+constructive-platform-seedverify_table_grant, column-level UPDATE usescolumn_privilegesinstead ofrole_table_grants.jsfiles,no-empty-object-typeoff for generated SDKTesting:
make kill && make up && make dev-compute→ jobs write toplatform_compute_log.pnpm test:workflow:schemaruns 31 schema verification tests against live stack.Link to Devin session: https://app.devin.ai/sessions/b2291a8e333e445aa125a2efd1996206
Requested by: @pyramation