Skip to content

Commit 1c88750

Browse files
pyramationpyramation
authored
Merge pull request #57 from constructive-io/feat/fbp-flow-graph
Feat/fbp flow graph
2 parents 7784149 + c0b5054 commit 1c88750

617 files changed

Lines changed: 39052 additions & 163 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/compute-worker/SKILL.md‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
---
2+
name: compute-worker
3+
description: Platform-aware compute worker and service for constructive-functions. Discovers functions from the database, tracks invocations, dispatches via HTTP. Use when working on the compute-worker, compute-service, function discovery, or invocation tracking.
4+
---
5+
6+
# Compute Worker & Service
7+
8+
The compute-worker is a platform-aware replacement for the legacy knative-job-worker. Instead of discovering functions from a static manifest or env vars, it queries `constructive_infra_public.platform_function_definitions` and tracks every invocation in `platform_function_invocations`.
9+
10+
## Architecture
11+
12+
```
13+
┌─────────────────────────────────────────────────────────┐
14+
│ compute-service (orchestrator) │
15+
│ │
16+
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
17+
│ │ HTTP callback │ │ ComputeWorker│ │ Scheduler │ │
18+
│ │ server (:8080)│ │ (polls jobs) │ │ (cron jobs) │ │
19+
│ └──────────────┘ └──────┬───────┘ └──────────────┘ │
20+
│ │ │
21+
│ ┌──────────────┼──────────────┐ │
22+
│ │ │ │ │
23+
│ ▼ ▼ ▼ │
24+
│ FunctionDiscovery InvocationTracker compute_request │
25+
│ (TTL-cached DB (INSERT/UPDATE (HTTP POST to │
26+
│ lookups) invocations) function URL) │
27+
└─────────────────────────────────────────────────────────┘
28+
│ │ │
29+
▼ ▼ ▼
30+
platform_function platform_function Function HTTP
31+
_definitions _invocations endpoint
32+
(read) (write) (send-email:8081)
33+
```
34+
35+
## Packages
36+
37+
### job/compute-worker (`@constructive-io/compute-worker`)
38+
39+
Core worker class and supporting modules:
40+
41+
| File | Purpose |
42+
|------|---------|
43+
| `src/index.ts` | `ComputeWorker` class — lifecycle, job polling, dispatch |
44+
| `src/discovery.ts` | `FunctionDiscovery` — lazy TTL-cached DB lookups |
45+
| `src/invocation.ts` | `InvocationTracker` — create/complete/fail invocation records |
46+
| `src/req.ts` | `compute_request()` — HTTP POST dispatch with X-* headers |
47+
| `src/cache.ts` | `TtlCache<T>` — generic TTL cache |
48+
| `src/types.ts` | TypeScript interfaces |
49+
50+
### job/compute-service (`@constructive-io/compute-service`)
51+
52+
Orchestrator that starts the callback server, ComputeWorker, and Scheduler:
53+
54+
| File | Purpose |
55+
|------|---------|
56+
| `src/index.ts` | `ComputeService` class + `bootCompute()` entry point |
57+
| `src/run.ts` | CLI entry point (`node dist/run.js`) |
58+
| `src/registry.ts` | Function registry loader (for optional in-process function servers) |
59+
| `src/types.ts` | TypeScript interfaces |
60+
61+
## Key differences from legacy worker
62+
63+
| Feature | knative-job-worker | compute-worker |
64+
|---------|-------------------|----------------|
65+
| Function discovery | Static manifest / `JOBS_SUPPORTED` env | DB query (TTL-cached) |
66+
| Invocation tracking | None | `platform_function_invocations` table |
67+
| Task filtering | `JOBS_SUPPORTED` allowlist | Accepts any registered task |
68+
| URL resolution | Gateway URL + dev map | `service_url` from DB → dev map → gateway fallback |
69+
| Infra requirement | Only needs `app_jobs` schema | Needs `app_jobs` + `constructive_infra_public` |
70+
71+
## Function discovery flow
72+
73+
```
74+
Job arrives (task_identifier = "send-email")
75+
│
76+
▼
77+
FunctionDiscovery.resolve("send-email")
78+
│
79+
├─ Cache hit? → return cached definition
80+
│
81+
└─ Cache miss? → SQL query:
82+
SELECT * FROM constructive_infra_public.platform_function_definitions
83+
WHERE task_identifier = 'send-email'
84+
│
85+
└─ Cache result (TTL default: 60s)
86+
│
87+
▼
88+
PlatformFunctionDefinition {
89+
id, name, task_identifier, service_url,
90+
is_invocable, max_attempts, priority, ...
91+
}
92+
```
93+
94+
## Invocation lifecycle
95+
96+
```
97+
1. create() → INSERT INTO platform_function_invocations
98+
(status='running', started_at=now())
99+
→ returns invocation_id
100+
101+
2a. complete() → UPDATE SET status='completed',
102+
completed_at=now(), duration_ms=X
103+
104+
2b. fail() → UPDATE SET status='failed',
105+
completed_at=now(), duration_ms=X, error='...'
106+
```
107+
108+
## Running locally
109+
110+
```bash
111+
# Tier 1: pgpm-local
112+
pgpm docker start --image docker.io/constructiveio/postgres-plus:18
113+
eval "$(pgpm env)"
114+
make setup-platform # deploy infra + seed functions
115+
make dev-compute # start compute-service + functions
116+
117+
# Tier 2: compose-local
118+
make dev # docker compose up
119+
make dev-compute # start compute-service + functions
120+
```
121+
122+
## Testing a job manually
123+
124+
```bash
125+
# Insert a test job
126+
eval "$(pgpm env)"
127+
psql -d constructive-functions-db1 -c "
128+
INSERT INTO app_jobs.jobs (task_identifier, payload)
129+
VALUES ('send-email', '{\"to\":\"test@example.com\",\"subject\":\"test\",\"html\":\"<p>hi</p>\"}'::json)
130+
"
131+
132+
# Check invocation records
133+
psql -d constructive-functions-db1 -c "
134+
SELECT id, task_identifier, status, duration_ms, error
135+
FROM constructive_infra_public.platform_function_invocations
136+
ORDER BY started_at DESC LIMIT 5
137+
"
138+
```
139+
140+
## Environment variables
141+
142+
| Variable | Default | Description |
143+
|----------|---------|-------------|
144+
| `COMPUTE_JOBS_ENABLED` | `true` | Enable/disable the compute worker |
145+
| `COMPUTE_CALLBACK_URL` | — | URL functions POST to on completion |
146+
| `COMPUTE_GATEWAY_URL` | — | Fallback gateway URL for functions without `service_url` |
147+
| `JOBS_SCHEMA` | `app_jobs` | PostgreSQL schema for the jobs table |
148+
| `INTERNAL_JOBS_CALLBACK_PORT` | `8080` | Port for the HTTP callback server |
149+
| `INTERNAL_GATEWAY_DEVELOPMENT_MAP` | — | JSON map of task→URL for local dev |
150+
151+
## Database requirements
152+
153+
The compute-service checks two things at boot:
154+
1. `app_jobs.jobs` table exists (deployed by `@pgpm/database-jobs`, a dependency of `constructive-infra`)
155+
2. `constructive_infra_public.platform_function_definitions` table exists (deployed by `constructive-infra`)
156+
157+
Both are deployed together via `make setup-platform` or the `platform-setup` Docker Compose service.

‎.agents/skills/dev-tiers/SKILL.md‎

Lines changed: 171 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,171 @@
1+
---
2+
name: dev-tiers
3+
description: Three-tier local development environment for constructive-functions. Covers pgpm-local (Tier 1), compose-local (Tier 2), and k8s-local (Tier 3). Use when setting up the dev environment, starting services, or debugging infrastructure.
4+
---
5+
6+
# Development Tiers
7+
8+
constructive-functions supports three tiers of local development, each adding more infrastructure while keeping the same function code. Choose based on what you need.
9+
10+
## Tier 1 — pgpm-local
11+
12+
**What it is:** Postgres only via `pgpm docker`. Functions + services run as bare Node.js processes on the host. Fastest edit-run cycle.
13+
14+
**When to use:** Day-to-day function development, quick iteration, debugging a single function.
15+
16+
**Setup:**
17+
```bash
18+
# Start Postgres
19+
pgpm docker start --image docker.io/constructiveio/postgres-plus:18
20+
eval "$(pgpm env)"
21+
22+
# Deploy infra schema + seed function definitions
23+
make setup-platform
24+
25+
# Generate function packages (if not done)
26+
pnpm generate && pnpm install && pnpm build
27+
28+
# Start functions + compute-service (platform-aware)
29+
make dev-compute
30+
31+
# Or start functions + legacy job-service
32+
make dev-fn
33+
```
34+
35+
**What runs where:**
36+
| Component | Where |
37+
|-----------|-------|
38+
| PostgreSQL | Docker container (via pgpm) |
39+
| MinIO (optional) | Docker container (via pgpm) |
40+
| Functions (send-email, etc.) | Local Node.js process |
41+
| compute-service / job-service | Local Node.js process |
42+
| GraphQL server | Not running (unless started manually) |
43+
44+
**Database:** `constructive-functions-db1` (configurable via `DB_NAME` env var)
45+
46+
**Ports:**
47+
| Service | Port |
48+
|---------|------|
49+
| PostgreSQL | 5432 |
50+
| compute-service / job-service | 8080 |
51+
| send-email | 8081 |
52+
| send-verification-link | 8082 |
53+
54+
---
55+
56+
## Tier 2 — compose-local
57+
58+
**What it is:** Docker Compose runs infrastructure (Postgres, db-setup, GraphQL server, mailpit, platform-setup). Functions still run as local Node.js processes.
59+
60+
**When to use:** Testing with full infrastructure, working with email, needing GraphQL API, integration testing.
61+
62+
**Setup:**
63+
```bash
64+
# Create .env from example
65+
cp .env.example .env
66+
67+
# Start infrastructure
68+
make dev # docker compose up -d
69+
70+
# Wait for db-setup + platform-setup to complete
71+
docker compose logs -f db-setup platform-setup
72+
73+
# Start functions + compute-service
74+
make dev-compute
75+
76+
# Or start functions + legacy job-service
77+
make dev-fn
78+
```
79+
80+
**What runs where:**
81+
| Component | Where |
82+
|-----------|-------|
83+
| PostgreSQL | Docker container |
84+
| db-setup (deploys constructive, metaschema) | Docker container (runs once) |
85+
| platform-setup (deploys constructive-infra, seeds functions) | Docker container (runs once) |
86+
| GraphQL server | Docker container (port 3002) |
87+
| Mailpit (email testing) | Docker container (SMTP 1025, UI 8025) |
88+
| Functions | Local Node.js process |
89+
| compute-service / job-service | Local Node.js process |
90+
91+
**Database:** `constructive` (full constructive stack)
92+
93+
**Ports:**
94+
| Service | Port |
95+
|---------|------|
96+
| PostgreSQL | 5432 |
97+
| GraphQL server | 3002 |
98+
| Mailpit SMTP | 1025 |
99+
| Mailpit UI | 8025 |
100+
| compute-service / job-service | 8080 |
101+
| send-email | 8081 |
102+
| send-verification-link | 8082 |
103+
104+
---
105+
106+
## Tier 3 — k8s-local
107+
108+
**What it is:** Everything runs in a local Kubernetes cluster via Skaffold. Two sub-profiles: `local-simple` (plain Deployments) and `local` (Knative Serving).
109+
110+
**When to use:** Testing K8s manifests, verifying production-like behavior, testing Knative scaling, pre-deployment validation.
111+
112+
**Setup (local-simple — no Knative):**
113+
```bash
114+
# One-time: install K8s tooling
115+
make setup-dev
116+
117+
# Start everything
118+
make skaffold-dev
119+
```
120+
121+
**Setup (local — Knative):**
122+
```bash
123+
# One-time: install Knative operators
124+
cd k8s && make operators-knative-only
125+
126+
# Start everything
127+
make skaffold-dev-knative
128+
```
129+
130+
**What runs where:**
131+
| Component | Where |
132+
|-----------|-------|
133+
| Everything | Kubernetes pods |
134+
135+
**Key manifests (local-simple overlay):**
136+
- `k8s/overlays/local-simple/postgres-local.yaml` — PostgreSQL StatefulSet
137+
- `k8s/overlays/local-simple/constructive-db-job.yaml` — DB setup Job
138+
- `k8s/overlays/local-simple/job-service.yaml` — Legacy job service Deployment
139+
- `k8s/overlays/local-simple/compute-service.yaml` — Platform-aware compute service Deployment
140+
- `k8s/overlays/local-simple/constructive-server.yaml` — GraphQL server
141+
142+
---
143+
144+
## Switching between tiers
145+
146+
The tiers are independent. Tear down one before starting another to avoid port conflicts:
147+
148+
```bash
149+
# Stop Tier 2
150+
make dev-down
151+
152+
# Stop Tier 3
153+
# Ctrl+C in the skaffold terminal
154+
155+
# Stop Tier 1
156+
pgpm docker stop
157+
```
158+
159+
## Environment variables
160+
161+
Common env vars across all tiers:
162+
163+
| Variable | Tier 1 Default | Tier 2 Default | Description |
164+
|----------|---------------|---------------|-------------|
165+
| `PGHOST` | localhost | localhost | PostgreSQL host |
166+
| `PGPORT` | 5432 | 5432 | PostgreSQL port |
167+
| `PGUSER` | postgres | postgres | PostgreSQL user |
168+
| `PGPASSWORD` | (from pgpm env) | (from .env) | PostgreSQL password |
169+
| `PGDATABASE` | constructive-functions-db1 | constructive | Database name |
170+
| `JOBS_SCHEMA` | app_jobs | app_jobs | Jobs table schema |
171+
| `COMPUTE_JOBS_ENABLED` | true | true | Enable compute worker |

0 commit comments

Comments
 (0)