Skip to content

Commit 0a44f4c

Browse files
committed
docs: fix CLAUDE.md casing, refresh root, add per-package guides
- Rename claude.md -> CLAUDE.md. Git tracked it lowercase, which works on macOS (case-insensitive filesystem) but means agents on Linux/CI never load it — they look for the uppercase name. The instructions were invisible on Linux. - Refresh the root file against the current codebase: Node 24 (was 18+), fix the typesync script name and the linker path, add `pnpm format`, split env vars by CLI vs worker, and add a short "Working with Projects" section covering the project subcommand. - Correct a misleading testing note. It claimed all tests run against dist/. In fact each package tests against its own src/ via @swc-node/register; only engine-multi (which runs its workers from dist/) and cross-package deps need a build first. - Add per-package CLAUDE.md for runtime, engine-multi and ws-worker — the packages with the least obvious invariants (VM sandboxing, the multi-process serialization boundary, strict event ordering). Kept short and focused on gotchas, not a rehash of the READMEs. These get loaded only when the agent is working with files in that package, keeping the context window leaner. - Remove .claude/command-refactor.md: looked like a plan for a refactor that already shipped, had drifted. - Trim .claude/event-processor.md to behavioural description, dropping variable-level detail that goes stale. Still accurate; stays as a doc companion to the worker's CLAUDE.md.
1 parent 89a41ed commit 0a44f4c

6 files changed

Lines changed: 111 additions & 266 deletions

File tree

.claude/command-refactor.md

Lines changed: 0 additions & 233 deletions
This file was deleted.

.claude/event-processor.md

Lines changed: 14 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -106,25 +106,14 @@ Bugs in one handler don't cascade to other events, but partial state may be visi
106106

107107
## Integration Points
108108

109-
### Initialization (execute.ts)
110-
111-
```
112-
eventProcessor(engine, context, {
113-
[WORKFLOW_START]: handleRunStart,
114-
[JOB_START]: handleStepStart,
115-
[JOB_COMPLETE]: handleStepComplete,
116-
[JOB_ERROR]: onJobError,
117-
[WORKFLOW_LOG]: handleRunLog,
118-
[WORKFLOW_COMPLETE]: handleRunComplete,
119-
[WORKFLOW_ERROR]: handleRunError,
120-
}, {
121-
batch: options.batchLogs ? { [WORKFLOW_LOG]: true } : {},
122-
batchInterval: options.batchInterval,
123-
batchLimit: options.batchLimit,
124-
})
125-
```
126-
127-
Processor set up before workflow starts, ensuring no events missed.
109+
### Initialization
110+
111+
The processor is created before the workflow starts (so no events are missed), and is given:
112+
113+
- a map of engine event name → handler (workflow start/complete/error, job/step start/complete/error, log)
114+
- batching options: which event types batch (currently only logs, and only when `batchLogs` is enabled), a batch size limit and interval, and a per-event fallback timeout
115+
116+
The exact option shape changes over time — read `src/api/execute.ts` for the current call, and `src/api/process-events.ts` for the loop itself.
128117

129118
### Websocket Layer (send-event.ts)
130119

@@ -172,12 +161,10 @@ Processor set up before workflow starts, ensuring no events missed.
172161

173162
5. **No explicit teardown**: Relies on engine event emitter lifecycle; queue drains naturally when workflow completes
174163

175-
## Critical Implementation Details
164+
## Where to Look
165+
166+
Core loop: `src/api/process-events.ts`. Websocket send wrapper: `src/util/send-event.ts`. Handlers: `src/events/`.
176167

177-
- Active batch tracked with `activeBatch` variable (event name or null)
178-
- Batch events stored in `batch` array, cleared after send
179-
- Batch timeout stored in `batchTimeout`, cleared when batch sends
180-
- Queue implemented as array, items are `{name, event}` objects
181-
- Processing triggered by `enqueue` when queue length becomes 1
182-
- Recursive `next()` call after queue shift creates continuous flow
183-
- Event handlers imported from events directory, mapped explicitly in execute function
168+
- The queue is an array of `{name, event}`; processing kicks off when the first item lands and drains one item at a time.
169+
- An open batch accumulates matching events (peek-ahead) and flushes on size limit, interval timeout, or arrival of a non-matching event.
170+
- Exact variable names and default values shift over time — trust the behavioural rules above and read the source for specifics.

claude.md renamed to CLAUDE.md

Lines changed: 28 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -37,11 +37,27 @@ The repository has three main packages: **CLI**, **Runtime**, and **Worker**. Th
3737

3838
The **Compiler** transforms job DSL code into standard ES modules with imports and operation arrays.
3939

40+
## Working with Projects
41+
42+
A **workspace** holds one or more local **projects** that sync with Lightning. The CLI `project` subcommand (package: `@openfn/project`) is the most actively-developed area of the repo; handlers live in `packages/cli/src/projects/`. Sync uses Lightning's v2 endpoint.
43+
44+
```bash
45+
openfn project list # List projects in the current workspace
46+
openfn project pull [project] # Pull from Lightning to the filesystem (fetch + checkout)
47+
openfn project fetch [project] # Download project state without expanding it
48+
openfn project checkout <project> # Switch the active project in the workspace
49+
openfn project deploy [project] # Deploy the checked-out project to Lightning
50+
openfn project clean # Delete the workflows folder and re-checkout the active project
51+
openfn project version [workflow] # Show the version hash of a workflow
52+
```
53+
54+
New subcommands follow the existing pattern in `packages/cli/src/projects/` (`list.ts` is the simplest reference).
55+
4056
## Development Setup
4157

4258
### Prerequisites
4359

44-
- Node.js 18+ (use `asdf`)
60+
- Node.js 24 (pinned in `.tool-versions`; `package.json` engines still allows `>=18`). Use `asdf`.
4561
- pnpm (enable with `corepack enable`)
4662

4763
### Common Commands
@@ -51,6 +67,7 @@ The **Compiler** transforms job DSL code into standard ES modules with imports a
5167
pnpm install # Install dependencies
5268
pnpm build # Build all packages
5369
pnpm test # Run all tests
70+
pnpm format # Format all packages with Prettier
5471
pnpm changeset # Add a changeset for your PR
5572

5673
# CLI
@@ -68,18 +85,22 @@ curl -X POST http://localhost:2222/claim # Manual claim
6885

6986
### Environment Variables
7087

88+
**CLI:**
7189
- `OPENFN_REPO_DIR` - CLI adaptor storage
7290
- `OPENFN_ADAPTORS_REPO` - Local adaptors monorepo path
7391
- `OPENFN_API_KEY` - API key for Lightning deployment
7492
- `OPENFN_ENDPOINT` - Lightning URL (default: app.openfn.org)
93+
94+
**Worker:**
7595
- `WORKER_SECRET` - Worker authentication secret
96+
- Many other `WORKER_*` vars (capacity, port, backoff, max payload, Sentry DSN, …) — see `packages/ws-worker/src/util/cli.ts`.
7697

7798
## Repository Structure
7899

79100
```
80101
packages/
81102
├── cli/ # CLI entry: cli.ts, commands.ts, projects/, options.ts
82-
├── runtime/ # Runtime entry: index.ts, runtime.ts, util/linker
103+
├── runtime/ # Runtime entry: index.ts, runtime.ts, modules/linker
83104
├── ws-worker/ # Worker entry: start.ts, server.ts, api/, events/
84105
├── compiler/ # Job DSL compiler
85106
├── engine-multi/ # Multi-process wrapper
@@ -109,15 +130,16 @@ cd packages/cli && pnpm test:watch # Watch mode
109130

110131
The [.claude](.claude) folder contains detailed guides:
111132

112-
- **[command-refactor.md](.claude/command-refactor.md)** - Refactoring CLI commands into project subcommand structure
113-
- **[event-processor.md](.claude/event-processor.md)** - Worker event processing architecture (batching, ordering)
133+
- **[event-processor.md](.claude/event-processor.md)** - Worker event processing deep-dive (ordering, batching) — companion to `packages/ws-worker/CLAUDE.md`
134+
135+
Key packages also carry their own `CLAUDE.md` (runtime, engine-multi, ws-worker), auto-loaded when you work in them.
114136

115137
## Code Standards
116138

117139
- **Formatting**: Use Prettier (`pnpm format`)
118140
- **TypeScript**: Required for all new code
119-
- **TypeSync**: Run `pnpm typesync` after modifying dependencies
120-
- **Tests**: Write tests and run `pnpm build` before testing (tests run against `dist/`)
141+
- **TypeSync**: Run `pnpm _typesync` after modifying dependencies
142+
- **Tests**: Each package's tests run against its own `src/` via `@swc-node/register` (no build of the package under test needed). BUT workspace `@openfn/*` deps resolve to their built `dist/`, and `engine-multi` runs its worker processes from `dist/` — so run `pnpm build` first when tests cross packages or use the engine. When unsure, build.
121143
- **Independence**: Keep packages loosely coupled where possible
122144

123145
## Architecture Principles

packages/engine-multi/CLAUDE.md

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Engine (@openfn/engine-multi)
2+
Wraps @openfn/runtime to execute each workflow in an isolated worker_thread nested inside a pooled child_process, so concurrent runs can't corrupt or starve each other.
3+
4+
## Invariants
5+
- Three-tier process tree: main process (engine) -> forked child_process (pool member) -> worker_thread (where @openfn/runtime actually runs). Each child hosts exactly one thread, spawned per task and terminated on completion.
6+
- Everything crossing main<->child<->thread goes over Node IPC / postMessage and is structured-cloned. Only serializable data survives: no functions, closures, class instances, or live handles. This is why resolvers/credentials are preloaded on the main thread and the fully-resolved plan is sent in — you cannot call back into the parent mid-run.
7+
- Errors must be plain serializable objects before crossing the boundary (use the error serializer); reconstruct Error instances on the parent side. Don't pass Error objects expecting stack/prototype to survive.
8+
- All messages over the boundary carry a `type` discriminator; the pool routes on the task-resolve / task-reject / run-task message types. New cross-boundary messages must follow this tagged shape.
9+
- The runtime's own timeout is disabled; the pool owns timeouts by killing the child. The runtime's module whitelist is the security boundary for what a job may import — regexes are stringified to cross the boundary and rehydrated in the thread.
10+
- Outgoing thread payloads are size-checked and oversized fields (state, logs) are redacted, not truncated transparently. Don't assume an emitted event carries the full state.
11+
12+
## Gotchas
13+
- Run the engine via the async factory and await it — it validates a live worker (handshake) before accepting work; a bad/missing worker file fails here, not at execute time.
14+
- Autoinstall and credential preloading happen on the MAIN thread before the task is dispatched; compilation happens INSIDE the thread. Don't move install/network work into the thread or expect lazy credential loading.
15+
- A clean worker exit uses a sentinel exit code; any other exit code is treated as a crash. OOM is detected by scraping the child's stderr for the V8 heap message — fragile, and distinct from the timeout path (which kills the child proactively).
16+
- Pool has fixed capacity; tasks beyond capacity queue rather than spawning unbounded children. Children are reused (returned to pool) unless killed by crash/OOM/timeout, so don't rely on fresh global state per run at the child level — isolation comes from the per-run thread, not the child.
17+
- Consumers get a per-workflow emitter; events are proxied main<-child<-thread. `listen()` can be called before `execute()` (handlers are deferred), and execute is dispatched on a tick so listeners can attach first. ws-worker's event processor consumes these.
18+
- `console.log` inside the thread is hijacked into the adaptor logger and won't reach stdout; use `console.debug` for raw stdout debugging.
19+
20+
## Testing
21+
- Tests run against `dist/` (worker entrypoints are resolved as built files, e.g. `dist/worker/...`, `dist/test/...`). Run `pnpm build` before `pnpm test` or paths won't resolve.
22+
- ava runs `--serial` (process/pool tests are stateful). Mock workers/runners exist as separate tsup entries (mock-run, worker-functions) — pass a custom worker path into the engine/pool to use them instead of the real runtime.

0 commit comments

Comments
 (0)