|
| 1 | +--- |
| 2 | +name: packages-worker-add-entrypoint |
| 3 | +description: > |
| 4 | + Scaffold a new sub-worker inside packages_worker (npm, deps.dev, osv, scorecard, |
| 5 | + etc.) following the single-service multi-entry-point structure. Use when: "add a |
| 6 | + new packages worker", "scaffold a sub-worker in packages_worker", "new worker for |
| 7 | + packages-db", "add npm worker", "add OSV worker", "add deps.dev worker". |
| 8 | +allowed-tools: Read, Write, Edit, Bash, AskUserQuestion, Glob |
| 9 | +--- |
| 10 | + |
| 11 | +# packages-worker — Add a New Sub-worker |
| 12 | + |
| 13 | +You are adding a new data-ingestion worker to `services/apps/packages_worker/`. |
| 14 | +The structure follows the same pattern as `backend/` (where `api.ts` and |
| 15 | +`job-generator.ts` share one Dockerfile): one npm package, one Docker image, |
| 16 | +each worker in its own `src/{worker}/` directory with its own entry point. |
| 17 | + |
| 18 | +``` |
| 19 | +services/apps/packages_worker/ |
| 20 | + src/ |
| 21 | + bin/ |
| 22 | + packages-worker.ts ← parent stub |
| 23 | + github-repos-enricher.ts ← existing worker |
| 24 | + <name>.ts ← entry point you will create |
| 25 | + github/ ← existing worker logic |
| 26 | + <worker>/ ← directory you will create |
| 27 | + index.ts ← main logic for this worker |
| 28 | + types.ts |
| 29 | + config.ts ← shared — add your config getter here |
| 30 | + db.ts ← shared — do not modify |
| 31 | +``` |
| 32 | + |
| 33 | +## Step 1 — Gather requirements |
| 34 | + |
| 35 | +Ask the engineer for: |
| 36 | + |
| 37 | +1. **Worker name** (kebab-case) — e.g. `npm-sync`, `osv-sync`, `scorecard-runner`. Used as the entry point filename (`src/bin/<name>.ts`) and docker-compose service name. |
| 38 | +2. **Worker directory name** (short, lowercase) — e.g. `npm`, `osv`, `scorecard`. Becomes `src/<worker>/`. |
| 39 | +3. **What it does** — what data it fetches/writes, what table(s) in packages-db it reads from and writes to. |
| 40 | +4. **External API or data source** (if any) — URL, auth method, rate-limit characteristics. |
| 41 | +5. **Required env vars** beyond the shared DB vars — e.g. `NPM_API_URL`, `OSV_API_KEY`. |
| 42 | + |
| 43 | +Do not proceed until you have answers to 1–3. |
| 44 | + |
| 45 | +## Step 2 — Read existing files first |
| 46 | + |
| 47 | +```bash |
| 48 | +cat services/apps/packages_worker/src/bin/github-repos-enricher.ts |
| 49 | +cat services/apps/packages_worker/src/config.ts |
| 50 | +cat services/apps/packages_worker/package.json |
| 51 | +cat scripts/services/github-repos-enricher.yaml |
| 52 | +``` |
| 53 | + |
| 54 | +These are the canonical references. Do not deviate from the patterns you see there. |
| 55 | + |
| 56 | +## Step 3 — Scaffold the files |
| 57 | + |
| 58 | +### 3a. Worker directory — `services/apps/packages_worker/src/<worker>/` |
| 59 | + |
| 60 | +Create the directory with at minimum: |
| 61 | + |
| 62 | +**`types.ts`** — types specific to this worker (input/output shapes, error kinds if calling an external API). |
| 63 | + |
| 64 | +**`index.ts`** — the main logic function(s) this worker runs. What goes here depends entirely on what the worker does — do not force a loop shape if it does not fit. Discuss with the engineer what the execution model should be (continuous loop, one-shot batch, event-driven, etc.) and implement accordingly. |
| 65 | + |
| 66 | +Add any additional files the worker needs (e.g. an API client, a DB query helper). All DB access uses inline pg-promise SQL via `qx.select` / `qx.result` / `qx.none` — do not add files to `services/libs/data-access-layer`. |
| 67 | + |
| 68 | +### 3b. Entry point — `services/apps/packages_worker/src/bin/<name>.ts` |
| 69 | + |
| 70 | +Follow the structure of `github-repos-enricher.ts`: |
| 71 | +- Import `getServiceLogger` from `@crowd/logging` |
| 72 | +- Import your worker's config getter from `../config` and `getPackagesDb` from `../db` |
| 73 | +- Import your worker's main function from `../<worker>/index` |
| 74 | +- Set `liveFilePath` / `readyFilePath` to `../tmp/<name>-live.tmp` / `../tmp/<name>-ready.tmp` |
| 75 | +- Handle SIGINT / SIGTERM with a `shuttingDown` flag |
| 76 | +- In `main()`: call config getter → validate any required tokens/keys → `await getPackagesDb()` → `await qx.selectOne('SELECT 1')` → `fs.mkdirSync` for the tmp dir → `setInterval` writing probe files every 5000ms → call your worker's main function → `clearInterval` → `process.exit(0)` |
| 77 | +- Fatal handler: `main().catch(err => { log.error({ err }, '<name> fatal error'); process.exit(1) })` |
| 78 | + |
| 79 | +### 3c. Config additions — `services/apps/packages_worker/src/config.ts` |
| 80 | + |
| 81 | +Read the file first, then add a `get<Worker>Config()` function: |
| 82 | +- Use `requireEnv(name)` for string vars, `requireEnvInt(name)` for integers |
| 83 | +- No defaults, no `?? undefined` — the process must refuse to start on missing config |
| 84 | + |
| 85 | +### 3d. Docker-compose service — `scripts/services/<name>.yaml` |
| 86 | + |
| 87 | +Copy `scripts/services/github-repos-enricher.yaml` and adapt: |
| 88 | +- Service names: `<name>` (prod) and `<name>-dev` (dev) |
| 89 | +- `command` (prod): `pnpm run start:<name>` |
| 90 | +- `command` (dev): `pnpm run dev:<name>` |
| 91 | +- `env_file`: keep the same four files (`backend/.env.dist.local`, `backend/.env.dist.composed`, `backend/.env.override.local`, `backend/.env.override.composed`) |
| 92 | +- `environment`: set any tuning var defaults inline (avoids requiring them in `.env.override.local` for local dev) |
| 93 | +- `volumes` (dev only): bind-mount `./services/apps/packages_worker/src` plus every `services/libs/*/src` directory (copy the full list from the enricher yaml for hot reload) |
| 94 | + |
| 95 | +### 3e. package.json scripts — `services/apps/packages_worker/package.json` |
| 96 | + |
| 97 | +Read the file first, then add: |
| 98 | +```json |
| 99 | +"start:<name>": "tsx src/bin/<name>.ts", |
| 100 | +"dev:<name>": "tsx watch src/bin/<name>.ts" |
| 101 | +``` |
| 102 | + |
| 103 | +### 3f. Env var files — `backend/.env.dist.local` and `backend/.env.dist.composed` |
| 104 | + |
| 105 | +Append new required vars with empty-string defaults (or sensible local values for non-secrets): |
| 106 | +``` |
| 107 | +NEW_WORKER_API_KEY= |
| 108 | +``` |
| 109 | + |
| 110 | +## Step 4 — TypeScript check |
| 111 | + |
| 112 | +```bash |
| 113 | +cd services/apps/packages_worker && pnpm tsc --noEmit |
| 114 | +``` |
| 115 | + |
| 116 | +Fix any errors before proceeding. |
| 117 | + |
| 118 | +## Checklist before committing |
| 119 | + |
| 120 | +- [ ] `src/<worker>/` directory created with `types.ts` and `index.ts` |
| 121 | +- [ ] `src/bin/<name>.ts` — probe files, SIGINT/SIGTERM handler, fail-fast config check, `SELECT 1` on startup |
| 122 | +- [ ] `config.ts` — new `get<Worker>Config()` using `requireEnv`/`requireEnvInt`, no defaults |
| 123 | +- [ ] `scripts/services/<name>.yaml` — prod + dev services with bind mounts |
| 124 | +- [ ] `package.json` — `start:<name>` and `dev:<name>` scripts added |
| 125 | +- [ ] `backend/.env.dist.local` and `.env.dist.composed` — new vars documented |
| 126 | +- [ ] No new files in `services/libs/data-access-layer` (packages-db uses inline SQL) |
| 127 | +- [ ] `pnpm tsc --noEmit` passes |
| 128 | + |
| 129 | +Use `/preflight` before opening a PR and `/commit` to sign off. |
0 commit comments