Skip to content

Commit 3e6292b

Browse files
Merge pull request #1289 from jitsucom/dev-productivity-improvements
Dev productivity improvements
2 parents 33cb29c + d61cee7 commit 3e6292b

24 files changed

Lines changed: 1377 additions & 2230 deletions

.npmrc

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,15 @@
11
auto-install-peers=true
22
strict-peer-dependencies=false
3+
4+
# Layered .env.local loading. pnpm exports node-options as NODE_OPTIONS for
5+
# every node process it spawns from a script. The preload module loads:
6+
# 1. ~/.jitsu/.env.local (shared across worktrees)
7+
# 2. <repo-root>/.env.local (per-worktree)
8+
# without overwriting existing process.env. Repo root is found by walking up
9+
# from process.cwd() to the nearest pnpm-workspace.yaml.
10+
#
11+
# Why a preload, not --env-file-if-exists directly: Node disallows --env-file*
12+
# in NODE_OPTIONS for security; --require is allowed.
13+
#
14+
# See CONTRIBUTING.md "Environment variables" for the full story.
15+
node-options=--require=env-preload

AGENTS.md

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,12 @@ Data ingestion engine for streaming events to warehouses.
5656
# Install JS dependencies
5757
pnpm install
5858

59+
# Generate Prisma client + zod schemas (required once after a fresh checkout
60+
# or worktree). Skipping this leaves Turbopack panicking in
61+
# ModuleGraphImportTracer::get_traces because it can't render the missing-
62+
# module error for `prisma/schema`.
63+
pnpm codegen
64+
5965
# Build all JS packages
6066
pnpm build:turbo
6167

@@ -76,6 +82,51 @@ pnpm dev
7682
pnpm console:dev
7783
```
7884

85+
## Running the app for the user
86+
87+
If the user asks you to run the app (console / ee-api / dev stack), use:
88+
89+
- `pnpm console:dev` — only console
90+
- `pnpm ee-api:dev` — only ee-api
91+
- `pnpm ui:dev` — both, in parallel (turbo)
92+
93+
These go through [portless](https://portless.sh) and serve the apps at
94+
`https://console.jitsu.localhost` and `https://ee.jitsu.localhost`.
95+
96+
**Branch hosting.** The dev wrapper auto-detects the current git branch and
97+
suffixes the dev host with it: `https://console-$BRANCH.jitsu.localhost` /
98+
`https://ee-$BRANCH.jitsu.localhost`. This avoids cookie / port collisions with
99+
whatever the user has running from another branch.
100+
101+
- The repo's default branch (resolved via `git rev-parse origin/HEAD`) gets no
102+
suffix.
103+
- The branch name is sanitized for DNS (lowercased, non-`[a-z0-9-]``-`,
104+
collapsed, capped at 30 chars).
105+
- `pnpm console:dev --no-branch` disables the suffix (use the bare
106+
`console.jitsu.localhost` host).
107+
108+
If the user explicitly asks you not to use a branch suffix, pass `--no-branch`.
109+
110+
> Implementation note: `dev-scripts/src/bin/run-app.ts` loads root `.env` /
111+
> `.env.local`, computes the slug, and runs portless from a non-git scratch dir
112+
> with `--name <slug>` and a `bash -c "cd <ws> && <cmd>"` wrapper — sidesteps
113+
> portless's hardcoded dot-style worktree prefix.
114+
115+
`portless` is a workspace devDependency — `pnpm install` is enough, no global
116+
install. First-run on a machine prompts once for `sudo` to bind port 443 and
117+
trust the local CA.
118+
119+
## Dev scripts
120+
121+
The `dev-scripts` package (`./dev-scripts`) hosts repo-wide developer tooling.
122+
Invoke via `pnpm dev <subcommand>`:
123+
124+
```bash
125+
pnpm dev # turbo run dev (start everything)
126+
pnpm dev copy-db --src URL --dst URL # rsync-style postgres copy ($ENV_VAR placeholders)
127+
pnpm dev help
128+
```
129+
79130
For Go (run inside `/bulker`):
80131

81132
```bash

CONTRIBUTING.md

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,39 @@ Run
2424
* `docker compose -f ./docker/docker-compose.yml up --profile jitsu-services-dev --force-recreate` - to run dependencies + all Jitsu services
2525
in a hot reload mode, see `docker/README.md`
2626

27+
# Environment variables
28+
29+
Every node process spawned by a pnpm script auto-loads two layered `.env.local` files
30+
(later wins; existing `process.env` always wins over both, missing files skipped silently):
31+
32+
1. `~/.jitsu/.env.local` — shared across all worktrees of all branches (Firebase, Stripe,
33+
OIDC, GitHub OAuth — anything that doesn't change per branch).
34+
2. `<repo>/.env.local` — per-worktree (`DATABASE_URL`, `NEXTAUTH_URL`, `AUTH_COOKIE_DOMAIN`,
35+
anything that should differ between two worktrees of two PRs).
36+
37+
**No wrapper.** `node --inspect script.js` and your debugger attach to the script's own
38+
process directly — there's no `dotenv-cli` parent in the tree.
39+
40+
`.env.example` documents the variables the apps expect. Runtime defaults belong in code
41+
(`process.env.FOO ?? "default"`), not in a tracked `.env`.
42+
43+
## How it works
44+
45+
The root [`.npmrc`](.npmrc) sets `node-options=--require=env-preload`,
46+
so pnpm exports `NODE_OPTIONS=--require=...` for every node process it spawns from a
47+
script. The preload ([`env-preload/preload-env.cjs`](env-preload/preload-env.cjs)) loads
48+
`~/.jitsu/.env.local`, then walks up from `process.cwd()` to find `pnpm-workspace.yaml`
49+
and load the repo-root `.env.local`. (Why a preload instead of `--env-file-if-exists`:
50+
Node disallows `--env-file*` in `NODE_OPTIONS` for security; `--require` is allowed.)
51+
52+
## Adding to the shared layer
53+
54+
```bash
55+
mkdir -p ~/.jitsu && chmod 700 ~/.jitsu
56+
touch ~/.jitsu/.env.local && chmod 600 ~/.jitsu/.env.local
57+
echo 'STRIPE_KEY=sk_live_xxx' >> ~/.jitsu/.env.local
58+
```
59+
2760
# Development Workflow
2861

2962
## Development Branch

dev-scripts/package.json

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{
2+
"name": "@jitsu-internal/dev-scripts",
3+
"version": "0.0.0",
4+
"private": true,
5+
"scripts": {
6+
"exec": "tsx src/index.ts",
7+
"typecheck": "tsc --noEmit"
8+
},
9+
"dependencies": {
10+
"juava": "workspace:*",
11+
"pg": "^8.18.0",
12+
"prompts": "^2.4.2"
13+
},
14+
"devDependencies": {
15+
"@jitsu/common-config": "workspace:*",
16+
"@types/node": "catalog:",
17+
"@types/pg": "^8.16.0",
18+
"@types/prompts": "^2.4.9",
19+
"tsx": "catalog:",
20+
"typescript": "catalog:"
21+
}
22+
}

dev-scripts/src/bin/run-app.ts

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
/**
2+
* Run a workspace dev command behind a portless https://{app}-{branch}.jitsu.localhost host.
3+
*
4+
* tsx run-app.ts <app> <cmd...>
5+
*
6+
* Example (from webapps/console):
7+
* tsx run-app.ts console next dev
8+
*
9+
* Responsibilities:
10+
* 1. Pick a slug — `<app>` plain on the repo's default branch, `<app>-<branch>`
11+
* otherwise. Default branch comes from `git rev-parse origin/HEAD`.
12+
* 2. Pass `--no-branch` to suppress the suffix.
13+
* 3. Spawn portless via the SHIM_DIR trick (see SHIM_DIR comment below).
14+
*
15+
* .env loading is handled by Node's `--env-file-if-exists` flag, set via
16+
* `node-options` in the root .npmrc — see CONTRIBUTING.md.
17+
*/
18+
import { spawn, spawnSync } from "node:child_process";
19+
import { mkdirSync } from "node:fs";
20+
import os from "node:os";
21+
import path from "node:path";
22+
import { fileURLToPath } from "node:url";
23+
24+
const __dirname = path.dirname(fileURLToPath(import.meta.url));
25+
26+
/**
27+
* Scratch dir we run portless from. Why:
28+
*
29+
* Portless inspects its own `cwd` with `git worktree list --porcelain`. When
30+
* cwd is inside a non-default git worktree, it prepends `<branch>.` to the
31+
* slug — DOT separator, hardcoded in three places in node_modules/portless/
32+
* dist/cli.js, no flag or env var to disable on the `run` / `<name> <cmd>`
33+
* code paths (`--no-worktree` exists only for `portless get`).
34+
*
35+
* That collides with our dash convention (`console-feat.jitsu.localhost`):
36+
* portless would turn it into `feat.console-feat.jitsu.localhost`. To keep
37+
* dash style we launch portless with cwd pointed at a path that is not in any
38+
* git repo, so both `detectWorktreeViaCli` (git command) and
39+
* `detectWorktreeViaFilesystem` (parent-dir .git walk) return null.
40+
*
41+
* The user's actual command still needs to run at the workspace cwd, so we
42+
* wrap it as `bash -c "cd <workspace> && <cmd>"`.
43+
*
44+
* Alternatives considered:
45+
* - Programmatic portless: the package's public API exposes RouteStore /
46+
* createProxyServer but not the ~200 LOC `runApp` / `ensureProxyRunning`
47+
* orchestration. Re-implementing means owning a parallel runner forever.
48+
* - Forking portless: too heavy for one-line behaviour.
49+
*/
50+
const SHIM_DIR = path.join(os.tmpdir(), "jitsu-portless-shim");
51+
52+
function gitOutput(args: string[]): string | null {
53+
const r = spawnSync("git", args, { encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] });
54+
if (r.status !== 0) return null;
55+
return (r.stdout ?? "").trim() || null;
56+
}
57+
58+
function defaultBranch(): string | null {
59+
const ref = gitOutput(["rev-parse", "--abbrev-ref", "origin/HEAD"]);
60+
return ref ? ref.replace(/^origin\//, "") : null;
61+
}
62+
63+
function currentBranch(): string | null {
64+
return gitOutput(["branch", "--show-current"]);
65+
}
66+
67+
function sanitizeBranch(name: string): string {
68+
return name
69+
.toLowerCase()
70+
.replace(/[^a-z0-9-]+/g, "-")
71+
.replace(/-+/g, "-")
72+
.replace(/^-+|-+$/g, "")
73+
.slice(0, 30)
74+
.replace(/-+$/, "");
75+
}
76+
77+
function shellQuote(s: string): string {
78+
return `'${s.replace(/'/g, "'\\''")}'`;
79+
}
80+
81+
function main(): void {
82+
const argv = process.argv.slice(2);
83+
const noBranch = argv.includes("--no-branch");
84+
const positional = argv.filter(a => a !== "--no-branch");
85+
const [appName, ...command] = positional;
86+
if (!appName || command.length === 0) {
87+
console.error("Usage: run-app <app> [--no-branch] <cmd...>");
88+
process.exit(2);
89+
}
90+
91+
let branch = "";
92+
let branchSource = "";
93+
if (!noBranch) {
94+
const current = currentBranch();
95+
if (current) {
96+
const def = defaultBranch();
97+
if (!def || current !== def) {
98+
const sanitized = sanitizeBranch(current);
99+
if (sanitized) {
100+
branch = sanitized;
101+
branchSource = def ? `git (default branch: ${def})` : "git";
102+
}
103+
}
104+
}
105+
}
106+
107+
const slug = (branch ? `${appName}-${branch}.jitsu` : `${appName}.jitsu`).toLowerCase();
108+
mkdirSync(SHIM_DIR, { recursive: true });
109+
110+
// Capture the actual workspace cwd before we redirect portless to SHIM_DIR;
111+
// the spawned bash will cd back here so `next dev` finds package.json etc.
112+
const workspace = process.cwd();
113+
const innerCmd = `cd ${shellQuote(workspace)} && exec ${command.map(shellQuote).join(" ")}`;
114+
115+
console.error(
116+
`[run-app] https://${slug}.localhost (branch=${branch || "<none>"}${
117+
branchSource ? ` from ${branchSource}` : noBranch ? " — --no-branch" : ""
118+
})`
119+
);
120+
121+
const child = spawn("portless", ["--name", slug, "bash", "-c", innerCmd], {
122+
cwd: SHIM_DIR,
123+
stdio: "inherit",
124+
});
125+
child.on("error", err => {
126+
if ((err as NodeJS.ErrnoException).code === "ENOENT") {
127+
console.error("[run-app] `portless` binary not on PATH. Run via pnpm so node_modules/.bin is on PATH.");
128+
process.exit(127);
129+
}
130+
throw err;
131+
});
132+
child.on("exit", (code, signal) => {
133+
if (signal) process.kill(process.pid, signal as NodeJS.Signals);
134+
else process.exit(code ?? 0);
135+
});
136+
}
137+
138+
main();

0 commit comments

Comments
 (0)