Commit 1baff25
authored
feat(umami-postgres): keploy compat lane sample (smoke-test only) (#96)
* feat(umami-postgres): keploy compat lane sample (scaffold)
Mirrors the doccano-django sample shape: the sample owns
orchestration (compose / bootstrap / traffic / coverage), keploy
CI lanes consume it as a thin wrapper.
This is a SCAFFOLD — the full traffic loop driven by the existing
keploy/enterprise lane (`run_api_flow` in
.ci/scripts/umami-linux.sh) needs to be ported into
flow.sh::umami_record_traffic in a follow-up. The current loop is
deliberately minimal (heartbeat / me / teams / websites CRUD)
which is enough to prove the sample boots end-to-end without
keploy.
Layout:
Dockerfile — pin to umami:postgresql-v2.18.1
docker-compose.yml — postgres-15 + umami v2, env-driven
flow.sh — bootstrap | record-traffic | coverage | list-routes
keploy.yml.template — globalNoise for createdAt/updatedAt/uuid id
README.md — handoff + status notes
Signed-off-by: Akash Kumar <meakash7902@gmail.com>
* feat(umami-postgres): port full umami v2 traffic loop
Replace the bootstrap-only stub in flow.sh::umami_record_traffic with the
complete umami v2 API drive that the keploy compat lanes need to gate
against on a record/replay round-trip. The sample now owns the entire
traffic loop end-to-end; consuming lanes wrap `bootstrap | record-traffic
| coverage` inside `keploy record` / `keploy test` and add no curls of
their own.
Surfaces driven by record-traffic:
* auth: /api/auth/login (via bootstrap), /api/auth/verify, /api/auth/logout
* identity: /api/me, /api/me/teams, /api/me/websites
* admin: /api/admin/users, /api/admin/websites, /api/admin/teams (incl.
paged + search variants)
* users CRUD: POST /api/users, GET /api/users/{id}, POST /api/users/{id}
(update), GET /api/users/{id}/websites, GET /api/users/{id}/teams
* websites CRUD: POST /api/websites, GET /api/websites (paged), GET
/api/websites/{id}, POST /api/websites/{id} (update), GET
/api/websites/{id}/active, GET /api/websites/{id}/daterange,
POST /api/websites/{id}/reset
* events ingest: POST /api/send (event + identify variants), POST /api/batch
* sessions deep-dive: GET /api/websites/{id}/sessions[, /stats, /weekly,
/{sessionId}, /{sessionId}/activity, /{sessionId}/properties,
/{sessionId}/replays], GET /api/websites/{id}/replays, GET
/api/websites/{id}/session-data/properties
* analytics: stats, pageviews (multiple unit/timezone variants), events
(series/stats), event-data[/stats], values, realtime, metrics (path /
referrer / browser / os / device / country / event + search/limit
variants), metrics/expanded
* reports: every type umami v2 ships — breakdown, goal, funnel, journey,
retention, utm, attribution, performance — plus saved-report CRUD
(create, read, update, delete) and the listing endpoints
* teams CRUD lifecycle: POST/GET/POST(update)/DELETE on /api/teams/{id},
member attach/list/detach via /api/teams/{id}/users[/{userId}]
* share tokens: POST /api/websites/{id}/shares + GET /api/share/{shareId}
(unauthenticated public-share access)
* boards: full CRUD + /api/boards/{id}/shares
* pixel tracker: GET /api/pixels
* heartbeat 405 path: POST /api/heartbeat
Total: 78 distinct (method, path) tuples fired per record-traffic run.
Resource ids/names are fixed UUIDs / deterministic strings so request
bodies stay byte-stable across record/replay (keeps keploy's body
equality check passing without per-field globalNoise entries). Each
call goes through a small umami_http() helper that logs the (method,
url) tuple to UMAMI_FIRED_ROUTES_FILE and tolerates non-2xx (|| true)
so a single endpoint regression in umami itself does not abort the
whole record run — keploy is the assertion layer at replay.
Also strips the SCAFFOLD/handoff/follow-up language from flow.sh and
README.md: the sample is now the complete reproducer, no out-of-tree
porting remains.
Signed-off-by: Akash Kumar <meakash7902@gmail.com>
* ci(umami-postgres): add per-sample coverage gate workflow
Adds a GitHub Actions workflow scoped via paths: filter to
umami-postgres/** so it triggers ONLY on PRs and main-branch
pushes that touch the umami-postgres sample (or the workflow
file itself). Other samples in this repo keep their orthogonal
CI; gating the whole repo on every umami change would slow them
all down for no benefit.
Three jobs:
* build-coverage — runs the sample end-to-end against the
PR's HEAD ref via flow.sh bootstrap +
record-traffic, captures the route-
coverage percentage from flow.sh
coverage.
* release-coverage — same end-to-end against the PR's base
ref. Has a first-PR bootstrap escape
hatch (sample-existed=false → coverage=0)
so the introducing PR doesn't fail for
lack of a baseline.
* coverage-gate — fails the PR if build-coverage drops
more than COVERAGE_THRESHOLD percentage
points below release-coverage. Default
1.0pp; overridable via the
UMAMI_COVERAGE_THRESHOLD repo variable.
Sticky PR comment summarises the diff.
The gate runs ONLY here, on the sample repo. The enterprise PR
pipeline (.woodpecker/umami-linux.yml) calls flow.sh coverage
informationally with || true and does NOT gate on coverage —
that separation keeps the enterprise lane decoupled from sample-
level coverage drift.
Helper script .github/workflows/scripts/run-and-measure.sh is
the keploy-independent measurement shared by both build- and
release-coverage jobs: two-phase compose boot
(UMAMI_SKIP_INIT=0 then =1) matching the lane scripts, then
flow.sh bootstrap + record-traffic + coverage with
UMAMI_FIRED_ROUTES_FILE wired in as the standalone numerator.
Signed-off-by: Akash Kumar <meakash7902@gmail.com>
* fix(umami-postgres): read route surface from compiled Next.js manifest
The upstream umami image (ghcr.io/umami-software/umami:postgresql-v2.18.1)
ships a compiled Next.js build, not the TypeScript source. The
prior implementation greped src/app/api/**/route.ts inside the
container, which doesn't exist there, so umami_list_routes returned
zero rows and umami_report_coverage skipped with
"WARNING: ...skipping coverage report".
The route surface is fully derivable from the build artefacts:
/app/.next/app-path-routes-manifest.json → URL paths
/app/.next/server/app<url>/route.js → compiled handlers
with method exports
({GET:...,POST:...})
Verified end-to-end against the running container: list-routes
now emits 93 (method, path) rows; coverage gate has a real
denominator.
Signed-off-by: Akash Kumar <meakash7902@gmail.com>
* ci(umami-postgres): drop coverage gate — upstream image is precompiled+minified
The upstream `ghcr.io/umami-software/umami:postgresql-v2.18.1`
image ships a heavily minified Next.js standalone build under
/app/.next/server/app/api/**/route.js. The source tree
(/app/src) and sourcemaps (.map) are stripped from the image.
V8 / c8 line coverage on minified code is structurally
meaningless — each "line" of the compiled output is many source
statements concatenated by the bundler, so a coverage
percentage doesn't map back to anything a reviewer can act on.
Rather than ship a misleading metric (the prior route-surface
"coverage" we removed elsewhere was exactly this kind of
proxy), the umami sample is now smoke-test-only:
- `flow.sh bootstrap` signs in as admin, persists the JWT
- `flow.sh record-traffic` exercises the v2 API surface
- `flow.sh coverage` is a no-op that prints an info message
and exits 0 (so consumers' `flow.sh
coverage || true` calls keep working)
The keploy/enterprise compat lane already uses the resulting
record/replay assertions as its correctness gate — that IS the
meaningful test here, not source coverage of umami's frontend.
If real source-line coverage becomes a hard requirement for
this sample, the path is to rebuild umami from source inside a
Dockerfile.coverage overlay (~5-10 min npm install + next build
without minification + with sourcemaps). That's a separate
~hours-of-work change.
Removed:
- .github/workflows/umami-postgres.yml (coverage gate workflow)
- .github/workflows/scripts/run-and-measure.sh (its helper)
- umami_list_routes / umami_list_recorded_routes / the
legacy route-surface umami_report_coverage in flow.sh.
- list-routes subcommand.
Replaced umami_report_coverage with a no-op stub.
Signed-off-by: Akash Kumar <meakash7902@gmail.com>
* docs(umami-postgres): document why coverage is not measured (precompiled image)
Signed-off-by: Akash Kumar <meakash7902@gmail.com>
* fix(umami-postgres): plumb SKIP_DB_CHECK + nest globalNoise schema
The umami compat lane (consumed by keploy/enterprise PR #1889 for
the parse-server / umami / restheart / doccano three-cell matrix)
was failing all three umami cells with `getaddrinfo EAI_AGAIN
binaries.prisma.sh` mid-startup. Two sample-side bugs feeding into
the lane:
1. SKIP_DB_CHECK wasn't surfaced through the compose
umami's scripts/check-db.js fetches the prisma `schema-engine`
binary on every startup unless SKIP_DB_CHECK is set. Once
keploy's DNS interception is on (record/replay phase), the fetch
fails and umami crashes before serving HTTP. The compose
exposed UMAMI_SKIP_INIT but not SKIP_DB_CHECK, so the lane had
no way to suppress the second-phase prisma fetch.
Fix: surface `SKIP_DB_CHECK: "${SKIP_DB_CHECK:-}"`. The default
is empty (not "0") because check-db.js does a JS truthiness
check `if (process.env.SKIP_DB_CHECK)` — the literal string "0"
is truthy in JS and skips, defeating the gate. Empty string is
falsy. Lane scripts can now render the bootstrap-phase compose
with SKIP_DB_CHECK="" (migrations run against the fresh volume)
and the keploy-phase compose with SKIP_DB_CHECK="1" (skip,
volume already has the migrations + admin user).
2. globalNoise was the wrong shape
keploy's GlobalNoise config type is
`map[string]map[string][]string` — outer key is the section
(`header` / `body`), inner key is the field name within that
section, value is the list of values to ignore (empty list ==
match anything). The flat dotted form `header.Date: []` /
`body.createdAt: []` doesn't parse into that shape — keploy's
yaml decoder silently dropped it, so the lane had no effective
noise filter. The only reason existing samples didn't notice
is that keploy auto-stamps Date as per-test noise on every
recording, masking the schema bug.
Restructured to the correct nested form. Caught
`body.timestamp` (Date.now() inside /api/realtime that
--freezeTime can't intercept through V8) along the way — was
the last failing test in the otherwise-clean replay run.
Validation against keploy/enterprise's lane (concurrent 3-cell
matrix):
* record-stable-replay-pr (cell A): 104/104 PASSED
* record-pr-replay-pr (cell B): 104/104 PASSED
* record-pr-replay-stable (cell C): 104/104 PASSED
Signed-off-by: Akash Kumar <meakash7902@gmail.com>
---------
Signed-off-by: Akash Kumar <meakash7902@gmail.com>1 parent 5094f03 commit 1baff25
5 files changed
Lines changed: 684 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
0 commit comments