Skip to content

Commit 9a03b3e

Browse files
Aditya-eddyclaudeslayerjain
authored
feat(postgres-wire-features): add Postgres v3 wire-protocol sample (#229)
* feat(postgres-wire-features): add Postgres v3 wire-protocol sample A stdlib-only Go HTTP service that drives a PostgreSQL connection directly over the v3 frontend/backend protocol (no driver), exposing ten scenarios under GET /run/all: - setup (DDL) - dml (INSERT ON CONFLICT, UPDATE, DELETE, MERGE) - catalog-cte / catalog-sub / catalog-setop (pg_catalog queries via CTE, subselect, UNION) - copy (COPY FROM STDIN + COPY TO STDOUT) - prepare (PREPARE, EXECUTE, DEALLOCATE) - cursor (BEGIN, DECLARE CURSOR, FETCH, CLOSE, COMMIT) - admin (ANALYZE, REINDEX, LOCK TABLE, DO, CREATE PROCEDURE, CALL) - validation-ping (SELECT 'ok', SELECT true, SELECT NULL, SELECT 1) Standard sample layout: Dockerfile, docker-compose.yml with api+db, test.sh traffic script that hits /run/all and each /case/<name>, and a README with record/test instructions. The wire-protocol traffic is intentionally low-level so Keploy's proxy sees every message class (CopyData, CopyDone, Parse/Bind/Execute, cursor portals, etc.) as the server sends them, rather than what a driver would hide behind its API. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Aditya-eddy <aditya282003@gmail.com> * feat(postgres-wire-features): simplify test.sh to /health + /run/all only Dropping the per-scenario /case/<name> loop from test.sh. /run/all already exercises every scenario inside one HTTP request, which is all a record pass needs; firing /case/* endpoints in quick succession afterwards created many back-to-back raw-socket Postgres connections that could push the recording proxy into an EOF regime in docker mode, producing noisy captures whose replay diffs drowned out the real signal. Signed-off-by: Aditya-eddy <aditya282003@gmail.com> * feat(postgres-wire-features): share one Postgres connection across /run/all Refactor: runXxx scenario functions now take *pgClient instead of opening their own. /run/all dials once and walks all ten scenarios on that single connection; /case/<name> keeps its own withClient wrapper so individual scenario endpoints still run standalone. Opening a fresh connection per scenario pushed Keploy's docker-mode proxy into a back-to-back connect/close regime where the agent dropped handshakes (visible in CI logs as 'failed to read client auth response: EOF' on every other connection during record, and 'replayer: session RecordedIndex missing PostgresV3Session mock' during replay). Sharing one connection drops the connect count from ~10 per /run/all to 1, which keeps the v3 wire surface exercised — CopyData/CopyDone frames are still sent on the wire; validation-ping SELECTs still collide on sqlAstHash — without tripping that unrelated instability. Signed-off-by: Aditya-eddy <aditya282003@gmail.com> * fix(postgres-wire-features): force trust auth so raw v3 client can handshake Postgres 16 defaults to scram-sha-256 (auth code 10) when a password is set. This sample's pure-stdlib client only implements codes 0/trust, 3/cleartext, and 5/md5 — SASL is out of scope for a wire-protocol repro. Pinning POSTGRES_HOST_AUTH_METHOD=trust keeps auth on code 0, which is what the client actually speaks. Observed in CI: dial was failing with 'unsupported postgres authentication code 10' before any scenario query ran, so the recorded test body was just the auth error — masking the real Postgres wire surface the sample is meant to exercise. Signed-off-by: Aditya-eddy <aditya282003@gmail.com> * feat(postgres-wire-features): add multistatement + empty-query scenarios and assertions Extend the sample with two new scenarios that exercise specific v3 recorder edges not covered by the existing suite: - /case/multistatement sends BEGIN; SELECT 1 AS a; SELECT 2 AS b; COMMIT as a SINGLE simple-Query packet. A recorder that splits the batch with pg_query emits four invocations; a recorder that tracks per-packet emits one. - /case/empty-query sends a Query packet with empty SQL body. The server answers with EmptyQueryResponse ('I') and ReadyForQuery only. A correct recorder suppresses this as a no-op; a naive recorder emits a ghost invocation with empty sqlNormalized and class=UNKNOWN. Both scenarios are added to /run/all so the single-connection path exercises them, and are also exposed as standalone /case endpoints so their recorded testcases are isolated by connection boundary. simpleQuery grows an 'I' (EmptyQueryResponse) case so the empty-query path returns cleanly rather than erroring on the unhandled tag. assertions.sh validates recorded mocks.yaml after a Keploy record run: 1. no invocation with class: UNKNOWN 2. no invocation with empty sqlNormalized (ghost event) 3. multistatement Q packet produced >=1 each of BEGIN, COMMIT, SELECT \$1 AS a, SELECT \$1 AS b invocations 4. at least some readable SQL is present (mocks aren't entirely base64) Use ./assertions.sh keploy after keploy record to fail the pipeline if the recorder regressed on any of those invariants. Signed-off-by: slayerjain <shubhamkjain@outlook.com> * fix(postgres-wire-features): bind Postgres to loopback and fail on HTTP errors Address Copilot review on #229: - docker-compose.yml: bind the container's 5433 port to 127.0.0.1 only instead of all interfaces. With POSTGRES_HOST_AUTH_METHOD=trust the DB accepts unauthenticated connections, so publishing on 0.0.0.0 left an open Postgres on whatever network the developer machine was attached to. Loopback-only keeps the sample usable for record/replay without that exposure. - docker-compose.yml: remove POSTGRES_PASSWORD / PGPASSWORD. Under trust auth the server never requests a password; keeping those env vars suggested the auth path in use required one and could mask an accidental reliance on trust mode. Trust is intentional here (see the adjacent comment block) so the password vars are now dropped. - test.sh: pass --fail-with-body to every curl invocation so an HTTP 5xx from /health, /run/all, or the /case endpoints exits the script non-zero. Without this curl exits 0 on 500s and the traffic script silently produces a bad recording. Signed-off-by: slayerjain <shubhamkjain@outlook.com> * fix(postgres-wire-features): bound Postgres ops and explain unsupported auth codes Address Copilot review on #229: - Add a 30s per-operation deadline around startup() and simpleQuery(). Before this, the raw net.Conn had no deadlines, so a stalled or half-open Postgres connection would wedge io.ReadFull inside readMessage and hang the HTTP handler (and leak goroutines) until the OS eventually gave up. setOpDeadline / clearOpDeadline arm the deadline on entry and clear it on exit, so the /run/all path (which shares one pgClient across scenarios) never inherits a stale deadline between operations. - Replace the opaque "unsupported postgres authentication code %d" error with a message that spells out the codes this client actually speaks (0/trust, 3/cleartext, 5/md5) and points at POSTGRES_HOST_AUTH_METHOD as the quick local remediation when the server negotiates scram-sha-256 (code 10, the Postgres 16 default). The prior message sent users digging through protocol docs to discover what the numeric code meant. Signed-off-by: slayerjain <shubhamkjain@outlook.com> --------- Signed-off-by: Aditya-eddy <aditya282003@gmail.com> Signed-off-by: slayerjain <shubhamkjain@outlook.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Co-authored-by: slayerjain <shubhamkjain@outlook.com>
1 parent 3ef38be commit 9a03b3e

8 files changed

Lines changed: 921 additions & 0 deletions

File tree

postgres-wire-features/.gitignore

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
keploy/
2+
keploy.yml
3+
keploy-logs.txt
4+
server
5+
postgres-wire-features
6+
*.log

postgres-wire-features/Dockerfile

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
FROM golang:1.24-alpine AS builder
2+
3+
WORKDIR /app
4+
COPY go.mod ./
5+
RUN go mod download
6+
COPY . .
7+
RUN CGO_ENABLED=0 go build -o server .
8+
9+
FROM alpine:3.19
10+
WORKDIR /app
11+
COPY --from=builder /app/server .
12+
EXPOSE 8080
13+
CMD ["./server"]

postgres-wire-features/README.md

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# postgres-wire-features
2+
3+
A small Go HTTP service that exercises a broad set of PostgreSQL wire-protocol features in one place — designed as a Keploy sample for recording and replaying Postgres interactions that go beyond plain `SELECT` / `INSERT`.
4+
5+
The service speaks the Postgres v3 frontend/backend protocol directly over a TCP socket (pure stdlib — no driver). That makes the recorded traffic a faithful representation of what Keploy sees on the wire, which matters for features like `COPY` whose payload is a stream of `CopyData` / `CopyDone` messages that driver libraries often abstract away.
6+
7+
## What it covers
8+
9+
A single call to `GET /run/all` walks through twelve scenarios in order:
10+
11+
| Scenario | Statements |
12+
| --- | --- |
13+
| `setup` | `DROP TABLE`, `CREATE TABLE`, `INSERT`, `CREATE TABLE IF NOT EXISTS` |
14+
| `dml` | `INSERT … ON CONFLICT DO UPDATE`, `UPDATE`, `DELETE`, `MERGE` |
15+
| `catalog-cte` | `WITH … SELECT FROM pg_catalog.pg_class` |
16+
| `catalog-sub` | `SELECT … FROM (SELECT …) AS s` |
17+
| `catalog-setop` | `SELECT … UNION SELECT …` |
18+
| `copy` | `TRUNCATE`, `COPY … FROM STDIN`, `COPY … TO STDOUT` |
19+
| `prepare` | `PREPARE`, `EXECUTE`, `DEALLOCATE` |
20+
| `cursor` | `BEGIN`, `DECLARE CURSOR`, `FETCH`×2, `CLOSE`, `COMMIT` |
21+
| `admin` | `ANALYZE`, `REINDEX`, `LOCK TABLE`, `DO $$ … $$`, `CREATE PROCEDURE`, `CALL` |
22+
| `validation-ping` | `SELECT 'ok'`, `SELECT true`, `SELECT NULL`, `SELECT 1` |
23+
| `multistatement` | `BEGIN; SELECT 1 AS a; SELECT 2 AS b; COMMIT` sent as a **single** simple-Query packet (exercises multi-statement split) |
24+
| `empty-query` | simple-Query packet with empty SQL body (exercises `EmptyQueryResponse` / ghost-event suppression) |
25+
26+
Each scenario is also reachable individually at `GET /case/<name>`. The response is JSON: per-statement `commands` (the Postgres command tag, e.g. `COPY 2`, `SELECT 1`), `rows`, and for `COPY TO STDOUT` a `copyData` array of the raw tab-separated rows.
27+
28+
## Endpoints
29+
30+
- `GET /health``200 ok`
31+
- `GET /run/all` → run every scenario in order, return the combined result
32+
- `GET /case/<name>` → run one scenario (see table above)
33+
34+
## Prerequisites
35+
36+
- Go `1.24+`
37+
- Docker and Docker Compose
38+
- Keploy CLI
39+
40+
Install Keploy:
41+
42+
```bash
43+
curl --silent -O -L https://keploy.io/install.sh
44+
source install.sh
45+
```
46+
47+
## Setup
48+
49+
```bash
50+
git clone https://github.com/keploy/samples-go.git
51+
cd samples-go/postgres-wire-features
52+
go mod download
53+
```
54+
55+
## Run with Docker
56+
57+
```bash
58+
docker compose up --build
59+
```
60+
61+
The API will be available at `http://localhost:8080`.
62+
63+
## Record Testcases with Keploy
64+
65+
```bash
66+
keploy record -c "docker compose up" --container-name api --delay 10
67+
```
68+
69+
In another terminal:
70+
71+
```bash
72+
./test.sh
73+
```
74+
75+
Keploy writes the recorded HTTP tests and Postgres mocks under `keploy/`.
76+
77+
## Validate recorded mocks
78+
79+
Run the assertions script after recording to verify v3 protocol invariants:
80+
81+
```bash
82+
./assertions.sh keploy
83+
```
84+
85+
It fails the pipeline if it sees any of:
86+
- an invocation with `class: UNKNOWN`
87+
- an invocation with an empty `sqlNormalized` (ghost event)
88+
- the `multistatement` batch collapsed into fewer than four invocations
89+
- all fields encoded as `!!binary` with no readable SQL in the mocks
90+
91+
## Replay Recorded Tests
92+
93+
```bash
94+
keploy test -c "docker compose up" --container-name api --delay 20
95+
```
96+
97+
## Run Natively
98+
99+
Start Postgres yourself (e.g. `docker run -p 5432:5432 -e POSTGRES_PASSWORD=postgres postgres:16-alpine`) and then:
100+
101+
```bash
102+
PGHOST=127.0.0.1 PGPORT=5432 PGUSER=postgres PGPASSWORD=postgres PGDATABASE=postgres go run .
103+
```
104+
105+
## Environment variables
106+
107+
| Variable | Default | Purpose |
108+
| --- | --- | --- |
109+
| `ADDR` | `:8080` | Listen address for the HTTP server |
110+
| `PGHOST` | `127.0.0.1` | Postgres host |
111+
| `PGPORT` | `5432` | Postgres port |
112+
| `PGUSER` | `postgres` | Postgres user |
113+
| `PGPASSWORD` | _(empty)_ | Postgres password |
114+
| `PGDATABASE` | `postgres` | Postgres database |
Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
#!/usr/bin/env bash
2+
# Post-record assertions for the v3 Postgres recorder against the
3+
# postgres-wire-features sample. Exits non-zero on the first violation
4+
# so the Woodpecker step shows a clean pass/fail.
5+
#
6+
# These assertions codify invariants that the *fixed* v3 recorder must
7+
# hold. A broken v3 (for example origin/main today) produces mocks that
8+
# violate at least one of them: empty-query ghost invocations with
9+
# class=UNKNOWN, or a multi-statement Q packet collapsed into one
10+
# invocation, or bind values emitted as base64 when the raw bytes are
11+
# valid UTF-8 text.
12+
#
13+
# Usage: ./assertions.sh [mocks_dir]
14+
# default mocks_dir: ./keploy
15+
16+
set -euo pipefail
17+
18+
MOCKS_DIR="${1:-./keploy}"
19+
20+
if [[ ! -d "$MOCKS_DIR" ]]; then
21+
echo "assertions: mocks directory not found at $MOCKS_DIR" >&2
22+
exit 2
23+
fi
24+
25+
# Collect every mocks.yaml beneath $MOCKS_DIR. Keploy writes one
26+
# mocks.yaml per test-set.
27+
mapfile -t MOCK_FILES < <(find "$MOCKS_DIR" -type f -name 'mocks.yaml' | sort)
28+
if [[ ${#MOCK_FILES[@]} -eq 0 ]]; then
29+
echo "assertions: no mocks.yaml found under $MOCKS_DIR" >&2
30+
exit 2
31+
fi
32+
33+
echo "assertions: scanning ${#MOCK_FILES[@]} mocks.yaml file(s) under $MOCKS_DIR"
34+
35+
fail() {
36+
echo "FAIL: $1" >&2
37+
exit 1
38+
}
39+
40+
pass() {
41+
echo "PASS: $1"
42+
}
43+
44+
# ------------------------------------------------------------------
45+
# Invariant 1: no invocation is classified as UNKNOWN.
46+
# Origin/main emits class=UNKNOWN for ghost events and for statements
47+
# its classifier does not recognize.
48+
# ------------------------------------------------------------------
49+
unknown_count=0
50+
for f in "${MOCK_FILES[@]}"; do
51+
c=$(grep -c -E '^\s*class:\s*UNKNOWN\s*$' "$f" || true)
52+
unknown_count=$((unknown_count + c))
53+
done
54+
if [[ $unknown_count -ne 0 ]]; then
55+
echo "--- first few UNKNOWN hits ---"
56+
grep -n -E '^\s*class:\s*UNKNOWN\s*$' "${MOCK_FILES[@]}" | head -n 10 || true
57+
fail "found $unknown_count invocation(s) with class: UNKNOWN"
58+
fi
59+
pass "no class: UNKNOWN invocations"
60+
61+
# ------------------------------------------------------------------
62+
# Invariant 2: no ghost invocation with an empty sqlNormalized.
63+
# A correct recorder suppresses EmptyQueryResponse paths instead of
64+
# emitting an invocation shell with sqlNormalized: "".
65+
# ------------------------------------------------------------------
66+
empty_sql_count=0
67+
for f in "${MOCK_FILES[@]}"; do
68+
# Match both sqlNormalized: "" and sqlNormalized:
69+
c=$(grep -c -E '^\s*sqlNormalized:\s*("")?\s*$' "$f" || true)
70+
empty_sql_count=$((empty_sql_count + c))
71+
done
72+
if [[ $empty_sql_count -ne 0 ]]; then
73+
echo "--- first few empty sqlNormalized hits ---"
74+
grep -n -E '^\s*sqlNormalized:\s*("")?\s*$' "${MOCK_FILES[@]}" | head -n 10 || true
75+
fail "found $empty_sql_count invocation(s) with empty sqlNormalized (ghost event)"
76+
fi
77+
pass "no empty sqlNormalized invocations"
78+
79+
# ------------------------------------------------------------------
80+
# Invariant 3: the multistatement Q packet produced distinct
81+
# invocations for each of its four statements.
82+
#
83+
# The sample sends `BEGIN; SELECT 1 AS a; SELECT 2 AS b; COMMIT`
84+
# as a single simple-Query packet. A recorder that splits the batch
85+
# with pg_query emits four invocations with distinct sqlNormalized
86+
# fragments; a recorder that tracks per-packet emits one.
87+
# ------------------------------------------------------------------
88+
begin_count=0
89+
commit_count=0
90+
select_a_count=0
91+
select_b_count=0
92+
for f in "${MOCK_FILES[@]}"; do
93+
begin_count=$((begin_count + $(grep -c -E '^\s*sqlNormalized:\s*["'"'"']?BEGIN["'"'"']?\s*$' "$f" || true)))
94+
commit_count=$((commit_count + $(grep -c -E '^\s*sqlNormalized:\s*["'"'"']?COMMIT["'"'"']?\s*$' "$f" || true)))
95+
select_a_count=$((select_a_count + $(grep -c -E 'sqlNormalized:.*SELECT\s+\$1\s+AS\s+a' "$f" || true)))
96+
select_b_count=$((select_b_count + $(grep -c -E 'sqlNormalized:.*SELECT\s+\$1\s+AS\s+b' "$f" || true)))
97+
done
98+
if [[ $begin_count -lt 1 || $commit_count -lt 1 || $select_a_count -lt 1 || $select_b_count -lt 1 ]]; then
99+
echo " BEGIN=$begin_count COMMIT=$commit_count SELECT_a=$select_a_count SELECT_b=$select_b_count"
100+
fail "multistatement Q packet was not split into 4 invocations (expected >=1 each of BEGIN, COMMIT, SELECT ... AS a, SELECT ... AS b)"
101+
fi
102+
pass "multistatement Q packet split into distinct invocations (BEGIN=$begin_count, COMMIT=$commit_count, SELECT a=$select_a_count, SELECT b=$select_b_count)"
103+
104+
# ------------------------------------------------------------------
105+
# Invariant 4: text bind values are emitted as plain YAML strings
106+
# (not wrapped in a !!binary tag) when the raw bytes are UTF-8 safe.
107+
# The prepare/execute scenario's bind contains ASCII integers, and
108+
# the COPY IN scenario's payload is ASCII text — both should survive
109+
# as readable strings.
110+
# ------------------------------------------------------------------
111+
binary_tag_count=0
112+
for f in "${MOCK_FILES[@]}"; do
113+
c=$(grep -c -E '!!binary' "$f" || true)
114+
binary_tag_count=$((binary_tag_count + c))
115+
done
116+
echo " info: !!binary tag occurrences across mocks = $binary_tag_count"
117+
# We don't fail on this count (truly-binary values like lsn/xid bytes
118+
# legitimately round-trip as !!binary); instead assert that a known-
119+
# textual bind ('seed-a' from COPY IN) appears as a plain string.
120+
if ! grep -R -l -E "seed-a" "$MOCKS_DIR" >/dev/null 2>&1; then
121+
# seed-a might only appear post-COPY; tolerate its absence, but
122+
# require that at least one readable SQL fragment is present so
123+
# we know the file isn't entirely base64-encoded.
124+
if ! grep -R -l -E 'sqlNormalized:.*SELECT' "$MOCKS_DIR" >/dev/null 2>&1; then
125+
fail "no readable sqlNormalized values found — every field may be base64 encoded"
126+
fi
127+
fi
128+
pass "readable text fields are present in mocks (not exclusively base64)"
129+
130+
echo "assertions: all invariants satisfied"
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
services:
2+
db:
3+
image: postgres:16-alpine
4+
environment:
5+
POSTGRES_USER: postgres
6+
POSTGRES_DB: postgres
7+
# Force trust auth (code 0). Postgres 16 defaults to scram-sha-256
8+
# (code 10), which this sample's raw-v3 client does not speak — we
9+
# only implement code 0 / 3 / 5. This keeps auth on a path the
10+
# sample can actually complete so the downstream Postgres wire
11+
# surface (CopyData, cursors, PREPARE/EXECUTE, etc.) is what the
12+
# record/replay cycle is actually exercising. With trust auth the
13+
# server never requests a password, so POSTGRES_PASSWORD/PGPASSWORD
14+
# are intentionally omitted to avoid implying otherwise.
15+
POSTGRES_HOST_AUTH_METHOD: trust
16+
ports:
17+
# Bind to loopback only so the trust-auth Postgres isn't reachable
18+
# from other hosts on the developer's network.
19+
- "127.0.0.1:5433:5432"
20+
volumes:
21+
- pgdata:/var/lib/postgresql/data
22+
healthcheck:
23+
test: ["CMD-SHELL", "pg_isready -U postgres"]
24+
interval: 2s
25+
timeout: 5s
26+
retries: 5
27+
28+
api:
29+
build: .
30+
ports:
31+
- "8080:8080"
32+
environment:
33+
PGHOST: db
34+
PGPORT: "5432"
35+
PGUSER: postgres
36+
PGDATABASE: postgres
37+
depends_on:
38+
db:
39+
condition: service_healthy
40+
41+
volumes:
42+
pgdata:

postgres-wire-features/go.mod

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
module postgres-wire-features
2+
3+
go 1.24

0 commit comments

Comments
 (0)