LLM Benchmark: Sequential Upgrades Test (#4817)

# Description of Changes

AI app generation benchmark comparing SpacetimeDB vs PostgreSQL (Express
+ Socket.io + Drizzle ORM). Same AI model (Claude Sonnet 4.6), same
prompts, same chat app, two backends. Upgraded through 12 feature
levels, manually graded at each level, bugs fixed, all costs measured
via OpenTelemetry.

Results viewable at:
https://spacetimedb.com/llms-benchmark-sequential-upgrade

## Benchmark harness (`tools/llm-sequential-upgrade/`)

- `run.sh`: orchestrates headless Claude Code sessions for code
generation, sequential upgrades, and bug fixes. Tracks all API costs via
OTel. Supports `--upgrade`, `--fix`, `--composed-prompt`,
`--resume-session` modes.
- `grade.sh` / `grade-agents.sh`: grading harnesses for manual testing
of generated apps.
- `docker-compose.otel.yaml`: OTel collector + PostgreSQL services.
- `generate-report.mjs` / `parse-telemetry.mjs`: aggregate per-session
telemetry into cost reports.
- Backend guidelines in `backends/`: SpacetimeDB SDK reference, config
templates, server setup docs, PostgreSQL setup with Drizzle/Socket.io
guidance.
**After #4740 merges,
we will likely want to update this so that it reads backend and SDK
guidance from SKILLS**

## Two complete benchmark runs

**Run 1 (20260403):** Original methodology.
**Run 2 (20260406):** Refined methodology with domain bias removed from
SpacetimeDB SDK docs and PostgreSQL instructions made
feature-spec-neutral.
**Note: no meaningful changes in results were observed with these
changes. Domain familiarity biases were very small and almost certainly
not the cause of STDB's major gains over PG stack.**

Each run contains full L1-L12 app source for both backends, level
snapshots preserving state before each upgrade, and per-session OTel
cost summaries.

## 12 feature levels

| Level | Feature |
|---|---|
| L1 | Basic Chat + Typing + Read Receipts + Unread Counts |
| L2 | Scheduled Messages |
| L3 | Ephemeral Messages |
| L4 | Message Reactions |
| L5 | Message Editing with History |
| L6 | Real-Time Permissions (kick, ban, promote) |
| L7 | Rich User Presence |
| L8 | Message Threading |
| L9 | Private Rooms + Direct Messages |
| L10 | Room Activity Indicators |
| L11 | Draft Sync |
| L12 | Anonymous to Registered Migration |

## Results

| | Run 1 (20260403) | Run 2 (20260406) |
|---|---|---|
| **SpacetimeDB total cost** | $13.33 | $12.62 |
| **PostgreSQL total cost** | $17.80 | $19.68 |
| **SpacetimeDB bugs** | 5 | 2 |
| **PostgreSQL bugs** | 19 | 8 |
| **SpacetimeDB fix sessions** | 4 | 1 |
| **PostgreSQL fix sessions** | 17 | 10 |

Both runs agree: SpacetimeDB apps are cheaper to build, have fewer bugs,
and require fewer fix iterations. The refined methodology (Run 2)
widened the cost gap and **confirmed the advantage is structural, not an
artifact of domain-biased SDK docs.**

## Performance benchmark (`perf-benchmark/`)

Stress throughput tool that fires concurrent writers at peak saturation
against the AI-generated `send_message` handlers.

| Tier | SpacetimeDB (avg) | PostgreSQL (avg) | Ratio |
|---|---|---|---|
| AI-generated (as-shipped) | 5,267 msgs/sec | 694 msgs/sec | 7.6x |
| PG rate limit removed | 5,267 msgs/sec | 1,070 msgs/sec | 4.9x |
| Optimized (same features kept) | 25,278 msgs/sec | 1,139 msgs/sec |
22x |

The gap widens with optimization because SpacetimeDB's bottleneck is
fixable code patterns in the reducer while PostgreSQL's bottleneck is
architectural (sequential network round-trips to an external database).

Optimized reference code with all features preserved is in
`perf-benchmark/results/optimized-reference/`.

## Data handling

Per-session cost summaries (`cost-summary.json`, `COST_REPORT.md`,
`metadata.json`) are committed. Raw OTel telemetry
(`raw-telemetry.jsonl`) containing PII is excluded via `.gitignore` and
stored privately.

# API and ABI breaking changes

None. All changes are in `tools/llm-sequential-upgrade/`. No production
code, library, or SDK changes.

# Expected complexity level and risk

**1 - Trivial.** Self-contained benchmarking tooling and data. No
interaction with production code.

# Testing

- [x] L1-L12 upgrades completed on all 4 apps (2 backends x 2 runs) with
OTel cost capture
- [x] All levels manually graded after each upgrade; bugs filed and
fixed via the harness
- [x] Methodology refinement between runs validated (domain bias
removal, feature-neutral instructions)
- [x] Stress benchmarks run across both runs x 3 tiers (as-shipped,
rate-limit-removed, optimized)
- [x] Optimized benchmarks verified to preserve all original features
- [x] Sensitive data (PII in raw telemetry) removed from repo and
gitignored
- [ ] Reviewer: spot-check that METRICS_DATA.json / METRICS_REPORT.json
numbers match the telemetry cost-summary.json files

---------

Co-authored-by: Tyler Cloutier <cloutiertyler@users.noreply.github.com>
Co-authored-by: clockwork-labs-bot <clockwork-labs-bot@users.noreply.github.com>

Author: 3 people

tools/llm-oneshot/.cursor/rules/patterns-typescript.mdc

@@ -60,7 +60,7 @@ spacetime publish chat-app-20260106-183045 --module-path backend/spacetimedb
   "type": "module",
   "version": "1.0.0",
   "dependencies": {
-    "spacetimedb": "^1.11.0"
+    "spacetimedb": "^2.0.0"
   }
 }
 ```
@@ -109,7 +109,7 @@ src/index.ts    → Import schema, define all reducers and lifecycle hooks
   "dependencies": {
     "react": "^18.3.1",
     "react-dom": "^18.3.1",
-    "spacetimedb": "^1.11.0"
+    "spacetimedb": "^2.0.0"
   },
   "devDependencies": {
     "@types/react": "^18.3.18",

tools/llm-oneshot/apps/chat-app/prompts/composed/01_basic.md

@@ -2,14 +2,45 @@
 
 Create a **real-time chat app**.
 
-**See `language/*.md` for language-specific setup, architecture, and constraints.**
 
-## UI Requirements
+## UI & Style Guide
 
-Use SpacetimeDB brand styling (dark theme).
+### Layout
+- **Sidebar** (left, ~220px fixed): app title/branding, user info with status, room list, online users
+- **Main area** (right, flex): room header bar, scrollable message list, input bar pinned to bottom
+- **Panels** (right slide-in or overlay): threads, pinned messages, profiles, settings
+
+### Visual Design
+- Dark theme using the brand colors from the language section below
+- Background: darkest shade for main bg, slightly lighter for sidebar and cards
+- Text: light on dark, muted color for timestamps and secondary info
+- Borders: subtle 1px, low contrast against background
+- Consistent spacing scale (8/12/16/24px)
+- Font: system font stack, clear hierarchy (bold headers, regular body, small muted metadata)
+- Rounded corners on inputs, buttons, cards, and message containers
+
+### Components
+- **Messages**: sender name (colored) + timestamp (muted) + text. Group consecutive messages from same sender. Action buttons appear on hover only (which buttons depend on the features below).
+- **Inputs**: full-width, rounded, subtle border, placeholder text, focus ring using primary color
+- **Buttons**: filled with primary color for main actions, outlined/ghost for secondary. Clear hover and active states.
+- **Badges**: small pill-shaped with count, contrasting color (e.g., unread count on rooms)
+- **Modals/panels**: slide-in from right with subtle backdrop, or dropdown overlays
+- **Status indicators**: small colored dots (green=online, yellow=away, red=DND, grey=offline)
+- **Room list**: room names with optional icon prefix (#), active room highlighted, unread badge
+
+### Interaction & UX
+- Show loading/connecting state while backend connects (spinner or skeleton, not blank screen)
+- Empty states: helpful text when no rooms, no messages, no results ("Create a room to get started")
+- Error feedback: inline error messages or toast notifications, never silent failures
+- Smooth transitions: fade/slide for panels, modals, and state changes
+- Hover reveals: message action buttons, tooltips on reactions, user profile cards
+- Keyboard support: Enter to send messages, Escape to close modals/panels
+- Auto-scroll to newest message, with scroll-to-bottom button when scrolled up
 
 ## Features
 
+**Important:** Each feature below includes a "UI contract" section specifying required element attributes for automated testing. You MUST follow these — they define the user-facing interface. Your architecture, state management, and backend design are entirely up to you.
+
 ### Basic Chat Features
 
 - Users can set a display name
@@ -18,20 +49,44 @@ Use SpacetimeDB brand styling (dark theme).
 - Show who's online
 - Include reasonable validation (e.g., don't let users spam, enforce sensible limits)
 
+**UI contract:**
+- Name input: `placeholder` contains "name" (case-insensitive)
+- Name submit: `button` with text "Join", "Register", "Set Name", or `type="submit"`
+- Room creation: `button` with text containing "Create" or "New" or "+"
+- Room name input: `placeholder` contains "room" or "name" (case-insensitive)
+- Message input: `placeholder` contains "message" (case-insensitive)
+- Send message: pressing Enter in the message input sends the message
+- Room list: room names visible as clickable text in a sidebar or list
+- Join room: clicking room name joins/enters it, or a `button` with text "Join"
+- Leave room: `button` with text "Leave"
+- Online users: user names displayed as text in a visible user list or member panel
+
 ### Typing Indicators
 
-- Show when other users are currently typing in a room
+- Show when other users are currently typing in the SAME room (typing must be scoped to room — do not broadcast typing to users in different rooms)
 - Typing indicator should automatically expire after a few seconds of inactivity
 - Display "User is typing..." or "Multiple users are typing..." in the UI
 
+**UI contract:**
+- Typing text: visible text containing "typing" (case-insensitive) when another user types
+- Auto-expiry: typing indicator text disappears within 6 seconds of inactivity
+
 ### Read Receipts
 
 - Track which users have seen which messages
-- Display "Seen by X, Y, Z" under messages (or a seen indicator)
+- Display "Seen by X, Y, Z" under messages — only show OTHER users who have seen it, not the sender
 - Update read status in real-time as users view messages
 
+**UI contract:**
+- Receipt text: text containing "seen" or "read" (case-insensitive) appears near messages after another user views them
+- Reader names: the receipt text includes the viewing user’s display name
+
 ### Unread Message Counts
 
 - Show unread message count badges on the room list
 - Track last-read position per user per room
 - Update counts in real-time as new messages arrive or are read
+
+**UI contract:**
+- Badge: a visible numeric badge (e.g., "3") appears next to room names in the sidebar when there are unread messages
+- Badge clears when the room is opened/entered

tools/llm-oneshot/apps/chat-app/prompts/composed/02_scheduled.md

@@ -2,14 +2,45 @@
 
 Create a **real-time chat app**.
 
-**See `language/*.md` for language-specific setup, architecture, and constraints.**
 
-## UI Requirements
+## UI & Style Guide
 
-Use SpacetimeDB brand styling (dark theme).
+### Layout
+- **Sidebar** (left, ~220px fixed): app title/branding, user info with status, room list, online users
+- **Main area** (right, flex): room header bar, scrollable message list, input bar pinned to bottom
+- **Panels** (right slide-in or overlay): threads, pinned messages, profiles, settings
+
+### Visual Design
+- Dark theme using the brand colors from the language section below
+- Background: darkest shade for main bg, slightly lighter for sidebar and cards
+- Text: light on dark, muted color for timestamps and secondary info
+- Borders: subtle 1px, low contrast against background
+- Consistent spacing scale (8/12/16/24px)
+- Font: system font stack, clear hierarchy (bold headers, regular body, small muted metadata)
+- Rounded corners on inputs, buttons, cards, and message containers
+
+### Components
+- **Messages**: sender name (colored) + timestamp (muted) + text. Group consecutive messages from same sender. Action buttons appear on hover only (which buttons depend on the features below).
+- **Inputs**: full-width, rounded, subtle border, placeholder text, focus ring using primary color
+- **Buttons**: filled with primary color for main actions, outlined/ghost for secondary. Clear hover and active states.
+- **Badges**: small pill-shaped with count, contrasting color (e.g., unread count on rooms)
+- **Modals/panels**: slide-in from right with subtle backdrop, or dropdown overlays
+- **Status indicators**: small colored dots (green=online, yellow=away, red=DND, grey=offline)
+- **Room list**: room names with optional icon prefix (#), active room highlighted, unread badge
+
+### Interaction & UX
+- Show loading/connecting state while backend connects (spinner or skeleton, not blank screen)
+- Empty states: helpful text when no rooms, no messages, no results ("Create a room to get started")
+- Error feedback: inline error messages or toast notifications, never silent failures
+- Smooth transitions: fade/slide for panels, modals, and state changes
+- Hover reveals: message action buttons, tooltips on reactions, user profile cards
+- Keyboard support: Enter to send messages, Escape to close modals/panels
+- Auto-scroll to newest message, with scroll-to-bottom button when scrolled up
 
 ## Features
 
+**Important:** Each feature below includes a "UI contract" section specifying required element attributes for automated testing. You MUST follow these — they define the user-facing interface. Your architecture, state management, and backend design are entirely up to you.
+
 ### Basic Chat Features
 
 - Users can set a display name
@@ -18,26 +49,56 @@ Use SpacetimeDB brand styling (dark theme).
 - Show who's online
 - Include reasonable validation (e.g., don't let users spam, enforce sensible limits)
 
+**UI contract:**
+- Name input: `placeholder` contains "name" (case-insensitive)
+- Name submit: `button` with text "Join", "Register", "Set Name", or `type="submit"`
+- Room creation: `button` with text containing "Create" or "New" or "+"
+- Room name input: `placeholder` contains "room" or "name" (case-insensitive)
+- Message input: `placeholder` contains "message" (case-insensitive)
+- Send message: pressing Enter in the message input sends the message
+- Room list: room names visible as clickable text in a sidebar or list
+- Join room: clicking room name joins/enters it, or a `button` with text "Join"
+- Leave room: `button` with text "Leave"
+- Online users: user names displayed as text in a visible user list or member panel
+
 ### Typing Indicators
 
-- Show when other users are currently typing in a room
+- Show when other users are currently typing in the SAME room (typing must be scoped to room — do not broadcast typing to users in different rooms)
 - Typing indicator should automatically expire after a few seconds of inactivity
 - Display "User is typing..." or "Multiple users are typing..." in the UI
 
+**UI contract:**
+- Typing text: visible text containing "typing" (case-insensitive) when another user types
+- Auto-expiry: typing indicator text disappears within 6 seconds of inactivity
+
 ### Read Receipts
 
 - Track which users have seen which messages
-- Display "Seen by X, Y, Z" under messages (or a seen indicator)
+- Display "Seen by X, Y, Z" under messages — only show OTHER users who have seen it, not the sender
 - Update read status in real-time as users view messages
 
+**UI contract:**
+- Receipt text: text containing "seen" or "read" (case-insensitive) appears near messages after another user views them
+- Reader names: the receipt text includes the viewing user’s display name
+
 ### Unread Message Counts
 
 - Show unread message count badges on the room list
 - Track last-read position per user per room
 - Update counts in real-time as new messages arrive or are read
 
+**UI contract:**
+- Badge: a visible numeric badge (e.g., "3") appears next to room names in the sidebar when there are unread messages
+- Badge clears when the room is opened/entered
+
 ### Scheduled Messages
 
 - Users can compose a message and schedule it to send at a future time
 - Show pending scheduled messages to the author (with option to cancel)
 - Message appears in the room at the scheduled time
+
+**UI contract:**
+- Schedule button: `button` with text "Schedule" or `aria-label` containing "schedule", or an icon button with `title` containing "schedule"
+- Time picker: an `input[type="datetime-local"]` or `input[type="time"]` or `input[type="number"]` for setting the send time
+- Pending list: text "Scheduled" or "Pending" visible when viewing scheduled messages
+- Cancel: `button` with text "Cancel" next to pending scheduled messages

tools/llm-oneshot/apps/chat-app/prompts/composed/03_realtime.md

@@ -2,14 +2,45 @@
 
 Create a **real-time chat app**.
 
-**See `language/*.md` for language-specific setup, architecture, and constraints.**
 
-## UI Requirements
-
-Use SpacetimeDB brand styling (dark theme).
+## UI & Style Guide
+
+### Layout
+- **Sidebar** (left, ~220px fixed): app title/branding, user info with status, room list, online users
+- **Main area** (right, flex): room header bar, scrollable message list, input bar pinned to bottom
+- **Panels** (right slide-in or overlay): threads, pinned messages, profiles, settings
+
+### Visual Design
+- Dark theme using the brand colors from the language section below
+- Background: darkest shade for main bg, slightly lighter for sidebar and cards
+- Text: light on dark, muted color for timestamps and secondary info
+- Borders: subtle 1px, low contrast against background
+- Consistent spacing scale (8/12/16/24px)
+- Font: system font stack, clear hierarchy (bold headers, regular body, small muted metadata)
+- Rounded corners on inputs, buttons, cards, and message containers
+
+### Components
+- **Messages**: sender name (colored) + timestamp (muted) + text. Group consecutive messages from same sender. Action buttons appear on hover only (which buttons depend on the features below).
+- **Inputs**: full-width, rounded, subtle border, placeholder text, focus ring using primary color
+- **Buttons**: filled with primary color for main actions, outlined/ghost for secondary. Clear hover and active states.
+- **Badges**: small pill-shaped with count, contrasting color (e.g., unread count on rooms)
+- **Modals/panels**: slide-in from right with subtle backdrop, or dropdown overlays
+- **Status indicators**: small colored dots (green=online, yellow=away, red=DND, grey=offline)
+- **Room list**: room names with optional icon prefix (#), active room highlighted, unread badge
+
+### Interaction & UX
+- Show loading/connecting state while backend connects (spinner or skeleton, not blank screen)
+- Empty states: helpful text when no rooms, no messages, no results ("Create a room to get started")
+- Error feedback: inline error messages or toast notifications, never silent failures
+- Smooth transitions: fade/slide for panels, modals, and state changes
+- Hover reveals: message action buttons, tooltips on reactions, user profile cards
+- Keyboard support: Enter to send messages, Escape to close modals/panels
+- Auto-scroll to newest message, with scroll-to-bottom button when scrolled up
 
 ## Features
 
+**Important:** Each feature below includes a "UI contract" section specifying required element attributes for automated testing. You MUST follow these — they define the user-facing interface. Your architecture, state management, and backend design are entirely up to you.
+
 ### Basic Chat Features
 
 - Users can set a display name
@@ -18,32 +49,68 @@ Use SpacetimeDB brand styling (dark theme).
 - Show who's online
 - Include reasonable validation (e.g., don't let users spam, enforce sensible limits)
 
+**UI contract:**
+- Name input: `placeholder` contains "name" (case-insensitive)
+- Name submit: `button` with text "Join", "Register", "Set Name", or `type="submit"`
+- Room creation: `button` with text containing "Create" or "New" or "+"
+- Room name input: `placeholder` contains "room" or "name" (case-insensitive)
+- Message input: `placeholder` contains "message" (case-insensitive)
+- Send message: pressing Enter in the message input sends the message
+- Room list: room names visible as clickable text in a sidebar or list
+- Join room: clicking room name joins/enters it, or a `button` with text "Join"
+- Leave room: `button` with text "Leave"
+- Online users: user names displayed as text in a visible user list or member panel
+
 ### Typing Indicators
 
-- Show when other users are currently typing in a room
+- Show when other users are currently typing in the SAME room (typing must be scoped to room — do not broadcast typing to users in different rooms)
 - Typing indicator should automatically expire after a few seconds of inactivity
 - Display "User is typing..." or "Multiple users are typing..." in the UI
 
+**UI contract:**
+- Typing text: visible text containing "typing" (case-insensitive) when another user types
+- Auto-expiry: typing indicator text disappears within 6 seconds of inactivity
+
 ### Read Receipts
 
 - Track which users have seen which messages
-- Display "Seen by X, Y, Z" under messages (or a seen indicator)
+- Display "Seen by X, Y, Z" under messages — only show OTHER users who have seen it, not the sender
 - Update read status in real-time as users view messages
 
+**UI contract:**
+- Receipt text: text containing "seen" or "read" (case-insensitive) appears near messages after another user views them
+- Reader names: the receipt text includes the viewing user’s display name
+
 ### Unread Message Counts
 
 - Show unread message count badges on the room list
 - Track last-read position per user per room
 - Update counts in real-time as new messages arrive or are read
 
+**UI contract:**
+- Badge: a visible numeric badge (e.g., "3") appears next to room names in the sidebar when there are unread messages
+- Badge clears when the room is opened/entered
+
 ### Scheduled Messages
 
 - Users can compose a message and schedule it to send at a future time
 - Show pending scheduled messages to the author (with option to cancel)
 - Message appears in the room at the scheduled time
 
+**UI contract:**
+- Schedule button: `button` with text "Schedule" or `aria-label` containing "schedule", or an icon button with `title` containing "schedule"
+- Time picker: an `input[type="datetime-local"]` or `input[type="time"]` or `input[type="number"]` for setting the send time
+- Pending list: text "Scheduled" or "Pending" visible when viewing scheduled messages
+- Cancel: `button` with text "Cancel" next to pending scheduled messages
+
 ### Ephemeral/Disappearing Messages
 
 - Users can send messages that auto-delete after a set duration (e.g., 1 minute, 5 minutes)
 - Show a countdown or indicator that the message will disappear
 - Message is permanently deleted from the database when time expires
+
+**UI contract:**
+- Ephemeral toggle: `select`, `button`, or `input` with text/label containing "ephemeral", "disappear", or "expire" (case-insensitive)
+- Duration options: selectable durations (e.g., 30s, 1m, 5m)
+- Indicator: visible text containing a countdown, "expires", or "disappearing" on ephemeral messages
+- Deletion: the message text is removed from the DOM after the duration expires