fix(benchmarks): seed Pro dummy DB so /posts_page stops 500ing (#3602)#3615
Conversation
WalkthroughReplaces nondeterministic dummy seeds with deterministic generators, adds RSpec tests for seeds determinism and the /posts_page route, and adds a conditional CI workflow step that prepares and verifies the Rails benchmark database for Pro Rails runs. ChangesDeterministic Benchmark Data and Testing
Sequence DiagramsequenceDiagram
participant GitHubCI as GitHub Actions
participant BenchmarkJob as benchmark job
participant RailsApp as Rails runner
participant DB as Database
participant RSpec as RSpec
GitHubCI->>BenchmarkJob: trigger benchmark job (Pro Rails matrix)
BenchmarkJob->>RailsApp: run db:prepare (RAILS_ENV=production NODE_ENV=production)
RailsApp->>DB: migrate/create schema and run db/seeds.rb
DB-->>RailsApp: return inserted Users/Posts/Comments
BenchmarkJob->>RailsApp: verify Post/User/Comment counts (abort if any zero)
BenchmarkJob->>RSpec: start test suite
RSpec->>RailsApp: clear and deterministically seed DB (tests)
RSpec->>RailsApp: GET /posts_page
RailsApp->>DB: query posts and comments
RailsApp-->>RSpec: HTTP 200 with rendered HTML (posts or "No posts found")
Estimated Code Review Effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related issues
Possibly related PRs
Suggested reviewers
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Greptile SummaryThis PR fixes the
Confidence Score: 4/5Safe to merge; the fix is well-scoped and targets a clear root cause without touching production application logic. The workflow condition, step ordering, and seeds logic are all correct. The two findings are robustness concerns that do not affect CI correctness for the benchmark fix. seeds.rb (constant redefinition warnings) and posts_page_spec.rb (schema guard scope) warrant a quick look. Important Files Changed
Reviews (1): Last reviewed commit: "fix(benchmarks): seed Pro dummy DB so /p..." | Re-trigger Greptile |
There was a problem hiding this comment.
🧹 Nitpick comments (1)
react_on_rails_pro/spec/dummy/db/seeds.rb (1)
46-49: ⚡ Quick winBreak the long create! call across multiple lines.
The combined line exceeds the 120-character limit. As per coding guidelines, Ruby code should have a max line length of 120 characters.
♻️ Proposed refactor to improve readability
- posts << user.posts.create!( - title: lorem_sentence.call(3, seed), - body: Array.new(3) { |p| lorem_paragraph.call(4, seed + p) }.join("\n\n") - ) + posts << user.posts.create!( + title: lorem_sentence.call(3, seed), + body: Array.new(3) { |p| lorem_paragraph.call(4, seed + p) } + .join("\n\n") + )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@react_on_rails_pro/spec/dummy/db/seeds.rb` around lines 46 - 49, The single-line user.posts.create! call is over 120 chars; split the arguments onto multiple lines to satisfy the line-length rule and improve readability: place the opening parenthesis after create!, put title: lorem_sentence.call(3, seed) on its own line, body: Array.new(3) { |p| lorem_paragraph.call(4, seed + p) }.join("\n\n") on its own line, and close the parenthesis on a new line; keep the surrounding posts << ... wrapper and preserve the use of create!, title, body, lorem_sentence, and lorem_paragraph.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@react_on_rails_pro/spec/dummy/db/seeds.rb`:
- Around line 46-49: The single-line user.posts.create! call is over 120 chars;
split the arguments onto multiple lines to satisfy the line-length rule and
improve readability: place the opening parenthesis after create!, put title:
lorem_sentence.call(3, seed) on its own line, body: Array.new(3) { |p|
lorem_paragraph.call(4, seed + p) }.join("\n\n") on its own line, and close the
parenthesis on a new line; keep the surrounding posts << ... wrapper and
preserve the use of create!, title, body, lorem_sentence, and lorem_paragraph.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: f713fefe-fd04-4c04-b8a1-4dc0524f4704
📒 Files selected for processing (3)
.github/workflows/benchmark.ymlreact_on_rails_pro/spec/dummy/db/seeds.rbreact_on_rails_pro/spec/dummy/spec/requests/posts_page_spec.rb
Code Review - SummaryClean, targeted fix for the benchmark 100% failure rate on /posts_page. Root cause diagnosis is accurate, changes are minimal and well-scoped. Three small items to consider before merge (see inline comments for specifics). What is good:
Issues (see inline comments):
No concerns on:
|
The Pro benchmark suite reported `/posts_page` at a 100% failure rate (5xx on every request) while every other route was green. Root cause: the benchmark workflow never creates/migrates/seeds the production database, so the `posts` table does not exist and the route's first query raises ActiveRecord::StatementInvalid (no such table: posts). `/posts_page` is the only synchronous benchmark route that reads the database directly. The streaming RSC posts routes hit the same broken DB but return HTTP 200 because the status line is flushed before the body errors, so the failure only surfaced here. Changes: - benchmark.yml: add a "Prepare and seed benchmark database" step (RAILS_ENV=production `rails db:prepare`) before the server starts, gated to the Pro Rails suite. db:prepare creates the DB, loads the schema, and runs the seeds. - db/seeds.rb: rewrite to be deterministic and free of the faker gem. faker is a development/test-only dependency and is not loaded under RAILS_ENV=production, so the previous seeds raised NameError there. Deterministic data also keeps benchmark runs reproducible. - spec/requests/posts_page_spec.rb: new integration test that seeds records and asserts /posts_page server-renders the posts and returns 200 (and 200, not 500, with an empty table), guarding against a regression. The spec self-loads the schema since the suite leaves maintain_test_schema! disabled. Fixes #3602 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Address PR review findings on #3615: - benchmark.yml: db:prepare only seeds on first DB creation, so an already-present-but-empty DB would silently skip seeding and serve a 200 "No posts found" payload that the benchmark scores as healthy. Replace the unconditional success echo with a `rails runner` row-count check that fails the job loudly if zero posts were seeded, and correct the step comment to state the seed-on-creation semantics. - posts_page_spec.rb: the before block always loads the schema, so the spec never reproduces the missing-table 500 its header claimed to guard. Reword the header to accurately describe what it covers (the render + empty-data paths, with the missing-table 500 prevented upstream by seeding the benchmark DB). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
24b50e8 to
0ed8ebd
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/benchmark.yml:
- Line 408: Replace direct expansion of ${{ matrix.suite_name }} in the echo
step by routing the matrix value through an environment variable: set an env key
(e.g., SUITE_NAME: ${{ matrix.suite_name }}) on the step and change the command
to echo "🌱 Preparing and seeding $SUITE_NAME benchmark database..." so the
workflow uses the env variable instead of inline template expansion (refer to
the existing pattern that uses env variables for matrix values and update the
step that currently references matrix.suite_name).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: aaccc71e-98cb-464c-aac8-1a423ff16481
📒 Files selected for processing (3)
.github/workflows/benchmark.ymlreact_on_rails_pro/spec/dummy/db/seeds.rbreact_on_rails_pro/spec/dummy/spec/requests/posts_page_spec.rb
🚧 Files skipped from review as they are similar to previous changes (2)
- react_on_rails_pro/spec/dummy/spec/requests/posts_page_spec.rb
- react_on_rails_pro/spec/dummy/db/seeds.rb
Code Review — PR #3615Root cause analysis and fix are solid. The PR correctly identifies that Workflow (
|
Apply non-blocking review feedback from PR #3615 (claude, greptile, coderabbit). No behavior change — seeded data is byte-for-byte identical. - seeds.rb: replace top-level constants with local variables so re-running `rails db:seed` in an already-loaded process no longer emits `warning: already initialized constant`; cycle per-record counts through precomputed arrays so the modulo stays correct if a range is ever made exclusive or non-unit-step. - benchmark.yml: route `matrix.suite_name` through an env var in the "Prepare and seed" step (injection hardening), matching the adjacent "Execute benchmark suite" step. - posts_page_spec.rb: broaden the self-loading schema guard to check all three tables the examples touch (users/posts/comments), and document why the H1 assertion parses with Nokogiri while the title checks use substrings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review follow-up — addressed in
|
|
The rationale for the two non-changes is also sound:
The PR looks good to merge. Approving. (ノ◕ヮ◕)ノ*:・゚✧ 🧠 Learnings used✅ Action performedComments resolved and changes approved. |
|
All addressed cleanly. The changes in
The rationale for the two non-changes is also sound: LGTM on |
Code ReviewOverall AssessmentSolid, well-scoped fix. The root cause analysis is accurate ( benchmark.yml — CI step
seeds.rb — deterministic rewriteClean rewrite. Using local variables (rather than constants) to avoid Nit: The block parameter body: Array.new(3) { |p| lorem_paragraph.call(4, seed + p) }.join("\n\n")Rename to posts_page_spec.rb — new integration test
SummaryNo blocking issues. The one actionable nit is the |
| seed = (user_index * 11) + post_index | ||
| posts << user.posts.create!( | ||
| title: lorem_sentence.call(3, seed), | ||
| body: Array.new(3) { |p| lorem_paragraph.call(4, seed + p) }.join("\n\n") |
There was a problem hiding this comment.
The block parameter p shadows Ruby's built-in p kernel method, which Rubocop flags under Naming/BlockParameterName. Consider renaming to para_idx or pi:
| body: Array.new(3) { |p| lorem_paragraph.call(4, seed + p) }.join("\n\n") | |
| body: Array.new(3) { |para_idx| lorem_paragraph.call(4, seed + para_idx) }.join("\n\n") |
Address review feedback on PR #3615: - benchmark.yml: use `set -euo pipefail` and guard every table /posts_page reads (users and comments, not just posts) so a partial seed can no longer pass the benchmark job green. - Add spec/dummy/spec/db/seeds_spec.rb: loads db/seeds.rb with the Faker constant hidden (reproducing RAILS_ENV=production) and asserts the tables populate deterministically. This catches a Faker reintroduction or empty seed on every Pro dummy spec run — the layer that actually broke in #3602 previously had no automated coverage. - Trim over-verbose comments in benchmark.yml, seeds.rb, and posts_page_spec.rb. The CI-gating gap that lets Pro-dummy-only changes skip these specs is pre-existing and tracked separately in #3633. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Addressed multi-agent review feedback in 7e4519b. I (Claude Code) made the following changes: Hardening (
New coverage (
Comment trims — removed over-verbose comments in Follow-up: the dummy integration jobs in |
| # green. `set -o pipefail` keeps this abort fatal even if the command | ||
| # is ever piped. | ||
| bundle exec rails runner 'counts = { posts: Post.count, users: User.count, comments: Comment.count }; empty = counts.select { |_, c| c.zero? }.keys; abort("Benchmark DB seeding incomplete after db:prepare (empty: #{empty.join(", ")}); seeds did not run") if empty.any?; puts "✅ Database seeded (#{counts.map { |k, v| "#{v} #{k}" }.join(", ")})"' | ||
|
|
There was a problem hiding this comment.
This ~250-character one-liner is correct but will be painful to read or edit later. rails runner - reads from stdin, so a heredoc keeps the YAML width manageable and the Ruby readable:
| bundle exec rails runner - <<'RUBY' | |
| counts = { posts: Post.count, users: User.count, comments: Comment.count } | |
| empty = counts.select { |_, c| c.zero? }.keys | |
| abort("Benchmark DB seeding incomplete after db:prepare (empty: #{empty.join(', ')}); seeds did not run") if empty.any? | |
| puts "✅ Database seeded (#{counts.map { |k, v| "#{v} #{k}" }.join(', ')})" | |
| RUBY |
| # the shell), matching the "Execute benchmark suite" step below. | ||
| env: | ||
| RAILS_ENV: production | ||
| NODE_ENV: production |
There was a problem hiding this comment.
NODE_ENV isn't read by rails db:prepare or rails runner. Dropping it avoids implying this step has a Node.js dependency.
| NODE_ENV: production | |
| NODE_ENV: production # not needed for db:prepare/rails runner; safe to remove |
Or simply omit the line entirely.
| # green. `set -o pipefail` keeps this abort fatal even if the command | ||
| # is ever piped. |
There was a problem hiding this comment.
set -euo pipefail was already set on line 410 of this block, so pipefail is already active for every subsequent command — including this one. The comment implies the flag is being applied specifically here, which is misleading. Consider removing it, since set -e at the top already covers the abort-on-error behavior for this command.
Code ReviewOverall: Well-structured fix with solid testing. Three minor nits in the workflow step (see inline comments). What this PR doesFixes a 100% 5xx rate on the Strengths
Issues (see inline comments for details)
Test coverage notes
SummaryThe fix is correct. The three issues above are all in the workflow step and are minor — the only one worth changing before merge is the one-liner (readability will matter the next time someone touches this step). |
Pro Node Renderer Benchmark Summary
🔴 significant regression · 🟢 significant improvement (vs baseline; tracked measures only) |
Pro (shard 1/2) Benchmark Summary
🔴 significant regression · 🟢 significant improvement (vs baseline; tracked measures only) |
Pro (shard 2/2) Benchmark Summary
🔴 significant regression · 🟢 significant improvement (vs baseline; tracked measures only) |
Core Benchmark Summary
🔴 significant regression · 🟢 significant improvement (vs baseline; tracked measures only) |
* origin/main: (21 commits) docs(benchmarks): archive RSC FOUC ShakaPerf investigation feat(generator): scaffold native RSCRspackPlugin for rspack RSC fix(pro): admit react-on-rails-rsc RC prereleases in peer range ci(benchmarks): richer summary table fix(benchmarks): seed Pro dummy DB so /posts_page stops 500ing (#3602) (#3615) fix(benchmark): detect backgrounded server/renderer crashes during readiness and benchmark (#3579) refactor: harden ShakapackerConfigHelpers module (#3626) (#3629) fix: classify renderer connection failures separately from bundle errors (#3614) Align docs demo naming and version guidance (#3598) Harden async props pending specs (#3600) docs(ci): link latest/minimum coupling comment to tracking issue #3622 (#3623) Use probe-only route for ShakaPerf FOUC gate (#3630) [codex] Add ShakaPerf RSC FOUC release gate (#3617) refactor: consolidate Shakapacker bundler/config diagnostics helpers (#3619) docs(pro): document CI pattern for renderer-backed Rails tests (#3612) Update RSC rollout to react-on-rails-rsc 19.0.5-rc.6 (#3577) fix(pro): polish rolling-deploy auto-route prefix + hash constraint (#3548) (#3611) fix(pro): log unmountComponentAtNode teardown failure at error level (#3592) (#3610) ci(rspack): decouple per-bundler build + integration jobs so one bundler's build failure doesn't skip the other (#3593) (#3613) ci: stop benchmarks/tests running on internal-docs & issue-template-only PRs (#3608) ... # Conflicts: # CHANGELOG.md
Summary
Fixes the Pro benchmark's
/posts_pageroute, which reported a 100% failure rate (5xx on every request) while every other route was green (see #3602 and the #3586 benchmark comment).Root cause
The benchmark workflow builds production assets and starts the server, but never creates, migrates, or seeds the production database. So
db/production.sqlite3has nopoststable, and the route's first query raises:/posts_pageis the only synchronous (non-streaming) benchmark route that reads the database directly, so it's the only one where the failure surfaces as a 5xx. The streaming RSC posts routes (rsc_posts_page_over_http,rsc_posts_page_over_redis) hit the same broken DB but return HTTP200because the status line is flushed before the body errors — which is why they showed0%fail.Verified locally: with the tables present but empty the route returns
200("No posts found"); only a missing table produces the 500.Changes
.github/workflows/benchmark.yml— add a Prepare and seed benchmark database step (RAILS_ENV=production bundle exec rails db:prepare) before the server starts, gated to the Pro Rails suite (server_kind == 'rails' && pro_env == 'true').db:preparecreates the DB, loads the schema, and runs the seeds on first creation.react_on_rails_pro/spec/dummy/db/seeds.rb— rewrite to be deterministic and free of thefakergem.fakerlives in the:development, :testbundler groups and is not loaded underRAILS_ENV=production, so the old seeds raisedNameError: uninitialized constant Fakerwhen run by the benchmark. Deterministic data also keeps benchmark runs reproducible run-to-run.react_on_rails_pro/spec/dummy/spec/requests/posts_page_spec.rb— new integration test (per the issue's request) that seeds records and asserts/posts_pageserver-renders the posts and returns200, plus a case asserting200(not500) with an empty table. The spec self-loads the schema because the suite intentionally leavesmaintain_test_schema!disabled.Side effect (one-time main re-baseline)
rsc_posts_page_over_redisfetches its data via the in-process Rails DB, so with a seeded DB it now renders real posts/comments/users instead of erroring out fast. Its timing will shift on the first main-push benchmark after merge (a one-time Bencher re-baseline).rsc_posts_page_over_httpis unaffected — it fetches from a hard-codedhttp://localhost:3000, which is not the benchmark server (port 3001), so it never fetched real data regardless.Testing
RAILS_ENV=production rails db:preparenow seeds cleanly (10 users / 50 posts / 173 comments) and theposts_pagecontroller assembles posts with nested comments + users.ssr-generated/server-bundle.js). The CI Pro rspec job builds the bundle and runs the node renderer, so SSR completes there.bundle exec rubocopclean on the changed Ruby files;actionlintclean on the new workflow step.Fixes #3602
🤖 Generated with Claude Code
Note
Low Risk
Benchmark CI and dummy-app seeds/tests only; no production app auth or payment paths.
Overview
Fixes Pro benchmark 100% 5xx on
/posts_pageby ensuring the production DB exists and is populated before the server starts, and by making seeds safe underRAILS_ENV=production.CI (
.github/workflows/benchmark.yml): For Pro Rails suites only, runsrails db:prepareafter asset build and before the server, then aborts ifposts,users, orcommentscounts are zero—so a missing table or skipped seed fails the job instead of looking like a healthy empty page.Seeds (
db/seeds.rb): Replaces Faker-based random data with deterministic lorem-style content sodb:prepareworks without the dev/test-onlyfakergem and benchmark payloads stay reproducible.Tests: Adds
db/seeds_spec.rb(loads seeds withFakerhidden; checks counts and determinism) andposts_page_spec.rb(SSR returns 200 with seeded posts; empty DB returns 200 with "No posts found", not 500).Reviewed by Cursor Bugbot for commit 7e4519b. Bugbot is set up for automated code reviews on this repo. Configure here.
Summary by CodeRabbit
Tests
Chores
Merge decision / follow-up
Minor benchmark cleanup nits are tracked in #3638 so this approved/green reliability fix can merge without restarting CI for readability-only changes.