-
-
Notifications
You must be signed in to change notification settings - Fork 629
fix(benchmarks): seed Pro dummy DB so /posts_page stops 500ing (#3602) #3615
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
2576776
0ed8ebd
4980451
7e4519b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -387,6 +387,36 @@ jobs: | |||||||||||||||
|
|
||||||||||||||||
| echo "✅ Production assets built successfully" | ||||||||||||||||
|
|
||||||||||||||||
| - name: Prepare and seed benchmark database | ||||||||||||||||
| # The Pro Rails suite serves DB-backed routes (e.g. /posts_page). Without | ||||||||||||||||
| # a migrated + seeded database those routes 500 on every request (the | ||||||||||||||||
| # `posts` table does not exist), so the benchmark must create and seed it | ||||||||||||||||
| # before the server starts. On a fresh CI runner `db:prepare` creates the | ||||||||||||||||
| # database, loads the schema, and — because the database is newly created — | ||||||||||||||||
| # runs db/seeds.rb (deterministic and faker-free so it works under | ||||||||||||||||
| # RAILS_ENV=production). Note: `db:prepare` only seeds on first creation, so | ||||||||||||||||
| # the row-count check below fails the job loudly if seeding was skipped (an | ||||||||||||||||
| # empty table would otherwise serve a 200 "No posts found" payload that the | ||||||||||||||||
| # benchmark scores as healthy). | ||||||||||||||||
| if: matrix.server_kind == 'rails' && matrix.pro_env == 'true' | ||||||||||||||||
| working-directory: ${{ matrix.app_directory }} | ||||||||||||||||
| # Pass the matrix value via env (treated as data, not interpolated into | ||||||||||||||||
| # the shell), matching the "Execute benchmark suite" step below. | ||||||||||||||||
| env: | ||||||||||||||||
| RAILS_ENV: production | ||||||||||||||||
| NODE_ENV: production | ||||||||||||||||
| SUITE_NAME: ${{ matrix.suite_name }} | ||||||||||||||||
| run: | | ||||||||||||||||
| set -euo pipefail | ||||||||||||||||
| echo "🌱 Preparing and seeding $SUITE_NAME benchmark database..." | ||||||||||||||||
| bundle exec rails db:prepare | ||||||||||||||||
| # Guard every table `/posts_page` reads (posts, plus the users and | ||||||||||||||||
| # comments it joins), not just posts: a partial seed would otherwise | ||||||||||||||||
| # let the route 500 (or render an empty page) while the job stayed | ||||||||||||||||
| # green. `set -o pipefail` keeps this abort fatal even if the command | ||||||||||||||||
| # is ever piped. | ||||||||||||||||
|
Comment on lines
+416
to
+417
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||||||||||||||||
| bundle exec rails runner 'counts = { posts: Post.count, users: User.count, comments: Comment.count }; empty = counts.select { |_, c| c.zero? }.keys; abort("Benchmark DB seeding incomplete after db:prepare (empty: #{empty.join(", ")}); seeds did not run") if empty.any?; puts "✅ Database seeded (#{counts.map { |k, v| "#{v} #{k}" }.join(", ")})"' | ||||||||||||||||
|
|
||||||||||||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This ~250-character one-liner is correct but will be painful to read or edit later.
Suggested change
|
||||||||||||||||
| - name: Start Pro node renderer | ||||||||||||||||
| if: matrix.server_kind == 'node-renderer' | ||||||||||||||||
| working-directory: ${{ matrix.app_directory }} | ||||||||||||||||
|
|
||||||||||||||||
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,48 +1,70 @@ | ||||||
| # frozen_string_literal: true | ||||||
|
|
||||||
| # This file should contain all the record creation needed to seed the database with its default values. | ||||||
| # The data can then be loaded with the rake db:seed (or created alongside the db with db:setup). | ||||||
| # Seed data for the dummy app. The benchmark suite's `/posts_page` route renders | ||||||
| # these records, and they are handy for local development too. | ||||||
| # | ||||||
| # Examples: | ||||||
| # | ||||||
| # cities = City.create([{ name: 'Chicago' }, { name: 'Copenhagen' }]) | ||||||
| # Mayor.create(name: 'Emanuel', city: cities.first) | ||||||
| # This file is intentionally deterministic and free of the `faker` gem: it must | ||||||
| # run under `RAILS_ENV=production` in CI (via `rails db:prepare`), where the | ||||||
| # development/test-only `faker` gem is not loaded. Deterministic data also keeps | ||||||
| # benchmark runs reproducible from one CI run to the next. | ||||||
|
|
||||||
| # Local variables (not top-level constants) so re-running `rails db:seed` in an | ||||||
| # already-loaded process — seeds.rb is `load`ed, not `require`d — does not emit | ||||||
| # `warning: already initialized constant`. Per-record counts index into these | ||||||
| # arrays, so cycling stays correct regardless of each range's bounds or step. | ||||||
| user_count = 10 | ||||||
| post_counts = (3..7).to_a | ||||||
| comment_counts = (2..5).to_a | ||||||
|
|
||||||
| lorem = %w[ | ||||||
| lorem ipsum dolor sit amet consectetur adipiscing elit sed do eiusmod tempor | ||||||
| incididunt ut labore et dolore magna aliqua enim ad minim veniam quis nostrud | ||||||
| exercitation ullamco laboris nisi aliquip ex ea commodo consequat duis aute | ||||||
| irure reprehenderit voluptate velit esse cillum fugiat nulla pariatur | ||||||
| ].freeze | ||||||
|
|
||||||
| lorem_words = lambda do |count, offset| | ||||||
| Array.new(count) { |i| lorem[(offset + i) % lorem.size] } | ||||||
| end | ||||||
| lorem_sentence = ->(word_count, offset) { "#{lorem_words.call(word_count, offset).join(' ').capitalize}." } | ||||||
| lorem_paragraph = lambda do |sentence_count, offset| | ||||||
| Array.new(sentence_count) { |i| lorem_sentence.call(6 + ((offset + i) % 6), offset + (i * 7)) }.join(" ") | ||||||
| end | ||||||
|
|
||||||
| # Clear existing data | ||||||
| puts "Clearing existing data..." | ||||||
| Comment.delete_all | ||||||
| Post.delete_all | ||||||
| User.delete_all | ||||||
|
|
||||||
| # Create Users | ||||||
| puts "Creating users..." | ||||||
| 10.times do | ||||||
| User.create!( | ||||||
| name: Faker::Name.name, | ||||||
| email: Faker::Internet.unique.email | ||||||
| ) | ||||||
| users = Array.new(user_count) do |i| | ||||||
| User.create!(name: "User #{i + 1}", email: "user-#{i + 1}@example.com") | ||||||
| end | ||||||
|
|
||||||
| # Create Posts | ||||||
| puts "Creating posts..." | ||||||
| User.all.each do |user| | ||||||
| rand(3..7).times do | ||||||
| user.posts.create!( | ||||||
| title: Faker::Lorem.sentence(word_count: 3), | ||||||
| body: Faker::Lorem.paragraphs(number: 3).join("\n\n") | ||||||
| posts = [] | ||||||
| users.each_with_index do |user, user_index| | ||||||
| post_count = post_counts[user_index % post_counts.size] | ||||||
| post_count.times do |post_index| | ||||||
| seed = (user_index * 11) + post_index | ||||||
| posts << user.posts.create!( | ||||||
| title: lorem_sentence.call(3, seed), | ||||||
| body: Array.new(3) { |p| lorem_paragraph.call(4, seed + p) }.join("\n\n") | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The block parameter
Suggested change
|
||||||
| ) | ||||||
| end | ||||||
| end | ||||||
|
|
||||||
| # Create Comments | ||||||
| puts "Creating comments..." | ||||||
| Post.all.each do |post| | ||||||
| rand(2..5).times do | ||||||
| posts.each_with_index do |post, post_index| | ||||||
| comment_count = comment_counts[post_index % comment_counts.size] | ||||||
| comment_count.times do |comment_index| | ||||||
| seed = (post_index * 7) + comment_index | ||||||
| post.comments.create!( | ||||||
| user: User.all.sample, | ||||||
| body: Faker::Lorem.paragraph | ||||||
| user: users[seed % users.size], | ||||||
| body: lorem_paragraph.call(2, seed) | ||||||
| ) | ||||||
| end | ||||||
| end | ||||||
|
|
||||||
| puts "Seed data created successfully!" | ||||||
| puts "Seed data created successfully! " \ | ||||||
| "(#{User.count} users, #{Post.count} posts, #{Comment.count} comments)" | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| # frozen_string_literal: true | ||
|
|
||
| require "rails_helper" | ||
|
|
||
| # Unit coverage for `db/seeds.rb`, the layer that actually broke in issue #3602. | ||
| # | ||
| # The Pro benchmark workflow runs `rails db:prepare` under RAILS_ENV=production, | ||
| # where the development/test-only `faker` gem is NOT in the bundle. If seeds.rb | ||
| # reintroduces a `Faker::...` call (or otherwise stops populating the tables), | ||
| # the production CI run leaves `/posts_page` with no data — exactly the #3602 | ||
| # failure. These examples reproduce that environment by hiding the `Faker` | ||
| # constant while loading the seed file, so such a regression fails here (on every | ||
| # Pro dummy spec run) instead of silently in the benchmark suite. | ||
| # | ||
| # Unlike posts_page_spec.rb this needs no node renderer — it only touches the DB. | ||
| RSpec.describe "db/seeds.rb" do | ||
| before do | ||
| # maintain_test_schema! is disabled in this suite (see posts_page_spec.rb), so | ||
| # load the schema if the tables the seeds touch are missing. | ||
| unless %i[users posts comments].all? { |t| ActiveRecord::Base.connection.table_exists?(t) } | ||
| ActiveRecord::Schema.verbose = false | ||
| load Rails.root.join("db/schema.rb") | ||
| end | ||
| end | ||
|
|
||
| after do | ||
| Comment.delete_all | ||
| Post.delete_all | ||
| User.delete_all | ||
| end | ||
|
|
||
| # seeds.rb is `load`ed (not `require`d) in production via db:prepare; do the | ||
| # same here. It clears the tables itself, so each load starts from a clean slate. | ||
| # Its progress `puts` output is captured to keep spec output quiet. | ||
| def load_seeds | ||
| original_stdout = $stdout | ||
| $stdout = StringIO.new | ||
| load Rails.root.join("db/seeds.rb") | ||
| ensure | ||
| $stdout = original_stdout | ||
| end | ||
|
|
||
| it "populates users, posts, and comments without the faker gem" do | ||
| hide_const("Faker") | ||
|
|
||
| expect { load_seeds }.not_to raise_error | ||
|
|
||
| expect(User.count).to be_positive | ||
| expect(Post.count).to be_positive | ||
| expect(Comment.count).to be_positive | ||
| end | ||
|
|
||
| it "produces identical data on every load (deterministic, reproducible benchmarks)" do | ||
| hide_const("Faker") | ||
|
|
||
| load_seeds | ||
| first = { users: User.count, posts: Post.count, comments: Comment.count, | ||
| titles: Post.order(:id).pluck(:title) } | ||
|
|
||
| load_seeds | ||
| second = { users: User.count, posts: Post.count, comments: Comment.count, | ||
| titles: Post.order(:id).pluck(:title) } | ||
|
|
||
| expect(second).to eq(first) | ||
| end | ||
| end |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| # frozen_string_literal: true | ||
|
|
||
| require "rails_helper" | ||
|
|
||
| # Integration coverage for the `/posts_page` benchmark route (issue #3602). | ||
| # | ||
| # `/posts_page` renders DB records synchronously (non-streaming), so when the | ||
| # `posts` table was missing in the Pro benchmark suite it surfaced as a hard 500 | ||
| # on every request. (The streaming RSC posts routes mask that class of failure | ||
| # because their HTTP 200 status line is flushed before the body errors.) | ||
| # | ||
| # These examples do not reproduce the missing-table case itself — the `before` | ||
| # block loads the schema (see below), so the table is always present here. The | ||
| # missing-table 500 is prevented upstream by seeding the benchmark DB before the | ||
| # server starts (.github/workflows/benchmark.yml). What these examples guard is | ||
| # the layer above that: with the table in place the route must server-render | ||
| # seeded posts and return 200, and an empty table must return 200 "No posts | ||
| # found" rather than 500. | ||
| # | ||
| # Requires the Pro node renderer to be running, like the other server-rendering | ||
| # request specs: the page is rendered with `prerender: true`. | ||
| RSpec.describe "Posts page", :server_rendering do | ||
| before do | ||
| # The suite intentionally leaves `maintain_test_schema!` disabled, so make | ||
| # this DB-backed spec self-sufficient: load the schema if it is not present | ||
| # yet. Check every table the examples touch (not just `posts`) so a partial | ||
| # schema can't slip past the guard. | ||
| unless %i[users posts comments].all? { |t| ActiveRecord::Base.connection.table_exists?(t) } | ||
| ActiveRecord::Schema.verbose = false | ||
| load Rails.root.join("db/schema.rb") | ||
| end | ||
|
|
||
| Comment.delete_all | ||
| Post.delete_all | ||
| User.delete_all | ||
|
|
||
| 2.times do |i| | ||
| user = User.create!(name: "User #{i + 1}", email: "user-#{i + 1}@example.com") | ||
| post = user.posts.create!(title: "Sentinel Post #{i + 1}", body: "Body of sentinel post #{i + 1}.") | ||
| post.comments.create!(user: user, body: "Comment on sentinel post #{i + 1}.") | ||
| end | ||
| end | ||
|
|
||
| after do | ||
| Comment.delete_all | ||
| Post.delete_all | ||
| User.delete_all | ||
| end | ||
|
|
||
| it "server-renders the seeded posts and returns 200" do | ||
| get "/posts_page" | ||
|
|
||
| expect(response).to have_http_status(:ok) | ||
|
|
||
|
justin808 marked this conversation as resolved.
|
||
| # Parse for the heading so we assert "Posts Page" renders specifically in an | ||
| # <h1> (the page chrome), not just anywhere in the body. The seeded titles | ||
| # below are plain-text content, so a raw substring match is sufficient there. | ||
| html = Nokogiri::HTML(response.body) | ||
| expect(html.css("h1").map(&:text)).to include("Posts Page") | ||
| expect(response.body).to include("Sentinel Post 1") | ||
| expect(response.body).to include("Sentinel Post 2") | ||
| end | ||
|
|
||
| it "returns 200 (not 500) when there are no posts to render" do | ||
| Comment.delete_all | ||
| Post.delete_all | ||
| User.delete_all | ||
|
|
||
| get "/posts_page" | ||
|
|
||
| expect(response).to have_http_status(:ok) | ||
| expect(response.body).to include("No posts found") | ||
| end | ||
| end | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NODE_ENVisn't read byrails db:prepareorrails runner. Dropping it avoids implying this step has a Node.js dependency.Or simply omit the line entirely.