Skip to content

Include prerelease gems in compact index during incremental update #108

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hsbt merged 3 commits into from
May 7, 2026
Merged

Include prerelease gems in compact index during incremental update #108

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d2a9a66
Include prerelease gems in compact index during incremental update
sudoremo Mar 9, 2026
c30dc55
Use assert_equal for info/d prerelease assertions
sudoremo Apr 8, 2026
6428295
Fix test for prerelease info line on Ruby < 3.5.0
sudoremo Apr 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion lib/rubygems/indexer.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -450,7 +450,7 @@ def update_index

if @build_compact
Gem.time "Updated compact index files" do
files += update_compact_index released, @dest_directory, @directory
files += update_compact_index specs, @dest_directory, @directory
end
end

Expand Down
3 changes: 2 additions & 1 deletion test/rubygems/test_gem_indexer.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -462,11 +462,12 @@ def test_update_index
assert_equal <<~INFO_FILE, File.read(File.join(infodir, "d"))
#{info_d_file.chomp}
2.1 |checksum:#{file_sha256(File.join(gems, "d-2.1.gem"))}
2.2.a |checksum:#{file_sha256(File.join(gems, "d-2.2.a.gem"))}#{",rubygems:> 1.3.1" if Gem::VERSION < "3.5.0"}
INFO_FILE

assert_equal <<~VERSIONS_FILE, File.read(File.join(@indexerdir, "versions"))
#{versions_file.chomp}
d 2.1 #{file_md5(File.join(@indexerdir, "info", "d"))}
d 2.1,2.2.a #{file_md5(File.join(@indexerdir, "info", "d"))}
Copy link
Copy Markdown
Contributor

@simi simi Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, test is super long and not simple to follow. I'm not 100% sure this is correct, since new version should be on separate line (appended) when modifying existing file to keep it partially cacheable. And compacted later on demand. Not sure this tests appends or does new index from scratch. Are there tests for both case? I can take a look later to understand the tests if needed.

Copy link
Copy Markdown
Contributor Author

@sudoremo sudoremo Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good question! This test exercises the incremental update_index path (not a full rebuild). Both d-2.1 (released) and d-2.2.a (prerelease) are added in the same update_index call, so compact_index_gems groups them into a single CompactIndex::Gem, and VersionsFile#contents appends one line per gem per update — hence d 2.1,2.2.a on a single line.

This is consistent with how a full generate_index works too — see test_build_indices where the versions file has d 2.0,2.0.a,2.0.a-x86-linux,2.0.b on a single line.

The info/d file does correctly have each version on its own line (appended to the existing content via update_compact_index at line 524).

So to summarize the test coverage:

  • Full rebuild: test_build_indices / test_generate_index
  • Incremental update (append): test_update_index — this is the one changed here

Copy link
Copy Markdown
Contributor

@simi simi Apr 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what happens on rubygems.org, in there new gems are appended (until compacting happening each month).

https://github.com/rubygems/rubygems.org/blob/edcc71a94163f50e9c95f7467a8d505600ce770d/app/jobs/upload_versions_file_job.rb#L27C34-L27C42

It is to make versions file cacheable at least for one calendar month and you're downloading only increments at that time. If this is testing appending, I assume it should create new line. But it seems this is not new issue with your addition, so maybe ok to accept this for now. I can check in separate PR to add also incremental additions (maybe as an option?).

e 1 #{file_md5(File.join(@indexerdir, "info", "e"))}
VERSIONS_FILE

Expand Down
Loading