Include prerelease gems in compact index during incremental update

[sudoremo]

Summary

Fixes #48.

In update_index, the call to update_compact_index only received released specs, excluding prerelease gems from the compact index files (versions, info/<gemname>). Since build_compact defaults to true and modern Bundler prefers the compact index over legacy specs files, prerelease gems were effectively invisible after an incremental gem generate_index --update — even though they were correctly added to prerelease_specs.4.8.gz.

The fix passes all specs (both released and prerelease) to update_compact_index, consistent with how build_compact_index already handles both during a full generate_index.

Changes

• lib/rubygems/indexer.rb: Pass specs instead of released to update_compact_index
• test/rubygems/test_gem_indexer.rb: Add assertions for prerelease gem d-2.2.a in compact index after update_index

[sudoremo]

The latest addition should fix all unit tests.

[hsbt]

Please keep the existing assert_equal for info/d and just append assertions for the pre-release entry (2.2.a).

The current change loosens the validation by replacing assert_equal with assert_match/start_with?, which reduces regression coverage.

[sudoremo]

Thanks @hsbt for your feedback. The requested changes have been implemented.

[simi]

Hmm, test is super long and not simple to follow. I'm not 100% sure this is correct, since new version should be on separate line (appended) when modifying existing file to keep it partially cacheable. And compacted later on demand. Not sure this tests appends or does new index from scratch. Are there tests for both case? I can take a look later to understand the tests if needed.

[sudoremo]

Good question! This test exercises the incremental update_index path (not a full rebuild). Both d-2.1 (released) and d-2.2.a (prerelease) are added in the same update_index call, so compact_index_gems groups them into a single CompactIndex::Gem, and VersionsFile#contents appends one line per gem per update — hence d 2.1,2.2.a on a single line.

This is consistent with how a full generate_index works too — see test_build_indices where the versions file has d 2.0,2.0.a,2.0.a-x86-linux,2.0.b on a single line.

The info/d file does correctly have each version on its own line (appended to the existing content via update_compact_index at line 524).

So to summarize the test coverage:

• Full rebuild: test_build_indices / test_generate_index
• Incremental update (append): test_update_index — this is the one changed here

[simi]

This is what happens on rubygems.org, in there new gems are appended (until compacting happening each month).

https://github.com/rubygems/rubygems.org/blob/edcc71a94163f50e9c95f7467a8d505600ce770d/app/jobs/upload_versions_file_job.rb#L27C34-L27C42

It is to make versions file cacheable at least for one calendar month and you're downloading only increments at that time. If this is testing appending, I assume it should create new line. But it seems this is not new issue with your addition, so maybe ok to accept this for now. I can check in separate PR to add also incremental additions (maybe as an option?).

[sudoremo]

@hsbt / @simi May I bump this issue again? The bug is currently causing massive slowdowns on our end, as we release multiple pre-releases daily and fully rebuilding the index takes 30-40 minutes. We would therefore really appreciate a timely release of this bugfix. Many thanks for your work.

[hsbt]

@sudoremo Can you fix test failure at first?

[sudoremo]

@hsbt Good catch — the test failure was caused by a missing version-conditional suffix on the d-2.2.a info line assertion. On Ruby < 3.5.0 (i.e. Ruby 3.0, 3.1, 3.2, JRuby 9.4), prerelease gems include rubygems:> 1.3.1 in their compact index info line, which wasn't accounted for in the new assertion.

Fixed in 6428295 by applying the same #{",rubygems:> 1.3.1" if Gem::VERSION < "3.5.0"} pattern already used at line 143 for the a-3.a entry. Tests should now pass across all CI Ruby versions.

Could you approve the CI run when you get a chance? Since each run requires maintainer approval, iterating on failures is a bit slow on our end.

[sudoremo]

@hsbt / @simi May I bump this issue again? We would be greatful for a timely release. Thanks for having a look at it.

[simi]

@sudoremo the situation is as follows: https://github.com/rubygems/compact_index got archived, probably because it was ported to rubygems/rubygems.org#6404, even it was generic implementation and it was beneficial to keep it independent and discuss potential changes there independent of rubygems.org IMHO. This gem depends (https://github.com/rubygems/rubygems-generate_index/blob/99d4d1f6d22d6e52720e53260b498f9998c76e45/rubygems-generate_index.gemspec#L32) on that one. So I assume this plugin is also soft-deprecated now.

I spoke to RubyGems maintainer and was told, this generator doesn't need to be part of RubyGems (or RubyGems plugin). I do actually agree on that. I have started extraction of it in https://github.com/compact-index/compact_index (for now covering only the original back) and I'm trying to speak to RubyGems.org maintainers to move back to shared solution (and move forward all together). Feel free to open issue in there, I have prototype (not release, pushed yet) for independent gem with CLI to generate compact index in similar way this plugin did.

[hsbt]

@sudoremo I released https://github.com/ruby/rubygems-generate_index/releases/tag/v1.2.0 with your fix. Thanks.

[sudoremo]

Many thanks @hsbt and @simi 🙌