bundler: Fix Bundler::Fetcher for PQC support, adding integration connection tests#9637
bundler: Fix Bundler::Fetcher for PQC support, adding integration connection tests#9637junaruga wants to merge 3 commits into
Conversation
26faa7a to
1f63f18
Compare
|
CI ruby-core cases failed. https://github.com/ruby/rubygems/actions/runs/28041447077/job/83008544582?pr=9637 |
I am testing by adding the 3rd commit. |
0284301 to
e578fea
Compare
I fixed the ruby-core cases. In CI Bundler on macOS ruby-4.0, ruby-3.2 cases, the following non-PQC/PQC tests failed. Investigating. https://github.com/ruby/rubygems/actions/runs/28050419021/job/83039282611?pr=9637#step:12:109 |
eca36c3 to
c245713
Compare
|
@hsbt To investigate the above issue that the testing SSL server returning HTTP status 403, I ran the CI on my fork repository with the following change to check the details. However, I couldn't find the failures on CI so far when running the CI 2 times. The logs are below.
I also see the CI on this PR is failing on Bunder on macOS-intel (ruby-4.0) cases. But I don't think the failures are related to this PR. https://github.com/ruby/rubygems/actions/runs/28117909971/job/83262168621?pr=9637 So, could you review this PR? When the above issues happen again, I will investigate again. |
c245713 to
defe1d5
Compare
|
The HTTP 403 issue appeared again after rebasing on the latest master branch. Let me investigate. |
3231648 to
40bd7b3
Compare
|
Maybe I found the cause of the HTTP status 403 issue and fixed it. I debugged with the following debug log on my fork repository. And the log is below. The full log is here. Added localhost to Artifice Because I cannot explain why the issue sometimes happens rather than always. |
40bd7b3 to
1d1474f
Compare
cf3cec0 to
e5c62af
Compare
|
After rebasing now, I see new failures with HTTP status 404 Not Found. Let me fix the issue. https://github.com/ruby/rubygems/actions/runs/28369192398/job/84042446245?pr=9637#step:12:122 |
…nection tests
Create spec/bundler/fetcher/gem_remote_fetcher_local_ssl_server_spec.rb
adding non-PQC and PQC server/client connection integration tests.
As "Bundler::Fetcher local SSL server #connection PQC connects with client cert
auth" failed with the following error due to hardcoded `OpenSSL::PKey::RSA.new` in
`Bundler::Fetcher#connection`, fixed it to support ML-DSA ssl_client_cert.
```
$ bin/rspec spec/bundler/fetcher/gem_remote_fetcher_local_ssl_server_spec.rb
...
Failures:
1) Bundler::Fetcher local SSL server #connection PQC connects with client cert auth
Failure/Error: fetcher = Bundler::Fetcher.new(remote)
OpenSSL::PKey::PKeyError:
incorrect pkey type: UNDEF
# /home/jaruga/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/openssl/pkey.rb:394:in 'OpenSSL::PKey::RSA#initialize'
# /home/jaruga/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/openssl/pkey.rb:394:in 'Class#new'
# /home/jaruga/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/openssl/pkey.rb:394:in 'OpenSSL::PKey::RSA.new'
# ./bundler/lib/bundler/fetcher.rb:321:in 'Bundler::Fetcher#connection'
# ./bundler/lib/bundler/fetcher.rb:140:in 'Bundler::Fetcher#initialize'
# ./spec/bundler/fetcher/gem_remote_fetcher_local_ssl_server_spec.rb:69:in 'RSpec::ExampleGroups::BundlerFetcherLocalSSLServer#fetch_path'
# ./spec/bundler/fetcher/gem_remote_fetcher_local_ssl_server_spec.rb:60:in 'block (4 levels) in <top (required)>'
...
```
Create test/rubygems/local_ssl_server_utilities.rb to manage utility methods
called by RubyGems test-unit and Bundler rspec tests.
Co-Authored-By: Claude <noreply@anthropic.com>
The hardcoded Gem::Net::HTTP in Artifice.deactivate, is actually
the replaced Artifice::Net::HTTP. This doesn't restore the original
Gem::Net::HTTP.
Restore the saved original Gem::Net::HTTP in Artifice.deactivate
This issue caused the tests sometimes failed in
spec/bundler/fetcher/gem_remote_fetcher_local_ssl_server_spec.rb
using Artifice::Net::HTTP unintentionally. Because localhost is not included in
spec/support/artifice/helpers/endpoint.rb - permitted_hosts caused HTTP status 403
(response.body: "Host not permitted"). But Gem::Net::HTTP should be used in the
tests instead of Artifice::Net::HTTP. Possibly this issue happens when
Artifice.deactivate is called in other tests such as spec/commands/ssl_spec.rb.
That's why this issue sometimes happened rather than always.
```
1) Bundler::Fetcher local SSL server #connection PQC connects
Failure/Error: expect(response.code).to eq("200")
expected: #<Encoding:UTF-8> "200"
got: #<Encoding:ASCII-8BIT> "403"
(compared using ==)
# ./spec/bundler/fetcher/gem_remote_fetcher_local_ssl_server_spec.rb:49:in 'block (4 levels) in <top (required)>'
# ./spec/spec_helper.rb:164:in 'block (4 levels) in <top (required)>'
# ./spec/spec_helper.rb:164:in 'block (3 levels) in <top (required)>'
# ./spec/support/helpers.rb:414:in 'block in Spec::Helpers#with_gem_path_as'
# ./spec/support/helpers.rb:428:in 'Spec::Helpers#without_env_side_effects'
# ./spec/support/helpers.rb:409:in 'Spec::Helpers#with_gem_path_as'
# ./spec/spec_helper.rb:163:in 'block (2 levels) in <top (required)>'
# ./lib/rubygems.rb:306:in 'Kernel#load'
# ./lib/rubygems.rb:306:in 'Gem.activate_and_load_bin_path'
```
spec/commands/ssl_spec.rb
```
...
after(:each) do
...
Artifice.deactivate
...
end
...
```
Co-Authored-By: Claude <noreply@anthropic.com>
e5c62af to
339165f
Compare
|
OK. Maybe I found and fixed the issue on the 2nd commit. The issue was in So, I don't need to add localhost to |
|
The following new test failures shows maybe the actual https://github.com/ruby/rubygems/actions/runs/28386154046/job/84101204150?pr=9637 |
This commit fixes the following failures.
The `require "support/artifice/compact_index"` is not enough
because the file is not loaded when it is called second time.
```
1) Bundler::ParallelInstaller connect to make jobserver takes all available slots
Failure/Error: raise HTTPError, e.message
Bundler::HTTPError:
Could not reach host gem.repo2. Check your network connection and try again.
# ./spec/bundler/installer/parallel_installer_spec.rb:151:in 'block (4 levels) in <top (required)>'
# ./spec/bundler/installer/parallel_installer_spec.rb:221:in 'RSpec::ExampleGroups::BundlerParallelInstaller::ConnectToMakeJobserver#redefine_build_jobs'
# ./spec/bundler/installer/parallel_installer_spec.rb:150:in 'block (3 levels) in <top (required)>'
# ./spec/spec_helper.rb:164:in 'block (4 levels) in <top (required)>'
# ./spec/spec_helper.rb:164:in 'block (3 levels) in <top (required)>'
# ./spec/support/helpers.rb:414:in 'block in Spec::Helpers#with_gem_path_as'
# ./spec/support/helpers.rb:428:in 'Spec::Helpers#without_env_side_effects'
# ./spec/support/helpers.rb:409:in 'Spec::Helpers#with_gem_path_as'
# ./spec/spec_helper.rb:163:in 'block (2 levels) in <top (required)>'
# ./lib/rubygems.rb:306:in 'Kernel#load'
# ./lib/rubygems.rb:306:in 'Gem.activate_and_load_bin_path'
# ------------------
# --- Caused by: ---
# Bundler::HTTPError:
# Could not reach host gem.repo2. Check your network connection and try again.
# ./spec/bundler/installer/parallel_installer_spec.rb:151:in 'block (4 levels) in <top (required)>'
...
```
Co-Authored-By: Claude <noreply@anthropic.com>
|
@hsbt I fixed the |
Summary
This PR is related to #9543. The first commit is the same with #9615. I want to see the #9615 is reviewed and merged. After that, I will rebase this PR on the latest master branch. The second commit is this PR's commit.
Created spec/bundler/fetcher/gem_remote_fetcher_local_ssl_server_spec.rb adding non-PQC and PQC server/client connection integration tests.
As "Bundler::Fetcher local SSL server #connection PQC connects with client cert auth" failed with the following error due to hardcoded
OpenSSL::PKey::RSA.newinBundler::Fetcher#connection, fixed it to support ML-DSA ssl_client_cert.Created test/rubygems/local_ssl_server_utilities.rb to manage utility methods
called by RubyGems test-unit and Bundler rspec tests.
Co-Authored-By: Claude noreply@anthropic.com
Files
spec/bundler/fetcher/gem_remote_fetcher_local_ssl_server_spec.rb
I aligned the file structure with
test/rubygems/test_gem_remote_fetcher_local_ssl_server.rb. I also referred tospec/bundler/fetcher/gem_remote_fetcher_spec.rb.test/rubygems/local_ssl_server_utilities.rb
For the file's naming, I referred to
test/rubygems/multifactor_auth_utilities.rb, there are following files as support files, I thinktest/rubygems/*_utilities.rbfile is suitable in this case in my opinion.What was the end-user or developer problem that led to this PR?
Bundler::Fetcher#connectiondoesn't work with ML-DSA ssl_client_cert due to hardcodedOpenSSL::PKey::RSA.new.What is your fix for the problem, implemented in this PR?
Fixed the
Bundler::Fetcher#connectionto support ML-DSA ssl_client_cert.Added integration HTTPS server/client connection tests.
Make sure the following tasks are checked