Fix installing gems with native extensions + transitive dependencies

[nicholasdower]

What was the end-user or developer problem that led to this PR?

I am seeing the following error during bundle install:

Gem::MissingSpecError: Could not find 'ffi' (>= 1.15.5) among 48 total gem(s) (Gem::MissingSpecError)

This is reproducible with:

source 'https://rubygems.org'

gem 'llhttp-ffi'

A binary search of this repo indicates the issue may have been introduced in #9381.

It seems only direct dependencies are checked when determining whether a Gem with native extensions is ready to install. I believe this can lead to a failure if a transitive dependency is not yet installed.

In the example above, llhttp-ffi depends on ffi-compiler, which depends on ffi. Since ffi-compiler has no extensions, it is installed immediately without waiting for ffi. When llhttp-ffi then checks its direct dependencies, ffi-compiler is already installed, so llhttp-ffi starts building its native extension. The build requires ffi, which may not have been installed yet.

What is your fix for the problem, implemented in this PR?

Check transitive dependencies for Gems with native extensions.

Make sure the following tasks are checked

• Describe the problem / feature
• Write tests for features and bug fixes
• Write code to solve the problem
• Make sure you follow the current code style and write meaningful commit messages without tags

[nicholasdower]

cc: @Edouard-chin

[Edouard-chin]

Thank you for catching this and opening a fix ! Not sure how I missed this case 🤦 .
The only concern I have with this solution is that all gems with native extensions now have to pay the cost of waiting for all their dependencies (direct + transitive) to be installed. I think native ext gems that needs a transitive dependencies to be installtable are the minority, and I'd like to explore a different solution.

What I have in mind is basically to install any gems without waiting (even the one with native extensions), and if an error arise during the installation of a gem with native exts, we put it back in the queue and set a flag so that we only retry the installation once all its dependencies have been installed. Many gems with native extensions don't need any dependencies to be installed, even direct ones, so that will not penalize them.

I'll try this experiment this week, and if that doesn't work for whatever reason, let's go with your approach !