EC2 Driver Relies on undefined refresh! method of Aws::Credentials in at least some cases

[tas50]

@timvisher commented on Thu Apr 29 2021

EC2 Driver Relies on undefined refresh! method of Aws::Credentials in at least some cases

Some credentials classes are non-refreshing and when those are in use
test-kitchen may trigger an inappropriate call.

Version

$ chef -v
Chef Workstation version: 21.4.365
Chef Infra Client version: 16.12.3
Chef InSpec version: 4.29.3
Chef CLI version: 4.0.0
Chef Habitat version: 1.6.288
Test Kitchen version: 2.11.2
Cookstyle version: 7.10.0

I tested this down to the latest 20 release as well.

Environment

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal

Doesn't seem to have anything to do with the contents of the kitchen run.

Scenario

Use EC2 test platforms and have a decent error message if the clock is skewed.

Steps to Reproduce

I'm happy to go to the trouble of providing a little Vagrantfile/.kitchen.yml reproduction repo if there's interest in fixing this. Didn't want to go that far until someone told me I should. :)

1. Launch an Ubuntu 20.04 instance

2. Install chef-workstation like:

   curl -L https://omnitruck.chef.io/install.sh |
     bash -s -- -c stable -P chef-workstation -v 21
   

3. Setup a .kitchen.yml file with an EC2 platform. Something like:

   - name: ec2-20.04
     driver:
       name: ec2
       image_search:
         owner-id: "099720109477"
         name: "ubuntu/images/*/ubuntu-*-20.04*"
   

4. Login to the Ubuntu instance, skew the clock, and go for a create.
   Something like:

   sudo timedatectl set-ntp no
   sudo timedatectl set-time 12:17:13 # skewed past 15 minutes
   aws sts get-caller-identity
   # An error occurred (SignatureDoesNotMatch) when calling the GetCallerIdentity operation: Signature expired: 20210429T121720Z is now earlier than 20210429T170527Z (20210429T172027Z - 15 min.)
   kitchen create test-ec2-2004
   # -----> Starting Test Kitchen (v2.11.2)
   # -----> Creating <test-ec2-2004>...
   # >>>>>> ------Exception-------
   # >>>>>> Class: Kitchen::ActionFailed
   # >>>>>> Message: 1 actions failed.
   # >>>>>>     Failed to complete #create action: [undefined method `refresh!' for #<Aws::SharedCredentials:0x0000000002b97290>] on test-ec2-2004
   # >>>>>> ----------------------
   # >>>>>> Please see .kitchen/logs/kitchen.log for more details
   # >>>>>> Also try running `kitchen diagnose --all` for configuration
   

5. Unskew the clock and retry the create action:

   sudo timedatectl set-ntp yes
   sudo systemctl restart systemd-timesyncd.service
   aws sts get-caller-identity
   # …
   kitchen create test-ec2-2004
   # -----> Starting Test Kitchen (v2.11.2)
   # -----> Creating <test-ec2-2004>...
   #        Detected platform: ubuntu version 16.04 on x86_64. Instance Type: m5.large. Default username: ubuntu (default).
   # …
   

Expected Result

Anything but a call to an undefined method by the AWS SDK.

Actual Result

E, [2021-04-29T12:17:23.206405 #25650] ERROR -- Kitchen: ------Exception-------
E, [2021-04-29T12:17:23.206671 #25650] ERROR -- Kitchen: Class: Kitchen::ActionFailed
E, [2021-04-29T12:17:23.206909 #25650] ERROR -- Kitchen: Message: 1 actions failed.
>>>>>>     Failed to complete #create action: [undefined method `refresh!' for #<Aws::SharedCredentials:0x0000000002ba4a58>] on test-ec2-2004
E, [2021-04-29T12:17:23.207086 #25650] ERROR -- Kitchen: ----------------------
E, [2021-04-29T12:17:23.207162 #25650] ERROR -- Kitchen: ------Backtrace-------
E, [2021-04-29T12:17:23.207218 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/command.rb:181:in `report_errors'
E, [2021-04-29T12:17:23.207274 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/command.rb:172:in `run_action'
E, [2021-04-29T12:17:23.207332 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/command/action.rb:35:in `block in call'
E, [2021-04-29T12:17:23.207385 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/2.7.0/benchmark.rb:293:in `measure'
E, [2021-04-29T12:17:23.207442 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/command/action.rb:33:in `call'
E, [2021-04-29T12:17:23.207617 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/cli.rb:52:in `perform'
E, [2021-04-29T12:17:23.207664 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/cli.rb:198:in `block (2 levels) in <class:CLI>'
E, [2021-04-29T12:17:23.207700 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/command.rb:27:in `run'
E, [2021-04-29T12:17:23.207798 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/invocation.rb:127:in `invoke_command'
E, [2021-04-29T12:17:23.208033 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor.rb:392:in `dispatch'
E, [2021-04-29T12:17:23.209106 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/thor-1.1.0/lib/thor/base.rb:485:in `start'
E, [2021-04-29T12:17:23.209277 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/bin/kitchen:11:in `block in <top (required)>'
E, [2021-04-29T12:17:23.209557 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/lib/kitchen/errors.rb:170:in `with_friendly_errors'
E, [2021-04-29T12:17:23.209841 #25650] ERROR -- Kitchen: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/test-kitchen-2.11.2/bin/kitchen:11:in `<top (required)>'
E, [2021-04-29T12:17:23.210094 #25650] ERROR -- Kitchen: /usr/bin/kitchen:354:in `load'
E, [2021-04-29T12:17:23.210174 #25650] ERROR -- Kitchen: /usr/bin/kitchen:354:in `<main>'
E, [2021-04-29T12:17:23.210233 #25650] ERROR -- Kitchen: ----End Backtrace-----

Additional context

I'm willing to bet dollars to pay that this is actually an upstream issue
in https://github.com/aws/aws-sdk-ruby but I'm not sure whether it would
make more sense to defend against it here while simultaneously opening a
bug report there.

[ashiqueps]

Hi,

The above issue is raised from the aws-sdk-core gem and I've created the above issue in their organization to handle this error. Since there is no option to refresh the credentials automatically, showing the relevant error message to the user will help him identify the issue, rather than showing the exception backtrace.

[ashiqueps]

The PR - aws/aws-sdk-ruby#2686 for fixing the issue in the aws-sdk-core has been merged and a new version of the aws-sdk-core gem has been released(3.130.1) containing a fix for this issue.

[timvisher]

@ashiqueps This is great! :)

Am I missing the release of kitchen-ec2 to bring in the latest dependency, though? It seems like closing the issue here before actually releasing the fix is premature?

[ashiqueps]

Hey @timvisher,

Sorry that I've closed this issue as soon as the aws-sdk-core gem was published. I'll reopen this issue and will close it once a new version of workstation is released.

[timvisher]

I more just assumed I'd missed something. I have no authority here. :)

Thanks again for chasing down the bug!