Retry CreateOnGithubJob on GitHub auth 401s

Deployment/status creation surfaces transient Octokit::Unauthorized when a
GitHub installation token is rejected or still propagating. CommitDeployment#create_on_github!
only rescues NotFound/Forbidden, so the 401 escaped the job unhandled and
reopened the Observe issue.

Add retry_on Octokit::Unauthorized to CreateOnGithubJob with polynomially_longer
backoff and attempts: 6 (~15m window) so transient auth failures settle before
giving up. On exhaustion, log and do not re-raise, matching the existing
NotFound/Forbidden give-up behavior. No token cache or client changes.

Fixes shop/issues#8801

Author: tdickers

CHANGELOG.md

@@ -1,5 +1,9 @@
 # Unreleased
 
+# 0.45.3
+* Retry CreateOnGithubJob on transient GitHub authentication failures.
+* Stabilize PerformTaskJob tests by stubbing the task execution strategy instead of Command#stream!.
+
 # 0.45.2
 * (bugfix) Fix 404 error when removing all permissions from an API client
 

app/jobs/shipit/create_on_github_job.rb

@@ -7,6 +7,16 @@ class CreateOnGithubJob < BackgroundJob
     queue_as :default
     on_duplicate :drop
 
+    # GitHub installation-token rejection / propagation lag surfaces as a transient 401.
+    # Retry on a backoff so it settles before we give up, instead of crashing the job.
+    retry_on Octokit::Unauthorized, wait: :polynomially_longer, attempts: 6 do |job, exception|
+      record = job.arguments.first
+      Rails.logger.warn(
+        "[CreateOnGithubJob] Giving up on #{record.class.name} #{record.id} " \
+          "after GitHub authentication failures: #{exception.class} #{exception.message}"
+      )
+    end
+
     # We observe that some objects regularly take longer than the default 10 seconds to create, e.g. deployments
     self.timeout = 40
     self.lock_timeout = 20

lib/shipit/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Shipit
-  VERSION = '0.45.2'
+  VERSION = '0.45.3'
 end

test/jobs/perform_task_job_test.rb

@@ -107,7 +107,7 @@ def success?
     end
 
     test "mark deploy as error an unexpected exception is raised" do
-      Command.any_instance.expects(:stream!).at_least_once.raises(Command::Denied)
+      Shipit::TaskExecutionStrategy::Default.any_instance.expects(:capture!).at_least_once.raises(Command::Denied)
 
       @job.perform(@deploy)
 
@@ -116,7 +116,7 @@ def success?
     end
 
     test "mark deploy as timedout if a command timeout" do
-      Command.any_instance.expects(:stream!).at_least_once.raises(Command::TimedOut)
+      Shipit::TaskExecutionStrategy::Default.any_instance.expects(:capture!).at_least_once.raises(Command::TimedOut)
 
       @job.perform(@deploy)
 
@@ -129,7 +129,7 @@ def success?
       begin
         Shipit.timeout_exit_codes = [70].freeze
 
-        Command.any_instance.expects(:stream!).at_least_once.raises(Command::Failed.new('Blah', 70))
+        Shipit::TaskExecutionStrategy::Default.any_instance.expects(:capture!).at_least_once.raises(Command::Failed.new('Blah', 70))
 
         @job.perform(@deploy)
 

test/jobs/shipit/create_on_github_job_test.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+require 'test_helper'
+
+module Shipit
+  class CreateOnGithubJobTest < ActiveSupport::TestCase
+    setup do
+      @deployment = shipit_commit_deployments(:shipit_pending_fourth)
+    end
+
+    test "#perform retries on GitHub authentication errors" do
+      CommitDeployment.any_instance.stubs(:create_on_github!).raises(Octokit::Unauthorized)
+
+      assert_enqueued_with(job: CreateOnGithubJob) do
+        CreateOnGithubJob.perform_now(@deployment)
+      end
+    end
+
+    test "#perform gives up without re-raising after exhausting authentication retries" do
+      CommitDeployment.any_instance.stubs(:create_on_github!).raises(Octokit::Unauthorized)
+      Rails.logger.stubs(:warn)
+
+      job = CreateOnGithubJob.new(@deployment)
+      job.exception_executions = { "[Octokit::Unauthorized]" => 5 }
+
+      assert_nothing_raised { job.perform_now }
+    end
+  end
+end