Retry CreateOnGithubJob on GitHub auth 401s

Deployment/status creation surfaces transient Octokit::Unauthorized when a
GitHub installation token is rejected or still propagating. CommitDeployment#create_on_github!
only rescues NotFound/Forbidden, so the 401 escaped the job unhandled and
reopened the Observe issue.

Add retry_on Octokit::Unauthorized to CreateOnGithubJob with polynomially_longer
backoff and attempts: 6 (~15m window) so transient auth failures settle before
giving up. On exhaustion, log and do not re-raise, matching the existing
NotFound/Forbidden give-up behavior. No token cache or client changes.

Fixes shop/issues#8801

Author: tdickers

app/jobs/shipit/create_on_github_job.rb

@@ -7,6 +7,16 @@ class CreateOnGithubJob < BackgroundJob
     queue_as :default
     on_duplicate :drop
 
+    # GitHub installation-token rejection / propagation lag surfaces as a transient 401.
+    # Retry on a backoff so it settles before we give up, instead of crashing the job.
+    retry_on Octokit::Unauthorized, wait: :polynomially_longer, attempts: 6 do |job, exception|
+      record = job.arguments.first
+      Rails.logger.warn(
+        "[CreateOnGithubJob] Giving up on #{record.class.name} #{record.id} " \
+          "after GitHub authentication failures: #{exception.class} #{exception.message}",
+      )
+    end
+
     # We observe that some objects regularly take longer than the default 10 seconds to create, e.g. deployments
     self.timeout = 40
     self.lock_timeout = 20

test/jobs/shipit/create_on_github_job_test.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+require 'test_helper'
+
+module Shipit
+  class CreateOnGithubJobTest < ActiveSupport::TestCase
+    setup do
+      @deployment = shipit_commit_deployments(:shipit_pending_fourth)
+    end
+
+    test "#perform retries on GitHub authentication errors" do
+      CommitDeployment.any_instance.stubs(:create_on_github!).raises(Octokit::Unauthorized)
+
+      assert_enqueued_with(job: CreateOnGithubJob) do
+        CreateOnGithubJob.perform_now(@deployment)
+      end
+    end
+
+    test "#perform gives up without re-raising after exhausting authentication retries" do
+      CommitDeployment.any_instance.stubs(:create_on_github!).raises(Octokit::Unauthorized)
+      Rails.logger.stubs(:warn)
+
+      job = CreateOnGithubJob.new(@deployment)
+      job.exception_executions = { "[Octokit::Unauthorized]" => 5 }
+
+      assert_nothing_raised { job.perform_now }
+    end
+  end
+end