feat(webhook): configurable queue selection for matching runners (#5190)

## Description

A `workflow_job` whose labels match several runner configurations is
always dispatched to the **first** matching queue (after the
`exactMatch` sort). When multiple pools intentionally share a generic
label — e.g. an "any architecture" or "this size or larger" label
spanning several runner configs — every cold scale-up funnels to a
single queue, overloading one pool while equally-valid pools sit idle.
There is currently no way to spread that load.

This adds a configurable **queue selection strategy**, applied to the
*equally-best* matches (those sharing the top `exactMatch` priority
tier):

- **`first`** (default): unchanged — deterministic first match.
- **`random`**: pick one uniformly, spreading jobs across the matching
queues so a single pool's queue does not become a bottleneck.
- **`all`**: dispatch to *every* matching queue — scaling up one runner
per matching pool and letting the first available runner take the job
(speed over cost). GitHub assigns the queued job to exactly one runner;
the losers are reaped by scale-down.

`exactMatch` priority is preserved: `random`/`all` only ever operate
within the highest-priority matching tier, never a lower-priority match.
The strategy applies to standard jobs; dynamic (`ghr-`) label jobs
continue to use the first compliant queue.

### Caveats for `all` (deliberate opt-in)

- Multiplies **instance launches** per job (losers idle until
scale-down's `minimum_running_time_in_minutes`).
- Multiplies **runner registrations** per job, increasing GitHub API
usage — relevant where API rate limits are already a concern.
- Only truly races when `enable_job_queued_check = false` (otherwise
later scale-ups see the job already taken and skip).

## Changes

**Lambda** — new `QUEUE_SELECTION_STRATEGY` env var (validated; defaults
to `first`), read by both the direct webhook and the EventBridge
dispatcher; `selectQueues()` implements `first`/`random`/`all` within
the top-priority matching tier.

**Terraform** — a `queue_selection_strategy` variable (validated
`first`/`random`/`all`, default `first`) on the root and `multi-runner`
modules, threaded through the `webhook` module `config` into the
`direct`/`eventbridge` lambda env var, plus regenerated terraform-docs.

> **RFC note:** Per CONTRIBUTING (discuss major changes first), open
questions for maintainers — happy to adjust:
> - global setting (as implemented) vs. per-runner-config option?
> - naming (`queue_selection_strategy`; values `first`/`random`/`all`)?
> - should `all` (and `random`) extend to the dynamic-label path?

## Test Plan

- Added unit tests in `dispatch.test.ts`: default picks first; `random`
spreads across equally-matching queues (`Math.random` mocked); `random`
preserves `exactMatch` priority; `all` dispatches to every top-tier
match but not lower-priority ones; invalid strategy rejected at config
load.
- `yarn test` (webhook): **40/40 pass**. `yarn build` (ncc typecheck)
passes. ESLint: 0 errors. Prettier: clean.
- `terraform fmt -check` and `terraform validate` pass on the root and
`multi-runner` modules; terraform-docs regenerated.

## Related Issues

Motivation is load distribution across pools that share generic labels
(avoiding single-queue hotspots), and a speed-over-cost option for
large-scale environments. No existing upstream issue — happy to open one
to track the discussion if preferred.

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Author: guicaulada

README.md

@@ -175,6 +175,7 @@ Join our discord community via [this invite link](https://discord.gg/bxgXW8jJGh)
 | <a name="input_pool_runner_owner"></a> [pool\_runner\_owner](#input\_pool\_runner\_owner) | The pool will deploy runners to the GitHub org ID, set this value to the org to which you want the runners deployed. Repo level is not supported. | `string` | `null` | no |
 | <a name="input_prefix"></a> [prefix](#input\_prefix) | The prefix used for naming resources | `string` | `"github-actions"` | no |
 | <a name="input_queue_encryption"></a> [queue\_encryption](#input\_queue\_encryption) | Configure how data on queues managed by the modules is encrypted at REST. Options are encrypted via SSE, non encrypted and via KMS. By default encrypted via SSE is enabled. See for more details the Terraform `aws_sqs_queue` resource https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue. | <pre>object({<br/>    kms_data_key_reuse_period_seconds = number<br/>    kms_master_key_id                 = string<br/>    sqs_managed_sse_enabled           = bool<br/>  })</pre> | <pre>{<br/>  "kms_data_key_reuse_period_seconds": null,<br/>  "kms_master_key_id": null,<br/>  "sqs_managed_sse_enabled": true<br/>}</pre> | no |
+| <a name="input_queue_selection_strategy"></a> [queue\_selection\_strategy](#input\_queue\_selection\_strategy) | Strategy used to pick a queue when multiple runner configurations match a job equally well. `first` keeps the historical deterministic behaviour (the first matching queue by priority). `random` spreads jobs across the matching queues to avoid concentrating load on a single one. `all` scales up one runner per matching queue and lets the first to become available take the job (favouring speed over cost; this multiplies instance launches and runner registrations per job). | `string` | `"first"` | no |
 | <a name="input_redrive_build_queue"></a> [redrive\_build\_queue](#input\_redrive\_build\_queue) | Set options to attach (optional) a dead letter queue to the build queue, the queue between the webhook and the scale up lambda. You have the following options. 1. Disable by setting `enabled` to false. 2. Enable by setting `enabled` to `true`, `maxReceiveCount` to a number of max retries. | <pre>object({<br/>    enabled         = bool<br/>    maxReceiveCount = number<br/>  })</pre> | <pre>{<br/>  "enabled": false,<br/>  "maxReceiveCount": null<br/>}</pre> | no |
 | <a name="input_repository_white_list"></a> [repository\_white\_list](#input\_repository\_white\_list) | List of github repository full names (owner/repo\_name) that will be allowed to use the github app. Leave empty for no filtering. | `list(string)` | `[]` | no |
 | <a name="input_role_path"></a> [role\_path](#input\_role\_path) | The path that will be added to role path for created roles, if not set the environment name will be used. | `string` | `null` | no |

lambdas/functions/webhook/src/ConfigLoader.ts

@@ -87,8 +87,14 @@ abstract class BaseConfig {
   }
 }
 
+export type QueueSelectionStrategy = 'first' | 'random' | 'all';
+
 abstract class MatcherAwareConfig extends BaseConfig {
   matcherConfig: RunnerMatcherConfig[] = [];
+  // How to pick a queue when several runner configs match a job equally well.
+  // 'first' keeps the historical deterministic behaviour; 'random' spreads jobs
+  // across the matching queues to avoid concentrating load on a single one.
+  queueSelectionStrategy: QueueSelectionStrategy = 'first';
 
   protected async loadMatcherConfig(paramPathsEnv: string) {
     if (!paramPathsEnv || paramPathsEnv === 'undefined' || paramPathsEnv === 'null' || !paramPathsEnv.includes(':')) {
@@ -133,6 +139,7 @@ export class ConfigWebhook extends MatcherAwareConfig {
 
   async loadConfig(): Promise<void> {
     this.loadEnvVar(process.env.REPOSITORY_ALLOW_LIST, 'repositoryAllowList', []);
+    this.loadEnvVar(process.env.QUEUE_SELECTION_STRATEGY, 'queueSelectionStrategy', 'first');
 
     await Promise.all([
       this.loadMatcherConfig(process.env.PARAMETER_RUNNER_MATCHER_CONFIG_PATH),
@@ -141,6 +148,7 @@ export class ConfigWebhook extends MatcherAwareConfig {
 
     validateWebhookSecret(this);
     validateRunnerMatcherConfig(this);
+    validateQueueSelectionStrategy(this);
   }
 }
 
@@ -165,9 +173,11 @@ export class ConfigDispatcher extends MatcherAwareConfig {
 
   async loadConfig(): Promise<void> {
     this.loadEnvVar(process.env.REPOSITORY_ALLOW_LIST, 'repositoryAllowList', []);
+    this.loadEnvVar(process.env.QUEUE_SELECTION_STRATEGY, 'queueSelectionStrategy', 'first');
     await this.loadMatcherConfig(process.env.PARAMETER_RUNNER_MATCHER_CONFIG_PATH);
 
     validateRunnerMatcherConfig(this);
+    validateQueueSelectionStrategy(this);
   }
 }
 
@@ -188,3 +198,12 @@ function validateRunnerMatcherConfig(config: ConfigDispatcher | ConfigWebhook):
     config.configLoadingErrors.push('Matcher config is empty');
   }
 }
+
+function validateQueueSelectionStrategy(config: ConfigDispatcher | ConfigWebhook): void {
+  const allowed: QueueSelectionStrategy[] = ['first', 'random', 'all'];
+  if (!allowed.includes(config.queueSelectionStrategy)) {
+    config.configLoadingErrors.push(
+      `Invalid queue selection strategy '${config.queueSelectionStrategy}', expected one of: ${allowed.join(', ')}`,
+    );
+  }
+}

lambdas/functions/webhook/src/modules.d.ts

@@ -4,6 +4,7 @@ declare namespace NodeJS {
     EVENT_BUS_NAME: string;
     PARAMETER_GITHUB_APP_WEBHOOK_SECRET: string;
     PARAMETER_RUNNER_MATCHER_CONFIG_PATH: string;
+    QUEUE_SELECTION_STRATEGY: string;
     REPOSITORY_ALLOW_LIST: string;
     RUNNER_LABELS: string;
     ACCEPT_EVENTS: string;

lambdas/functions/webhook/src/runners/dispatch.test.ts

@@ -179,6 +179,70 @@ describe('Dispatcher', () => {
     });
   });
 
+  describe('queue selection strategy', () => {
+    const twoExactMatches: RunnerConfig = [
+      { ...runnerConfig[0], id: 'q1', matcherConfig: { labelMatchers: [['self-hosted', 'any']], exactMatch: true } },
+      { ...runnerConfig[0], id: 'q2', matcherConfig: { labelMatchers: [['self-hosted', 'any']], exactMatch: true } },
+    ];
+    const jobEvent = (labels: string[]) =>
+      ({
+        ...workFlowJobEvent,
+        workflow_job: { ...workFlowJobEvent.workflow_job, labels },
+      }) as unknown as WorkflowJobEvent;
+
+    it('defaults to the first matching queue', async () => {
+      config = await createConfig(undefined, twoExactMatches);
+      await dispatch(jobEvent(['self-hosted', 'any']), 'workflow_job', config);
+      expect(sendActionRequest).toHaveBeenCalledWith(expect.objectContaining({ queueId: 'q1' }));
+    });
+
+    it('random spreads across equally-matching queues', async () => {
+      process.env.QUEUE_SELECTION_STRATEGY = 'random';
+      config = await createConfig(undefined, twoExactMatches);
+      const rand = vi.spyOn(Math, 'random').mockReturnValue(0.99);
+      await dispatch(jobEvent(['self-hosted', 'any']), 'workflow_job', config);
+      expect(sendActionRequest).toHaveBeenCalledWith(expect.objectContaining({ queueId: 'q2' }));
+      rand.mockRestore();
+    });
+
+    it('random still respects exactMatch priority (never a lower-priority match)', async () => {
+      process.env.QUEUE_SELECTION_STRATEGY = 'random';
+      config = await createConfig(undefined, [
+        { ...runnerConfig[0], id: 'loose', matcherConfig: { labelMatchers: [['self-hosted']], exactMatch: false } },
+        {
+          ...runnerConfig[0],
+          id: 'exact',
+          matcherConfig: { labelMatchers: [['self-hosted', 'any']], exactMatch: true },
+        },
+      ]);
+      const rand = vi.spyOn(Math, 'random').mockReturnValue(0.99);
+      await dispatch(jobEvent(['self-hosted', 'any']), 'workflow_job', config);
+      expect(sendActionRequest).toHaveBeenCalledWith(expect.objectContaining({ queueId: 'exact' }));
+      rand.mockRestore();
+    });
+
+    it('all dispatches to every equally-matching queue but not lower-priority ones', async () => {
+      process.env.QUEUE_SELECTION_STRATEGY = 'all';
+      config = await createConfig(undefined, [
+        { ...runnerConfig[0], id: 'loose', matcherConfig: { labelMatchers: [['self-hosted']], exactMatch: false } },
+        { ...runnerConfig[0], id: 'q1', matcherConfig: { labelMatchers: [['self-hosted', 'any']], exactMatch: true } },
+        { ...runnerConfig[0], id: 'q2', matcherConfig: { labelMatchers: [['self-hosted', 'any']], exactMatch: true } },
+      ]);
+      await dispatch(jobEvent(['self-hosted', 'any']), 'workflow_job', config);
+      expect(sendActionRequest).toHaveBeenCalledTimes(2);
+      expect(sendActionRequest).toHaveBeenCalledWith(expect.objectContaining({ queueId: 'q1' }));
+      expect(sendActionRequest).toHaveBeenCalledWith(expect.objectContaining({ queueId: 'q2' }));
+      expect(sendActionRequest).not.toHaveBeenCalledWith(expect.objectContaining({ queueId: 'loose' }));
+    });
+
+    it('rejects an invalid strategy at config load', async () => {
+      process.env.QUEUE_SELECTION_STRATEGY = 'bogus';
+      ConfigDispatcher.reset();
+      mockSSMResponse(twoExactMatches);
+      await expect(ConfigDispatcher.load()).rejects.toThrow(/queue selection strategy/i);
+    });
+  });
+
   describe('decides can run job based on label and config (canRunJob)', () => {
     it('should accept job with an exact match and identical labels.', () => {
       const workflowLabels = ['self-hosted', 'linux', 'x64', 'ubuntu-latest'];

lambdas/functions/webhook/src/runners/dispatch.ts

@@ -4,7 +4,7 @@ import { WorkflowJobEvent } from '@octokit/webhooks-types';
 import { Response } from '../lambda';
 import { RunnerMatcherConfig, sendActionRequest } from '../sqs';
 import ValidationError from '../ValidationError';
-import { ConfigDispatcher, ConfigWebhook } from '../ConfigLoader';
+import { ConfigDispatcher, ConfigWebhook, QueueSelectionStrategy } from '../ConfigLoader';
 import { violationsAgainstPolicy } from './dynamic-labels-policy';
 
 const logger = createChildLogger('handler');
@@ -19,7 +19,7 @@ export async function dispatch(
 ): Promise<Response> {
   validateRepoInAllowList(event, config);
 
-  return await handleWorkflowJob(event, eventType, config.matcherConfig!);
+  return await handleWorkflowJob(event, eventType, config.matcherConfig!, config.queueSelectionStrategy);
 }
 
 function validateRepoInAllowList(event: WorkflowJobEvent, config: ConfigDispatcher) {
@@ -33,6 +33,7 @@ async function handleWorkflowJob(
   body: WorkflowJobEvent,
   githubEvent: string,
   matcherConfig: Array<RunnerMatcherConfig>,
+  queueSelectionStrategy: QueueSelectionStrategy = 'first',
 ): Promise<Response> {
   if (body.action !== 'queued') {
     return {
@@ -67,17 +68,22 @@ async function handleWorkflowJob(
     return notAccepted(body);
   }
 
-  // 2. Pick a queue.
-  let chosen: RunnerMatcherConfig;
+  // 2. Pick the target queue(s).
+  let targets: RunnerMatcherConfig[];
   let labelsToSend: string[];
 
   if (!hasDynamicLabels) {
-    // No dynamic labels in the job: take the first match, forward as-is.
-    chosen = matches[0];
+    // No dynamic labels in the job: select among the equally-best matches (those
+    // sharing the top priority, i.e. the same exactMatch as the first match)
+    // according to the configured strategy, and forward as-is.
+    const topMatches = matches.filter((q) => q.matcherConfig.exactMatch === matches[0].matcherConfig.exactMatch);
+    targets = selectQueues(topMatches, queueSelectionStrategy);
     labelsToSend = nonGhrLabels;
   } else {
     // Dynamic labels present: prefer the first match that has dynamic labels
-    // enabled AND accepts these labels under its policy.
+    // enabled AND accepts these labels under its policy. The queue selection
+    // strategy applies to standard jobs only; dynamic-label jobs always use the
+    // first compliant queue.
     let compliant: RunnerMatcherConfig | undefined;
     for (const q of matches) {
       if (!q.matcherConfig.enableDynamicLabels) {
@@ -95,7 +101,7 @@ async function handleWorkflowJob(
     }
 
     if (compliant) {
-      chosen = compliant;
+      targets = [compliant];
       labelsToSend = [...nonGhrLabels, ...sanitizedGhrLabels];
     } else {
       // No queue accepts the dynamic labels under its policy: refuse the job.
@@ -106,27 +112,52 @@ async function handleWorkflowJob(
     }
   }
 
-  await sendActionRequest({
-    id: body.workflow_job.id,
-    repositoryName: body.repository.name,
-    repositoryOwner: body.repository.owner.login,
-    eventType: githubEvent,
-    installationId: body.installation?.id ?? 0,
-    queueId: chosen.id,
-    repoOwnerType: body.repository.owner.type,
-    labels: labelsToSend,
-  });
+  await Promise.all(
+    targets.map((queue) =>
+      sendActionRequest({
+        id: body.workflow_job.id,
+        repositoryName: body.repository.name,
+        repositoryOwner: body.repository.owner.login,
+        eventType: githubEvent,
+        installationId: body.installation?.id ?? 0,
+        queueId: queue.id,
+        repoOwnerType: body.repository.owner.type,
+        labels: labelsToSend,
+      }),
+    ),
+  );
 
+  const queueIds = targets.map((q) => q.id).join(', ');
   logger.info(
-    `Successfully dispatched job for ${body.repository.full_name} to the queue ${chosen.id} - ` +
+    `Successfully dispatched job for ${body.repository.full_name} to the queue(s) ${queueIds} - ` +
       `Job ID: ${body.workflow_job.id}, Job Name: ${body.workflow_job.name}, Run ID: ${body.workflow_job.run_id}`,
   );
   return {
     statusCode: 201,
-    body: `Successfully queued job for ${body.repository.full_name} to the queue ${chosen.id}`,
+    body: `Successfully queued job for ${body.repository.full_name} to the queue(s) ${queueIds}`,
   };
 }
 
+/**
+ * Select the target queue(s) from a set of equally-matching candidates.
+ * - 'first'  keeps the historical deterministic choice (the first candidate).
+ * - 'random' picks one uniformly random candidate, spreading jobs across queues
+ *   so a single pool's queue does not become a bottleneck.
+ * - 'all'    returns every candidate, scaling up one runner per matching pool and
+ *   letting the first to become available take the job (speed over cost). Note
+ *   this multiplies instance launches and runner registrations per job.
+ */
+function selectQueues(candidates: RunnerMatcherConfig[], strategy: QueueSelectionStrategy): RunnerMatcherConfig[] {
+  switch (strategy) {
+    case 'all':
+      return candidates;
+    case 'random':
+      return [candidates[Math.floor(Math.random() * candidates.length)]];
+    default:
+      return [candidates[0]];
+  }
+}
+
 function notAccepted(body: WorkflowJobEvent): Response {
   const notAcceptedErrorMsg = `Received event contains runner labels '${body.workflow_job.labels}' from '${
     body.repository.full_name

main.tf

@@ -143,6 +143,7 @@ module "webhook" {
   role_path                 = var.role_path
   role_permissions_boundary = var.role_permissions_boundary
   repository_white_list     = var.repository_white_list
+  queue_selection_strategy  = var.queue_selection_strategy
 
   lambda_subnet_ids         = var.lambda_subnet_ids
   lambda_security_group_ids = var.lambda_security_group_ids

modules/multi-runner/README.md

@@ -158,6 +158,7 @@ module "multi-runner" {
 | <a name="input_pool_lambda_timeout"></a> [pool\_lambda\_timeout](#input\_pool\_lambda\_timeout) | Time out for the pool lambda in seconds. | `number` | `60` | no |
 | <a name="input_prefix"></a> [prefix](#input\_prefix) | The prefix used for naming resources | `string` | `"github-actions"` | no |
 | <a name="input_queue_encryption"></a> [queue\_encryption](#input\_queue\_encryption) | Configure how data on queues managed by the modules in ecrypted at REST. Options are encrypted via SSE, non encrypted and via KMSS. By default encryptes via SSE is enabled. See for more details the Terraform `aws_sqs_queue` resource https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue. | <pre>object({<br/>    kms_data_key_reuse_period_seconds = number<br/>    kms_master_key_id                 = string<br/>    sqs_managed_sse_enabled           = bool<br/>  })</pre> | <pre>{<br/>  "kms_data_key_reuse_period_seconds": null,<br/>  "kms_master_key_id": null,<br/>  "sqs_managed_sse_enabled": true<br/>}</pre> | no |
+| <a name="input_queue_selection_strategy"></a> [queue\_selection\_strategy](#input\_queue\_selection\_strategy) | Strategy used to pick a queue when multiple runner configurations match a job equally well. `first` keeps the historical deterministic behaviour (the first matching queue by priority). `random` spreads jobs across the matching queues to avoid concentrating load on a single one. `all` scales up one runner per matching queue and lets the first to become available take the job (favouring speed over cost; this multiplies instance launches and runner registrations per job). | `string` | `"first"` | no |
 | <a name="input_repository_white_list"></a> [repository\_white\_list](#input\_repository\_white\_list) | List of github repository full names (owner/repo\_name) that will be allowed to use the github app. Leave empty for no filtering. | `list(string)` | `[]` | no |
 | <a name="input_role_path"></a> [role\_path](#input\_role\_path) | The path that will be added to the role; if not set, the environment name will be used. | `string` | `null` | no |
 | <a name="input_role_permissions_boundary"></a> [role\_permissions\_boundary](#input\_role\_permissions\_boundary) | Permissions boundary that will be added to the created role for the lambda. | `string` | `null` | no |

modules/multi-runner/variables.tf

@@ -396,6 +396,16 @@ variable "repository_white_list" {
   default     = []
 }
 
+variable "queue_selection_strategy" {
+  description = "Strategy used to pick a queue when multiple runner configurations match a job equally well. `first` keeps the historical deterministic behaviour (the first matching queue by priority). `random` spreads jobs across the matching queues to avoid concentrating load on a single one. `all` scales up one runner per matching queue and lets the first to become available take the job (favouring speed over cost; this multiplies instance launches and runner registrations per job)."
+  type        = string
+  default     = "first"
+  validation {
+    condition     = contains(["first", "random", "all"], var.queue_selection_strategy)
+    error_message = "`queue_selection_strategy` value not valid. Valid values are 'first', 'random', 'all'."
+  }
+}
+
 variable "log_level" {
   description = "Logging level for lambda logging. Valid values are  'silly', 'trace', 'debug', 'info', 'warn', 'error', 'fatal'."
   type        = string

modules/multi-runner/webhook.tf

@@ -34,6 +34,7 @@ module "webhook" {
   role_path                 = var.role_path
   role_permissions_boundary = var.role_permissions_boundary
   repository_white_list     = var.repository_white_list
+  queue_selection_strategy  = var.queue_selection_strategy
 
   lambda_subnet_ids         = var.lambda_subnet_ids
   lambda_security_group_ids = var.lambda_security_group_ids