perf(webapp): parallelize streaming batch-item ingest (#3777)

matt-aitken · claude · web-flow · commit f48c89752cac · 2026-06-12T13:01:29.000+01:00
## Problem The item-streaming endpoint of the two-phase batch API (`POST /api/v3/batches/:batchId/items`) processed streamed items strictly sequentially. For a batch of many large payloads, each offloaded to object storage inline, this serialized N object-store round-trips inside a single request and could exceed Node's default `server.requestTimeout` (300s). The webapp then returned `408`, which the SDK reads as `408 terminated` and retries up to 5 times, turning a slow ingest into a failure that takes tens of minutes to surface. ## Fix Ingest now runs through `p-map` over the NDJSON async iterable with bounded concurrency (`STREAMING_BATCH_INGEST_CONCURRENCY`, default 10): - `p-map` pulls lazily from the stream, so at most `concurrency` items are read and in-flight at once. Peak memory stays bounded to roughly `concurrency × STREAMING_BATCH_ITEM_MAXIMUM_SIZE` and request-body backpressure is preserved. - Set the env to `1` for fully sequential ingestion (escape hatch). ## Why this is safe (ordering and idempotency unchanged) - Ordering derives from each item's index (enqueue `timestamp = batch.createdAt + index`), not enqueue order. - Dedup is atomic per index in `enqueueBatchItem`. - The NDJSON parser now stamps oversized-item markers with their emit position, removing the consumer's sequential `lastIndex` assumption (the only order-dependent bit). - The count-check and conditional-seal path is untouched. ## Scope This speeds up every batch ingested through the streaming endpoint, not just large-payload batches. Each item does a per-item Redis enqueue regardless of size, and those now overlap. Large payloads benefit most because they add an object-store offload round-trip on top of the enqueue. ## Verification Added an integration test (`streamBatchItems.test.ts`) that drives the real service against Postgres + Redis + RunEngine and times a 150-item batch at increasing concurrency. Object-store offload is modelled as a fixed per-item latency (local round-trips are too small to compare meaningfully): ``` runCount=150 large payloads (10ms/item offload): concurrency=1 1739ms concurrency=10 192ms (9.1x faster) concurrency=50 57ms (30.7x faster) small payloads (Redis enqueue only, no offload): concurrency=1 90ms concurrency=10 24ms (3.7x faster) ``` The test asserts correctness at every concurrency (all items accepted, sealed, enqueued exactly once), that parallel ingest beats the sequential floor, and that the small-payload case is strictly faster than sequential, so the win is not specific to large payloads. Also exercised end-to-end over real HTTP against a local server: a 20-item batch (12MB body) ingests and seals, a re-stream of the sealed batch returns `sealed: true` with zero re-accepted items (idempotent retry), and an oversized item still seals at its correct index. Existing coverage stays green: concurrent ingest of a 100-item batch, in-flight processing never exceeding the configured concurrency, concurrent dedup on streaming retry, and emit-position marker indexing. ## Follow-ups (not in this PR) - SDK pre-offload of large item payloads (send `application/store` refs instead of raw blobs) to remove object-store work from the request hot path and shrink the request body. - Optional `server.requestTimeout` bump as a safety net. ## CI fix Added `.github/workflows/codeql.yml` to replace GitHub's automatic ("dynamic") CodeQL scanning. The dynamic setup was failing to upload SARIF results because the auto-generated `GITHUB_TOKEN` lacked the `security-events: write` permission. The explicit workflow grants that permission at the job level and pins all actions to commit SHAs, consistent with the repo's security conventions. ## ✅ Checklist - [ ] I have followed every step in the [contributing guide](https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md) - [ ] The PR title follows the convention. - [ ] I ran and tested the code works --- ## Testing - Integration test (`streamBatchItems.test.ts`) validates correctness and performance at concurrency 1, 10, and 50 for both large and small payloads. - End-to-end verified over real HTTP: 20-item/12MB batch ingests and seals, idempotent retry returns `sealed: true`, oversized item seals at correct index. --- ## Changelog Streaming batch ingest now processes items with bounded concurrency instead of one at a time, so batches of many large payloads ingest far faster and no longer time out. Concurrency is configurable via `STREAMING_BATCH_INGEST_CONCURRENCY` (default 10); set it to 1 for fully sequential ingestion. --- ## Screenshots _[Screenshots]_ 💯 --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
diff --git a/.server-changes/parallel-batch-item-ingest.md b/.server-changes/parallel-batch-item-ingest.md
@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+Streaming batch ingest now processes items with bounded concurrency instead of one at a time, so batches of many large payloads ingest far faster and no longer time out. Concurrency is configurable via `STREAMING_BATCH_INGEST_CONCURRENCY` (default 10); set it to 1 for fully sequential ingestion.
diff --git a/apps/webapp/app/env.server.ts b/apps/webapp/app/env.server.ts
@@ -768,6 +768,10 @@ const EnvironmentSchema = z
     // 2-phase batch API settings
     STREAMING_BATCH_MAX_ITEMS: z.coerce.number().int().default(1_000), // Max items in streaming batch
     STREAMING_BATCH_ITEM_MAXIMUM_SIZE: z.coerce.number().int().default(3_145_728),
+    // Number of streamed batch items ingested concurrently in Phase 2. Peak
+    // in-flight memory per request ≈ this × STREAMING_BATCH_ITEM_MAXIMUM_SIZE,
+    // so raise with care. Set to 1 for fully sequential ingestion.
+    STREAMING_BATCH_INGEST_CONCURRENCY: z.coerce.number().int().positive().default(10),
     BATCH_RATE_LIMIT_REFILL_RATE: z.coerce.number().int().default(100),
     BATCH_RATE_LIMIT_MAX: z.coerce.number().int().default(1200),
     BATCH_RATE_LIMIT_REFILL_INTERVAL: z.string().default("10s"),
diff --git a/apps/webapp/app/routes/api.v3.batches.$batchId.items.ts b/apps/webapp/app/routes/api.v3.batches.$batchId.items.ts
@@ -84,6 +84,7 @@ export async function action({ request, params }: ActionFunctionArgs) {
     const service = new StreamBatchItemsService();
     const result = await service.call(authResult.environment, batchId, itemsIterator, {
       maxItemBytes: env.STREAMING_BATCH_ITEM_MAXIMUM_SIZE,
+      concurrency: env.STREAMING_BATCH_INGEST_CONCURRENCY,
     });
 
     return json(result, { status: 200 });
diff --git a/apps/webapp/app/runEngine/services/streamBatchItems.server.ts b/apps/webapp/app/runEngine/services/streamBatchItems.server.ts
@@ -4,6 +4,7 @@ import {
 } from "@trigger.dev/core/v3";
 import { BatchId } from "@trigger.dev/core/v3/isomorphic";
 import type { BatchItem, RunEngine } from "@internal/run-engine";
+import pMap from "p-map";
 import type { BatchTaskRunStatus } from "@trigger.dev/database";
 import { prisma, type PrismaClientOrTransaction } from "~/db.server";
 import type { AuthenticatedEnvironment } from "~/services/apiAuth.server";
@@ -55,6 +56,8 @@ export function isIdempotentRetrySuccess(
 
 export type StreamBatchItemsServiceOptions = {
   maxItemBytes: number;
+  /** Max items processed concurrently. The route wires this to STREAMING_BATCH_INGEST_CONCURRENCY. */
+  concurrency: number;
 };
 
 export type OversizedItemMarker = {
@@ -68,6 +71,8 @@ export type OversizedItemMarker = {
 export type StreamBatchItemsServiceConstructorOptions = {
   prisma?: PrismaClientOrTransaction;
   engine?: RunEngine;
+  /** Override the payload processor (used in tests to observe ingest concurrency). */
+  payloadProcessor?: BatchPayloadProcessor;
 };
 
 /**
@@ -88,7 +93,7 @@ export class StreamBatchItemsService extends WithRunEngine {
 
   constructor(opts: StreamBatchItemsServiceConstructorOptions = {}) {
     super({ prisma: opts.prisma ?? prisma, engine: opts.engine });
-    this.payloadProcessor = new BatchPayloadProcessor();
+    this.payloadProcessor = opts.payloadProcessor ?? new BatchPayloadProcessor();
   }
 
   /**
@@ -170,94 +175,28 @@ export class StreamBatchItemsService extends WithRunEngine {
           );
         }
 
+        // Process items from the stream with bounded concurrency.
+        //
+        // Ordering and idempotency do NOT depend on processing order:
+        //  - The BatchQueue derives run order from each item's index
+        //    (enqueue timestamp = batch.createdAt + itemIndex), not enqueue order.
+        //  - enqueueBatchItem() dedups atomically per index.
+        // We cap concurrency to bound peak in-flight memory (≈ concurrency ×
+        // maxItemBytes) and to keep backpressure on the request body stream.
+        // p-map pulls lazily from the async iterator — at most `concurrency`
+        // items are read and in flight at once. stopOnError aborts ingestion on
+        // the first failure (the batch is left unsealed; the SDK's retry
+        // re-streams and dedups already-enqueued items).
+        const outcomes = await pMap(
+          itemsIterator,
+          (rawItem) => this.#processItem(rawItem, batchId, environment, batch.runCount),
+          { concurrency: options.concurrency, stopOnError: true }
+        );
+
         let itemsAccepted = 0;
         let itemsDeduplicated = 0;
-        let lastIndex = -1;
-
-        // Process items from the stream
-        for await (const rawItem of itemsIterator) {
-          // Check for oversized item markers from the NDJSON parser
-          if (rawItem && typeof rawItem === "object" && "__batchItemError" in rawItem) {
-            const marker = rawItem as OversizedItemMarker;
-            const itemIndex = marker.index >= 0 ? marker.index : lastIndex + 1;
-
-            const errorMessage = `Batch item payload is too large (${(marker.actualSize / 1024).toFixed(1)} KB). Maximum allowed size is ${(marker.maxSize / 1024).toFixed(1)} KB. Reduce the payload size or offload large data to external storage.`;
-
-            // Enqueue with __error metadata - processItemCallback will detect this
-            // and use TriggerFailedTaskService to create a pre-failed run
-            const batchItem: BatchItem = {
-              task: marker.task,
-              payload: "{}",
-              payloadType: "application/json",
-              options: {
-                __error: errorMessage,
-                __errorCode: "PAYLOAD_TOO_LARGE",
-              },
-            };
-
-            const result = await this._engine.enqueueBatchItem(
-              batchId,
-              environment.id,
-              itemIndex,
-              batchItem
-            );
-
-            if (result.enqueued) {
-              itemsAccepted++;
-            } else {
-              itemsDeduplicated++;
-            }
-            lastIndex = itemIndex;
-            continue;
-          }
-
-          // Parse and validate the item
-          const parseResult = BatchItemNDJSONSchema.safeParse(rawItem);
-          if (!parseResult.success) {
-            throw new ServiceValidationError(
-              `Invalid item at index ${lastIndex + 1}: ${parseResult.error.message}`
-            );
-          }
-
-          const item = parseResult.data;
-          lastIndex = item.index;
-
-          // Validate index is within expected range
-          if (item.index >= batch.runCount) {
-            throw new ServiceValidationError(
-              `Item index ${item.index} exceeds batch runCount ${batch.runCount}`
-            );
-          }
-
-          // Get the original payload type
-          const originalPayloadType = (item.options?.payloadType as string) ?? "application/json";
-
-          // Process payload - offload to R2 if it exceeds threshold
-          const processedPayload = await this.payloadProcessor.process(
-            item.payload,
-            originalPayloadType,
-            batchId,
-            item.index,
-            environment
-          );
-
-          // Convert to BatchItem format with potentially offloaded payload
-          const batchItem: BatchItem = {
-            task: item.task,
-            payload: processedPayload.payload,
-            payloadType: processedPayload.payloadType,
-            options: item.options,
-          };
-
-          // Enqueue the item
-          const result = await this._engine.enqueueBatchItem(
-            batchId,
-            environment.id,
-            item.index,
-            batchItem
-          );
-
-          if (result.enqueued) {
+        for (const outcome of outcomes) {
+          if (outcome === "accepted") {
             itemsAccepted++;
           } else {
             itemsDeduplicated++;
@@ -446,6 +385,112 @@ export class StreamBatchItemsService extends WithRunEngine {
       }
     );
   }
+
+  /**
+   * Process a single streamed batch item: validate it, offload its payload to
+   * object storage if oversized, and enqueue it. Returns whether the item was
+   * newly enqueued ("accepted") or was a duplicate ("deduplicated"). Throws
+   * ServiceValidationError for invalid items, which aborts the stream.
+   *
+   * Safe to run concurrently: enqueueBatchItem() is atomic and order-independent
+   * per item index, and each item carries its own index (real items from the
+   * SDK; oversized markers are stamped by the NDJSON parser).
+   */
+  async #processItem(
+    rawItem: unknown,
+    batchId: string,
+    environment: AuthenticatedEnvironment,
+    runCount: number
+  ): Promise<"accepted" | "deduplicated"> {
+    // Oversized item marker emitted by the NDJSON parser
+    if (rawItem && typeof rawItem === "object" && "__batchItemError" in rawItem) {
+      const marker = rawItem as OversizedItemMarker;
+
+      // Same out-of-range guard as normal items: an oversized item with an
+      // out-of-range index must 4xx rather than create a stray pre-failed run.
+      if (marker.index >= runCount) {
+        throw new ServiceValidationError(
+          `Item index ${marker.index} exceeds batch runCount ${runCount}`
+        );
+      }
+
+      const errorMessage = `Batch item payload is too large (${(marker.actualSize / 1024).toFixed(
+        1
+      )} KB). Maximum allowed size is ${(marker.maxSize / 1024).toFixed(
+        1
+      )} KB. Reduce the payload size or offload large data to external storage.`;
+
+      // Enqueue with __error metadata - processItemCallback will detect this
+      // and use TriggerFailedTaskService to create a pre-failed run
+      const batchItem: BatchItem = {
+        task: marker.task,
+        payload: "{}",
+        payloadType: "application/json",
+        options: {
+          __error: errorMessage,
+          __errorCode: "PAYLOAD_TOO_LARGE",
+        },
+      };
+
+      const result = await this._engine.enqueueBatchItem(
+        batchId,
+        environment.id,
+        marker.index,
+        batchItem
+      );
+
+      return result.enqueued ? "accepted" : "deduplicated";
+    }
+
+    // Parse and validate the item
+    const parseResult = BatchItemNDJSONSchema.safeParse(rawItem);
+    if (!parseResult.success) {
+      const rawIndex = (rawItem as { index?: unknown } | null)?.index;
+      const where = typeof rawIndex === "number" ? `index ${rawIndex}` : "unknown index";
+      throw new ServiceValidationError(
+        `Invalid item at ${where}: ${parseResult.error.message}`
+      );
+    }
+
+    const item = parseResult.data;
+
+    // Validate index is within expected range
+    if (item.index >= runCount) {
+      throw new ServiceValidationError(
+        `Item index ${item.index} exceeds batch runCount ${runCount}`
+      );
+    }
+
+    // Get the original payload type
+    const originalPayloadType = (item.options?.payloadType as string) ?? "application/json";
+
+    // Process payload - offload to object storage if it exceeds threshold
+    const processedPayload = await this.payloadProcessor.process(
+      item.payload,
+      originalPayloadType,
+      batchId,
+      item.index,
+      environment
+    );
+
+    // Convert to BatchItem format with potentially offloaded payload
+    const batchItem: BatchItem = {
+      task: item.task,
+      payload: processedPayload.payload,
+      payloadType: processedPayload.payloadType,
+      options: item.options,
+    };
+
+    // Enqueue the item
+    const result = await this._engine.enqueueBatchItem(
+      batchId,
+      environment.id,
+      item.index,
+      batchItem
+    );
+
+    return result.enqueued ? "accepted" : "deduplicated";
+  }
 }
 
 /**
@@ -587,12 +632,29 @@ export function createNdjsonParserStream(
   let chunks: Uint8Array[] = [];
   let totalBytes = 0;
   let lineNumber = 0;
+  // 0-based position of the next object we emit (parsed item or oversized
+  // marker). The parser is the single sequential point in the pipeline, so this
+  // is the authoritative source of item ordering — downstream consumers can
+  // process items concurrently and must not rely on processing order to derive
+  // an item's index. Used to back-fill an oversized marker's index when it
+  // couldn't be extracted from the (truncated) raw bytes.
+  let emittedCount = 0;
   // When an oversized incomplete line is detected (Case 2), we must discard
   // all remaining bytes of that line until the next newline delimiter.
   let skipUntilNewline = false;
 
   const NEWLINE_BYTE = 0x0a; // '\n'
 
+  /**
+   * Emit a parsed object or marker downstream and advance the emit position.
+   * Every emitted object MUST go through here so `emittedCount` stays aligned
+   * with item position (empty/skipped lines never emit, so they don't count).
+   */
+  function emit(controller: TransformStreamDefaultController<unknown>, obj: unknown): void {
+    controller.enqueue(obj);
+    emittedCount++;
+  }
+
   /**
    * Concatenate all chunks into a single Uint8Array
    */
@@ -675,7 +737,7 @@ export function createNdjsonParserStream(
 
     try {
       const obj = JSON.parse(trimmed);
-      controller.enqueue(obj);
+      emit(controller, obj);
     } catch (err) {
       throw new Error(`Invalid JSON at line ${lineNumber}: ${(err as Error).message}`);
     }
@@ -715,12 +777,12 @@ export function createNdjsonParserStream(
           const extracted = extractIndexAndTask(lineBytes);
           const marker: OversizedItemMarker = {
             __batchItemError: "OVERSIZED",
-            index: extracted.index,
+            index: extracted.index >= 0 ? extracted.index : emittedCount,
             task: extracted.task,
             actualSize: newlineIndex,
             maxSize: maxItemBytes,
           };
-          controller.enqueue(marker);
+          emit(controller, marker);
           lineNumber++;
           continue;
         }
@@ -736,12 +798,12 @@ export function createNdjsonParserStream(
         const extracted = extractIndexAndTask(concatenateChunks());
         const marker: OversizedItemMarker = {
           __batchItemError: "OVERSIZED",
-          index: extracted.index,
+          index: extracted.index >= 0 ? extracted.index : emittedCount,
           task: extracted.task,
           actualSize: totalBytes,
           maxSize: maxItemBytes,
         };
-        controller.enqueue(marker);
+        emit(controller, marker);
         lineNumber++;
         // Clear buffer and skip remaining bytes of this oversized line
         // until the next newline delimiter is found in a subsequent chunk
@@ -768,12 +830,12 @@ export function createNdjsonParserStream(
         const extracted = extractIndexAndTask(concatenateChunks());
         const marker: OversizedItemMarker = {
           __batchItemError: "OVERSIZED",
-          index: extracted.index,
+          index: extracted.index >= 0 ? extracted.index : emittedCount,
           task: extracted.task,
           actualSize: totalBytes,
           maxSize: maxItemBytes,
         };
-        controller.enqueue(marker);
+        emit(controller, marker);
         return;
       }
 
diff --git a/apps/webapp/test/engine/streamBatchItems.test.ts b/apps/webapp/test/engine/streamBatchItems.test.ts
diff --git a/docs/self-hosting/env/webapp.mdx b/docs/self-hosting/env/webapp.mdx
