Skip to content

Commit b92d4d0

Browse files
committed
docs: sync deps incremental byte-ceiling note for rubygems (CM-1296)
Signed-off-by: Uroš Marolt <uros@marolt.me>
1 parent ec0e676 commit b92d4d0

1 file changed

Lines changed: 4 additions & 3 deletions

File tree

services/apps/packages_worker/src/deps-dev/workflows/ingestDependencies.ts

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -285,9 +285,10 @@ export async function ingestDependencies(opts: {
285285
syncMode: fullScan ? 'full' : opts.syncMode,
286286
snapshotAt: opts.today,
287287
// Full/fill scan the *Latest views (everything) → 25000. Incremental is a snapshot edge-diff
288-
// (today vs watermark partitions of DependencyGraphEdges + GoRequirements + NuGetRequirements);
289-
// measured ~4.1TB for Option A. 10000 leaves ~2.4x headroom and still trips a runaway full-table
290-
// scan. Overridable via BQ_DATASET_INGEST_PACKAGE_DEPENDENCIES[_INCREMENTAL]_MAX_BQ_GB (see README).
288+
// (today vs watermark partitions of DependencyGraphEdges + GoRequirements + NuGetRequirements +
289+
// RubyGemsRequirements); measured ~4.1TB for Option A (pre-RubyGems, which adds a small increment).
290+
// 10000 leaves ~2.4x headroom and still trips a runaway full-table scan. Overridable via
291+
// BQ_DATASET_INGEST_PACKAGE_DEPENDENCIES[_INCREMENTAL]_MAX_BQ_GB (see README).
291292
maxBytesGb: fullScan ? 25000 : 10000,
292293
reuseExports: opts.reuseExports,
293294
exportName: opts.exportName,

0 commit comments

Comments
 (0)