Skip to content

Commit c7a6dd0

Browse files
committed
fix: comments
Signed-off-by: Uroš Marolt <uros@marolt.me>
1 parent 053aefe commit c7a6dd0

4 files changed

Lines changed: 21 additions & 16 deletions

File tree

services/apps/packages_worker/src/deps-dev/README.md

Lines changed: 9 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -68,13 +68,12 @@ The override logic lives in `src/deps-dev/activities/bqExportToGcs.ts`.
6868

6969
## Environment variables
7070

71-
| Variable | Required | Purpose |
72-
| ------------------------------ | --------- | -------------------------------------------------------------------------------------------------- |
73-
| `OSSPCKGS_GCP_PROJECT` | yes | GCP project ID for BQ and GCS |
74-
| `OSSPCKGS_GCS_BUCKET` | yes | GCS bucket for Parquet exports |
75-
| `OSSPCKGS_GCP_CREDENTIALS_B64` | yes | Base64-encoded GCP service account JSON |
76-
| `OSSPCKGS_DEPS_TABLE` | no | Set to `B` to use `DependenciesLatest` (ADR-0003 Option B) instead of `DependencyGraphEdgesLatest` |
77-
| `CROWD_TEMPORAL_SERVER_URL` | yes | Temporal server address |
78-
| `CROWD_TEMPORAL_NAMESPACE` | yes | Temporal namespace (overrides `backend-config` default) |
79-
| `CROWD_TEMPORAL_CERTIFICATE` | prod only | Base64-encoded mTLS client certificate |
80-
| `CROWD_TEMPORAL_PRIVATE_KEY` | prod only | Base64-encoded mTLS private key |
71+
| Variable | Required | Purpose |
72+
| ------------------------------ | --------- | ------------------------------------------------------- |
73+
| `OSSPCKGS_GCP_PROJECT` | yes | GCP project ID for BQ and GCS |
74+
| `OSSPCKGS_GCS_BUCKET` | yes | GCS bucket for Parquet exports |
75+
| `OSSPCKGS_GCP_CREDENTIALS_B64` | yes | Base64-encoded GCP service account JSON |
76+
| `CROWD_TEMPORAL_SERVER_URL` | yes | Temporal server address |
77+
| `CROWD_TEMPORAL_NAMESPACE` | yes | Temporal namespace (overrides `backend-config` default) |
78+
| `CROWD_TEMPORAL_CERTIFICATE` | prod only | Base64-encoded mTLS client certificate |
79+
| `CROWD_TEMPORAL_PRIVATE_KEY` | prod only | Base64-encoded mTLS private key |

services/apps/packages_worker/src/deps-dev/activities/bqExportToGcs.ts

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -198,7 +198,10 @@ export async function bqExportToGcs(input: BqExportToGcsInput): Promise<BqExport
198198
const dryRunBytes = Number(dryRunJob.metadata.statistics.totalBytesProcessed ?? 0)
199199
// Log the effective ceiling (env override may differ from the default maxBytesGb) and the
200200
// computed byte ceiling, so ops can see what the abort decision is actually compared against.
201-
log.info({ jobKind, dryRunBytes, maxBytesGb, effectiveMaxBytesGb, ceiling }, 'BQ dry-run complete')
201+
log.info(
202+
{ jobKind, dryRunBytes, maxBytesGb, effectiveMaxBytesGb, ceiling },
203+
'BQ dry-run complete',
204+
)
202205
if (dryRunBytes > ceiling) {
203206
throw new Error(
204207
`BQ dry-run for ${jobKind} reports ${dryRunBytes} bytes > ceiling ${ceiling} — aborting`,

services/apps/packages_worker/src/deps-dev/config.ts

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,6 @@ export const GCS_BUCKET = requireEnv('OSSPCKGS_GCS_BUCKET')
1212
export const DEPS_DEV_DATASET = 'bigquery-public-data.deps_dev_v1'
1313
export const SCORECARD_DATASET = 'openssf.scorecardcron'
1414

15-
// ADR-0003: Option A = DependencyGraphEdgesLatest (prod default, has version_constraint).
16-
// Set OSSPCKGS_DEPS_TABLE=B locally to use DependenciesLatest (cheaper, no version_constraint).
17-
export const DEPS_TABLE_OPTION: 'A' | 'B' = process.env.OSSPCKGS_DEPS_TABLE === 'B' ? 'B' : 'A'
18-
1915
const credentials = JSON.parse(
2016
Buffer.from(requireEnv('OSSPCKGS_GCP_CREDENTIALS_B64'), 'base64').toString('utf8'),
2117
)

services/apps/packages_worker/src/deps-dev/workflows/ingestDependencies.ts

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -233,7 +233,14 @@ export async function ingestDependencies(opts: {
233233
// scanning the live snapshot. Both full (*Latest = newest snapshot) and incremental can hit
234234
// a bad snapshot, so the guard runs for both. Probes only resolved-graph ecosystems; a clean
235235
// GO/NUGET-only run finds no canaries and passes through.
236-
if (!opts.reuseExports) {
236+
//
237+
// Option A only: the guard's canary ratios are DependencyGraphEdges-schema-specific. Option B
238+
// ingests the separate Dependencies/DependenciesLatest table, which the 2026-06 corruption was
239+
// never observed in and has no calibrated baseline — probing Edges there would "validate" a table
240+
// we don't ingest (false confidence) and could abort a healthy Option B run when only Edges is bad.
241+
// Option B is a manual, non-scheduled cost-experiment path (--deps-table-b); leave it unguarded by
242+
// design rather than invent a guard for an unproven threat. Option A is the production default.
243+
if (!opts.reuseExports && tableOption === 'A') {
237244
const guard = await checkEdgeSnapshotQuality({ snapshotDate: opts.today, ecosystems, fullScan })
238245
if (!guard.ok) {
239246
throw ApplicationFailure.nonRetryable(

0 commit comments

Comments
 (0)