fix(ent): retry HasSBOM include-edge and PackageVersion bulk inserts on FK race#3042
Open
dgellman wants to merge 5 commits into
Open
fix(ent): retry HasSBOM include-edge and PackageVersion bulk inserts on FK race#3042dgellman wants to merge 5 commits into
dgellman wants to merge 5 commits into
Conversation
… 23503 Signed-off-by: Daniel Gellman <dgellman8@gmail.com>
Signed-off-by: Daniel Gellman <dgellman8@gmail.com>
… and upsertPackage Signed-off-by: Daniel Gellman <dgellman8@gmail.com>
2337bfa to
b936869
Compare
mihaimaruseac
approved these changes
May 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two ingest paths fail intermittently with PostgreSQL FK violations (SQLSTATE 23503) when concurrent transactions are racing on the same parent rows:
IngestHasSBOMswrites M2M edges intobill_of_materials_included_software_packagesetc. that referencepackage_version_idrows committed in a siblingIngestPackagestransaction. When the sibling has not yet committed, the M2M insert fails. Production telemetry has shown ~30–40% of resultingHasSBOMrows ending up with emptyincludedSoftware.IngestPackagesitself fails on itsPackageVersionupsert when a concurrent transaction is mid-flight on the parentPackageNamerow.Both paths now wrap their per-batch
Exec(ctx)in a smallretryOnFKViolationhelper. On a PGforeign_key_violationit retries with bounded backoff (500 ms, then 1 s — 3 attempts total) and honors context cancellation. Non-FK errors propagate immediately.What this does and does not address
This absorbs the race within typical commit-latency windows (~1.5 s). It does not eliminate the race architecturally. A deferred FK constraint (
DEFERRABLE INITIALLY DEFERRED) or atomic package + SBOM ingest in a single transaction would remove it at the source. I deliberately did not bundle a schema migration or transaction-boundary refactor into this PR — those involve maintainer-level decisions about migration sequencing and the public ingest API. Happy to follow up with whichever direction maintainers prefer in a separate PR or issue.Detection
isPGForeignKeyViolationmatches*pq.Error.Code == "23503"specifically, not ent's genericIsConstraintError— unique and check-constraint violations are not retried because they are not expected to become valid on retry.Files
pkg/assembler/backends/ent/backend/retry.go(new): detector + bounded-backoff helper, no ent dependencypkg/assembler/backends/ent/backend/retry_test.go(new): unit tests for both helpers using fabricated*pq.Errorvalues, no DBpkg/assembler/backends/ent/backend/sbom.go: wrap the fourupdateHasSBOMWithInclude*per-batchExeccallspkg/assembler/backends/ent/backend/package.go: wrapPackageVersion.CreateBulkinupsertBulkPackageandPackageVersion.CreateinupsertPackageTest plan
Run on this branch (rebased onto current main):
go build ./...clean (exit 0)go vet ./...clean (exit 0)go test -race -timeout=30s ./... -count=1— 79 packages OK, 0 fail, 0 data racesgolangci-lint run ./pkg/assembler/backends/ent/backend/...— 0 issues in changed files (the 2 staticcheck warnings reported are pre-existing insearch.go, unrelated to this PR)retry_test.goall pass: detection covers wrapped/by-value/doubly-wrapped*pq.Error, plain errors, nil, unique-violation negative case, check-violation negative case; retry behavior covers success-first-attempt, non-retryable propagation, recovery after transient FK, exhaustion after max attempts, and context cancellation