Skip to content

Fix vulndb indexes and migrations#1938

Merged
timbastin merged 3 commits into
mainfrom
fix-vulndb-indexes-and-migrations
May 6, 2026
Merged

Fix vulndb indexes and migrations#1938
timbastin merged 3 commits into
mainfrom
fix-vulndb-indexes-and-migrations

Conversation

@Hubtrick-Git
Copy link
Copy Markdown
Collaborator

  • Removed obsolete index on cve_affected_component
  • Added osv test coverage to handle index mismatches in the future
  • Removed all down migration scripts

Copilot AI review requested due to automatic review settings May 6, 2026 13:20
@timbastin timbastin merged commit 9160568 into main May 6, 2026
9 of 11 checks passed
@timbastin timbastin deleted the fix-vulndb-indexes-and-migrations branch May 6, 2026 13:25
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aligns the vulndb OSV bulk-import path with the current database index/migration state by removing an obsolete composite index, updating the bulk-import “drop indexes/constraints” SQL, and adding a regression test to catch future index/constraint mismatches. It also removes rollback (down) migration scripts from the repository.

Changes:

  • Drop the obsolete idx_cve_affected_component_cve_id_aff_comp_id index via a new migration and ensure the OSV bulk-import prep SQL drops it too.
  • Extend OSV bulk-import testing to assert the expected minimal index/constraint state during import and full restoration afterward.
  • Remove all *.down.sql migration scripts.

Reviewed changes

Copilot reviewed 67 out of 74 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
vulndb/osv_service.go Drops the removed composite index during PrepareBulkInsert and clarifies why certain PK indexes must remain for ON CONFLICT.
tests/osv_service_test.go Adds assertions verifying which indexes/constraints remain after PrepareBulkInsert, and that everything is restored after AddIndexesAndConstraints.
database/migrations/20260506094859_remove_obsolete_index_on_cve_affected_components.up.sql Drops the obsolete composite index from cve_affected_component.
database/migrations/20260504101108_drop_updated_created_at.down.sql Removed down migration script.
database/migrations/20260504000000_add_btree_index_cve_affected_component.down.sql Removed down migration script.
database/migrations/20260415090935_refactor-vulndb-tables.down.sql Removed down migration script.
database/migrations/20260410163018_add_dependency_proxy_secret.down.sql Removed down migration script.
database/migrations/20260402110000_add_fixable_distribution_to_artifact_risk_history.down.sql Removed down migration script.
database/migrations/20260323134661_refactor_vuln_ids.down.sql Removed down migration script.
database/migrations/20260323134660_add_trusted_entities_model.down.sql Removed down migration script.
database/migrations/20260323134659_add_direct_dependency_fixed_version.down.sql Removed down migration script.
database/migrations/20260323134658_fix-duplicates.down.sql Removed down migration script.
database/migrations/20260306000001_convert_btree_to_hash_indices.down.sql Removed down migration script.
database/migrations/20260306000000_add_vuln_id_index_to_vuln_events.down.sql Removed down migration script.
database/migrations/20260304170435_recover_missing_migration.down.sql Removed down migration script.
database/migrations/20260218171852_add-state-property.down.sql Removed down migration script.
database/migrations/20260206102932_add_csaf_package_scope.down.sql Removed down migration script.
database/migrations/20260204084100_add_keep_original_sbom_root_component_to_assets.down.sql Removed down migration script.
database/migrations/20260203084000_add_created_by_vex_rule_column.down.sql Removed down migration script.
database/migrations/20260203083743_vex_rule_enabled_drop_upstream.down.sql Removed down migration script.
database/migrations/20260202000000_add_vex_rules_and_external_references.down.sql Removed down migration script.
database/migrations/20260127100000_add_cve_purl_distribution_to_risk_history.down.sql Removed down migration script.
database/migrations/20260125100000_add_path_pattern_to_vuln_events.down.sql Removed down migration script.
database/migrations/20260122083658_add_vulnerability_path.down.sql Removed down migration script.
database/migrations/20260119124407_delete_badge_secret_column.down.sql Removed down migration script.
database/migrations/20260117090614_cleanup.down.sql Removed down migration script.
database/migrations/20260116192251_malicious_components_jsonb.down.sql Removed down migration script.
database/migrations/20260116174850_remove_fk_dependency_vulns_component.down.sql Removed down migration script.
database/migrations/20260116162639_delete_is_license_risk_overwritten.down.sql Removed down migration script.
database/migrations/20260115100000_change_qualifiers_to_jsonb.down.sql Removed down migration script.
database/migrations/20260114160930_make_dependency_vuln_cve_required.down.sql Removed down migration script.
database/migrations/20260114155810_add_cve_relationships_foreign_keys.down.sql Removed down migration script.
database/migrations/20260114145241_delete_invalid_license_risks.down.sql Removed down migration script.
database/migrations/20260114143739_rename_component_columns.down.sql Removed down migration script.
database/migrations/20260114142952_drop_depth_column.down.sql Removed down migration script.
database/migrations/20260114142512_fix_foreign_key_cascades.down.sql Removed down migration script.
database/migrations/20260114131136_remove_version_column.down.sql Removed down migration script.
database/migrations/20251230150234_cve_relationships.down.sql Removed down migration script.
database/migrations/20251214112317_add_malicious_packages.down.sql Removed down migration script.
database/migrations/20251214112316_artifact_dependency_vuln_constraint.down.sql Removed down migration script.
database/migrations/20251213134055_pipeline_errors.down.sql Removed down migration script.
database/migrations/20251112102935_cascade_dependency_vuln_foreign_key_update.down.sql Removed down migration script.
database/migrations/20251103152514_fix_default_migration.down.sql Removed down migration script.
database/migrations/20251031092922_asset_shares_information.down.sql Removed down migration script.
database/migrations/20251023111226_add_upstream_and_paranoia_columns.down.sql Removed down migration script.
database/migrations/20251023110850_add_default_node_to_sbom.down.sql Removed down migration script.
database/migrations/20251023110603_add_artifact_links_table.down.sql Removed down migration script.
database/migrations/20251011142707_asset_permissions.down.sql Removed down migration script.
database/migrations/20251009074354_public_asset.down.sql Removed down migration script.
database/migrations/20250926115507_nil_opaque_id.down.sql Removed down migration script.
database/migrations/20250923094634_delete_deleted_at_column.down.sql Removed down migration script.
database/migrations/20250919112335_add_metadata_column_to_assets.down.sql Removed down migration script.
database/migrations/20250918112540_drop_attestation_scanner_id.down.sql Removed down migration script.
database/migrations/20250904072950_delete_dep_vulns_without_cve.down.sql Removed down migration script.
database/migrations/20250904072716_fingerprint_idx.down.sql Removed down migration script.
database/migrations/20250903072552_drop_cpe_matches.down.sql Removed down migration script.
database/migrations/20250829095702_add_missing_indices.down.sql Removed down migration script.
database/migrations/20250829073721_add_external_entity_parent_id.down.sql Removed down migration script.
database/migrations/20250828161853_add_attestations_pkey.down.sql Removed down migration script.
database/migrations/20250828155058_add_last_accessed_at.down.sql Removed down migration script.
database/migrations/20250827071338_add_releases.down.sql Removed down migration script.
database/migrations/20250822131113_add_component_foreign_key.down.sql Removed down migration script.
database/migrations/20250821102935_add_artifacts.down.sql Removed down migration script.
database/migrations/20250821102442_remove_cves_columns.up.down.sql Removed down migration script.
database/migrations/20250819171907_add_risk_distribution_to_history.down.sql Removed down migration script.
database/migrations/20250818164407_adds_avatars.down.sql Removed down migration script.
database/migrations/20250813113628_update_debian_licenses.down.sql Removed down migration script.
database/migrations/20250811103154_add_repository_provider.down.sql Removed down migration script.
database/migrations/20250802184646_add_vuln_auto_reopen_after_days.down.sql Removed down migration script.
database/migrations/20250801170509_add_shares_vuln_information_column.down.sql Removed down migration script.
database/migrations/20250801115325_add_fingerprint_column.down.sql Removed down migration script.
database/migrations/20250801111615_code_risks_multiple_snippets.down.sql Removed down migration script.
database/migrations/20250801103405_add_cascade_delete_project_enabled_policies.down.sql Removed down migration script.
database/migrations/20250801103400_migration.down.sql Removed down migration script.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/osv_service_test.go
Comment on lines 35 to +40
err = vulndb.PrepareBulkInsert(ctx, tx)
assert.NoError(t, err)

cleanedConstraints, cleanedIndexes, err := getCurrentIndexAndConstraintState(ctx, tx)
assert.NoError(t, err)
assert.Len(t, cleanedIndexes, 2, "only the primary key indexes of cves and cve_relationships should remain for the import to detect ON CONFLICT triggers")
@@ -0,0 +1 @@
DROP INDEX public.idx_cve_affected_component_cve_id_aff_comp_id;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants