fix: cargo dump failure due to schema changes [CM-1282]

[mbani01]

This pull request refactors how staging tables are created and validated when loading cargo dump CSVs. Instead of relying on a static DDL string, the code now dynamically constructs the SQL table definitions based on the CSV headers and enforces that required columns are present. This makes the process more robust to schema changes in the input data.

Key changes:

Dynamic Table Creation and Validation

• Replaces the static STAGING_DDL with logic that reads CSV headers and dynamically builds the CREATE TABLE statements for each staging table. This ensures the table schema matches the actual CSV columns and types, and adapts automatically to changes in the dump format.
• Introduces validation to check for required columns in each CSV file before table creation. If any required columns are missing, an error is thrown with details, preventing silent data issues.

Supporting Utilities

• Adds a readCsvHeader utility to efficiently read the header row from a CSV file.
• Adds COLUMN_TYPES and REQUIRED_COLUMNS definitions to specify expected types and required fields for each table, used in dynamic DDL generation and validation.

Integration with Load Process

• Updates the loadDump function to call the new dynamic table creation logic (buildStagingTable) for each CSV, instead of executing a static DDL block.

These changes make the CSV import process more maintainable and resilient to upstream schema changes.

---

Note

Medium Risk
Changes the packages worker cargo ingest path; failures are clearer but header parsing is naive (comma-split) and downstream SQL still assumes specific column names exist in staging.

Overview
Fixes cargo dump loads breaking when crates.io adds or reorders CSV columns by replacing the fixed STAGING_DDL with per-file staging tables built from each file’s header row.

loadDump now only ensures the cargo_sync schema exists, then for each dump CSV calls buildStagingTable (read header → validate REQUIRED_COLUMNS → DROP/CREATE with COLUMN_TYPES where defined, else text) before COPY. Missing required columns fail fast with a message that points at a crates.io schema change.

New columns in upstream dumps are accepted as extra text fields without code changes, while core ingest columns stay explicitly required.

^{Reviewed by Cursor Bugbot for commit cb185ec. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Copilot]

Pull request overview

Refactors the Cargo dump CSV ingest in packages_worker to make staging table creation resilient to upstream schema changes by deriving the staging DDL from each CSV’s header and validating required columns before loading.

Changes:

• Replace the static staging DDL with per-CSV dynamic DROP/CREATE TABLE built from the CSV header.
• Add required-column validation to fail fast when crates.io’s dump schema changes.
• Add a lightweight CSV header reader used during the load loop.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.