Implement BigQuery Atomic View Swapping & High-Performance GCS Streaming

[BLMgithub]

Transitions pipeline publishing from file-based pointers to a dual-system architecture utilizing BigQuery Authorized Views. Hardens core I/O to ensure zero-downtime BI delivery and 8GB memory footprint.

Implemented

• Atomic View Swapping: Added swap_bigquery_view logic to redirect stable published_ views via versioned BigQuery External Tables.
• Native GCS Streaming: Replaced legacy BigQuery Storage API with direct Polars scan_parquet via BigQuery metadata URI resolution (scan_gcs_uris_from_bigquery).
• Discovery-First ID Mapping: Refactored the ID Registrar to resolve all entity mappings in a pre-contract Discovery Phase, consolidating I/O into a single surgical pass across the 40M-row historical registry regardless of raw file format (CSV/Parquet).
• Temporal Normalization: Implemented terminal us (microsecond) resolution enforcement at source, transformation, and sink layers to prevent ns schema panics.
• Primitive Integer Pipeline: Integrated UInt32 ID Registrar for UUID-to-Int mapping; decoupled Assembly/Semantic I/O to reduce join-memory overhead by >60%.
• Infra-as-Code: Provisioned Semantic Datasets, BigLake Connections, and 1-month auto-expiration policies via Terraform.
• Environmental Parity: Unified local/cloud I/O paths using disk volume mounts and regional context injection (GCP_REGION).

Impact

• BI Stability: Eliminates Power BI "Dynamic Data Source" refresh blockers by providing static,native SQL endpoints.
• I/O Efficiency: Eliminates the "Multiplier Trap" by reducing registry read-redundancy by 66% for entities appearing in multiple tables (e.g., order_id). Maximizes Parquet row-group pruning by consolidating UUID predicates into a unified global Hash-Set.
• Reliability: Guarantees environment neutrality; resolves all "ns vs us" data type mismatches across local and cloud environments.
• High-Throughput Scaling: Successfully defends the 8GB memory ceiling. Benchmarks confirm high-efficiency processing of 40M rows (~5.5GB Parquet) at a throughput of ~307k rows/sec, achieving full pipeline execution in 130 seconds.
• Auditability: Maintains immutable, versioned GCS snapshots with corresponding BigQuery metadata pointers for point-in-time recovery.