feat(intermediate): architect transactional silver layer firewalls with multi-locale parsing and defensive metrics bounding

[Ritik574-coder]

Summary

Why is this change needed?

Our raw downstream transaction ingestion stream (stg_transactions) was structurally compromised by systemic upstream data anomalies, high-variance formatting distributions, and business logic violations. Processing this raw layer directly created massive data-quality regressions in downstream BI layers and predictive science datasets.

Specific problems resolved by this intermediate layer include:

1. String-Polluted Financial Metrics: Ingestion of raw currency artifacts ($, ,) in financial fields caused silent failures or structural calculation drops during mathematical aggregations.
2. Temporal & Regional Disparities: Inconsistent multi-locale date layouts (US vs. European vs. ISO formats) across multi-channel web/retail terminals broke standard chronologies.
3. Logistics & Supply Chain Anomalies: Flawed upstream event emitters recorded impossible fulfillment loops—such as items marked shipped before the transaction order date or delivery lags breaching critical service level agreements (SLAs).
4. Out-of-Bounds Metric Noise: Form-submission bugs occasionally registered impossible values for cart item positions or item unit volumes.

What problem does it solve?

This Pull Request introduces a componentized Silver Layer (Intermediate) Architecture. By decomposing a monolithic transaction table vertically into four domain-specific intermediate tables, we establish an immutable data-quality firewall. Downstream consumption models are now shielded from raw input corruption, query performance is optimized by minimizing scan volume contention, and system operations are tightly bound to strict corporate thresholds without crashing production pipelines.

---

Changes made

1. Data Engineering Core Layer Implementation

• int_transaction_financial.sql: Configured an aggregation firewall. Sanitizes textual noise ($, ,) out of unit_list_price, line_total_before_tax, tax_amount, and line_total_with_tax via nested REPLACE string manipulations before applying high-precision TRY_CONVERT(DECIMAL(10,2)). Enforces strict metric boundaries (quantity_ordered bounded strictly to 1-30, and percentages limited to 0-100%), gracefully coercing structural outliers to NULL to preserve reporting averages.
• int_transaction_fulfillment.sql: Established a robust logistics engine. Built an explicit, multi-locale pattern-matching ladder using regex-like SQL LIKE conditions to deterministically parse US, ISO, and international textual date fields into structured DATE formats. Resolved supply chain timeline inversions using DATEDIFF logic; automatically re-aligns reversed day/month inputs using DATEFROMPARTS and substitutes missing values with an artificial corporate SLA benchmark (DATEADD(DAY, 7, order_date)).
• int_transaction_order.sql: Stabilized our dimensional relational star-schema mapping topology. This table isolates and structures fundamental enterprise composite keys (transaction_id, order_id, customer_id, product_id, store_id, employee_id). Validates shopping cart sequences by binding order_line_number between 1 and 20.
• int_transaction_status.sql: Executed uniform categorical smoothing to fix data fragmentation. Normalizes multi-channel status variants into explicit Title Case values ('Pending', 'Shipped', etc.) and maps irregular Boolean entries (1, yes, y, true) into standardized corporate state strings ('True', 'False', 'Unknown').

2. Analytics Governance & Testing Documentation

• schema.yml: Authored an end-to-end dbt configuration mapping tracking model schema constraints. Embedded strict enterprise validations including primary identity uniqueness (unique, not_null), numeric range assertions (expression_is_true), and rigid structural domain bounds (accepted_values) to trap edge-case failures automatically during materialization passes.

---

Validation

• Commands tested locally (if applicable)
  • Executed dbt clean && dbt compile && dbt run --select path:models/intermediate/transactions across local sandbox container layers. All models compiled cleanly to target dialect syntax.
• Docs links checked
  • Generated local server maps via dbt docs generate to ensure target graph dependencies and contextual descriptions trace accurately across the intermediate lineage visualization.
• No secrets committed
  • Verified git staging areas contain no plain-text database connections, infrastructure endpoints, or warehouse profile credentials.
• CI passes
  • Pre-hook mock warehouse run successfully passed all strict unit boundaries, data constraints, and integration checks.

---

Related issues

Closes #402 (Issue regarding string formatting corrupting downstream dashboard revenue aggregations)
Closes #415 (Issue tracking broken timeline metrics on supply chain logistics dashboards)

---

Notes for reviewers

Technical Tradeoffs & Architectural Mindset

1. Vertical Domain Splitting vs. Monolithic Table: We purposely chose to partition the intermediate transactional layer into four separate components (financial, fulfillment, order, status) instead of outputting a single massive table. This isolates domain modifications, preserves data segregation principles, and prevents memory-intensive data warehouse locks when downstream analysts are querying logistics timelines versus gross margins.
2. Coercion to NULL over Pipeline Interruption: When encountering wild anomalies (e.g., an order line item > 30), the SQL engine uses a defensive CASE switch to force the value to NULL or map it to 'Unknown'. This intentional design choice prevents the enterprise overnight warehouse orchestration from crashing, while exposing anomalies to automated dbt alert tests (warn severity thresholds) for immediate profiling by the Data Governance squad.
3. Future Optimization Note: As ingestion volume grows over the coming quarters, we plan to implement downstream incremental materialization predicates on the fulfillment model based on the last_modified metadata delta cursor to bypass heavy historical date rescans.

[qodo-code-review]

Qodo reviews are paused for this user.

Troubleshooting steps vary by plan Learn more →

On a Teams plan?
Reviews resume once this user has a paid seat and their Git account is linked in Qodo.
Link Git account →

Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center?
These require an Enterprise plan - Contact us
Contact us →

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[Ritik574-coder]

• Establish components: int_transaction_financial, int_transaction_fulfillment, int_transaction_order, and int_transaction_status.
• Implement multi-locale temporal date-parsing ladder handling US, ISO, and international string variances dynamically.
• Enforce logistics chronological matrices utilizing DATEDIFF logic to resolve retroactive shipping anomalies and automatically inject fallback SLAs.
• Inject strict data quality firewalls with unit-level bounding matrices (quantities bounded 1-30, line items 1-20) and safe NULL coercions.
• Strip textual currency artifacts from gross revenue fields and cast to DECIMAL(10,2) to safeguard downstream BI aggregations.
• Standardize categorical taxonomies across promotion naming layouts and boolean vectors.
• Deploy enterprise-grade schema.yml documenting programmatic transformations and data model boundaries.