Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 3 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,2 +1,5 @@
# Auto detect text files and perform LF normalization
* text=auto

# Power BI LFS
*.pbix filter=lfs diff=lfs merge=lfs -text
43 changes: 40 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,40 @@
## Overview
Small to mid-sized organizations are trapped in a cycle where they outgrow the capabilities of spreadsheets but lack the technical infrastructure to migrate to databases. This creates a "structural debt" where the very tools that allow a business to be agile (spreadsheets) are the same tools that make their data fundamentally untrustworthy for scaling or reporting.

### The Solution
## The Result: Operational Decision Systems
The ultimate purpose of this defensive architecture is to serve the **Presentation Layer** with absolute reliability. By the time data reaches Power BI, it has been stripped of anomalies, structurally validated, and semantically flattened.

Because the pipeline utilizes **Atomic View Swaps** in BigQuery, the semantic models are updated instantaneously. This guarantees that Power BI scheduled refreshes (Import Mode) never encounter locked tables or partial data, resulting in zero downtime for business consumers.

> **Dynamic Sensitivity Calibration**
>
>The reporting suite features interactive "Smoke Detectors." Built with dynamic What-If parameters, these dashboards allow operators to manually adjust alert sensitivity thresholds to match changing business realities (e.g., loosening thresholds during peak holiday seasons).
>
> Explore the **[Power BI Directory](/power_bi)** to read detailed [operational guides](/power_bi/docs) or download the `.pbix` [releases](/power_bi/releases/).

### Customer Experience & Revenue Exposure
A decision-support system designed to monitor financial risk driven by fulfillment failures. It correlates delivery delays with buyer drop-off rates, allowing leadership to quantify the "cost of friction."

* **Calibration Mechanism:** Utilizes a **Delay Threshold Parameter** to dynamically define what constitutes a critical "Danger Zone" for segment and regional prioritization.

![Customer Experince Image](/assets/screenshots/customer-experience-monitor.png)

### Fulfillment Decision Monitor
An operational early-warning system that identifies logistics partners and sellers requiring immediate intervention by focusing on statistical deviations in network speed rather than absolute failure.

* **Calibration Mechanism:** Controlled via **Standard Deviation & Slippage Thresholds** to filter out normal variance and flag true systemic slowdowns.

![Fulfillment Decision Monitor](/assets/screenshots/seller-fulfillment-monitor.png)

### Product Friction Monitor

Designed to identify physical and structural fulfillment bottlenecks driven by product specifications (e.g., weight-driven outliers).

* **Calibration Mechanism:** Leverages **Standard Deviation & Slippage Thresholds** applied to *lead-time volatility*, alerting operations teams to route oversize items to specialized freight before delays impact the customer.

![Product Friction Monitor](/assets/screenshots/product-friction-monitor.png)

## The Solution
This project solves that challenge by delivering a highly resilient, event-driven data pipeline on Google Cloud Platform for reliable operational analytics. It guarantees data integrity through a strict Medallion architecture (Bronze, Silver, Gold) that relies on rigid data contracts and validation gates to catch and isolate bad data early in the lifecycle.

### Defensive Pipeline Architecture
Expand Down Expand Up @@ -182,16 +215,20 @@ operations-analytics-pipeline/
│ ├── shared/ # Extractor logic and core I/O utilities
│ └── run_extract.py # The Drive extractor orchestrator
├── data_pipeline/
│ ├── .shared/ # Storage adapters, IO wrappers, and registry configurations
│ ├── assembly/ # Delta merging and event mapping logic (Gold Pre-processing)
│ ├── contract/ # Subtractive filtering logic (Silver Layer)
│ ├── publish/ # Manages the atomic publish lifecycle of semantic datasets
│ ├── semantic/ # Fact/Dimension table builders (Gold Layer)
│ ├── shared/ # Storage adapters, IO wrappers, and registry configurations
│ ├── validation/ # Dual-pass structural data validation gates
│ └── run_pipeline.py # The pipeline orchestrator and state manager
├── docs/ # Detailed architectural and stage-level system contracts
├── runtime/ # Git-ignored ephemeral workspace used by the local pipeline executor
└── tests/ # Pytest suite for pipeline logic and validation rules
├── tests/ # Pytest suite for pipeline logic and validation rules
└── power_bi/
├── .shared/ # Global Dashboards BI assets (e.g. Themes, .json files, etc.)
├── dashboards/ # Source Control (PBIP)
└──releases # Deliverables (PBIX)
```

## CI/CD & Security
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added assets/screenshots/product-friction-monitor.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
23 changes: 23 additions & 0 deletions power_bi/.shared/.m_queries/calendar_date.pq
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
let
SourceFact = // fact table,
MaxDateInFact = Date.From(List.Max(SourceFact[week_start_date])),

// List of dates
StartDate = #date(2022, 12, 26),
EndDate = MaxDateInFact,
CalendarList = List.Dates(StartDate, Duration.Days(EndDate - StartDate) + 1, #duration(1, 0, 0, 0)),

// Convert to table
#"Converted to Table" = Table.FromList(CalendarList, Splitter.SplitByNothing(), {"Date"}, null, ExtraValues.Error),
#"Changed Type" = Table.TransformColumnTypes(#"Converted to Table",{{"Date", type date}}),

// Date format columns
#"Added Year" = Table.AddColumn(#"Changed Type", "Year", each Date.Year([Date]), Int64.Type),
#"Added Month" = Table.AddColumn(#"Added Year", "Month", each Date.Month([Date]), Int64.Type),
#"Added Month Name" = Table.AddColumn(#"Added Month", "Month Name", each Date.MonthName([Date]), type text),
#"Added Year Month" = Table.AddColumn(#"Added Month Name", "Year Month", each Text.From([Year]) & "-" & Text.PadStart(Text.From([Month]), 2, "0"), type text),
#"Added Week Number" = Table.AddColumn(#"Added Year Month", "Week Number", each " W" & Text.PadStart(Text.From(Date.WeekOfYear([Date])), 2, "0"), type text),
// #"Added Year Week" = Table.AddColumn(#"Added Year Month", "Year Week", each Text.From([Year]) & " W" & Text.PadStart(Text.From(Date.WeekOfYear([Date])), 2, "0"), type text),
#"Added Week Start Date" = Table.AddColumn(#"Added Week Number", "Week Start Date", each Date.StartOfWeek([Date], Day.Monday), type date)
in
#"Added Week Start Date"
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading