repo

Author: JaskRendix

banking_data_pipeline_python/README.md

@@ -0,0 +1,111 @@
+# Banking Data Pipeline — Credit Risk (ETL)
+
+## Overview
+This repository contains a compact, production‑style ETL pipeline for calculating **Risk‑Weighted Assets (RWA)** using standard Credit Risk parameters:
+
+- **EAD** — Exposure at Default  
+- **PD** — Probability of Default  
+- **LGD** — Loss Given Default  
+
+The pipeline mirrors the type of operational data processing performed within **BNP Paribas Finance & Risk Solutions**.
+
+The core Basel‑style formula implemented is:
+
+\[
+RWA = EAD \times PD \times LGD \times 12.5
+\]
+
+---
+
+## 🔍 Business Logic Highlights
+
+### **Data Quality**
+- Negative exposures are automatically filtered out.
+- All transformations are logged for traceability.
+
+### **Operational Reliability**
+- Extraction failures (e.g., missing files) raise clear, support‑friendly errors.
+- Logging provides a full audit trail of ETL steps.
+
+### **Database Layer**
+- Results are written to SQLite for demo purposes.
+- `db_setup.sql` includes indexing to support large‑scale reporting workloads.
+
+---
+
+## 🚀 How to Run the Pipeline
+
+### **1. Install the package (editable mode recommended during development)**
+
+From the project root:
+
+```bash
+pip install -e banking_data_pipeline_python
+```
+
+This makes the `banking_data_pipeline` package importable from anywhere.
+
+---
+
+### **2. Run the ETL pipeline**
+
+Because the project is now a proper Python package, you run it using the module syntax:
+
+```bash
+python -m banking_data_pipeline.rwa_calculator
+```
+
+This works from **any directory**, thanks to the `pathlib`‑based data resolution inside the script.
+
+---
+
+## Running Tests
+
+Tests are located under:
+
+```
+banking_data_pipeline_python/tests/
+```
+
+Run them with:
+
+```bash
+pytest
+```
+
+All tests validate:
+
+- RWA calculation correctness  
+- Negative exposure filtering  
+- Error handling for missing files  
+
+---
+
+## Project Structure
+
+```
+bnp/
+│
+├── banking_data_pipeline_python/
+│   ├── banking_data_pipeline/
+│   │   ├── rwa_calculator.py
+│   │   ├── data/
+│   │   │   └── mock_loans.csv
+│   │   └── __init__.py
+│   └── tests/
+│       └── test_rwa_calculator.py
+│
+└── pyproject.toml
+```
+
+---
+
+## 🛠️ Tech Stack
+
+- **Python 3.11**
+- **NumPy 1.26.4**
+- **Pandas 2.1.4**
+- **SQLite**
+- **Pytest**
+
+All versions are pinned in `pyproject.toml` for reproducibility.

banking_data_pipeline_python/banking_data_pipeline.egg-info/PKG-INFO

@@ -0,0 +1,6 @@
+Metadata-Version: 2.4
+Name: banking-data-pipeline
+Version: 0.1.0
+Requires-Dist: numpy==1.26.4
+Requires-Dist: pandas==2.1.4
+Requires-Dist: pytest==9.0.2

banking_data_pipeline_python/banking_data_pipeline.egg-info/SOURCES.txt

@@ -0,0 +1,11 @@
+README.md
+pyproject.toml
+requirements.txt
+banking_data_pipeline/__init__.py
+banking_data_pipeline/rwa_calculator.py
+banking_data_pipeline.egg-info/PKG-INFO
+banking_data_pipeline.egg-info/SOURCES.txt
+banking_data_pipeline.egg-info/dependency_links.txt
+banking_data_pipeline.egg-info/requires.txt
+banking_data_pipeline.egg-info/top_level.txt
+tests/test_rwa_calculator.py

banking_data_pipeline_python/banking_data_pipeline.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

banking_data_pipeline_python/banking_data_pipeline.egg-info/requires.txt

@@ -0,0 +1,3 @@
+numpy==1.26.4
+pandas==2.1.4
+pytest==9.0.2

banking_data_pipeline_python/banking_data_pipeline.egg-info/top_level.txt

@@ -0,0 +1 @@
+banking_data_pipeline

banking_data_pipeline_python/banking_data_pipeline/__init__.py

@@ -0,0 +1,64 @@
+import logging
+import os
+import sqlite3
+from pathlib import Path
+
+import pandas as pd
+from pandas import DataFrame
+
+logging.basicConfig(
+    level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s"
+)
+
+
+class CreditRiskETL:
+    def __init__(self, db_name: str = "banking_risk.db"):
+        self.db_name = db_name
+
+    def extract(self, file_path: str) -> DataFrame:
+        logging.info(f"Extracting data from {file_path}")
+        if not os.path.exists(file_path):
+            raise FileNotFoundError(f"Source file {file_path} missing.")
+        return pd.read_csv(file_path)
+
+    def transform(self, df: DataFrame) -> DataFrame:
+        logging.info("Starting transformation: Calculating RWA")
+
+        # 1. Data Cleaning: Remove invalid exposure
+        valid_df = df[df["exposure_at_default"] > 0].copy()
+        dropped = len(df) - len(valid_df)
+        if dropped > 0:
+            logging.warning(f"Dropped {dropped} rows with invalid exposure values.")
+
+        # 2. RWA Calculation: RWA = EAD * PD * LGD * 12.5
+        # Using 12.5 as the standard Basel multiplier for demo purposes
+        valid_df["rwa"] = (
+            valid_df["exposure_at_default"]
+            * valid_df["probability_of_default"]
+            * valid_df["loss_given_default"]
+            * 12.5
+        )
+
+        return valid_df
+
+    def load(self, df: DataFrame) -> None:
+        logging.info(f"Loading {len(df)} records to database {self.db_name}")
+        conn = sqlite3.connect(self.db_name)
+
+        # Mapping DataFrame columns to SQL table
+        df.to_sql("risk_reports", conn, if_exists="append", index=False)
+        conn.close()
+        logging.info("Load complete.")
+
+
+if __name__ == "__main__":
+    PACKAGE_ROOT = Path(__file__).resolve().parent
+    DATA_FILE = PACKAGE_ROOT / "data" / "mock_loans.csv"
+
+    etl = CreditRiskETL()
+    try:
+        raw_data = etl.extract(str(DATA_FILE))
+        processed_data = etl.transform(raw_data)
+        etl.load(processed_data)
+    except Exception as e:
+        logging.error(f"Pipeline Failed: {e}")

banking_data_pipeline_python/banking_data_pipeline/rwa_calculator.py

@@ -0,0 +1,6 @@
+loan_id,customer_name,exposure_at_default,probability_of_default,loss_given_default,sector
+L001,Global Tech Corp,1500000.00,0.02,0.45,Technology
+L002,Main Street Retail,500000.00,0.08,0.35,Retail
+L003,BioHealth Inc,2200000.00,0.01,0.40,Healthcare
+L004,BuildIt Construction,850000.00,0.12,0.50,Construction
+L005,Invalid Data Co,-100.00,0.05,0.40,Finance

banking_data_pipeline_python/data/mock_loans.csv

@@ -0,0 +1,18 @@
+[build-system]
+requires = ["setuptools"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "banking-data-pipeline"
+version = "0.1.0"
+dependencies = [
+    "numpy==1.26.4",
+    "pandas==2.1.4",
+    "pytest==9.0.2"
+]
+
+[tool.setuptools]
+packages = ["banking_data_pipeline"]
+
+[tool.pytest.ini_options]
+pythonpath = ["."]