Production-Style PySpark ETL Pipeline with Incremental Loading, Historical Tracking Foundations (SCD) and Audit Monitoring.
DeltaFlow is a data engineering project designed to demonstrate how production ETL pipelines avoid duplicate processing while maintaining reliable, repeatable data loads.
The framework extracts user data from a paginated REST API, applies data quality transformations using PySpark, identifies new records through incremental loading logic, stores analytics-ready datasets in partitioned Parquet format and generates audit logs for operational monitoring.
Unlike traditional full-refresh pipelines, DeltaFlow processes only newly arriving records, reducing unnecessary computation and making pipeline reruns safe.
In many data platforms, source systems are refreshed continuously while ETL pipelines run on schedules.
Reprocessing the same records on every execution can:
- Create duplicate data
- Increase processing costs
- Slow downstream analytics
- Reduce trust in reporting systems
DeltaFlow addresses this problem by implementing incremental data loading using Left Anti Join logic, ensuring that only unseen records are loaded into the curated layer.
The framework follows a production-inspired ETL architecture commonly used in modern data engineering workflows.
DummyJSON Users API
→ Raw Landing Zone (JSON)
→ PySpark Data Quality Layer
→ Incremental Loading Engine
→ Curated Parquet Storage
→ Audit & Monitoring Layer
The design focuses on reliability, rerun safety, data quality and operational visibility.
- REST API extraction
- Pagination using limit and skip parameters
- Retry mechanism with exponential backoff
- Raw JSON landing file generation
- Schema projection
- Null handling
- Deduplication
- Data standardization
- Invalid record filtering
- User-level duplicate detection
- Left Anti Join implementation
- New records only processing
- Rerun-safe architecture
- Analytics-ready Parquet datasets
- Partitioned by state
- Optimized storage layout
Every execution generates operational metrics including:
- Rows extracted
- Rows cleaned
- New rows loaded
- Execution timestamp
- Python
- SQL
- PySpark
- JSON
- Parquet
- Incremental Loading
- Data Quality Validation
- Audit Logging
- Git
- GitHub
deltaflow/
│
├── screenshots/
│ ├── initial_pipeline_run.png
│ ├── incremental_run.png
│ └── audit_logs.png
│
├── output/
│ ├── users_cleaned/
│ └── audit_logs.csv
│
├── architecture.png
├── users_raw.json
├── DE_project.py
├── README.md
└── .gitignore
The first execution extracts records from the API, applies transformations and loads all valid records into the curated Parquet layer.
On subsequent executions, DeltaFlow identifies previously processed records and loads only new data.
This prevents duplicate inserts and keeps the curated layer clean.
Pipeline execution metrics are automatically captured for every run.
The audit framework tracks operational statistics and provides visibility into pipeline performance.
| Metric | Value |
|---|---|
| Records Extracted | 208 |
| Records Cleaned | 208 |
| Incremental Logic | Left Anti Join |
| Storage Format | Parquet |
| Partition Strategy | State-Based |
| Audit Logging | Enabled |
Planned DeltaFlow v2 improvements:
- SCD Type 2 historical tracking
- Active / Inactive record versioning
- Apache Airflow orchestration
- AWS S3 Data Lake integration
- AWS Glue ETL deployment
- Athena analytics layer
- Redshift warehouse integration
- Data quality monitoring with Great Expectations
Through this project I gained hands-on experience with:
- API-based ingestion pipelines
- Incremental ETL design patterns
- Data cleansing with PySpark
- Partitioned Parquet storage
- Audit logging frameworks
- Production-style pipeline development
- Rerun-safe data processing architectures
DeltaFlow demonstrates core data engineering concepts commonly used in production environments:
- Incremental data processing
- ETL pipeline development
- Data quality enforcement
- Operational monitoring
- Analytics-ready storage design
The project serves as a foundation for larger-scale cloud implementations involving Airflow, AWS Glue, S3, Athena and Redshift.
Vinit Vajani
Data Engineer | AWS | PySpark | SQL | ETL Pipelines
LinkedIn: https://www.linkedin.com/in/vinit-vajani-8672b8208/
GitHub: https://github.com/vinitvajani