Malware Detection CI/CD Pipeline

Overview

This project is a comprehensive data processing and model deployment pipeline designed to handle data ingestion from MongoDB, perform validation, apply transformations, and train machine learning models. It includes an automated evaluation process that compares model performance against a benchmark stored in an S3 bucket, replacing the existing model if a better-performing one is identified.

The pipeline showcases the integration of various data processing techniques and machine learning methodologies, aligning with Continuous Integration/Continuous Deployment (CI/CD) principles to ensure efficient processing and deployment of superior models.

Data

Data used for this project was taken from UCI ML Repository created by Parthajit Borah (parthajit@tezu.ernet.in) and Dhruba K. Bhattacharyya (dkb@tezu.ernet.in)

Installation

1. Clone this repository: git clone https://github.com/MercMayhem/malware-detection-pipeline.git

2. Create Conda Environment with Python 3.12.0: conda create -n malwareDetection python=3.12

3. Activate environment: conda activate malwareDetection

4. Install requirements.txt file pip install -r requirements.txt

Configure

1. Install AWS CLI

2. Configure AWS CLI: aws configure

3. Add MongoDB Server URI to environment variable: export SERVER_URI="<add Server URI>"

4. Configure config.yaml file

Run

1. Activate the conda environment
2. Run the main file: python3 main.py

Project Directory Structure

malware-detection-pipeline
├── README.md
├── artifact (Contains files produced at the end of each stage of pipeline)
│   ├── ...
│   ├── ...
│   └── ...
├── config.yaml (contains configurations that need to be set by the user)
├── data (data which was used for testing the pipeline)
│   └── TUANDROMD.csv
├── main.py (main file to be run)
├── notebooks (prototyping stages of the pipelines)
│   ├── ...
│   ├── ...
│   └── ...
├── params.yaml (hyperparameters for the model)
├── requirements.txt
├── schema.yaml (schema of input data)
├── setup.py (installing malwareDetection as a module)
└── src
│   └── malwareDetection
│       ├── __init__.py
│       ├── components (contains logic for each stage)
│       │   ├── __init__.py
│		│   ├── ...
│		│   ├── ...
│		│   └── ...
│       ├── config (contains builder which builds config from config.yaml for each stage)
│       │   ├── __init__.py
│       │   └── config_builder.py
│       ├── constants
│       │   ├── __init__.py
│       ├── entity (data models for config validation)
│       │   ├── __init__.py
│       │   └── entities.py
│       ├── pipeline (sub-pipelines for each stage)
│       │   ├── data_ingestion_pipeline.py
│       │   ├── data_transformation_pipeline.py
│       │   ├── data_validation_pipeline.py
│       │   ├── model_evaluation_pipeline.py
│       │   ├── model_pusher_pipeline.py
│       │   └── model_training_pipeline.py
│       └── utils
│		    └── __init__.py
└── upload_csv.py (script used to upload malware data to the database)