Deep Learning Framework from Scratch

A lightweight deep learning framework built entirely with NumPy. This project implements the core components of modern neural networks including forward propagation, backpropagation, optimizers, regularization techniques, and training utilities without relying on existing deep learning libraries such as TensorFlow or PyTorch.

Features

Neural Network Components

• Sequential model API
• Dense (Fully Connected) layers
• Modular layer architecture

Activation Functions

• ReLU
• Leaky ReLU
• Sigmoid
• Tanh
• Softmax

Loss Functions

• Mean Squared Error (MSE)
• Binary Cross Entropy (BCE)
• Categorical Cross Entropy (CCE)

Optimizers

• Stochastic Gradient Descent (SGD)
• SGD with Momentum
• Adam Optimizer

Regularization

• L1 Regularization
• L2 Regularization
• Dropout Layer

Training Utilities

• Mini-batch Gradient Descent
• Validation Split
• Early Stopping
• Xavier Initialization
• He Initialization

Project Structure

dl_framework/
│
├── core/
│   └── model.py
│
├── nn/
│   ├── dense.py
│   ├── activations.py
│   └── dropout.py
|
├── optimizer/
│   ├── losses.py
│   └── optimizers.py
│
├── examples/
│   ├── regression.py
│   ├── binary_classification.py
│   └── multiclass_classification.py
│
├── notebooks/
│
└── README.md

Installation

Clone the repository:

git clone https://github.com/abacts/dl-framework.git
cd dl-framework

Install dependencies:

pip install -r requirements.txt

Example Usage

from core.model import Sequential
from nn.dense import Dense
from nn.activations import ReLU, Softmax
from optimizer.losses import CategoricalCrossEntropy
from optimizer.optimizers import Adam

model = Sequential()

model.add(Dense(784, 128))
model.add(ReLU())

model.add(Dense(128, 10))
model.add(Softmax())

model.compile(
    loss=CategoricalCrossEntropy(),
    optimizer=Adam()
)

model.fit(
    X_train,
    y_train,
    epochs=50,
    batch_size=32
)

Results

The framework has been tested on:

• Regression datasets
• Binary Classification datasets
• Multi-class Classification datasets
• MNIST handwritten digit classification

Learning Goals

This project was created to gain a deeper understanding of:

• Neural Network Fundamentals
• Backpropagation
• Optimization Algorithms
• Weight Initialization
• Regularization Techniques
• Deep Learning Framework Design

License

MIT License