Vision Transformer (ViT) Implementation on STL10

Overview

This repository contains a Vision Transformer (ViT) implementation trained from scratch on the STL10 image classification dataset using PyTorch.

The aim of this project is to empirically explore how Vision Transformers behave on a medium-scale image dataset, and to analyze how they generalize without pretraining. This includes model implementation, training routines, evaluation, and logging.

STL10 serves as an intermediate benchmark between simple datasets (like MNIST) and large-scale datasets (like ImageNet), making it a good testbed for understanding Vision Transformer capabilities.

---

Motivation

Vision Transformers have gained popularity due to their ability to model long-range dependencies via self-attention. Unlike Convolutional Neural Networks (CNNs), they do not use convolutional inductive bias. This project investigates:

• How a ViT learns visual representations on a dataset with real object classes
• How performance differs from simpler datasets
• The effects of model configuration and training dynamics on generalization

STL10 provides real-world images at moderate resolution, challenging ViT to capture meaningful spatial structure without the benefit of extensive pretrained knowledge.

---

Model Description

This repository implements a standard Vision Transformer architecture:

1. Patch Embedding

   • Images are split into non-overlapping patches
   • Each patch is projected to an embedding vector

2. Positional Embeddings

   • Learnable positional encodings preserve patch order information

3. Transformer Encoder

   • Multi-head self-attention
   • MLP blocks with GELU activation
   • Residual connections with Pre-LayerNorm

4. Classification Head

   • Final pooling or [CLS] token representation
   • Linear projection to class logits for prediction

Attention layers process all token pairs, enabling global context modeling.

---

Dataset

• Dataset: STL10
• Images: 96 × 96 RGB images
• Classes: 10 object categories
• Total Labeled Images:
  • Train: 5,000
  • Test: 8,000
• Preprocessing:
  • Normalization
  • Resizing / transformations
  • Data augmentation (if applied)

STL10 bridges the gap between simple toy datasets and complex large-scale datasets, making it ideal for architectural experimentation.