📊 Embedding Space Analysis: BERT vs Contrastive Models with Fine-Tuning

🧠 Overview

This project analyzes and compares text embedding spaces generated by different models and studies how fine-tuning reshapes embedding geometry.

It combines:

• Embedding comparison (BERT vs Contrastive model)
• Embedding improvement (before vs after fine-tuning)

The goal is to understand how embedding representations affect clustering, semantic similarity, and downstream task performance.

---

🎯 Objectives

• Compare embedding quality of BERT and Sentence Transformers
• Visualize embeddings using:
  • PCA
  • t-SNE
  • UMAP
• Study the effect of fine-tuning on embedding space
• Analyze:
  • Cluster separability
  • Anisotropy vs isotropy
• Evaluate embeddings using:
  • Silhouette score
  • Linear probe classification

---

📂 Dataset

• Source: sklearn.datasets.fetch_20newsgroups
• Categories used:
  • rec.sport.baseball
  • rec.sport.hockey
  • sci.space
  • comp.graphics
• Sample size: 500 texts

---

⚙️ Methodology

1. Embedding Generation

🔹 BERT (Baseline)

• Model: bert-base-uncased
• Approach: Mean pooling over token embeddings

🔹 Contrastive Model

• Model: all-MiniLM-L6-v2
• Trained using contrastive learning for semantic similarity

---

2. Dimensionality Reduction

To visualize high-dimensional embeddings:

• PCA → Linear projection
• t-SNE → Preserves local structure
• UMAP → Preserves local + global structure

---

3. Fine-Tuning

• Model: distilbert-base-uncased
• Task: Text classification
• Dataset: Same 20 Newsgroups subset
• Output: Task-specific embeddings

---

4. Post Fine-Tuning Embeddings

• Extracted using [CLS] token representation
• Compared with original BERT embeddings

---

5. Evaluation Metrics

📊 Visualization

• PCA and UMAP plots before and after fine-tuning

📈 Clustering Quality

• Silhouette Score

🤖 Linear Probe

• Logistic Regression classifier trained on embeddings

---

📊 Results & Observations

🔹 BERT vs Contrastive Embeddings

• BERT embeddings are scattered and overlapping
• Contrastive embeddings form clear, separable clusters
• Indicates better semantic structure in contrastive models

---

🔹 Dimensionality Reduction Insights

• PCA: Basic separation (linear)
• t-SNE: Best visual cluster separation
• UMAP: Shows both structure and overlap between classes

---

🔹 Effect of Fine-Tuning

• Embeddings become:
  • More compact
  • More separable
• Transition from:
  • Anisotropic → More isotropic space

---

🔹 Quantitative Improvements

• Silhouette score increases
• Linear probe accuracy improves

---

🧪 Key Insights

• Pretrained BERT is not optimized for sentence similarity
• Contrastive learning significantly improves embedding geometry
• Fine-tuning aligns embeddings with task-specific objectives
• Better embedding structure → better downstream performance

---

🚀 Conclusion

This project demonstrates that:

• Embedding quality depends heavily on training objectives
• Contrastive models outperform vanilla BERT in semantic tasks
• Fine-tuning reshapes embedding space to be more structured and discriminative
• Visualization + metrics together provide a complete understanding of embedding behavior

---

🛠️ Tech Stack

• Python
• PyTorch
• Hugging Face Transformers
• Sentence Transformers
• Scikit-learn
• UMAP
• Matplotlib

---

▶️ How to Run

1. Initialize project

uv init
uv sync