Add v1.4 training results: full entity coverage, 241K examples, best T3/T4 recall

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: Sid Mohan

README.md

@@ -2,7 +2,7 @@
 
 Open research and development for lightweight PII detection models. This repo contains the full training code, experiment history, and research behind [DataFog](https://datafog.ai)'s PII-NER model family.
 
-**Latest checkpoint:** [DataFog/pii-small-en](https://huggingface.co/DataFog/pii-small-en) on HuggingFace (v1.3, Apache 2.0)
+**Latest checkpoint:** [DataFog/pii-small-en](https://huggingface.co/DataFog/pii-small-en) on HuggingFace (v1.4, Apache 2.0)
 
 ## PII-NER v1
 
@@ -28,31 +28,30 @@ The gating fusion dynamically weights character-level features (for structured P
 
 ### Results
 
-Best results from v1.3 training on H100 (20 hours, 10 epochs):
+| Metric | V1.0 | V1.1 | V1.2 | V1.3 | V1.4 |
+|--------|------|------|------|------|------|
+| **Overall F1** | 0.904 | 0.901 | 0.901 | 0.907 | **0.889** |
+| Precision | 0.907 | 0.906 | 0.905 | 0.898 | 0.870 |
+| Recall | 0.902 | 0.895 | 0.896 | 0.916 | **0.910** |
+| Tier 1 Recall (SSN, Credit Card, ...) | 0.722 | 0.771 | **0.841** | 0.823 | 0.814 |
+| Tier 2 Recall (Person, Email, Phone, ...) | 0.934 | 0.933 | 0.936 | **0.945** | 0.937 |
+| Tier 3 Recall (Username, Date, Location, ...) | 0.919 | 0.908 | 0.911 | 0.930 | **0.945** |
+| Tier 4 Recall (Employee ID, IBAN, ...) | 0.866 | 0.844 | 0.845 | 0.868 | **0.937** |
 
-| Metric | V1.0 | V1.1 | V1.2 | V1.3 |
-|--------|------|------|------|------|
-| **Overall F1** | 0.904 | 0.901 | 0.901 | **0.907** |
-| Precision | 0.907 | 0.906 | 0.905 | 0.898 |
-| Recall | 0.902 | 0.895 | 0.896 | **0.916** |
-| Tier 1 Recall (SSN, Credit Card, ...) | 0.722 | 0.771 | **0.841** | 0.823 |
-| Tier 2 Recall (Person, Email, Phone, ...) | 0.934 | 0.933 | 0.936 | **0.945** |
-| Tier 3 Recall (Username, Date, Location, ...) | 0.919 | 0.908 | 0.911 | **0.930** |
-| Tier 4 Recall (Employee ID, IBAN, ...) | 0.866 | 0.844 | 0.845 | **0.868** |
+V1.3 has the best overall F1 (0.907). V1.4 expanded training data from 169K to 241K examples with 4 new data sources and 22K synthetic examples covering 11 entity types that previously had zero training data. V1.4 achieves the best Tier 3 and Tier 4 recall — all previously-zero entity types now produce results. The F1 drop vs v1.3 reflects the broader entity coverage at the cost of some precision on existing types.
 
-V1.3 has the best overall F1 and recall. V1.2 is better for Tier 1-critical deployments (0.841 vs 0.823).
-
-### Top entity F1 scores (v1.3)
+### Top entity F1 scores (v1.4)
 
 | Entity | F1 | Entity | F1 |
 |--------|------|--------|------|
-| URL | 0.994 | License Plate | 0.952 |
-| Biometric | 0.992 | Gender | 0.946 |
-| IP Address | 0.988 | Employee ID | 0.940 |
-| Date of Birth | 0.981 | IBAN | 0.935 |
-| Vehicle ID | 0.976 | Username | 0.930 |
-| Email | 0.968 | SSN | 0.930 |
-| Phone | 0.966 | Location | 0.929 |
+| URL | 0.995 | Nationality | 0.993 |
+| Religion | 1.000 | Crypto Wallet | 0.987 |
+| Health Condition | 0.988 | Insurance Number | 0.986 |
+| Student ID | 1.000 | Political Affiliation | 0.995 |
+| Marital Status | 1.000 | Salary | 0.995 |
+| Sexual Orientation | 1.000 | Criminal Record | 1.000 |
+| Biometric | 0.975 | Gender | 0.957 |
+| Email | 0.976 | Phone | 0.947 |
 
 ### Quick start
 
@@ -100,11 +99,13 @@ See [eval_benchmark.md](pii-ner-v1/docs/eval_benchmark.md) for flags and options
 
 | Dataset | Size | License |
 |---------|------|---------|
-| [AI4Privacy](https://huggingface.co/datasets/ai4privacy/pii-masking-200k) | ~200K examples | Apache 2.0 |
+| [AI4Privacy](https://huggingface.co/datasets/ai4privacy/pii-masking-200k) | ~43K examples (English) | Apache 2.0 |
 | [NVIDIA Nemotron-PII](https://huggingface.co/datasets/nvidia/Nemotron-PII) | ~100K examples | CC-BY-4.0 |
-| [Gretel Synthetic PII Finance](https://huggingface.co/datasets/gretelai/synthetic_pii_finance_multilingual) | ~56K examples | Apache 2.0 |
+| [Gretel Synthetic PII Finance](https://huggingface.co/datasets/gretelai/synthetic_pii_finance_multilingual) | ~26K examples | Apache 2.0 |
+| [Gretel PII Masking EN v1](https://huggingface.co/datasets/gretelai/gretel-pii-masking-en-v1) | ~50K examples | Apache 2.0 |
+| Synthetic (generated) | ~22K examples | Apache 2.0 |
 
-Combined: ~169K English examples after filtering and dedup. 41 canonical entity types across 4 sensitivity tiers, unified into 83 BIO labels. The dataset has a 323x frequency imbalance (DATE: 170K occurrences vs PASSPORT: 526), which drives many of the training innovations below.
+Combined: ~241K English examples after filtering and dedup. 41 canonical entity types across 4 sensitivity tiers, unified into 83 BIO labels. The synthetic data covers 11 entity types that had zero examples in the open-source datasets: NATIONALITY, RELIGION, MARITAL_STATUS, STUDENT_ID, CRYPTO_WALLET, INSURANCE_NUMBER, SALARY, CRIMINAL_RECORD, POLITICAL_AFFILIATION, SEXUAL_ORIENTATION, and HEALTH_CONDITION.
 
 ## Documentation
 
@@ -125,7 +126,8 @@ Key finding: no published work combines differentiable character-level pattern r
 
 | Date | Version | What changed |
 |------|---------|-------------|
-| 2026-02-07 | **v1.3** | Best F1 (0.907). Early backbone freeze (epoch 3) + progressive tier weight reduction. Discovered training spikes originate in head components, not backbone. |
+| 2026-02-08 | **v1.4** | Full entity coverage. Added 4 new data sources (241K total), synthetic data for 11 zero-occurrence types. All 41 entity types now produce results. Best T3 (0.945) and T4 (0.937) recall. Backbone freeze confirmed harmful — epoch 3 best, stopped early. |
+| 2026-02-07 | v1.3 | Best F1 (0.907). Early backbone freeze (epoch 3) + progressive tier weight reduction. Discovered training spikes originate in head components, not backbone. |
 | 2026-02-05 | v1.2 | Best Tier 1 recall (0.841). Backbone freezing after epoch 4. Epoch 3 identified as consistent sweet spot. |
 | 2026-02-04 | v1.1 | Tier-weighted CRF loss (3x for Tier 1), rare entity oversampling, inference pipeline. Tier 1 recall +4.9pts. |
 | 2026-02-04 | — | Training chronicle, entity frequency audit (323x imbalance discovered). |
@@ -142,15 +144,17 @@ Key finding: no published work combines differentiable character-level pattern r
 
 3. **The training spike is a head problem, not backbone.** V1.3 proved this definitively: the spike occurred at epoch 5 with the backbone already frozen since epoch 3. The CharCNN/GatingFusion/CRF destabilize under continued training.
 
-4. **Epoch 3 is consistently the best checkpoint.** Across v1.2 and v1.3, the model peaks at epoch 3 then destabilizes. Earlier representations generalize better.
+4. **Epoch 3 is consistently the best checkpoint.** Across v1.2, v1.3, and v1.4, the model peaks at epoch 3 then destabilizes. Earlier representations generalize better.
 
 5. **Tier-weighted loss works but amplifies instability.** 3x weight + 3x oversampling = ~9x gradient signal for Tier 1, which accelerates learning but accumulates damage.
 
+6. **Backbone freeze hurts more than it helps.** V1.4 confirmed: freezing the backbone after epoch 3 causes immediate F1 regression (0.889→0.806) and eval loss spike (2.4→8.3). The head cannot adapt without backbone co-training.
+
 ## Open problems
 
-- **Tier 1 recall gap**: 0.823 vs 0.98 target. Passport number (0.426 F1) has only 526 training examples.
-- **13 zero-occurrence entity types**: NATIONALITY, RELIGION, MARITAL_STATUS, etc. exist in the taxonomy but no training data covers them.
-- **Head instability**: Root cause of the epoch 3+ training spike is unknown. Gradient clipping, per-component LR decay, or early stopping are candidate fixes.
+- **Tier 1 recall gap**: 0.814 vs 0.98 target. Passport number (0.447 F1) and PIN (0.556 F1) remain weak due to limited training examples.
+- **Head instability**: Backbone freeze causes immediate regression (F1 0.889→0.806, loss 2.4→8.3). Root cause is in the CharCNN/GatingFusion/CRF head. Gradient clipping, per-component LR decay, or early stopping are candidate fixes.
+- **Precision vs coverage trade-off**: v1.4 expanded entity coverage at cost of ~2pts F1 vs v1.3. Better synthetic data quality or curriculum learning could close this gap.
 - **ONNX export**: CRF Viterbi decode doesn't export cleanly; needs pure-PyTorch reimplementation.
 
 ## Project structure