Remove ETHNICITY, GENETIC_DATA, TRADE_UNION from entity taxonomy

These types are not span-extractable by NER: ethnicity is inferred
from context, genetic data is structured lab results, and trade union
membership is a document-level fact. Reduces taxonomy from 44 to 41
entity types (83 BIO labels).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: Sid Mohan

README.md

@@ -6,7 +6,7 @@ Open research and development for lightweight PII detection models. This repo co
 
 ## PII-NER v1
 
-A 22.7M parameter model for detecting 44 types of personally identifiable information in English text. Combines a pretrained DeBERTa-v3-xsmall backbone with a character CNN encoder, adaptive gating fusion, and CRF output layer.
+A 22.7M parameter model for detecting 41 types of personally identifiable information in English text. Combines a pretrained DeBERTa-v3-xsmall backbone with a character CNN encoder, adaptive gating fusion, and CRF output layer.
 
 ```
 Input Text
@@ -104,7 +104,7 @@ See [eval_benchmark.md](pii-ner-v1/docs/eval_benchmark.md) for flags and options
 | [NVIDIA Nemotron-PII](https://huggingface.co/datasets/nvidia/Nemotron-PII) | ~100K examples | CC-BY-4.0 |
 | [Gretel Synthetic PII Finance](https://huggingface.co/datasets/gretelai/synthetic_pii_finance_multilingual) | ~56K examples | Apache 2.0 |
 
-Combined: ~169K English examples after filtering and dedup. 44 canonical entity types across 4 sensitivity tiers, unified into 89 BIO labels. The dataset has a 323x frequency imbalance (DATE: 170K occurrences vs PASSPORT: 526), which drives many of the training innovations below.
+Combined: ~169K English examples after filtering and dedup. 41 canonical entity types across 4 sensitivity tiers, unified into 83 BIO labels. The dataset has a 323x frequency imbalance (DATE: 170K occurrences vs PASSPORT: 526), which drives many of the training innovations below.
 
 ## Documentation
 
@@ -149,7 +149,7 @@ Key finding: no published work combines differentiable character-level pattern r
 ## Open problems
 
 - **Tier 1 recall gap**: 0.823 vs 0.98 target. Passport number (0.426 F1) has only 526 training examples.
-- **16 zero-occurrence entity types**: NATIONALITY, ETHNICITY, RELIGION, etc. exist in the taxonomy but no training data covers them.
+- **13 zero-occurrence entity types**: NATIONALITY, RELIGION, MARITAL_STATUS, etc. exist in the taxonomy but no training data covers them.
 - **Head instability**: Root cause of the epoch 3+ training spike is unknown. Gradient clipping, per-component LR decay, or early stopping are candidate fixes.
 - **ONNX export**: CRF Viterbi decode doesn't export cleanly; needs pure-PyTorch reimplementation.
 

pii-ner-v1/src/datafog_pii_ner/data/label_schema.py

@@ -35,7 +35,6 @@
     "AGE",
     "NATIONALITY",
     "GENDER",
-    "ETHNICITY",
     "RELIGION",
     "MARITAL_STATUS",
 ]
@@ -60,8 +59,6 @@
     "POLITICAL_AFFILIATION",
     "SEXUAL_ORIENTATION",
     "HEALTH_CONDITION",
-    "GENETIC_DATA",
-    "TRADE_UNION",
 ]
 
 ALL_ENTITY_TYPES = TIER_1 + TIER_2 + TIER_3 + TIER_4
@@ -150,7 +147,6 @@ def build_label_weights(tier_weights: dict[int, float] | None = None) -> list[fl
     "GENDER": "GENDER",
     "AGE": "AGE",
     "NATIONALITY": "NATIONALITY",
-    "ETHNICITY": "ETHNICITY",
     "RELIGION": "RELIGION",
     "MARITAL_STATUS": "MARITAL_STATUS",
     "POLITICAL_AFFILIATION": "POLITICAL_AFFILIATION",