You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fix incorrect entity/label counts in all docs: 81→44 types, 163→89 BIO labels
The label_schema.py defines 44 entity types (6+6+12+20 across 4 tiers),
producing 89 BIO labels (1 O + 44 B- + 44 I-). All documentation
incorrectly referenced 81 entity types and 163 BIO labels. Updated
counts, transition matrix sizes (89x89 = 7,921 params), and derived
calculations across 6 documentation files.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
<pstyle="font-size: 1em;">100+ source label names must map to 81 canonical entity types. "FIRSTNAME", "FIRST_NAME", "first_name", "PER" → all become PERSON.</p>
148
+
<pstyle="font-size: 1em;">100+ source label names must map to 44 canonical entity types. "FIRSTNAME", "FIRST_NAME", "first_name", "PER" → all become PERSON.</p>
<pstyle="margin-bottom: 20px;">Each dataset uses its own label vocabulary. We maintain per-dataset mapping dictionaries to a canonical set.</p>
292
292
<divclass="two-col">
293
293
<div>
@@ -326,7 +326,7 @@ <h3>Why Not Automatic Fuzzy Matching?</h3>
326
326
<divclass="slide" id="s7">
327
327
<div>
328
328
<h2>The 4-Tier PII Taxonomy</h2>
329
-
<pstyle="margin-bottom: 20px;">81 entity types organized by sensitivity. Each tier has a different recall target — miss tolerance decreases with severity.</p>
329
+
<pstyle="margin-bottom: 20px;">44 entity types organized by sensitivity. Each tier has a different recall target — miss tolerance decreases with severity.</p>
<h2>Mixed Precision + CRF: A Delicate Balance</h2>
461
-
<pstyle="margin-bottom: 20px;">The CRF head computes log-sum-exp over 163×163 transition matrices. This is numerically unstable in half precision.</p>
461
+
<pstyle="margin-bottom: 20px;">The CRF head computes log-sum-exp over 89×89 transition matrices. This is numerically unstable in half precision.</p>
<spanclass="yellow">FP16 CRF failure mode:</span> Transition scores for 163 labels easily exceed FP16 range (65504) in the log-sum-exp computation, producing <code>inf</code> → <code>NaN</code> loss.
492
+
<spanclass="yellow">FP16 CRF failure mode:</span> Transition scores for 89 labels can exceed FP16 range (65504) in the log-sum-exp computation, producing <code>inf</code> → <code>NaN</code> loss.
<li><strong>50x LR ratio:</strong> Validated in smoke test — DeBERTa representations must be preserved while CRF transitions (163×163 matrix) need aggressive learning</li>
530
+
<li><strong>50x LR ratio:</strong> Validated in smoke test — DeBERTa representations must be preserved while CRF transitions (89×89 matrix) need aggressive learning</li>
531
531
<li><strong>10 epochs:</strong> With 360K examples and effective batch 32, each epoch is ~11,250 steps. 10 epochs = ~112K steps — well within the typical fine-tuning budget</li>
532
532
<li><strong>256 seq len:</strong> Covers 95%+ of PII-containing text. Longer sequences have diminishing returns and quadratic attention cost</li>
533
533
<li><strong>Linear schedule:</strong> Standard for NER fine-tuning. Cosine provides marginal benefit for <20epochruns</li>
<td>Single dataset (gaps in coverage), 5+ datasets (diminishing returns, more label noise)</td>
699
699
</tr>
700
700
</table>
@@ -754,7 +754,7 @@ <h2>FAQ: Optimizer & Training</h2>
754
754
</div>
755
755
<divclass="callout callout-blue">
756
756
<strongstyle="color: #60a5fa;">Q: Why 50x LR ratio specifically?</strong><br>
757
-
Derived empirically from the smoke test. Standard NER fine-tuning uses 2e-5–5e-5 for the backbone. The CRF head starts from random initialization with a 163×163 transition matrix — it needs aggressive learning. 1e-3 is a standard "train from scratch" LR. The 50x ratio ensures the backbone is gently tuned while the head learns fast.
757
+
Derived empirically from the smoke test. Standard NER fine-tuning uses 2e-5–5e-5 for the backbone. The CRF head starts from random initialization with an 89×89 transition matrix — it needs aggressive learning. 1e-3 is a standard "train from scratch" LR. The 50x ratio ensures the backbone is gently tuned while the head learns fast.
758
758
</div>
759
759
</div>
760
760
<div>
@@ -800,8 +800,8 @@ <h2>FAQ: Model Architecture</h2>
800
800
Constrained decoding (e.g., masking invalid transitions at inference) prevents illegal sequences but doesn't learn transition preferences. The CRF learns that <code>B-PERSON → I-PERSON</code> is much more likely than <code>B-PERSON → I-EMAIL</code>. This soft constraint improves boundary detection even when emission scores are ambiguous.
<strongstyle="color: #60a5fa;">Q: 163 BIO tags seems like a lot. Is the CRF transition matrix too large?</strong><br>
804
-
The transition matrix is 163×163 = 26,569 parameters. This is tiny compared to the model's 71M total. Many transitions are effectively zero (B-SSN → I-PERSON never happens), so the matrix is very sparse in practice. The Viterbi decode is O(T×K²) where T=seq_len, K=163 — still fast at ~2ms per sequence.
803
+
<strongstyle="color: #60a5fa;">Q: Is the CRF transition matrix large enough to matter?</strong><br>
804
+
The transition matrix is 89×89 = 7,921 parameters. This is tiny compared to the model's 71M total. Many transitions are effectively zero (B-SSN → I-PERSON never happens), so the matrix is very sparse in practice. The Viterbi decode is O(T×K²) where T=seq_len, K=89 — still fast at ~2ms per sequence.
805
805
</div>
806
806
<divclass="callout callout-blue">
807
807
<strongstyle="color: #60a5fa;">Q: What's the <code>prediction_step</code> override doing?</strong><br>
<strong>Our differentiator:</strong> Widest open-source PII entity coverage (81 types), CRF-enforced boundary consistency, CharCNN for format pattern detection, and ONNX-exportable for sub-5ms inference. The combination of breadth + speed + open source doesn't exist yet.
864
+
<strong>Our differentiator:</strong> Widest open-source PII entity coverage (44 types), CRF-enforced boundary consistency, CharCNN for format pattern detection, and ONNX-exportable for sub-5ms inference. The combination of breadth + speed + open source doesn't exist yet.
Copy file name to clipboardExpand all lines: pii-ner-v1/docs/model_comparison.html
+7-7Lines changed: 7 additions & 7 deletions
Original file line number
Diff line number
Diff line change
@@ -170,7 +170,7 @@ <h3>Sections</h3>
170
170
<divclass="section-title" id="tldr">0. The 30-Second Version</div>
171
171
172
172
<divclass="a">
173
-
<p><spanclass="model-tag tag-spacy">SpaCy</span> is fast and cheap but can only find about 7 out of 81 PII types. It needs regex bolted on for everything else. Good baseline, not a real PII solution on its own.</p>
173
+
<p><spanclass="model-tag tag-spacy">SpaCy</span> is fast and cheap but can only find about 7 out of 44 PII types. It needs regex bolted on for everything else. Good baseline, not a real PII solution on its own.</p>
174
174
<p><spanclass="model-tag tag-gliner">GLiNER 2</span> is flexible and can find any entity type you describe, without retraining. But it's 9x larger, 5-8x slower, degrades past 30 entity types, and has no character-level features for structured PII like SSNs.</p>
175
175
<p><spanclass="model-tag tag-datafog">DataFog v1</span> is purpose-built for PII. Smaller, faster, with character-level pattern detection and CRF sequence constraints. But it's unproven — the smoke test passed, but full training hasn't happened yet.</p>
176
176
<divclass="callout">
@@ -204,7 +204,7 @@ <h3>Sections</h3>
204
204
<divclass="q">What is DataFog PII-NER v1?</div>
205
205
<divclass="a">
206
206
<p>Our model. DeBERTa-v3-xsmall (70.7M params, 384-dim) combined with a Character CNN (catches patterns like XXX-XX-XXXX), a gating fusion layer (dynamically blends character and context signals per token), and a CRF head (enforces valid BIO tag sequences).</p>
207
-
<p>It's designed from the ground up for one task: finding PII in English text. Fixed taxonomy of 81 entity types, 4 severity tiers, trained on 360K PII-specific examples. Target: ONNX INT8, ~25 MB, runs on CPU at ~10ms per 128 tokens.</p>
207
+
<p>It's designed from the ground up for one task: finding PII in English text. Fixed taxonomy of 44 entity types, 4 severity tiers, trained on 360K PII-specific examples. Target: ONNX INT8, ~25 MB, runs on CPU at ~10ms per 128 tokens.</p>
208
208
<divclass="callout warn">
209
209
<strong>Important caveat:</strong> The smoke test passed (F1 0.992 on 100 memorized examples). Full training on 360K examples hasn't happened yet. The performance targets below are projections based on architecture design and literature, not measured results. SpaCy and GLiNER numbers are real. Ours are a bet.
210
210
</div>
@@ -244,7 +244,7 @@ <h3>Sections</h3>
244
244
<td><strong>PII types (native)</strong></td>
245
245
<tdclass="orange">~7 (PERSON, ORG, GPE, LOC, DATE, TIME, MONEY)</td>
<divclass="q">What about that "custom transformer" that got 0.95 F1 in the arxiv benchmark?</div>
553
553
<divclass="a">
554
554
<p>That number is real but misleading for our purposes. It was measured on <strong>CoNLL-2003</strong>, which has only 4 entity types: PERSON, LOCATION, ORGANIZATION, and MISC. The training data is ~14K sentences of Reuters newswire — real text, not synthetic.</p>
555
-
<p>Detecting 4 entity types in clean newswire text is a fundamentally easier task than detecting 46-81 PII types in diverse synthetic or real-world documents. A 0.95 on CoNLL-2003 doesn't tell us anything about how that model would perform on SSNs, credit cards, or medical record numbers — because those entities don't exist in CoNLL-2003.</p>
555
+
<p>Detecting 4 entity types in clean newswire text is a fundamentally easier task than detecting 44-55 PII types in diverse synthetic or real-world documents. A 0.95 on CoNLL-2003 doesn't tell us anything about how that model would perform on SSNs, credit cards, or medical record numbers — because those entities don't exist in CoNLL-2003.</p>
556
556
<p>The useful takeaway: fine-tuned transformers <em>with CRF</em> consistently outperform other approaches on NER. The CRF in that same benchmark scored 0.93, nearly matching the custom transformer. Our architecture includes both.</p>
<p><strong>1. See character patterns.</strong> GLiNER and SpaCy operate on subword tokens. They never see the raw characters. Our CharCNN looks at the actual characters of each token and can learn that "three digits, dash, two digits, dash, four digits" is an SSN pattern. GLiNER sees <code>["_555", "-12", "-34", "56"]</code> and has to learn the pattern indirectly from subword token co-occurrence. This is why NVIDIA's GLiNER-PII gets only 0.64 F1 on AI4Privacy despite being 460M parameters.</p>
698
698
<p><strong>2. Enforce sequence validity.</strong> Our CRF head physically cannot output <code>I-EMAIL</code> after <code>B-PERSON</code>. The transition matrix makes that transition score negative infinity. GLiNER uses span matching with greedy decoding — it can produce contradictory extractions (overlapping spans with different types) and has no mechanism to enforce sequential BIO constraints.</p>
699
-
<p><strong>3. Run without entity type tokens in the input.</strong> GLiNER prepends entity type names as tokens, which means: (a) the effective sequence gets longer, (b) with 50+ types, you're adding 50+ tokens to every input, and (c) performance degrades beyond ~30 types because the entity type tokens compete with the actual text for attention. Our model always predicts over all 81 types simultaneously with zero additional input overhead.</p>
699
+
<p><strong>3. Run without entity type tokens in the input.</strong> GLiNER prepends entity type names as tokens, which means: (a) the effective sequence gets longer, (b) with 50+ types, you're adding 50+ tokens to every input, and (c) performance degrades beyond ~30 types because the entity type tokens compete with the actual text for attention. Our model always predicts over all 44 types simultaneously with zero additional input overhead.</p>
700
700
</div>
701
701
702
702
<divclass="q">What can GLiNER do that our model fundamentally can't?</div>
0 commit comments