|
| 1 | +--- |
| 2 | +viewer: false |
| 3 | +license: apache-2.0 |
| 4 | +tags: |
| 5 | + - uv-script |
| 6 | + - ner |
| 7 | + - zero-shot |
| 8 | + - gliner |
| 9 | + - hf-jobs |
| 10 | +--- |
| 11 | + |
| 12 | +# GLiNER UV Scripts |
| 13 | + |
| 14 | +Zero-shot named-entity recognition over Hugging Face datasets using [GLiNER](https://github.com/urchade/GLiNER). Pass a list of entity types at runtime — no fine-tuning required. |
| 15 | + |
| 16 | +| Script | What it does | Output | |
| 17 | +|---|---|---| |
| 18 | +| `extract-entities.py` | Extract entities from a text column with a custom set of types | New `entities` column (list of `{start, end, text, label, score}`) | |
| 19 | + |
| 20 | +## Quick start |
| 21 | + |
| 22 | +Run on any HF dataset with a text column. No setup — `uv` resolves dependencies inline. |
| 23 | + |
| 24 | +```bash |
| 25 | +# Local CPU (small samples) |
| 26 | +uv run extract-entities.py \ |
| 27 | + librarian-bots/model_cards_with_metadata \ |
| 28 | + yourname/model-cards-entities \ |
| 29 | + --text-column card \ |
| 30 | + --entity-types Person Organization Dataset Model Framework \ |
| 31 | + --max-samples 100 |
| 32 | +``` |
| 33 | + |
| 34 | +## On HF Jobs |
| 35 | + |
| 36 | +```bash |
| 37 | +# CPU job — fine for small/medium datasets, free or near-free |
| 38 | +hf jobs uv run --flavor cpu-basic --secrets HF_TOKEN \ |
| 39 | + https://huggingface.co/datasets/uv-scripts/gliner/raw/main/extract-entities.py \ |
| 40 | + librarian-bots/model_cards_with_metadata \ |
| 41 | + yourname/model-cards-entities \ |
| 42 | + --text-column card \ |
| 43 | + --entity-types Person Organization Dataset Model Framework \ |
| 44 | + --max-samples 1000 |
| 45 | + |
| 46 | +# GPU job — worth it once you're processing >~1000 samples |
| 47 | +hf jobs uv run --flavor t4-small --secrets HF_TOKEN \ |
| 48 | + https://huggingface.co/datasets/uv-scripts/gliner/raw/main/extract-entities.py \ |
| 49 | + librarian-bots/model_cards_with_metadata \ |
| 50 | + yourname/model-cards-entities \ |
| 51 | + --text-column card \ |
| 52 | + --entity-types Person Organization Dataset Model Framework \ |
| 53 | + --device cuda \ |
| 54 | + --batch-size 32 |
| 55 | +``` |
| 56 | + |
| 57 | +## Reading from local files or a mounted bucket |
| 58 | + |
| 59 | +The `input_dataset` argument also accepts local file paths (parquet, jsonl, json, csv). Useful when the input is staged in a [Storage Bucket](https://huggingface.co/docs/hub/storage-buckets) — typical pattern for multi-stage pipelines where an upstream Job has prepared the data: |
| 60 | + |
| 61 | +```bash |
| 62 | +hf jobs uv run --flavor t4-small --secrets HF_TOKEN \ |
| 63 | + -v hf://buckets/yourname/working-data:/input \ |
| 64 | + https://huggingface.co/datasets/uv-scripts/gliner/raw/main/extract-entities.py \ |
| 65 | + /input/data.parquet \ |
| 66 | + yourname/output-entities \ |
| 67 | + --text-column text --entity-types Person Organization Location \ |
| 68 | + --device cuda --batch-size 32 |
| 69 | +``` |
| 70 | + |
| 71 | +Local paths are detected heuristically — anything starting with `/`, `./`, `../`, or ending in a known data extension is treated as a file path; otherwise the argument is interpreted as a HF dataset ID. |
| 72 | + |
| 73 | +## Recommended entity-type vocabularies |
| 74 | + |
| 75 | +GLiNER is open-vocabulary, so any string works. Some starting points: |
| 76 | + |
| 77 | +- **General news/web text**: `Person Organization Location Date Event` |
| 78 | +- **ML/AI text (e.g. model cards)**: `Person Organization Dataset Model Framework Metric License` |
| 79 | +- **Legal/policy**: `Person Organization Court Statute Date Jurisdiction` |
| 80 | +- **Biomedical**: `Drug Disease Gene Protein Symptom` |
| 81 | + |
| 82 | +Quality drops on very abstract or polysemous types — start simple, iterate. |
| 83 | + |
| 84 | +## Models |
| 85 | + |
| 86 | +Default: `urchade/gliner_multi-v2.1` (multilingual, ~600 MB). Override with `--gliner-model`. |
| 87 | + |
| 88 | +Other useful checkpoints: |
| 89 | +- `urchade/gliner_small-v2.1` — English, faster |
| 90 | +- `urchade/gliner_large-v2.1` — English, larger / higher quality |
| 91 | +- `knowledgator/gliner-multitask-large-v0.5` — multitask (NER + classification + relation) |
| 92 | + |
| 93 | +See the [Knowledgator org](https://huggingface.co/knowledgator) and [urchade's models](https://huggingface.co/urchade) for the full set. |
| 94 | + |
| 95 | +## Pairing with Label Studio |
| 96 | + |
| 97 | +Output of this script is a Hugging Face dataset of texts + extracted entities. To put those entities in front of human reviewers, see the `bootstrap-labels` skill (or the workflow it documents): pull this dataset's predictions into a Label Studio project for review, then export a corrected dataset back to the Hub. |
| 98 | + |
| 99 | +## Caveats |
| 100 | + |
| 101 | +- GLiNER predictions are **bootstrap labels** — useful as a starting point, not as ground truth. Plan a review pass before downstream training. |
| 102 | +- Texts longer than `--max-text-chars` (default 8000) are truncated. Long-form documents may need chunking + reassembly. |
| 103 | +- Entity types are case-sensitive labels in output. Pass them as you want them to appear. |
0 commit comments