Skip to content

Commit 831d774

Browse files
davanstrienclaude
andauthored
provenance stamp v0: canonical convention + retrofit 4 hero recipes (#81)
- AGENTS.md: one canonical provenance-stamp convention (copy-block) — 'Produced on HF Jobs' gated on JOB_ID (set in-container by Jobs; a local run must not claim it), hardware from ACCELERATOR, reproduce command is 'hf jobs uv run', org-level credit, tags uv-script + hf-jobs (when on Jobs) as the adoption meters. - glm-ocr, nanonets-ocr, rolm-ocr: align existing card Reproduction sections to the convention (bare 'uv run' -> 'hf jobs uv run', conditional origin line, conditional hf-jobs tag, fold the footer into the origin line). - generate-embeddings: fix unconditional 'Produced on Hugging Face Jobs' claim (false for local runs), add card frontmatter tags (had none), add canonical Reproduction section. Meter baseline 2026-07-08: 431 public datasets tagged uv-script (280 davanstrien, 151 external across 40+ owners). Claude-Session: https://claude.ai/code/session_01QyEUD557s6w9CaJoNQo1yY Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent a279051 commit 831d774

5 files changed

Lines changed: 80 additions & 15 deletions

File tree

AGENTS.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,34 @@ workflow's optional `repo_id` / `repo_type` inputs.)
4949
dataset repos hosting code).
5050
- Every script's docstring shows a runnable **`hf jobs uv run`** example. README examples use the **HF raw
5151
URL** `https://huggingface.co/datasets/uv-scripts/<repo>/raw/main/<script>.py` (the trackable invocation).
52+
- **Provenance stamp (canonical — copy this, don't invent variants).** Every output dataset card a recipe
53+
pushes ends with a `## Reproduction` section in exactly this shape (inline it in the card f-string;
54+
no shared helper):
55+
56+
```python
57+
on_jobs = os.environ.get("JOB_ID") is not None # set by HF Jobs in-container
58+
hw = os.environ.get("ACCELERATOR") or "" # e.g. "a10g-small"; empty on CPU
59+
origin = (
60+
f"Produced on [Hugging Face Jobs](https://huggingface.co/docs/huggingface_hub/guides/jobs)"
61+
+ (f" (`{hw}`)" if hw else "")
62+
) if on_jobs else "Generated"
63+
```
64+
65+
```markdown
66+
## Reproduction
67+
68+
{origin} with the [`<script>.py`](<script-raw-url>) recipe from [uv-scripts](https://huggingface.co/uv-scripts). Run it yourself:
69+
70+
```bash
71+
hf jobs uv run <script-raw-url> <args>
72+
```
73+
```
74+
75+
Rules: the reproduce command is **`hf jobs uv run`** (never bare `uv run` — the card is the Jobs
76+
advertisement); the "Produced on HF Jobs" claim is **gated on `JOB_ID`** (a local run must not claim it);
77+
credit goes to the **org**, never a personal name; no badges. Card frontmatter tags: always `uv-script`,
78+
plus **`hf-jobs` when `JOB_ID` is set** — these are the adoption meters
79+
(`list_datasets(tags="uv-script")` / `tags="hf-jobs"` count stamped public outputs).
5280
- GPU scripts check `torch.cuda.is_available()` and exit with a clear message. Write outputs to the Hub
5381
(`push_to_hub`) or a bucket (`-v hf://…`), never local paths (Jobs disk is ephemeral). Name by task, not tool.
5482

embeddings/generate-embeddings.py

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -315,14 +315,28 @@ def main():
315315
effective = prompt_str if prompt_str is not None else next(
316316
(v for k in side_keys if (v := (getattr(model, "prompts", {}) or {}).get(k))), "")
317317
prompt_line = f"`{effective}`" if effective else "(none)"
318+
# Canonical provenance stamp (see AGENTS.md): Jobs claim gated on JOB_ID, set by HF Jobs in-container.
319+
script_url = "https://huggingface.co/datasets/uv-scripts/embeddings/raw/main/generate-embeddings.py"
320+
on_jobs = os.environ.get("JOB_ID") is not None
321+
hw = os.environ.get("ACCELERATOR") or ""
322+
origin = (
323+
"Produced on [Hugging Face Jobs](https://huggingface.co/docs/huggingface_hub/guides/jobs)"
324+
+ (f" (`{hw}`)" if hw else "")
325+
) if on_jobs else "Generated"
326+
jobs_tag = "\n- hf-jobs" if on_jobs else ""
318327
card = DatasetCard(
328+
f"---\ntags:\n- embeddings\n- uv-script\n- generated{jobs_tag}\n---\n\n"
319329
f"# {args.output_dataset}\n\n"
320330
f"Embeddings of [`{args.input_dataset}`](https://huggingface.co/datasets/{args.input_dataset}) "
321331
f"column `{args.column}`.\n\n"
322332
f"- Model: [`{args.model}`](https://huggingface.co/{args.model}) (dim {dim})\n"
323333
f"- Column: `{args.output_column}` · normalized: {args.normalize}\n"
324334
f"- Prompt prepended ({'query' if args.query_mode else 'document'} side): {prompt_line}\n\n"
325-
f"Produced on Hugging Face Jobs with `uv-scripts/embeddings/generate-embeddings.py`.\n"
335+
f"## Reproduction\n\n"
336+
f"{origin} with the [`generate-embeddings.py`]({script_url}) recipe "
337+
f"from [uv-scripts](https://huggingface.co/uv-scripts). Run it yourself:\n\n"
338+
f"```bash\nhf jobs uv run {script_url} \\\n"
339+
f" {args.input_dataset} <output-dataset> --column {args.column} --model {args.model}\n```\n"
326340
)
327341
# Retry the push with an XET-disable fallback: a transient upload failure here would
328342
# otherwise lose the whole (paid) embedding run.

ocr/glm-ocr.py

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -199,14 +199,23 @@ def create_dataset_card(
199199
"table": "table recognition",
200200
}
201201

202+
# Canonical provenance stamp (see AGENTS.md): Jobs claim gated on JOB_ID, set by HF Jobs in-container.
203+
on_jobs = os.environ.get("JOB_ID") is not None
204+
hw = os.environ.get("ACCELERATOR") or ""
205+
origin = (
206+
"Produced on [Hugging Face Jobs](https://huggingface.co/docs/huggingface_hub/guides/jobs)"
207+
+ (f" (`{hw}`)" if hw else "")
208+
) if on_jobs else "Generated"
209+
jobs_tag = "\n- hf-jobs" if on_jobs else ""
210+
202211
return f"""---
203212
tags:
204213
- ocr
205214
- document-processing
206215
- glm-ocr
207216
- markdown
208217
- uv-script
209-
- generated
218+
- generated{jobs_tag}
210219
---
211220
212221
# Document OCR using {model_name}
@@ -252,16 +261,16 @@ def create_dataset_card(
252261
253262
## Reproduction
254263
264+
{origin} with the [`glm-ocr.py`](https://huggingface.co/datasets/uv-scripts/ocr/raw/main/glm-ocr.py) recipe from [uv-scripts](https://huggingface.co/uv-scripts). Run it yourself:
265+
255266
```bash
256-
uv run https://huggingface.co/datasets/uv-scripts/ocr/raw/main/glm-ocr.py \\
267+
hf jobs uv run https://huggingface.co/datasets/uv-scripts/ocr/raw/main/glm-ocr.py \\
257268
{source_dataset} \\
258269
<output-dataset> \\
259270
--image-column {image_column} \\
260271
--batch-size {batch_size} \\
261272
--task {task}
262273
```
263-
264-
Generated with [UV Scripts](https://huggingface.co/uv-scripts)
265274
"""
266275

267276

ocr/nanonets-ocr.py

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,15 @@ def create_dataset_card(
130130
"""Create a dataset card documenting the OCR process."""
131131
model_name = model.split("/")[-1]
132132

133+
# Canonical provenance stamp (see AGENTS.md): Jobs claim gated on JOB_ID, set by HF Jobs in-container.
134+
on_jobs = os.environ.get("JOB_ID") is not None
135+
hw = os.environ.get("ACCELERATOR") or ""
136+
origin = (
137+
"Produced on [Hugging Face Jobs](https://huggingface.co/docs/huggingface_hub/guides/jobs)"
138+
+ (f" (`{hw}`)" if hw else "")
139+
) if on_jobs else "Generated"
140+
jobs_tag = "\n- hf-jobs" if on_jobs else ""
141+
133142
return f"""---
134143
viewer: false
135144
tags:
@@ -138,7 +147,7 @@ def create_dataset_card(
138147
- nanonets
139148
- markdown
140149
- uv-script
141-
- generated
150+
- generated{jobs_tag}
142151
---
143152
144153
# Document OCR using {model_name}
@@ -202,10 +211,10 @@ def create_dataset_card(
202211
203212
## Reproduction
204213
205-
This dataset was generated using the [uv-scripts/ocr](https://huggingface.co/datasets/uv-scripts/ocr) Nanonets OCR script:
214+
{origin} with the [`nanonets-ocr.py`](https://huggingface.co/datasets/uv-scripts/ocr/raw/main/nanonets-ocr.py) recipe from [uv-scripts](https://huggingface.co/uv-scripts). Run it yourself:
206215
207216
```bash
208-
uv run https://huggingface.co/datasets/uv-scripts/ocr/raw/main/nanonets-ocr.py \\
217+
hf jobs uv run https://huggingface.co/datasets/uv-scripts/ocr/raw/main/nanonets-ocr.py \\
209218
{source_dataset} \\
210219
<output-dataset> \\
211220
--image-column {image_column} \\
@@ -219,8 +228,6 @@ def create_dataset_card(
219228
220229
- **Processing Speed**: ~{num_samples / (float(processing_time.split()[0]) * 60):.1f} images/second
221230
- **GPU Configuration**: vLLM with {gpu_memory_utilization:.0%} GPU memory utilization
222-
223-
Generated with 🤖 [UV Scripts](https://huggingface.co/uv-scripts)
224231
"""
225232

226233

ocr/rolm-ocr.py

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -131,14 +131,23 @@ def create_dataset_card(
131131
"""Create a dataset card documenting the OCR process."""
132132
model_name = model.split("/")[-1]
133133

134+
# Canonical provenance stamp (see AGENTS.md): Jobs claim gated on JOB_ID, set by HF Jobs in-container.
135+
on_jobs = os.environ.get("JOB_ID") is not None
136+
hw = os.environ.get("ACCELERATOR") or ""
137+
origin = (
138+
"Produced on [Hugging Face Jobs](https://huggingface.co/docs/huggingface_hub/guides/jobs)"
139+
+ (f" (`{hw}`)" if hw else "")
140+
) if on_jobs else "Generated"
141+
jobs_tag = "\n- hf-jobs" if on_jobs else ""
142+
134143
return f"""---
135144
viewer: false
136145
tags:
137146
- ocr
138147
- text-extraction
139148
- rolmocr
140149
- uv-script
141-
- generated
150+
- generated{jobs_tag}
142151
---
143152
144153
# OCR Text Extraction using {model_name}
@@ -195,10 +204,10 @@ def create_dataset_card(
195204
196205
## Reproduction
197206
198-
This dataset was generated using the [uv-scripts/ocr](https://huggingface.co/datasets/uv-scripts/ocr) RolmOCR script:
207+
{origin} with the [`rolm-ocr.py`](https://huggingface.co/datasets/uv-scripts/ocr/raw/main/rolm-ocr.py) recipe from [uv-scripts](https://huggingface.co/uv-scripts). Run it yourself:
199208
200209
```bash
201-
uv run https://huggingface.co/datasets/uv-scripts/ocr/raw/main/rolm-ocr.py \\
210+
hf jobs uv run https://huggingface.co/datasets/uv-scripts/ocr/raw/main/rolm-ocr.py \\
202211
{source_dataset} \\
203212
<output-dataset> \\
204213
--image-column {image_column} \\
@@ -212,8 +221,6 @@ def create_dataset_card(
212221
213222
- **Processing Speed**: ~{num_samples / (float(processing_time.split()[0]) * 60):.1f} images/second
214223
- **GPU Configuration**: vLLM with {gpu_memory_utilization:.0%} GPU memory utilization
215-
216-
Generated with 🤖 [UV Scripts](https://huggingface.co/uv-scripts)
217224
"""
218225

219226

0 commit comments

Comments
 (0)