Backfill parquet versions for 4 datasets added after #241

README.md

@@ -166,3 +166,58 @@ maintainer PRs that touch dataset files but should *not* be re-processed
 this way — e.g., format conversions or mass migrations of already-finalized
 data — apply the `skip-update-submission` label to the PR. The validation
 side of the workflow still runs.
+
+### Converting datasets to Parquet
+
+Datasets are stored as `.pb.gz`; most also have a Parquet sibling. New
+submissions arrive as `.pb.gz` only, so their Parquet versions are backfilled
+with [`scripts/convert_to_parquet.py`](scripts/convert_to_parquet.py). The
+script globs every `data/**/*.pb.gz`, merges the known de-shard groups (the
+`uspto-grants-YYYY_MM` monthly buckets and the `C8SC04228D` shards) into single
+outputs, converts everything else 1:1 (carrying the existing `dataset_id`), and
+skips any output that already exists — so it is safe to re-run and writes only
+what is missing.
+
+It needs `ord_schema` at the pinned `ORD_SCHEMA_TAG` (see the workflows) and
+Python ≥3.11. Because it reads every `.pb.gz` to classify by name, pull the
+inputs first:
+
+```bash
+uv venv --python 3.11 && source .venv/bin/activate  # or: python -m venv .venv
+pip install "ord-schema==0.6.3"  # match ORD_SCHEMA_TAG
+
+git lfs pull --include="data/**/*.pb.gz"  # the converter reads pb.gz content
+python scripts/convert_to_parquet.py --dry-run  # preview what it will write
+python scripts/convert_to_parquet.py  # write the Parquet siblings
+```
+
+Commit the new `.parquet` files (they become LFS objects), push them (see
+[Pushing new LFS objects](#pushing-new-lfs-objects)), and open the PR with the
+`skip-update-submission` label. Validation runs against the full dataset on
+merge to `main`.
+
+### Pushing new LFS objects
+
+[`.lfsconfig`](.lfsconfig) routes LFS **reads** to the Hugging Face mirror and
+deliberately sets no `pushurl`, so a plain push would try to upload new objects
+to HF — which you cannot write. Point LFS uploads at GitHub for the push, and
+make sure git can authenticate to `github.com` over **HTTPS** for the LFS API
+(the LFS endpoint is HTTPS even when your `git` remote is SSH). The simplest
+auth is the GitHub CLI:
+
+```bash
+git config lfs.pushurl https://github.com/open-reaction-database/ord-data.git/info/lfs
+gh auth setup-git  # let git use your gh token for github.com over HTTPS
+git push -u origin <branch>
+```
+
+Or, as a one-off without persisting any config:
+
+```bash
+git -c lfs.pushurl=https://github.com/open-reaction-database/ord-data.git/info/lfs \
+    -c 'credential.https://github.com.helper=!gh auth git-credential' \
+    push -u origin <branch>
+```
+
+Reads stay on the mirror; only your uploads go to GitHub. On merge to `main`,
+`huggingface_mirror.yml` copies the new objects to Hugging Face.