Skip to content

Commit ddb0d25

Browse files
skearnesclaude
andauthored
Backfill parquet versions for 4 datasets added after #241 (#245)
* Backfill parquet versions for 3 datasets added after #241 The submissions merged after the parquet conversion (#235 catechol transient flow, #238 decarboxylative olefination x2) only had .pb.gz. Generated 1:1 parquet siblings with scripts/convert_to_parquet.py (all singletons; no de-shard merge applies). Reaction counts match the originals (1227/120/136) and validate_dataset.py passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Document Parquet conversion and LFS push setup for maintainers Add README maintainer notes for backfilling Parquet with convert_to_parquet.py and for routing new-LFS-object pushes to GitHub (since .lfsconfig points reads at the HF mirror and has no pushurl). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Use two-space inline comments in README code blocks Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Backfill parquet version for Cernak C–N coupling dataset (#243) The Cernak 50,688-reaction Pd/Ni/Cu C–N coupling dataset (#243) was merged after the parquet conversion (#241), so it only had .pb.gz. Generated its 1:1 parquet sibling with scripts/convert_to_parquet.py (singleton path; the name matches no MERGE_SPEC, so it keeps its existing dataset_id). Reaction count matches the original (50688) and the parquet round-trips. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent bfea35a commit ddb0d25

5 files changed

Lines changed: 67 additions & 0 deletions

README.md

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,3 +166,58 @@ maintainer PRs that touch dataset files but should *not* be re-processed
166166
this way — e.g., format conversions or mass migrations of already-finalized
167167
data — apply the `skip-update-submission` label to the PR. The validation
168168
side of the workflow still runs.
169+
170+
### Converting datasets to Parquet
171+
172+
Datasets are stored as `.pb.gz`; most also have a Parquet sibling. New
173+
submissions arrive as `.pb.gz` only, so their Parquet versions are backfilled
174+
with [`scripts/convert_to_parquet.py`](scripts/convert_to_parquet.py). The
175+
script globs every `data/**/*.pb.gz`, merges the known de-shard groups (the
176+
`uspto-grants-YYYY_MM` monthly buckets and the `C8SC04228D` shards) into single
177+
outputs, converts everything else 1:1 (carrying the existing `dataset_id`), and
178+
skips any output that already exists — so it is safe to re-run and writes only
179+
what is missing.
180+
181+
It needs `ord_schema` at the pinned `ORD_SCHEMA_TAG` (see the workflows) and
182+
Python ≥3.11. Because it reads every `.pb.gz` to classify by name, pull the
183+
inputs first:
184+
185+
```bash
186+
uv venv --python 3.11 && source .venv/bin/activate # or: python -m venv .venv
187+
pip install "ord-schema==0.6.3" # match ORD_SCHEMA_TAG
188+
189+
git lfs pull --include="data/**/*.pb.gz" # the converter reads pb.gz content
190+
python scripts/convert_to_parquet.py --dry-run # preview what it will write
191+
python scripts/convert_to_parquet.py # write the Parquet siblings
192+
```
193+
194+
Commit the new `.parquet` files (they become LFS objects), push them (see
195+
[Pushing new LFS objects](#pushing-new-lfs-objects)), and open the PR with the
196+
`skip-update-submission` label. Validation runs against the full dataset on
197+
merge to `main`.
198+
199+
### Pushing new LFS objects
200+
201+
[`.lfsconfig`](.lfsconfig) routes LFS **reads** to the Hugging Face mirror and
202+
deliberately sets no `pushurl`, so a plain push would try to upload new objects
203+
to HF — which you cannot write. Point LFS uploads at GitHub for the push, and
204+
make sure git can authenticate to `github.com` over **HTTPS** for the LFS API
205+
(the LFS endpoint is HTTPS even when your `git` remote is SSH). The simplest
206+
auth is the GitHub CLI:
207+
208+
```bash
209+
git config lfs.pushurl https://github.com/open-reaction-database/ord-data.git/info/lfs
210+
gh auth setup-git # let git use your gh token for github.com over HTTPS
211+
git push -u origin <branch>
212+
```
213+
214+
Or, as a one-off without persisting any config:
215+
216+
```bash
217+
git -c lfs.pushurl=https://github.com/open-reaction-database/ord-data.git/info/lfs \
218+
-c 'credential.https://github.com.helper=!gh auth git-credential' \
219+
push -u origin <branch>
220+
```
221+
222+
Reads stay on the mirror; only your uploads go to GitHub. On merge to `main`,
223+
`huggingface_mirror.yml` copies the new objects to Hugging Face.
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
version https://git-lfs.github.com/spec/v1
2+
oid sha256:b1fd81bb3034644117b7d07e0471ede7370597bb4bd2433dd4b5c20bf02fad99
3+
size 99755
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
version https://git-lfs.github.com/spec/v1
2+
oid sha256:31b7c4762145e5dad973050373cd967bdd85caf3a9858767c83a17fed6c4af36
3+
size 5088296
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
version https://git-lfs.github.com/spec/v1
2+
oid sha256:159472bff84a548cb2def33ae92ff36c27139271372e38e70bba8eea152760d3
3+
size 16939
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
version https://git-lfs.github.com/spec/v1
2+
oid sha256:baaa0748a4a561705df275c3152a67ef52605a33249f674129f24c1abecfb2b9
3+
size 20160

0 commit comments

Comments
 (0)