Skip to content

Backfill parquet versions for 4 datasets added after #241#245

Merged
skearnes merged 5 commits into
mainfrom
backfill-parquet-new-datasets
Jun 20, 2026
Merged

Backfill parquet versions for 4 datasets added after #241#245
skearnes merged 5 commits into
mainfrom
backfill-parquet-new-datasets

Conversation

@skearnes

@skearnes skearnes commented Jun 12, 2026

Copy link
Copy Markdown
Member

What

Backfills parquet versions for the four datasets submitted after the parquet conversion (#241), which therefore only had .pb.gz:

How

Generated 1:1 parquet siblings with scripts/convert_to_parquet.py (the singleton path — none of the four match a de-shard MERGE_SPEC, so no merging applies and each keeps its existing dataset_id). Reaction counts match the originals exactly (1227 / 120 / 136 / 50688), and the parquet outputs round-trip with matching row counts. main is merged in so the branch carries the #243 .pb.gz alongside its new parquet sibling.

The skip-update-submission label is set because these are format conversions of already-finalized datasets that should not be re-processed by process_dataset --update. Validation runs against the full dataset on merge to main.

🤖 Generated with Claude Code

Greptile Summary

This PR backfills Parquet siblings for datasets that were merged to main after the initial Parquet-conversion PR (#241) and therefore only had .pb.gz files. It also adds a maintainer-facing README section documenting the conversion and LFS-push workflow.

  • Four new LFS-tracked .parquet files are added: the catechol transient flow dataset (1227 reactions), two decarboxylative olefination datasets (120 and 136 reactions), and the Cernak C–N coupling dataset — all 1:1 singletons carrying their existing dataset_ids.
  • The README additions are accurate: the --dry-run flag exists in scripts/convert_to_parquet.py, and ord-schema==0.6.3 matches ORD_SCHEMA_TAG: v0.6.3 used in the CI workflow files.
  • The PR title and description say "3 datasets" but the diff includes a fourth parquet file (data/80/ord_dataset-805ad863…) for the Cernak dataset; the description should be updated before merge.

Confidence Score: 5/5

Safe to merge — adds LFS-tracked Parquet siblings for four already-finalised datasets and accurate maintainer documentation; no logic changes.

All changes are additive: four new Parquet LFS pointers and README documentation. The conversion script is idempotent and was run with validate_dataset.py passing locally. The README instructions accurately reflect the script flags and pinned schema version used in CI.

No files require special attention; the discrepancy between the PR description ('3 datasets') and the four added files is cosmetic.

Important Files Changed

Filename Overview
README.md Adds two new maintainer sections: 'Converting datasets to Parquet' and 'Pushing new LFS objects'. Both are accurate — --dry-run flag exists in the script, ord-schema==0.6.3 matches ORD_SCHEMA_TAG: v0.6.3 in the workflow files.
data/1e/ord_dataset-1ec2807f02fa4beda27d2a81b86eb843.parquet New LFS pointer for the catechol transient flow dataset (1227 reactions) — parquet sibling for the existing .pb.gz.
data/c7/ord_dataset-c703268ea43a4c7e802c6048b6166b34.parquet New LFS pointer for the Bayesian optimization decarboxylative olefination dataset (120 reactions) — parquet sibling for the existing .pb.gz.
data/dc/ord_dataset-dc0249930af34d17a3c76881a762aebf.parquet New LFS pointer for the transfer learning decarboxylative olefination dataset (136 reactions) — parquet sibling for the existing .pb.gz.
data/80/ord_dataset-805ad863feef48579d95d86a728035f4.parquet New LFS pointer for the Cernak C–N coupling dataset — parquet sibling added in the HEAD commit, but not mentioned in the PR title or description.

Reviews (4): Last reviewed commit: "Backfill parquet version for Cernak C–N ..." | Re-trigger Greptile

The submissions merged after the parquet conversion (#235 catechol transient
flow, #238 decarboxylative olefination x2) only had .pb.gz. Generated 1:1
parquet siblings with scripts/convert_to_parquet.py (all singletons; no
de-shard merge applies). Reaction counts match the originals (1227/120/136)
and validate_dataset.py passes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@skearnes skearnes added the skip-update-submission Skip Update step in Submission workflow; validation still runs. label Jun 12, 2026
skearnes and others added 2 commits June 11, 2026 22:10
Add README maintainer notes for backfilling Parquet with convert_to_parquet.py
and for routing new-LFS-object pushes to GitHub (since .lfsconfig points reads
at the HF mirror and has no pushurl).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@skearnes skearnes requested a review from bdeadman June 12, 2026 02:15
skearnes and others added 2 commits June 19, 2026 20:41
The Cernak 50,688-reaction Pd/Ni/Cu C–N coupling dataset (#243) was merged
after the parquet conversion (#241), so it only had .pb.gz. Generated its 1:1
parquet sibling with scripts/convert_to_parquet.py (singleton path; the name
matches no MERGE_SPEC, so it keeps its existing dataset_id). Reaction count
matches the original (50688) and the parquet round-trips.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@skearnes skearnes changed the title Backfill parquet versions for 3 datasets added after #241 Backfill parquet versions for 4 datasets added after #241 Jun 20, 2026
@skearnes skearnes merged commit ddb0d25 into main Jun 20, 2026
5 checks passed
@skearnes skearnes deleted the backfill-parquet-new-datasets branch June 20, 2026 01:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-update-submission Skip Update step in Submission workflow; validation still runs.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants