Bug
_from_exported_parquet_files_and_dataset_infos() uses itertools.groupby() on unsorted exported_parquet_files.
If the same config appears again after another config, the earlier shard URLs are lost.
Repro
from itertools import groupby
from operator import itemgetter
exported = [
{"config": "default", "split": "train", "url": ".../train-0000.parquet"},
{"config": "other", "split": "train", "url": ".../other/train.parquet"},
{"config": "default", "split": "train", "url": ".../train-0001.parquet"},
]
metadata_configs = {
config_name: [p["url"] for _, ps in groupby(rows, itemgetter("split")) for p in ps]
for config_name, rows in groupby(exported, itemgetter("config"))
}
# default keeps only train-0001Proof (failing before fix)
FAILED - default only had 1 URL instead of 2
Fix
Sort by (config, split) before both groupby() calls.
I'll follow up with a PR + regression test.
Bug
_from_exported_parquet_files_and_dataset_infos()usesitertools.groupby()on unsortedexported_parquet_files.If the same
configappears again after another config, the earlier shard URLs are lost.Repro
Proof (failing before fix)
Fix
Sort by
(config, split)before bothgroupby()calls.I'll follow up with a PR + regression test.