Skip to content

Commit afb8e1f

Browse files
baogorekclaude
andcommitted
chore: format code and update changelog for parallel publishing
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent ae0237c commit afb8e1f

3 files changed

Lines changed: 16 additions & 7 deletions

File tree

changelog_entry.yaml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,3 +14,6 @@
1414
added:
1515
- Added CPS_2024_Full class for full-sample 2024 CPS generation
1616
- Added raw_cache utility for Census data caching
17+
- Added atomic parallel local area H5 publishing with Modal Volume staging
18+
- Added manifest validation with SHA256 checksums for versioned uploads
19+
- Added HuggingFace retry logic with exponential backoff to fix timeout errors

modal_app/local_area.py

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -361,8 +361,12 @@ def coordinate_publish(
361361
calibration_dir.mkdir(parents=True, exist_ok=True)
362362

363363
# hf_hub_download preserves directory structure, so files are in calibration/ subdir
364-
weights_path = calibration_dir / "calibration" / "w_district_calibration.npy"
365-
dataset_path = calibration_dir / "calibration" / "stratified_extended_cps.h5"
364+
weights_path = (
365+
calibration_dir / "calibration" / "w_district_calibration.npy"
366+
)
367+
dataset_path = (
368+
calibration_dir / "calibration" / "stratified_extended_cps.h5"
369+
)
366370
db_path = calibration_dir / "calibration" / "policy_data.db"
367371

368372
if not all(p.exists() for p in [weights_path, dataset_path, db_path]):

modal_app/worker_script.py

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -95,11 +95,13 @@ def main():
9595

9696
except Exception as e:
9797
results["failed"].append(f"{item_type}:{item_id}")
98-
results["errors"].append({
99-
"item": f"{item_type}:{item_id}",
100-
"error": str(e),
101-
"traceback": traceback.format_exc(),
102-
})
98+
results["errors"].append(
99+
{
100+
"item": f"{item_type}:{item_id}",
101+
"error": str(e),
102+
"traceback": traceback.format_exc(),
103+
}
104+
)
103105
print(f"FAILED {item_type}:{item_id}: {e}", file=sys.stderr)
104106

105107
print(json.dumps(results))

0 commit comments

Comments
 (0)