Skip to content

Commit 93f59b7

Browse files
authored
Add memory guard to dasymetric validate_disaggregation (#1127)
* Add sweep-performance design spec Parallel subagent triage + ralph-loop workflow for auditing all xrspatial modules for performance bottlenecks, OOM risk under 30TB dask workloads, and backend-specific anti-patterns. * Add sweep-performance implementation plan 7 tasks covering command scaffold, module scoring, parallel subagent dispatch, report merging, ralph-loop generation, and smoke tests. * Add sweep-performance slash command * Add memory guard to dasymetric validate_disaggregation (#1126)
1 parent 9edd073 commit 93f59b7

File tree

1 file changed

+16
-0
lines changed

1 file changed

+16
-0
lines changed

xrspatial/dasymetric.py

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -685,6 +685,22 @@ def validate_disaggregation(
685685
"""
686686
values_dict = _normalize_values(values)
687687

688+
# Memory guard: validation requires both arrays in RAM.
689+
for arr, label in [(result.data, 'result'), (zones.data, 'zones')]:
690+
if has_dask_array() and isinstance(arr, da.Array):
691+
est = np.prod(arr.shape) * arr.dtype.itemsize
692+
try:
693+
from xrspatial.zonal import _available_memory_bytes
694+
avail = _available_memory_bytes()
695+
except ImportError:
696+
avail = 2 * 1024**3
697+
if est * 2 > 0.8 * avail:
698+
raise MemoryError(
699+
f"validate_disaggregation needs to materialize the "
700+
f"{label} array (~{est / 1e9:.1f} GB) but only "
701+
f"~{avail / 1e9:.1f} GB available."
702+
)
703+
688704
# extract numpy arrays from any backend
689705
result_np = _to_numpy(result.data)
690706
zones_np = _to_numpy(zones.data)

0 commit comments

Comments
 (0)