dev notes

Author: wpbonelli

docs/dimension-resolution.md

@@ -0,0 +1,254 @@
+# Dimension Resolution
+
+## Current Problem
+
+When loading components with array fields that depend on dimensions (e.g., TDIS with `perlen` array of shape `(nper,)`), the dimension resolution fails:
+
+```
+ValueError: Couldn't resolve dims: ['nper']
+```
+
+This happens because `_resolve_dimensions()` tries to find `nper` on `self_` during attrs initialization, but `nper` hasn't been set yet.
+
+## Planned Architecture (Issue #167)
+
+The data model refactor introduces explicit dimension handling via protocols:
+
+### DimensionProvider Protocol
+
+Components that define dimensions implement this:
+
+```python
+class DimensionProvider(Protocol):
+    def get_dimensions(self) -> dict[str, int]: ...
+    def get_dimension_scope(self, dim_name: str) -> str | type: ...
+```
+
+- **TDIS** provides `nper` at global/simulation scope
+- **DIS/DISV/DISU** provide `nlay`, `nrow`, `ncol`, `nodes` at model scope
+
+### DimensionRegistry Protocol
+
+Containers that aggregate dimensions:
+
+```python
+class DimensionRegistry(Protocol):
+    def register_dimension_provider(self, provider: DimensionProvider): ...
+    def resolve_dimension(self, dim_name: str) -> int | None: ...
+    def get_all_dimensions(self) -> dict[str, int]: ...
+```
+
+- **Simulation** registry handles global scope (time dimensions)
+- **Model** registry handles model scope (grid dimensions)
+- Resolution walks up parent chain: package → model → simulation
+
+### Loading Order
+
+With this design, dimension resolution naturally works if we load in the right order:
+
+```
+1. Load Simulation (creates registry)
+   ↓
+2. Load TDIS → registers nper with simulation
+   ↓
+3. Load Model (creates registry, links to simulation)
+   ↓
+4. Load DIS → registers nlay, nrow, ncol with model
+   ↓
+5. Load other packages → resolve dims from parent chain
+```
+
+## Design Considerations
+
+### Self-Contained Components (TDIS, DIS)
+
+For components that DEFINE their own dimensions, like TDIS:
+
+```python
+@xattree
+class Tdis(Package):
+    nper: int = dim(block="dimensions", default=1)
+    perlen: NDArray = array(dims=("nper",), ...)
+```
+
+The `nper` field is defined ON the same class that uses it. The issue is attrs field initialization order - `perlen`'s converter runs before `nper` is set.
+
+**Solution**: For self-contained dimension providers, we can:
+1. Extract dimensions from parsed data BEFORE calling `__init__`
+2. Pass them explicitly to converters
+3. Or: defer array validation until after `__init__` completes
+
+### Cross-Component Dimensions
+
+For packages that depend on dimensions from OTHER components:
+
+```python
+@xattree
+class Npf(Package):
+    k: NDArray = array(dims=("nlay", "nrow", "ncol"), ...)  # From DIS
+```
+
+**Solution**: These dimensions come from the parent chain:
+1. Package has `parent` field pointing to Model
+2. Model registry has dimensions from DIS
+3. `resolve_dimension("nlay")` walks up to find it
+
+### Partial/Incomplete Simulations
+
+Sometimes we want to load components without their dimension providers:
+- Loading a single package for inspection
+- Testing with mock data
+- Partial simulations
+
+**Solution**: `strict=False` mode that:
+- Skips dimension validation
+- Accepts arrays of any shape
+- Logs warnings instead of raising errors
+
+```python
+def structure(data, path, component_type, *, strict=True):
+    if not strict:
+        # Skip dimension validation, accept any array shapes
+        ...
+```
+
+## Recommended Implementation
+
+### Phase 1: Make Current Loading Work
+
+For now, extract dimensions from parsed data and pass to converters:
+
+```python
+def structure(data, path, component_type):
+    # Extract dimensions from the data itself
+    dims = {}
+    if "dimensions" in data:
+        dims.update(data["dimensions"])
+
+    # Make dims available during structuring
+    # Option A: Thread-local context
+    # Option B: Pass via __init__ parameter
+    # Option C: Store on a context object passed to converters
+```
+
+### Consider: attrs → pydantic Migration
+
+The self-contained bootstrapping problem (TDIS needs `nper` set before `perlen` converter runs) could be solved by migrating from attrs to pydantic.
+
+**Why pydantic helps:**
+
+Pydantic provides explicit control over validation ordering:
+
+```python
+from pydantic import BaseModel, model_validator
+
+class Tdis(BaseModel):
+    nper: int = 1
+    perlen: list[float]  # Raw data, not yet structured
+    nstp: list[int]
+    tsmult: list[float]
+
+    @model_validator(mode='after')
+    def structure_arrays(self) -> 'Tdis':
+        # Runs AFTER all fields are set, so self.nper is available
+        self.perlen = np.array(self.perlen).reshape((self.nper,))
+        self.nstp = np.array(self.nstp).reshape((self.nper,))
+        self.tsmult = np.array(self.tsmult).reshape((self.nper,))
+        return self
+```
+
+**Key pydantic features:**
+- `model_validator(mode='before')` - transform input data before field assignment
+- `model_validator(mode='after')` - validate/transform after ALL fields are set
+- `field_validator` - per-field validation with ordering control
+- `model_config` - fine-grained control over validation behavior
+
+**With pydantic, the flow becomes:**
+1. Set `nper=3` (scalar, no validation needed)
+2. Set `perlen=[1.0, 2.0, 3.0]` (raw list, no structuring yet)
+3. `model_validator(mode='after')` runs
+4. Now `self.nper` is available → structure arrays with correct shape
+
+This cleanly separates field assignment from array structuring, solving the bootstrapping problem without workarounds.
+
+### Phase 2: Explicit Dimension Registration (Issue #167)
+
+Once the refactor is complete:
+
+1. Simulation/Model implement `DimensionRegistry`
+2. TDIS/DIS implement `DimensionProvider`
+3. Auto-register providers when assigned as children
+4. Resolution walks parent chain naturally
+
+### Phase 3: Loading Order Enforcement
+
+For full simulations, enforce loading order:
+
+```python
+def load_simulation(path):
+    sim = Simulation._load_header(path)  # Just options, no children
+
+    # Load dimension providers first
+    sim.tdis = Tdis.load(...)  # Registers nper
+
+    for model_binding in sim.model_bindings:
+        model = Model._load_header(...)
+        model.dis = Dis.load(...)  # Registers grid dims
+
+        # Now load packages that need dimensions
+        for pkg_binding in model.package_bindings:
+            pkg = Package.load(...)  # Can resolve dims from parent
+            model.add_package(pkg)
+
+        sim.add_model(model)
+```
+
+## Loading Order: Detect Dimension Providers First
+
+During binding resolution, detect dimension-provider components and load them before consumers:
+
+```python
+def _resolve_bindings_in_dict(data, workspace, target_type):
+    # Partition bindings into dimension providers vs consumers
+    dim_providers = []  # TDIS, DIS, DISV, DISU
+    dim_consumers = []  # Everything else
+
+    for field_name, bindings in binding_fields:
+        for binding in bindings:
+            component_type = _resolve_component_class(binding[0])
+            if is_dimension_provider(component_type):
+                dim_providers.append((field_name, binding))
+            else:
+                dim_consumers.append((field_name, binding))
+
+    # Load dimension providers FIRST
+    for field_name, binding in dim_providers:
+        component = Binding.to_component(binding, workspace)
+        resolved[field_name] = component
+        # Dimensions now registered with parent
+
+    # Load consumers - they can now resolve dimensions
+    for field_name, binding in dim_consumers:
+        component = Binding.to_component(binding, workspace)
+        resolved[field_name] = component
+```
+
+**Detecting dimension providers:**
+- Check for `dim()` fields in the component class
+- Or maintain a registry: `{Tdis, Dis, Disv, Disu, ...}`
+- Or use a marker: `_is_dimension_provider = True`
+
+## Example Flow (Future State)
+
+```
+mfsim.nam
+├── simulation.tdis  → Tdis(nper=3) → registers nper=3
+└── gwf.nam
+    ├── gwf.dis      → Dis(nlay=5, nrow=10, ncol=10) → registers grid dims
+    ├── gwf.npf      → Npf(k=array(5,10,10)) → resolves from parent
+    └── gwf.chd      → Chd(spd=array(3,...)) → resolves nper from sim
+```
+
+Each package can resolve dimensions by walking up:
+- `Npf.parent` → `GwfModel` → has `nlay`, `nrow`, `ncol`
+- `GwfModel.parent` → `Simulation` → has `nper`