Skip to content

Commit 8190a77

Browse files
coroaclaudeFabianHofmannFBumannpre-commit-ci[bot]
authored
perf: matrix accessor rewrite (#630)
* perf: add to_matrix_via_csr * perf: improve per-constraint csr matrix construction * Add conversion functions * feat: add ability to freeze constraints into csr * Add io.to_netcdf support for frozen Constraint * fix: re-implement matrices * Move sum_duplicates * feat: VariableLabelIndex * fix: until solve * fix: disentangle range and ncons * fix: don't freeze if model is chunked * fix typing errors * fix: bring back forward-refs * fix issues in tests * fix: add doc strings to VariableLabelIndex * test: relax dtype assertions for Windows np.int32 compatibility Use np.issubdtype() instead of exact dtype comparisons to allow np.int32/np.float32 that Windows may produce. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: review fixes for #630 (matrix accessor rewrite) (#632) * fix: review fixes for PR #630 (matrix accessor rewrite) - Fix __repr__ passing CSR positions instead of variable labels - Fix set_blocks failing on frozen Constraint - Extract _active_to_dataarray helper to reduce DRY violations - Simplify reset_dual to direct mutation instead of reconstruction - Add tests for freeze/mutable roundtrip, VariableLabelIndex, to_matrix_with_rhs, from_mutable mixed signs, repr correctness * feat: support mixed per-element signs in frozen Constraint, add rhs/lhs setters - Store _sign as str (uniform, fast) or np.ndarray (mixed, per-row) - Add rhs/lhs setters on Constraint via _refreeze_after pattern - Invalidate _dual on mutation; update netcdf serialization for array signs - Add tests for setters, mixed-sign freeze/roundtrip/sanitize/repr/netcdf * feat: mixed per-element signs in frozen Constraint, rhs/lhs setters, DRY helpers * rename Constraint to CSRConstraint * rename MutableConstraint to Constraint (original name) * fix: xpress crash with zero constraints and remove_variables not removing variable without referencing constraints * Fix MultiIndex deprecation warning in CSRConstraint by using assign_multiindex_safe * Add freeze_constraints option, default freeze to None (resolves from global setting) * bench: add pypsa carbon_management benchmark for direct solver path * Add set_names parameter to skip solver name-setting in direct IO, use polars for 3x faster name generation * perf: use polars to list logic in print_variables and print_constraints * Use non-deprecated formatting APIs in tests * Tighten direct-solver naming tests and repr formatting * fix mypy * Fix mypy failures in constraint and IO tests * Move freeze and direct IO naming settings to Model * perf: speed up mutable direct solver export * docs: add CSRConstraint documentation to release notes, API reference, and constraints notebook * perf: direct CSR-to-LP writer for frozen constraints (#631) * perf: direct CSR-to-LP writer for frozen constraints Override Constraint.to_polars() to expand CSR data directly into a polars DataFrame, bypassing the expensive mutable() → xarray Dataset reconstruction. Also override iterate_slices() to yield CSR row-batches instead of relying on xarray's isel(). Move eliminate_zeros() to freeze time (from_mutable) so the cleanup happens once rather than on every to_polars() call. LP write is now 20-40% faster than master across all benchmark models. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: handle mixed per-row signs in CSR-to-LP writer When _sign is a numpy array (per-row signs from from_rule with mixed <=/>=/= constraints), expand it per-nonzero via _sign[rows] instead of using pl.lit() which only works for scalar strings. Also slice _sign in iterate_slices when it's an array. Add test for frozen mixed-sign constraint LP output equivalence. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> * merge: move bench_pypsa_carbon_management into benchmarks/ suite, fix MatrixAccessor compat * fix: align CSRConstraint.iterate_slices return type with base class for mypy * fix: make assert_linequal compare semantic equality of expressions Sort both sides by variable labels along _term before comparing, so expressions with different term orderings (e.g. from CSR round-trip with freeze_constraints=True) are correctly recognized as equal. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix types from merge conflict * docs: note CSRConstraint API differences from Constraint * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * refac(CSRConstraint): simplify iterate_slices with _equal_nnz_slices helper Replace the convoluted cumsum/diff/range loop with a clean while-loop helper that uses searchsorted directly on indptr. Batch slices pass coords=[] since batches cover contiguous active rows, not a contiguous slice of the coordinate grid. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * deprecate(Constraint): add DeprecationWarning to .flat property Use to_polars() instead. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refac(ConstraintBase): make ncons/lhs/to_matrix abstract; move to Constraint _matrix_export_data becomes a method on Constraint instead of a module-level function. ncons, lhs, and to_matrix are now abstract in ConstraintBase, with xarray-based implementations on Constraint and CSR-based implementations on CSRConstraint. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(Model): add __weakref__ slot and drop stale matrices.clean_cached_properties call * refac(CSRConstraint): direct rhs setter, read-only lhs setter; drop xarray round-trip - rhs setter writes _rhs directly, rejects expressions - lhs setter raises AttributeError (call .mutable() to modify terms) - lhs getter skips mutable() wrapper, builds LinearExpression from _to_dataset - to_polars uses pl.lit for scalar sign * fix(types): drop duplicate MaskLike/PathLike; annotate sign_expr for mypy * refac(CSRConstraint): make rhs setter read-only; call .mutable() to modify --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Fabian Hofmann <fab.hof@gmx.de> Co-authored-by: Felix <117816358+FBumann@users.noreply.github.com> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent 22b5ca6 commit 8190a77

28 files changed

Lines changed: 2549 additions & 885 deletions

benchmarks/test_matrices.py

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -17,15 +17,15 @@
1717

1818
def _access_matrices(m):
1919
"""Access all matrix properties to force computation."""
20-
m.matrices.clean_cached_properties()
21-
_ = m.matrices.A
22-
_ = m.matrices.b
23-
_ = m.matrices.c
24-
_ = m.matrices.lb
25-
_ = m.matrices.ub
26-
_ = m.matrices.sense
27-
_ = m.matrices.vlabels
28-
_ = m.matrices.clabels
20+
matrices = m.matrices
21+
_ = matrices.A
22+
_ = matrices.b
23+
_ = matrices.c
24+
_ = matrices.lb
25+
_ = matrices.ub
26+
_ = matrices.sense
27+
_ = matrices.vlabels
28+
_ = matrices.clabels
2929

3030

3131
@pytest.mark.parametrize("n", BASIC_SIZES, ids=[f"n={n}" for n in BASIC_SIZES])
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
import pypsa
2+
import pytest
3+
4+
import linopy as lp
5+
6+
7+
@pytest.fixture(scope="module")
8+
def network():
9+
return pypsa.examples.carbon_management()
10+
11+
12+
def test_create_model_frozen(benchmark, network):
13+
benchmark(network.optimize.create_model, freeze_constraints=True)
14+
15+
16+
def test_create_model_mutable(benchmark, network):
17+
benchmark(network.optimize.create_model, freeze_constraints=False)
18+
19+
20+
@pytest.fixture(scope="module")
21+
def model_frozen(network):
22+
return network.optimize.create_model(freeze_constraints=True)
23+
24+
25+
@pytest.fixture(scope="module")
26+
def model_mutable(network):
27+
return network.optimize.create_model(freeze_constraints=False)
28+
29+
30+
def test_to_highspy_frozen(benchmark, model_frozen):
31+
benchmark(lp.io.to_highspy, model_frozen)
32+
33+
34+
def test_to_highspy_mutable(benchmark, model_mutable):
35+
benchmark(lp.io.to_highspy, model_mutable)
36+
37+
38+
def test_to_highspy_mutable_no_names(benchmark, model_mutable):
39+
benchmark(lp.io.to_highspy, model_mutable, set_names=False)
40+
41+
42+
def test_to_highspy_frozen_no_names(benchmark, model_frozen):
43+
benchmark(lp.io.to_highspy, model_frozen, set_names=False)

doc/api.rst

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,27 @@ Constraint
9797
constraints.Constraint.sign
9898
constraints.Constraint.rhs
9999
constraints.Constraint.flat
100+
constraints.Constraint.freeze
101+
constraints.Constraint.mutable
102+
103+
104+
CSRConstraint
105+
-------------
106+
107+
``CSRConstraint`` is a memory-efficient, immutable constraint representation backed by a scipy CSR sparse matrix. See the :doc:`creating-constraints` guide for usage.
108+
109+
.. autosummary::
110+
:toctree: generated/
111+
112+
constraints.CSRConstraint
113+
constraints.CSRConstraint.coeffs
114+
constraints.CSRConstraint.vars
115+
constraints.CSRConstraint.sign
116+
constraints.CSRConstraint.rhs
117+
constraints.CSRConstraint.ncons
118+
constraints.CSRConstraint.nterm
119+
constraints.CSRConstraint.freeze
120+
constraints.CSRConstraint.mutable
100121

101122

102123
Constraints

doc/release_notes.rst

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,11 @@ Release Notes
44
Upcoming Version
55
----------------
66

7+
* Increase speed of direct solver communication (~10x) in conversion functions like `to_highspy` through faster matrix creation (see below), leading to significant overall speed-up when setting `io_api="direct"`.
8+
* Add ``CSRConstraint``, a memory-efficient immutable constraint representation backed by scipy CSR sparse matrices. Provides up to 90% memory savings for constraints with many terms and 30–120x faster matrix generation for direct solver APIs.
9+
- Add ``freeze_constraints`` parameter to ``Model`` for globally storing constraints in CSR format on ``add_constraints``.
10+
- Add ``freeze`` parameter to ``Model.add_constraints`` for per-constraint opt-in to CSR storage.
11+
- Add ``freeze()`` and ``mutable()`` methods on ``Constraint`` and ``CSRConstraint`` for lossless conversion between xarray-backed and CSR-backed representations.
712

813
Version 0.7.0
914
-------------

examples/creating-constraints.ipynb

Lines changed: 126 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -235,8 +235,133 @@
235235
{
236236
"cell_type": "markdown",
237237
"id": "r0wxi7v1m7l",
238-
"source": "## Coordinate Alignment in Constraints\n\nAs an alternative to the ``<=``, ``>=``, ``==`` operators, linopy provides ``.le()``, ``.ge()``, and ``.eq()`` methods on variables and expressions. These methods accept a ``join`` parameter (``\"inner\"``, ``\"outer\"``, ``\"left\"``, ``\"right\"``) for explicit control over how coordinates are aligned when creating constraints. See the :doc:`coordinate-alignment` guide for details.",
238+
"metadata": {},
239+
"source": "## Coordinate Alignment in Constraints\n\nAs an alternative to the ``<=``, ``>=``, ``==`` operators, linopy provides ``.le()``, ``.ge()``, and ``.eq()`` methods on variables and expressions. These methods accept a ``join`` parameter (``\"inner\"``, ``\"outer\"``, ``\"left\"``, ``\"right\"``) for explicit control over how coordinates are aligned when creating constraints. See the :doc:`coordinate-alignment` guide for details."
240+
},
241+
{
242+
"cell_type": "markdown",
243+
"id": "csr-backend-intro",
244+
"metadata": {},
245+
"source": [
246+
"## CSR Backend (Advanced)\n",
247+
"\n",
248+
"By default, linopy stores each constraint as an `xarray.Dataset` (`Constraint`). This is flexible and allows full label-based indexing, but can use significant memory when constraints have many terms.\n",
249+
"\n",
250+
"For large models, linopy provides an alternative **CSR backend** via the `CSRConstraint` class. It stores the constraint coefficients as a [scipy CSR sparse matrix](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.csr_array.html) with flat numpy arrays for the right-hand side and signs. This can reduce memory usage by up to **90%** and speeds up matrix generation for direct solver APIs by **30–120x**.\n",
251+
"\n",
252+
"`CSRConstraint` is **immutable** — once frozen, the constraint data cannot be modified in place. You can always convert back to the mutable xarray-backed `Constraint` if needed."
253+
]
254+
},
255+
{
256+
"cell_type": "markdown",
257+
"id": "csr-per-constraint",
258+
"metadata": {},
259+
"source": [
260+
"### Freezing individual constraints\n",
261+
"\n",
262+
"Pass `freeze=True` to `add_constraints` to store a single constraint in CSR format:"
263+
]
264+
},
265+
{
266+
"cell_type": "code",
267+
"execution_count": null,
268+
"id": "csr-per-constraint-code",
269+
"metadata": {},
270+
"outputs": [],
271+
"source": [
272+
"import numpy as np\n",
273+
"\n",
274+
"from linopy import Model\n",
275+
"\n",
276+
"m2 = Model()\n",
277+
"y = m2.add_variables(coords=[np.arange(100)], name=\"y\")\n",
278+
"\n",
279+
"m2.add_constraints(y <= 10, name=\"upper\", freeze=True)\n",
280+
"\n",
281+
"print(type(m2.constraints[\"upper\"]))\n",
282+
"m2.constraints[\"upper\"]"
283+
]
284+
},
285+
{
286+
"cell_type": "markdown",
287+
"id": "csr-global",
288+
"metadata": {},
289+
"source": [
290+
"### Freezing all constraints globally\n",
291+
"\n",
292+
"Set `freeze_constraints=True` on the `Model` to automatically freeze every constraint added via `add_constraints`:"
293+
]
294+
},
295+
{
296+
"cell_type": "code",
297+
"execution_count": null,
298+
"id": "csr-global-code",
299+
"metadata": {},
300+
"outputs": [],
301+
"source": [
302+
"m3 = Model(freeze_constraints=True)\n",
303+
"z = m3.add_variables(coords=[np.arange(50)], name=\"z\")\n",
304+
"m3.add_constraints(z >= 0, name=\"lower\")\n",
305+
"m3.add_constraints(z <= 100, name=\"upper\")\n",
306+
"\n",
307+
"print(type(m3.constraints[\"lower\"]))\n",
308+
"print(type(m3.constraints[\"upper\"]))"
309+
]
310+
},
311+
{
312+
"cell_type": "markdown",
313+
"id": "csr-roundtrip",
314+
"metadata": {},
315+
"source": [
316+
"### Converting between representations\n",
317+
"\n",
318+
"Use `.freeze()` and `.mutable()` to convert between the two representations. The conversion is lossless:"
319+
]
320+
},
321+
{
322+
"cell_type": "code",
323+
"execution_count": null,
324+
"id": "csr-roundtrip-code",
325+
"metadata": {},
326+
"outputs": [],
327+
"source": [
328+
"frozen = m3.constraints[\"lower\"]\n",
329+
"print(f\"Frozen type: {type(frozen).__name__}\")\n",
330+
"\n",
331+
"thawed = frozen.mutable()\n",
332+
"print(f\"Mutable type: {type(thawed).__name__}\")\n",
333+
"\n",
334+
"refrozen = thawed.freeze()\n",
335+
"print(f\"Re-frozen type: {type(refrozen).__name__}\")"
336+
]
337+
},
338+
{
339+
"cell_type": "markdown",
340+
"id": "7843d42c",
341+
"source": "### API differences from `Constraint`\n\n`CSRConstraint` deliberately exposes a narrower API than the xarray-backed `Constraint`:\n\n- **No in-place mutation.** Setters such as `con.coeffs = ...`, `con.vars = ...`, `con.sign = ...`, `con.rhs = ...`, and `con.lhs = ...` are only available on `Constraint`.\n- **No label-based indexing.** `con.loc[...]` is only available on `Constraint`.\n- **Accessing `.coeffs` / `.vars` triggers reconstruction.** On a `CSRConstraint` these properties rebuild the full xarray `Dataset` on demand and emit a `PerformanceWarning`. For solver-oriented workflows prefer `con.to_matrix()` or work with the CSR data directly.\n\nIf you need any of the above, call `.mutable()` first to get a `Constraint`:\n\n```python\ncon = m.constraints[\"my_constraint\"].mutable()\ncon.loc[{\"time\": 0}] # label-based indexing now available\ncon.rhs = 5 # mutation now available\n```",
239342
"metadata": {}
343+
},
344+
{
345+
"cell_type": "markdown",
346+
"id": "csr-when-to-use",
347+
"metadata": {},
348+
"source": [
349+
"### When to use the CSR backend\n",
350+
"\n",
351+
"The CSR backend is most beneficial when:\n",
352+
"\n",
353+
"- Your model has **many constraints with many terms**.\n",
354+
"- **Memory** is a bottleneck.\n",
355+
"- You use a **direct solver API** (e.g. HiGHS, Gurobi Python bindings) rather than file-based I/O.\n",
356+
"\n",
357+
"For small models the overhead is negligible and the default xarray-backed `Constraint` is perfectly fine.\n",
358+
"\n",
359+
"Additionally, if you don't need variable and constraint names in the solver (e.g. for batch solves), you can disable name export for extra speed:\n",
360+
"\n",
361+
"```python\n",
362+
"m = Model(freeze_constraints=True, set_names_in_solver_io=False)\n",
363+
"```"
364+
]
240365
}
241366
],
242367
"metadata": {

linopy/__init__.py

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -19,8 +19,14 @@
1919
GREATER_EQUAL,
2020
LESS_EQUAL,
2121
EvolvingAPIWarning,
22+
PerformanceWarning,
23+
)
24+
from linopy.constraints import (
25+
Constraint,
26+
ConstraintBase,
27+
Constraints,
28+
CSRConstraint,
2229
)
23-
from linopy.constraints import Constraint, Constraints
2430
from linopy.expressions import LinearExpression, QuadraticExpression, merge
2531
from linopy.io import read_netcdf
2632
from linopy.model import Model, Variable, Variables, available_solvers
@@ -40,9 +46,12 @@
4046
pass
4147

4248
__all__ = (
43-
"Constraint",
49+
"CSRConstraint",
50+
"ConstraintBase",
4451
"Constraints",
52+
"Constraint",
4553
"EQUAL",
54+
"PerformanceWarning",
4655
"EvolvingAPIWarning",
4756
"GREATER_EQUAL",
4857
"LESS_EQUAL",

0 commit comments

Comments
 (0)