Skip to content

Commit 04c7cb4

Browse files
FBumannclaude
andcommitted
Merge master into feat/arithmetic-convention
Brings in the coords-as-truth stack (#732), the alignment.py module split (#742), has_terms (#743), and the MatrixAccessor caching (#716). Conflict resolutions (consistent rule: keep this branch's v1/legacy dispatch structure, use master's conversion calls inside it): - expressions.py: _add_constant, _apply_constant_op_{v1,legacy}, and to_constraint use broadcast_to_coords(..., strict=False) instead of as_dataarray; SUPPORTED_CONSTANT_TYPES -> CONSTANT_TYPES. - variables.py: to_linexpr converts via broadcast_to_coords(strict=False), then applies the v1/legacy absence handling. - __init__.py: align from linopy.alignment + LinopySemanticsWarning export. Test adaptations for post-#732 APIs and semantics: - test_legacy_violations: name the MultiIndex coords entry (required since #732). - test_linear_expression: the masked-addend tails of test_nterm and test_variable_names pin legacy absence behavior; split into @pytest.mark.legacy / @pytest.mark.v1 pairs (section 6 divergence). Full suite under both semantics: 6446 passed, 514 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 parents a91b901 + d5a82a2 commit 04c7cb4

29 files changed

Lines changed: 3026 additions & 1074 deletions

.github/workflows/test-models.yml

Lines changed: 27 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,7 @@ on:
55
branches:
66
- master
77
pull_request:
8-
branches:
9-
- master
8+
branches: ["*"]
109
schedule:
1110
- cron: "0 5 * * *"
1211

@@ -24,7 +23,7 @@ jobs:
2423
matrix:
2524
version:
2625
- master
27-
# - latest # Activate when v0.14.0 is released
26+
# - latest
2827

2928
defaults:
3029
run:
@@ -43,11 +42,23 @@ jobs:
4342
latest_tag=$(git describe --tags `git rev-list --tags --max-count=1`)
4443
git checkout $latest_tag
4544
45+
- name: Setup Pixi
46+
uses: prefix-dev/setup-pixi@v0.9.6
47+
with:
48+
pixi-version: v0.68.1
49+
cache: true
50+
# Do not cache in branches
51+
cache-write: ${{ github.event_name == 'push' && github.ref_name == 'master' }}
52+
53+
- name: Setup cache keys
54+
run: |
55+
echo "WEEK=$(date +'%Y%U')" >> $GITHUB_ENV # data and cutouts
56+
4657
# Only run check if package is not pinned
47-
- name: Check if inhouse package is pinned
58+
- name: Check if linopy package is pinned
4859
run: |
49-
grep_line=$(grep -- '- pypsa' envs/environment.yaml)
50-
if [[ $grep_line == *"<"* || $grep_line == *"=="* ]]; then
60+
grep_line=$(grep -- '${{ github.event.repository.name }} = ' pixi.toml)
61+
if [[ $grep_line == *"<* || $grep_line == *"==* ]]; then
5162
echo "pinned=true" >> $GITHUB_ENV
5263
else
5364
echo "pinned=false" >> $GITHUB_ENV
@@ -67,37 +78,23 @@ jobs:
6778
cutouts
6879
key: data-cutouts-${{ env.week }}
6980

70-
- uses: conda-incubator/setup-miniconda@v4
71-
if: env.pinned == 'false'
72-
with:
73-
activate-environment: pypsa-eur
74-
75-
- name: Cache Conda env
76-
if: env.pinned == 'false'
77-
uses: actions/cache@v5
78-
with:
79-
path: ${{ env.CONDA }}/envs
80-
key: conda-pypsa-eur-${{ env.week }}-${{ hashFiles('envs/linux-64.lock.yaml') }}
81-
id: cache-env
82-
83-
- name: Update environment
84-
if: env.pinned == 'false' && steps.cache-env.outputs.cache-hit != 'true'
85-
run: conda env update -n pypsa-eur -f envs/linux-64.lock.yaml
86-
8781
- name: Install package from ref
8882
if: env.pinned == 'false'
8983
run: |
90-
python -m pip install git+https://github.com/${{ github.repository }}@${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
84+
pixi remove pypsa
85+
pixi remove linopy
86+
pixi add --pypi --git https://github.com/${{ github.repository }}.git ${{ github.event.repository.name }} --rev ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
87+
pixi add --pypi pypsa
9188
9289
- name: Run snakemake test workflows
9390
if: env.pinned == 'false'
9491
run: |
95-
make test
92+
pixi run integration-tests
9693
9794
- name: Run unit tests
9895
if: env.pinned == 'false'
9996
run: |
100-
make unit-test
97+
pixi run unit-tests
10198
10299
- name: Upload artifacts
103100
if: env.pinned == 'false'
@@ -109,3 +106,7 @@ jobs:
109106
.snakemake/log
110107
results
111108
retention-days: 3
109+
110+
- name: Show remaining disk space
111+
if: always()
112+
run: df -h

doc/api.rst

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -248,6 +248,7 @@ Structure
248248
expressions.LinearExpression.coeffs
249249
expressions.LinearExpression.const
250250
expressions.LinearExpression.nterm
251+
expressions.LinearExpression.has_terms
251252

252253
Conversion
253254
----------
@@ -288,6 +289,7 @@ Structure
288289
expressions.QuadraticExpression.coeffs
289290
expressions.QuadraticExpression.const
290291
expressions.QuadraticExpression.nterm
292+
expressions.QuadraticExpression.has_terms
291293

292294
Conversion
293295
----------

doc/release_notes.rst

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,9 @@ Release Notes
44
Upcoming Version
55
----------------
66

7+
* Add documentation about `LinearExpression.where` with `drop=True`. Add `BaseExpression.variable_names` property.
8+
* Add ``BaseExpression.has_terms`` property: boolean array, true at slots with at least one live term (`#741 <https://github.com/PyPSA/linopy/issues/741>`_).
9+
710
**Features**
811

912
*Inspect the solver after solving*
@@ -49,20 +52,29 @@ Most users should keep calling ``model.solve(...)``. If you want more control, y
4952
**Deprecations**
5053

5154
* ``Solver.solve_problem``, ``Solver.solve_problem_from_model``, and ``Solver.solve_problem_from_file`` still work but emit a ``DeprecationWarning``. Use ``Solver.from_name(...).solve()`` (or simply ``model.solve(...)``) instead. They will be removed in a future release.
55+
* **Implicit MultiIndex-level projection is deprecated.** Passing an input indexed by a *level* of a stacked-``MultiIndex`` dimension (e.g. per-``period`` bounds onto a ``(period, timestep)`` ``snapshot`` index) emits an ``EvolvingAPIWarning`` — in arithmetic and in ``add_variables`` / ``add_constraints`` — and will raise under the upcoming v1 convention. Project the input onto the dimension explicitly (select with the dimension's level values) to keep current behavior. Affects PyPSA multi-investment models. See Bug Fixes below for details.
5256

5357
**Bug Fixes**
5458

59+
* ``add_variables`` / ``add_constraints``: extends 0.7.0's coords-as-truth rule to ``lower``, ``upper`` and ``mask`` for every bound type and dim order. Pandas ``Series`` / ``DataFrame`` bounds or masks missing a dimension are broadcast to ``coords`` instead of being silently dropped (`#709 <https://github.com/PyPSA/linopy/issues/709>`__); the variable's dimension order always follows ``coords`` (`#706 <https://github.com/PyPSA/linopy/issues/706>`__); bare-tuple coord entries (``coords=[(0, 1, 2)]``) now behave like lists. Mismatched values or extra dims raise ``ValueError`` with a labelled message; sparse-coord masks (formerly a v0.6.3 ``FutureWarning``, #580) raise ``ValueError``, and masks with dims not in the data raise ``ValueError`` instead of ``AssertionError``.
60+
* Pandas inputs whose index names *levels* of a stacked-``MultiIndex`` ``coords`` dimension are now projected onto that dimension: a level subset broadcasts across the others, the full set aligns element-wise. This fixes PyPSA multi-investment arithmetic (e.g. an expression over a ``(period, timestep)`` ``snapshot`` MultiIndex times a ``period``-indexed weighting). In ``add_variables`` / ``add_constraints`` the input must provide a value for every level combination of the MultiIndex or a ``ValueError`` is raised (the error lists the missing combinations). **Implicit level projections are deprecated**: they emit an ``EvolvingAPIWarning`` everywhere — in arithmetic *and* in ``add_variables`` / ``add_constraints`` — and will raise under the upcoming v1 convention. Project the input onto the dimension explicitly (select with the dimension's level values) to keep current behavior. Aligning the full level set with full coverage stays silent. Strict validation also rejects a ``MultiIndex`` input with *unnamed* levels whose combinations don't match ``coords`` (previously a silent bypass, as such inputs can't be projected by level name).
61+
* ``add_piecewise_formulation`` now produces a reproducible dimension order in the broadcast breakpoint array. The previous set-based expansion gave a hash-randomized order that varied between processes.
5562
* SOS constraints on masked variables no longer cause solver-specific failures (Gurobi ``IndexError``, Xpress ``?404 Invalid column number``, LP parse errors, silent set corruption). ``Model.solve()`` and ``Model.to_file()`` now raise a clear ``NotImplementedError`` referring users to `#688 <https://github.com/PyPSA/linopy/issues/688>`__; pass ``reformulate_sos=True`` as a workaround.
5663
* ``Model.solve(..., reformulate_sos=True)`` now actually reformulates SOS constraints even when the solver supports them natively. Previously it was silently ignored with a warning.
64+
* Fix Mosek interface to inspect both the basic and IPM solutions and pick the one with the better status, so that an optimal crossover solution is not discarded when IPM terminates with a (near-)Farkas certificate.
5765

5866
**Breaking Changes**
5967

68+
* ``add_variables`` / ``add_constraints``: the v0.6.3 ``mask`` deprecations (#580) are now hard ``ValueError``\ s; an unnamed ``pd.MultiIndex`` in sequence-form ``coords`` raises ``TypeError`` unless paired with ``dims=[i]``. See Bug Fixes above.
69+
* Sequence-form ``coords`` entries can no longer be ``xarray.DataArray`` objects — they raise ``TypeError``. Pass the underlying index instead: ``variable.indexes[dim]`` (a ``pd.Index``).
6070
* ``available_solvers`` now lists all *installed* solvers, even ones without a working license. If you used it to decide "can I actually solve with X?", switch to ``linopy.licensed_solvers`` or ``SolverClass.license_status()``.
6171
* ``Model.solver_model`` and ``Model.solver_name`` are now read-only properties that delegate to ``model.solver``. You can't reassign them (only ``= None`` is allowed, which closes the solver), and ``solver_name`` is ``None`` before the first solve.
6272
* ``result.solution.primal`` and ``result.solution.dual`` are now ``numpy`` arrays indexed by linopy's integer labels (with ``NaN`` for slots without a value), instead of pandas Series keyed by variable/constraint name. If you accessed them by name, use ``model.variables[name].solution`` (or ``model.constraints[name].dual``) instead.
73+
* Drop Python 3.10 support. Minimum supported version is now Python 3.11.
6374

6475
**Internal**
6576

77+
* New module ``linopy.alignment`` owns conversion, broadcasting, and alignment of user input against coordinates (moved out of ``linopy.common``): ``as_dataarray`` (convert only), ``broadcast_to_coords`` (convert and broadcast against ``coords``; ``strict=True`` by default raises on any mismatch, naming ``label`` in the error), ``validate_alignment``, and ``align``.
6678
* Each ``Solver`` subclass now overrides at most three hooks: ``_build_direct`` (build the native model), ``_run_direct`` (run it), and ``_run_file`` (run the solver on an LP/MPS file). File-only solvers (CBC, GLPK, CPLEX, SCIP, Knitro, COPT, MindOpt) only override ``_run_file``.
6779
* New ``ConstraintLabelIndex`` cached on ``Model.constraints`` (mirrors the existing ``Variables.label_index``); ``ConstraintBase`` gains ``active_labels()`` and a ``range`` property; ``CSRConstraint`` exposes ``coords``.
6880
* ``linopy.common`` gains ``values_to_lookup_array``; the legacy pandas-based helpers ``series_to_lookup_array`` and ``lookup_vals`` are removed.

examples/creating-expressions.ipynb

Lines changed: 102 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -321,10 +321,106 @@
321321
]
322322
},
323323
{
324-
"attachments": {},
325324
"cell_type": "markdown",
326325
"id": "29",
327326
"metadata": {},
327+
"source": [
328+
"Sometimes `.where` may lead to a situation where some of the variables are completely masked"
329+
]
330+
},
331+
{
332+
"cell_type": "code",
333+
"execution_count": null,
334+
"id": "30",
335+
"metadata": {},
336+
"outputs": [],
337+
"source": [
338+
"mask_a = xr.DataArray(False, coords=[time])\n",
339+
"mask_b = xr.DataArray(time > 2, coords=[time])\n",
340+
"\n",
341+
"z = (x.where(mask_a) + y).where(mask_b)\n",
342+
"z"
343+
]
344+
},
345+
{
346+
"cell_type": "markdown",
347+
"id": "31",
348+
"metadata": {},
349+
"source": [
350+
"In this example you can see that many of the elements of the LinearExpression are None. If you want to remove all the None terms, you can use `.where(.., drop=True)`"
351+
]
352+
},
353+
{
354+
"cell_type": "code",
355+
"execution_count": null,
356+
"id": "32",
357+
"metadata": {},
358+
"outputs": [],
359+
"source": [
360+
"z = z.where(mask_b, drop=True)\n",
361+
"z"
362+
]
363+
},
364+
{
365+
"cell_type": "markdown",
366+
"id": "33",
367+
"metadata": {},
368+
"source": [
369+
"That looks nicer!<br>"
370+
]
371+
},
372+
{
373+
"cell_type": "markdown",
374+
"id": "34",
375+
"metadata": {},
376+
"source": [
377+
"You may notice that the variable `x` is not used at all. The expression still contains two terms (one of them is unused) but it only has one variable `y`"
378+
]
379+
},
380+
{
381+
"cell_type": "code",
382+
"execution_count": null,
383+
"id": "35",
384+
"metadata": {},
385+
"outputs": [],
386+
"source": [
387+
"z.nterm"
388+
]
389+
},
390+
{
391+
"cell_type": "code",
392+
"execution_count": null,
393+
"id": "36",
394+
"metadata": {},
395+
"outputs": [],
396+
"source": [
397+
"z.variable_names"
398+
]
399+
},
400+
{
401+
"cell_type": "markdown",
402+
"id": "37",
403+
"metadata": {},
404+
"source": [
405+
"You can get rid of the unused term with `.simplify()`"
406+
]
407+
},
408+
{
409+
"cell_type": "code",
410+
"execution_count": null,
411+
"id": "38",
412+
"metadata": {},
413+
"outputs": [],
414+
"source": [
415+
"z = z.simplify()\n",
416+
"z.nterm"
417+
]
418+
},
419+
{
420+
"attachments": {},
421+
"cell_type": "markdown",
422+
"id": "39",
423+
"metadata": {},
328424
"source": [
329425
"## Using `.shift` to shift the Variable along one dimension\n",
330426
"\n",
@@ -336,7 +432,7 @@
336432
{
337433
"cell_type": "code",
338434
"execution_count": null,
339-
"id": "30",
435+
"id": "40",
340436
"metadata": {},
341437
"outputs": [],
342438
"source": [
@@ -346,7 +442,7 @@
346442
{
347443
"attachments": {},
348444
"cell_type": "markdown",
349-
"id": "31",
445+
"id": "41",
350446
"metadata": {},
351447
"source": [
352448
"## Using `.groupby` to group by a key and apply operations on the groups\n",
@@ -359,7 +455,7 @@
359455
{
360456
"cell_type": "code",
361457
"execution_count": null,
362-
"id": "32",
458+
"id": "42",
363459
"metadata": {},
364460
"outputs": [],
365461
"source": [
@@ -370,7 +466,7 @@
370466
{
371467
"attachments": {},
372468
"cell_type": "markdown",
373-
"id": "33",
469+
"id": "43",
374470
"metadata": {},
375471
"source": [
376472
"## Using `.rolling` to perform a rolling operation\n",
@@ -383,7 +479,7 @@
383479
{
384480
"cell_type": "code",
385481
"execution_count": null,
386-
"id": "34",
482+
"id": "44",
387483
"metadata": {},
388484
"outputs": [],
389485
"source": [

linopy/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
# Note: For intercepting multiplications between xarray dataarrays, Variables and Expressions
1313
# we need to extend their __mul__ functions with a quick special case
1414
import linopy.monkey_patch_xarray # noqa: F401
15-
from linopy.common import align
15+
from linopy.alignment import align
1616
from linopy.config import LinopySemanticsWarning, options
1717
from linopy.constants import (
1818
EQUAL,

0 commit comments

Comments
 (0)