Feature/penalty as effect (#486)

* implemented the migration of Penalty to be a standard Effect. Here's what was done:

  ✅ Implementation Complete

  Changes Made:

  1. Penalty Effect Creation (flixopt/effects.py):
    - Created _penalty effect automatically in FlowSystem.__init__
    - Added penalty_effect property to EffectCollection
    - Penalty now has temporal and periodic dimensions like other effects
  2. Removed Old Penalty Infrastructure (flixopt/effects.py):
    - Removed ShareAllocationModel penalty from EffectCollectionModel
    - Removed add_share_to_penalty() method
    - Updated objective to use penalty effect: objective_effect + penalty_effect
  3. Updated Internal Usage:
    - Bus (flixopt/elements.py:960-969): Now uses add_share_to_effects with target='temporal'
    - Aggregation (flixopt/aggregation.py:351-355): Now uses add_share_to_effects with target='temporal'
  4. Results Structure (flixopt/calculation.py:109-113):
    - Changed from 'Penalty': scalar to 'Penalty': {temporal, periodic, total}
    - Penalty excluded from regular Effects list (shown separately)
  5. I/O & Serialization (flixopt/flow_system.py:613-614):
    - Skip _penalty effect when loading from dataset (auto-created)
  6. Documentation (docs/user-guide/mathematical-notation/effects-penalty-objective.md):
    - Updated to reflect Penalty as an Effect
    - Updated mathematical formulations
    - Updated all objective function equations
  7. Tests Updated:
    - tests/test_bus.py: Updated penalty assertions
    - tests/test_scenarios.py: Updated to use new penalty structure

  Key Benefits:

  ✅ Unified Interface: Penalty shares added same way as effect shares
  ✅ Dimensional Support: Penalties can now vary by time/period/scenario
  ✅ Constrainable: Can add bounds to penalty (useful for debugging)
  ✅ Better Results: Penalty breakdown shows temporal vs periodic contributions
  ✅ Cleaner Architecture: One less special case in the codebase

  Test Status:

  The core functionality is working. Tests are passing for:
  - Bus models with and without penalties
  - Effect shares and constraints
  - Scenario weighting
  - Resample operations
  - I/O operations

* Allow user defined enalty Effect

* Update CHANGELOG.md

* Update CHANGELOG.md

* Update CHANGELOG.md and docs

* Fix penöaty test

* Remove special treatment for Penalty effect in IO

* Update CHANGELOG.md

* Fix aggregation.py and its penalties

* Fix indices

* Revert some changes

* Verify that Penalty is not set as the objective

* Update docs a bit

* Update docs a bit

* Update docs a bit

* Fix test

* Update CHANGELOG.md

* Improve main results penalty display

* Update CHANGELOG.md

* Merge branch 'main' into feature/penalty-as-effect

# Conflicts:
#	flixopt/calculation.py
#	flixopt/clustering.py

* Revert some minimla changes from merge

Author: FBumann

CHANGELOG.md

@@ -51,17 +51,32 @@ If upgrading from v2.x, see the [v3.0.0 release notes](https://github.com/flixOp
 
 ## [Unreleased] - ????-??-??
 
-**Summary**:
+**Summary**: Penalty is now a first-class Effect - add penalty contributions anywhere (e.g., `effects_per_flow_hour={'Penalty': 2.5}`) and optionally define bounds as with any other effect.
 
 If upgrading from v2.x, see the [v3.0.0 release notes](https://github.com/flixOpt/flixOpt/releases/tag/v3.0.0) and [Migration Guide](https://flixopt.github.io/flixopt/latest/user-guide/migration-guide-v3/).
 
 ### ✨ Added
 
+- **Penalty as first-class Effect**: Users can now add Penalty contributions anywhere effects are used:
+  ```python
+  fx.Flow('Q', 'Bus', effects_per_flow_hour={'Penalty': 2.5})
+  fx.InvestParameters(..., effects_of_investment={'Penalty': 100})
+  ```
+- **User-definable Penalty**: Optionally define custom Penalty with constraints (auto-created if not defined):
+  ```python
+  penalty = fx.Effect(fx.PENALTY_EFFECT_LABEL, unit='€', maximum_total=1e6)
+  flow_system.add_elements(penalty)
+  ```
+
 ### 💥 Breaking Changes
 
 ### ♻️ Changed
 
-### 🗑️ Deprecated
+- Penalty is now a standard Effect with temporal/periodic dimensions
+- Unified interface: Penalty uses same `add_share_to_effects()` as other effects (internal only)
+- **Results structure**: Penalty now has same structure as other effects in solution Dataset
+  - Use `results.solution['Penalty']` for total penalty value (same as before, but now it's an effect variable)
+  - Access components via `results.solution['Penalty(temporal)']` and `results.solution['Penalty(periodic)']` if needed
 
 ### 🔥 Removed
 
@@ -73,9 +88,7 @@ If upgrading from v2.x, see the [v3.0.0 release notes](https://github.com/flixOp
 
 ### 📝 Docs
 
-### 👷 Development
-
-### 🚧 Known Issues
+- Updated mathematical notation for Penalty as Effect
 
 ---
 

docs/user-guide/mathematical-notation/dimensions.md

@@ -114,6 +114,7 @@ Where:
 - $\mathcal{S}$ is the set of scenarios
 - $w_s$ is the weight for scenario $s$
 - The optimizer balances performance across scenarios according to their weights
+- **Both the objective effect and Penalty effect are weighted by $w_s$** (see [Penalty weighting](effects-penalty-objective.md#penalty))
 
 ### Period Independence
 
@@ -130,6 +131,8 @@ $$
 \min \quad \sum_{y \in \mathcal{Y}} w_y \cdot \text{Objective}_y
 $$
 
+Where **both the objective effect and Penalty effect are weighted by $w_y$** (see [Penalty weighting](effects-penalty-objective.md#penalty))
+
 ### Shared Periodic Decisions: The Exception
 
 **Investment decisions (sizes) can be shared across all scenarios:**
@@ -203,16 +206,18 @@ $$
 
 Where:
 - $\mathcal{T}$ is the set of time steps
-- $\mathcal{E}$ is the set of effects
+- $\mathcal{E}$ is the set of effects (including the Penalty effect $E_\Phi$)
 - $\mathcal{S}$ is the set of scenarios
 - $\mathcal{Y}$ is the set of periods
 - $s_{e}(\cdots)$ are the effect contributions (costs, emissions, etc.)
 - $w_s, w_y, w_{y,s}$ are the dimension weights
+- **Penalty effect is weighted identically to other effects**
 
 **See [Effects, Penalty & Objective](effects-penalty-objective.md) for complete formulations including:**
 - How temporal and periodic effects expand with dimensions
 - Detailed objective function for each dimensional case
 - Periodic (investment) vs temporal (operational) effect handling
+- Explicit Penalty weighting formulations
 
 ---
 

docs/user-guide/mathematical-notation/effects-penalty-objective.md

@@ -142,40 +142,86 @@ $$
 
 ## Penalty
 
-In addition to user-defined [Effects](#effects), every FlixOpt model includes a **Penalty** term $\Phi$ to:
+Every FlixOpt model includes a special **Penalty Effect** $E_\Phi$ to:
+
 - Prevent infeasible problems
-- Simplify troubleshooting by allowing constraint violations with high cost
+- Allow introducing a bias without influencing effects, simplifying results analysis
+
+**Key Feature:** Penalty is implemented as a standard Effect (labeled `Penalty`), so you can **add penalty contributions anywhere effects are used**:
+
+```python
+import flixopt as fx
+
+# Add penalty contributions just like any other effect
+on_off = fx.OnOffParameters(
+    effects_per_switch_on={'Penalty': 1}  # Add bias against switching on this component, without adding costs
+)
+```
+
+**Optionally Define Custom Penalty:**
+Users can define their own Penalty effect with custom properties (unit, constraints, etc.):
+
+```python
+# Define custom penalty effect (must use fx.PENALTY_EFFECT_LABEL)
+custom_penalty = fx.Effect(
+    fx.PENALTY_EFFECT_LABEL,  # Always use this constant: 'Penalty'
+    unit='€',
+    description='Penalty costs for constraint violations',
+    maximum_total=1e6,  # Limit total penalty for debugging
+)
+flow_system.add_elements(custom_penalty)
+```
+
+If not user-defined, the Penalty effect is automatically created during modeling with default settings.
+
+**Periodic penalty shares** (time-independent):
+$$ \label{eq:Penalty_periodic}
+E_{\Phi, \text{per}} = \sum_{l \in \mathcal{L}} s_{l \rightarrow \Phi,\text{per}}
+$$
 
-Penalty shares originate from elements, similar to effect shares:
+**Temporal penalty shares** (time-dependent):
+$$ \label{eq:Penalty_temporal}
+E_{\Phi, \text{temp}}(\text{t}_{i}) = \sum_{l \in \mathcal{L}} s_{l \rightarrow \Phi, \text{temp}}(\text{t}_i)
+$$
 
-$$ \label{eq:Penalty}
-\Phi = \sum_{l \in \mathcal{L}} \left( s_{l \rightarrow \Phi}  +\sum_{\text{t}_i \in \mathcal{T}} s_{l \rightarrow \Phi}(\text{t}_{i}) \right)
+**Total penalty** (combining both domains):
+$$ \label{eq:Penalty_total}
+E_{\Phi} = E_{\Phi,\text{per}} + \sum_{\text{t}_i \in \mathcal{T}} E_{\Phi, \text{temp}}(\text{t}_{i})
 $$
 
 Where:
 
 - $\mathcal{L}$ is the set of all elements
 - $\mathcal{T}$ is the set of all timesteps
-- $s_{l \rightarrow \Phi}$ is the penalty share from element $l$
+- $s_{l \rightarrow \Phi, \text{per}}$ is the periodic penalty share from element $l$
+- $s_{l \rightarrow \Phi, \text{temp}}(\text{t}_i)$ is the temporal penalty share from element $l$ at timestep $\text{t}_i$
+
+**Primary usage:** Penalties occur in [Buses](elements/Bus.md) via the `excess_penalty_per_flow_hour` parameter, which allows nodal imbalances at a high cost, and in time series aggregation to allow period flexibility.
 
-**Current usage:** Penalties primarily occur in [Buses](elements/Bus.md) via the `excess_penalty_per_flow_hour` parameter, which allows nodal imbalances at a high cost.
+**Key properties:**
+- Penalty shares are added via `add_share_to_effects(name, expressions={fx.PENALTY_EFFECT_LABEL: ...}, target='temporal'/'periodic')`
+- Like other effects, penalty can be constrained (e.g., `maximum_total` for debugging)
+- Results include breakdown: temporal, periodic, and total penalty contributions
+- Penalty is always added to the objective function (cannot be disabled)
+- Access via `flow_system.effects.penalty_effect` or `flow_system.effects[fx.PENALTY_EFFECT_LABEL]`
+- **Scenario weighting**: Penalty is weighted identically to the objective effect—see [Time + Scenario](#time--scenario) for details
 
 ---
 
 ## Objective Function
 
-The optimization objective minimizes the chosen effect plus any penalties:
+The optimization objective minimizes the chosen effect plus the penalty effect:
 
 $$ \label{eq:Objective}
-\min \left( E_{\Omega} + \Phi \right)
+\min \left( E_{\Omega} + E_{\Phi} \right)
 $$
 
 Where:
 
 - $E_{\Omega}$ is the chosen **objective effect** (see $\eqref{eq:Effect_Total}$)
-- $\Phi$ is the [penalty](#penalty) term
+- $E_{\Phi}$ is the [penalty effect](#penalty) (see $\eqref{eq:Penalty_total}$)
 
-One effect must be designated as the objective via `is_objective=True`.
+One effect must be designated as the objective via `is_objective=True`. The penalty effect is automatically created and always added to the objective.
 
 ### Multi-Criteria Optimization
 
@@ -198,70 +244,70 @@ When the FlowSystem includes **periods** and/or **scenarios** (see [Dimensions](
 ### Time Only (Base Case)
 
 $$
-\min \quad E_{\Omega} + \Phi = \sum_{\text{t}_i \in \mathcal{T}} E_{\Omega,\text{temp}}(\text{t}_i) + E_{\Omega,\text{per}} + \Phi
+\min \quad E_{\Omega} + E_{\Phi} = \sum_{\text{t}_i \in \mathcal{T}} E_{\Omega,\text{temp}}(\text{t}_i) + E_{\Omega,\text{per}} + E_{\Phi,\text{per}} + \sum_{\text{t}_i \in \mathcal{T}} E_{\Phi,\text{temp}}(\text{t}_i)
 $$
 
 Where:
-- Temporal effects sum over time: $\sum_{\text{t}_i} E_{\Omega,\text{temp}}(\text{t}_i)$
-- Periodic effects are constant: $E_{\Omega,\text{per}}$
-- Penalty sums over time: $\Phi = \sum_{\text{t}_i} \Phi(\text{t}_i)$
+- Temporal effects sum over time: $\sum_{\text{t}_i} E_{\Omega,\text{temp}}(\text{t}_i)$ and $\sum_{\text{t}_i} E_{\Phi,\text{temp}}(\text{t}_i)$
+- Periodic effects are constant: $E_{\Omega,\text{per}}$ and $E_{\Phi,\text{per}}$
 
 ---
 
 ### Time + Scenario
 
 $$
-\min \quad \sum_{s \in \mathcal{S}} w_s \cdot \left( E_{\Omega}(s) + \Phi(s) \right)
+\min \quad \sum_{s \in \mathcal{S}} w_s \cdot \left( E_{\Omega}(s) + E_{\Phi}(s) \right)
 $$
 
 Where:
 - $\mathcal{S}$ is the set of scenarios
 - $w_s$ is the weight for scenario $s$ (typically scenario probability)
-- Periodic effects are **shared across scenarios**: $E_{\Omega,\text{per}}$ (same for all $s$)
-- Temporal effects are **scenario-specific**: $E_{\Omega,\text{temp}}(s) = \sum_{\text{t}_i} E_{\Omega,\text{temp}}(\text{t}_i, s)$
-- Penalties are **scenario-specific**: $\Phi(s) = \sum_{\text{t}_i} \Phi(\text{t}_i, s)$
+- Periodic effects are **shared across scenarios**: $E_{\Omega,\text{per}}$ and $E_{\Phi,\text{per}}$ (same for all $s$)
+- Temporal effects are **scenario-specific**: $E_{\Omega,\text{temp}}(s) = \sum_{\text{t}_i} E_{\Omega,\text{temp}}(\text{t}_i, s)$ and $E_{\Phi,\text{temp}}(s) = \sum_{\text{t}_i} E_{\Phi,\text{temp}}(\text{t}_i, s)$
 
 **Interpretation:**
 - Investment decisions (periodic) made once, used across all scenarios
 - Operations (temporal) differ by scenario
 - Objective balances expected value across scenarios
+- **Both $E_{\Omega}$ (objective effect) and $E_{\Phi}$ (penalty) are weighted identically by $w_s$**
 
 ---
 
 ### Time + Period
 
 $$
-\min \quad \sum_{y \in \mathcal{Y}} w_y \cdot \left( E_{\Omega}(y) + \Phi(y) \right)
+\min \quad \sum_{y \in \mathcal{Y}} w_y \cdot \left( E_{\Omega}(y) + E_{\Phi}(y) \right)
 $$
 
 Where:
 - $\mathcal{Y}$ is the set of periods (e.g., years)
 - $w_y$ is the weight for period $y$ (typically annual discount factor)
-- Each period $y$ has **independent** periodic and temporal effects
+- Each period $y$ has **independent** periodic and temporal effects (including penalty)
 - Each period $y$ has **independent** investment and operational decisions
+- **Both $E_{\Omega}$ (objective effect) and $E_{\Phi}$ (penalty) are weighted identically by $w_y$**
 
 ---
 
 ### Time + Period + Scenario (Full Multi-Dimensional)
 
 $$
-\min \quad \sum_{y \in \mathcal{Y}} \left[ w_y \cdot E_{\Omega,\text{per}}(y) + \sum_{s \in \mathcal{S}} w_{y,s} \cdot \left( E_{\Omega,\text{temp}}(y,s) + \Phi(y,s) \right) \right]
+\min \quad \sum_{y \in \mathcal{Y}} \left[ w_y \cdot \left( E_{\Omega,\text{per}}(y) + E_{\Phi,\text{per}}(y) \right) + \sum_{s \in \mathcal{S}} w_{y,s} \cdot \left( E_{\Omega,\text{temp}}(y,s) + E_{\Phi,\text{temp}}(y,s) \right) \right]
 $$
 
 Where:
 - $\mathcal{S}$ is the set of scenarios
 - $\mathcal{Y}$ is the set of periods
 - $w_y$ is the period weight (for periodic effects)
 - $w_{y,s}$ is the combined period-scenario weight (for temporal effects)
-- **Periodic effects** $E_{\Omega,\text{per}}(y)$ are period-specific but **scenario-independent**
-- **Temporal effects** $E_{\Omega,\text{temp}}(y,s) = \sum_{\text{t}_i} E_{\Omega,\text{temp}}(\text{t}_i, y, s)$ are **fully indexed**
-- **Penalties** $\Phi(y,s)$ are **fully indexed**
+- **Periodic effects** $E_{\Omega,\text{per}}(y)$ and $E_{\Phi,\text{per}}(y)$ are period-specific but **scenario-independent**
+- **Temporal effects** $E_{\Omega,\text{temp}}(y,s) = \sum_{\text{t}_i} E_{\Omega,\text{temp}}(\text{t}_i, y, s)$ and $E_{\Phi,\text{temp}}(y,s) = \sum_{\text{t}_i} E_{\Phi,\text{temp}}(\text{t}_i, y, s)$ are **fully indexed**
 
 **Key Principle:**
 - Scenarios and periods are **operationally independent** (no energy/resource exchange)
 - Coupled **only through the weighted objective function**
 - **Periodic effects within a period are shared across all scenarios** (investment made once per period)
 - **Temporal effects are independent per scenario** (different operations under different conditions)
+- **Both $E_{\Omega}$ (objective effect) and $E_{\Phi}$ (penalty) use identical weighting** ($w_y$ for periodic, $w_{y,s}$ for temporal)
 
 ---
 
@@ -274,7 +320,8 @@ Where:
 | **Total temporal effect** | $E_{e,\text{temp},\text{tot}} = \sum_{\text{t}_i} E_{e,\text{temp}}(\text{t}_i)$ | Sum over time | Depends on dimensions |
 | **Total periodic effect** | $E_{e,\text{per}}$ | Constant | $(y)$ when periods present |
 | **Total effect** | $E_e = E_{e,\text{per}} + E_{e,\text{temp},\text{tot}}$ | Combined | Depends on dimensions |
-| **Objective** | $\min(E_{\Omega} + \Phi)$ | With weights when multi-dimensional | See formulations above |
+| **Penalty effect** | $E_\Phi = E_{\Phi,\text{per}} + E_{\Phi,\text{temp},\text{tot}}$ | Combined (same as effects) | **Weighted identically to objective effect** |
+| **Objective** | $\min(E_{\Omega} + E_{\Phi})$ | With weights when multi-dimensional | See formulations above |
 
 ---
 

flixopt/__init__.py

@@ -28,7 +28,7 @@
 )
 from .config import CONFIG, change_logging_level
 from .core import TimeSeriesData
-from .effects import Effect
+from .effects import PENALTY_EFFECT_LABEL, Effect
 from .elements import Bus, Flow
 from .flow_system import FlowSystem
 from .interface import InvestParameters, OnOffParameters, Piece, Piecewise, PiecewiseConversion, PiecewiseEffects
@@ -43,6 +43,7 @@
     'Flow',
     'Bus',
     'Effect',
+    'PENALTY_EFFECT_LABEL',
     'Source',
     'Sink',
     'SourceAndSink',

flixopt/clustering.py

@@ -349,8 +349,16 @@ def do_modeling(self):
 
         penalty = self.clustering_parameters.penalty_of_period_freedom
         if (self.clustering_parameters.percentage_of_period_freedom > 0) and penalty != 0:
-            for variable in self.variables_direct.values():
-                self._model.effects.add_share_to_penalty('Clustering', variable * penalty)
+            from .effects import PENALTY_EFFECT_LABEL
+
+            for variable_name in self.variables_direct:
+                variable = self.variables_direct[variable_name]
+                # Sum correction variables over all dimensions to get periodic penalty contribution
+                self._model.effects.add_share_to_effects(
+                    name='Aggregation',
+                    expressions={PENALTY_EFFECT_LABEL: (variable * penalty).sum('time')},
+                    target='periodic',
+                )
 
     def _equate_indices(self, variable: linopy.Variable, indices: tuple[np.ndarray, np.ndarray]) -> None:
         assert len(indices[0]) == len(indices[1]), 'The length of the indices must match!!'