Skip to content

Commit d55581a

Browse files
committed
fix: extend zero_division parameter to percentage and range-based metrics
Percentage and range-based metrics (`wmape`, `ope`, `arre`, `marre`, `coefficient_of_variation`) previously either raised a hard `ValueError` or silently returned `nan`/`inf` when their denominator was zero. This made batch evaluation pipelines brittle for constant or all-zero components. Mirrors the `zero_division` design introduced in #3059 for the scaled error family: `"warn"` (default) returns `np.nan` and emits a warning, `"raise"` preserves the legacy `ValueError`. A new `_safe_pct_divide` helper sits next to `_safe_scaled_divide`; the two differ only in fill semantics — percentage metrics multiply the ratio by 100 so a `1.0` fill for the 0/0 case (the scaled-metric "on par with naive") would surface as `100 %` error and be misleading, hence `np.nan` instead. Two adjacent bugs surface and are fixed in the same change: * `ope` previously checked `sum > 0` and rejected `actual_series` with a strictly negative sum (e.g. financial return series). The check is now `sum != 0` via the helper. * `wmape`'s docstring claimed `ValueError if actual_series contains some zeros`, but the implementation divides by `sum(|y_true|)` and only the all-zero case ever triggered the path. Docstring corrected. The CHANGELOG entry for the parameter addition carries the breaking- change marker per the convention discussed in #3080 (the post-mortem on #3059), since the default behavior flips from raising to warning. Adds a parametrized regression test covering all five metrics and an explicit OPE-with-negative-sum test. Existing `test_ope_zero` and the arre/marre legacy raise check are updated to opt into the legacy behavior with `zero_division="raise"`.
1 parent b0ccd3a commit d55581a

4 files changed

Lines changed: 208 additions & 31 deletions

File tree

CHANGELOG.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,12 +15,14 @@ but cannot always guarantee backwards compatibility. Changes that may **break co
1515
- Added `use_longer_projection_head` to `TimesFM2p5Model` to enable longer non-autoregressive prediction horizons (up to 1024 steps for `output_chunk_length + output_chunk_shift`). [#3121](https://github.com/unit8co/darts/pull/3121) by [Zhihao Dai](https://github.com/daidahao).
1616
- `TimeSeries.from_dataframe()` now supports time columns of type `pl.Date` for `polars.DataFrame`. [#3124](https://github.com/unit8co/darts/pull/3124) by [Dennis Bader](https://github.com/dennisbader)
1717
- Custom encoders now support functions that return multiple components. Simply pass such a function via the `"custom"` encoder key in the `add_encoders` model input parameter. [#3069](https://github.com/unit8co/darts/pull/3069) by [Moritz Waldleben](https://github.com/mwaldleben).
18+
- 🔴 Percentage and range-based metrics (`wmape`, `ope`, `arre`, `marre`, `coefficient_of_variation`) now expose a `zero_division` parameter (mirroring [#3059](https://github.com/unit8co/darts/pull/3059)) controlling the behavior when the denominator is zero: `"warn"` (default) returns `np.nan` and emits a warning, `"raise"` preserves the legacy `ValueError`. [#3122](https://github.com/unit8co/darts/pull/3122) by [Mahimn](https://github.com/mahimn01).
1819

1920
**Fixed**
2021

2122
- Fixed `_ScaledDotProductAttention` float16 overflow in `masked_fill` under mixed precision training. [#3087](https://github.com/unit8co/darts/pull/3087) by [Robert Ruidisch](https://github.com/robrui).
2223
- Fixed a bug in `TimeSeries.quantile()` where the output dtype did not match the input series dtype for dtypes `float32` or `float16`. Now the dtype is correctly propagated. [#3124](https://github.com/unit8co/darts/pull/3124) by [Dennis Bader](https://github.com/dennisbader)
2324
- Optuna integration's `PyTorchLightningPruningCallback` for hyperparameter optimization of torch models is now natively available in Darts via `darts.utils.callbacks`. [#3114](https://github.com/unit8co/darts/pull/3114) by [Jakub Chłapek](https://github.com/jakubchlapek).
25+
- Fixed `ope` to accept `actual_series` with a strictly negative sum; the previous `sum > 0` check incorrectly rejected valid inputs such as financial return series. Also corrected the `wmape` docstring which inaccurately claimed it raised on zeros in `actual_series`. [#3122](https://github.com/unit8co/darts/pull/3122) by [Mahimn](https://github.com/mahimn01).
2426

2527
**Dependencies**
2628

darts/metrics/metrics.py

Lines changed: 62 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@
2424
_get_values_or_raise,
2525
_get_wrapped_metric,
2626
_LabelReduction,
27+
_safe_pct_divide,
2728
_safe_scaled_divide,
2829
classification_support,
2930
interval_support,
@@ -1725,6 +1726,7 @@ def wmape(
17251726
intersect: bool = True,
17261727
*,
17271728
q: float | list[float] | tuple[np.ndarray, pd.Index] | None = None,
1729+
zero_division: str = "warn",
17281730
component_reduction: Callable[[np.ndarray], float] | None = np.nanmean,
17291731
series_reduction: Callable[[np.ndarray], float | np.ndarray] | None = None,
17301732
n_jobs: int = 1,
@@ -1753,6 +1755,12 @@ def wmape(
17531755
will consider the values only over their common time interval (intersection in time).
17541756
q
17551757
Optionally, the quantile (float [0, 1]) or list of quantiles of interest to compute the metric on.
1758+
zero_division
1759+
Controls behavior when the denominator :math:`\\sum_{t=1}^T |y_t|` is zero (i.e. ``actual_series`` is
1760+
all zeros for a given component).
1761+
1762+
* ``"warn"`` (default) – returns ``np.nan`` and emits a warning.
1763+
* ``"raise"`` – raises a ``ValueError``.
17561764
component_reduction
17571765
Optionally, a function to aggregate the metrics over the component/column axis. It must reduce a `np.ndarray`
17581766
of shape `(t, c)` to a `np.ndarray` of shape `(t,)`. The function takes as input a ``np.ndarray`` and a
@@ -1776,7 +1784,7 @@ def wmape(
17761784
Raises
17771785
------
17781786
ValueError
1779-
If `actual_series` contains some zeros.
1787+
If `zero_division="raise"` and the denominator :math:`\\sum_{t=1}^T |y_t|` is zero for some component.
17801788
17811789
Returns
17821790
-------
@@ -1812,10 +1820,10 @@ def wmape(
18121820
q=q,
18131821
)
18141822

1815-
return (
1816-
100.0
1817-
* np.nansum(np.abs(y_true - y_pred), axis=TIME_AX)
1818-
/ np.nansum(np.abs(y_true), axis=TIME_AX)
1823+
return 100.0 * _safe_pct_divide(
1824+
np.nansum(np.abs(y_true - y_pred), axis=TIME_AX),
1825+
np.nansum(np.abs(y_true), axis=TIME_AX),
1826+
zero_division=zero_division,
18191827
)
18201828

18211829

@@ -2029,6 +2037,7 @@ def ope(
20292037
intersect: bool = True,
20302038
*,
20312039
q: float | list[float] | tuple[np.ndarray, pd.Index] | None = None,
2040+
zero_division: str = "warn",
20322041
component_reduction: Callable[[np.ndarray], float] | None = np.nanmean,
20332042
series_reduction: Callable[[np.ndarray], float | np.ndarray] | None = None,
20342043
n_jobs: int = 1,
@@ -2058,6 +2067,14 @@ def ope(
20582067
will consider the values only over their common time interval (intersection in time).
20592068
q
20602069
Optionally, the quantile (float [0, 1]) or list of quantiles of interest to compute the metric on.
2070+
zero_division
2071+
Controls behavior when the denominator :math:`\\sum_{t=1}^{T}{y_t}` is zero.
2072+
2073+
* ``"warn"`` (default) – returns ``np.nan`` and emits a warning.
2074+
* ``"raise"`` – raises a ``ValueError``.
2075+
2076+
Note: a negative sum is a valid denominator (e.g. financial return series). Only an exact
2077+
zero sum triggers the zero-division handling.
20612078
component_reduction
20622079
Optionally, a function to aggregate the metrics over the component/column axis. It must reduce a `np.ndarray`
20632080
of shape `(t, c)` to a `np.ndarray` of shape `(t,)`. The function takes as input a ``np.ndarray`` and a
@@ -2081,7 +2098,7 @@ def ope(
20812098
Raises
20822099
------
20832100
ValueError
2084-
If :math:`\\sum_{t=1}^{T}{y_t} = 0`.
2101+
If `zero_division="raise"` and :math:`\\sum_{t=1}^{T}{y_t} = 0` for some component.
20852102
20862103
Returns
20872104
-------
@@ -2116,14 +2133,16 @@ def ope(
21162133
np.nansum(y_true, axis=TIME_AX),
21172134
np.nansum(y_pred, axis=TIME_AX),
21182135
)
2119-
if not (y_true_sum > 0).all():
2120-
raise_log(
2121-
ValueError(
2122-
"The series of actual value cannot sum to zero when computing OPE."
2123-
),
2124-
logger=logger,
2136+
return (
2137+
np.abs(
2138+
_safe_pct_divide(
2139+
y_true_sum - y_pred_sum,
2140+
y_true_sum,
2141+
zero_division=zero_division,
2142+
)
21252143
)
2126-
return np.abs((y_true_sum - y_pred_sum) / y_true_sum) * 100.0
2144+
* 100.0
2145+
)
21272146

21282147

21292148
@multi_ts_support
@@ -2134,6 +2153,7 @@ def arre(
21342153
intersect: bool = True,
21352154
*,
21362155
q: float | list[float] | tuple[np.ndarray, pd.Index] | None = None,
2156+
zero_division: str = "warn",
21372157
time_reduction: Callable[..., np.ndarray] | None = None,
21382158
component_reduction: Callable[[np.ndarray], float] | None = np.nanmean,
21392159
series_reduction: Callable[[np.ndarray], float | np.ndarray] | None = None,
@@ -2163,6 +2183,12 @@ def arre(
21632183
will consider the values only over their common time interval (intersection in time).
21642184
q
21652185
Optionally, the quantile (float [0, 1]) or list of quantiles of interest to compute the metric on.
2186+
zero_division
2187+
Controls behavior when the denominator :math:`\\max_t{y_t} - \\min_t{y_t}` is zero (i.e.
2188+
``actual_series`` is constant for a given component).
2189+
2190+
* ``"warn"`` (default) – returns ``np.nan`` for affected components and emits a warning.
2191+
* ``"raise"`` – raises a ``ValueError``.
21662192
time_reduction
21672193
Optionally, a function to aggregate the metrics over the time axis. It must reduce a `np.ndarray`
21682194
of shape `(t, c)` to a `np.ndarray` of shape `(c,)`. The function takes as input a ``np.ndarray`` and a
@@ -2191,7 +2217,7 @@ def arre(
21912217
Raises
21922218
------
21932219
ValueError
2194-
If :math:`\\max_t{y_t} = \\min_t{y_t}`.
2220+
If `zero_division="raise"` and :math:`\\max_t{y_t} = \\min_t{y_t}` for some component.
21952221
21962222
Returns
21972223
-------
@@ -2226,16 +2252,10 @@ def arre(
22262252
q=q,
22272253
)
22282254
y_max, y_min = np.nanmax(y_true, axis=TIME_AX), np.nanmin(y_true, axis=TIME_AX)
2229-
if not (y_max > y_min).all():
2230-
raise_log(
2231-
ValueError(
2232-
"The difference between the max and min values must "
2233-
"be strictly positive to compute the MARRE."
2234-
),
2235-
logger=logger,
2236-
)
22372255
true_range = y_max - y_min
2238-
return 100.0 * np.abs((y_true - y_pred) / true_range)
2256+
return 100.0 * np.abs(
2257+
_safe_pct_divide(y_true - y_pred, true_range, zero_division=zero_division)
2258+
)
22392259

22402260

22412261
@multi_ts_support
@@ -2246,6 +2266,7 @@ def marre(
22462266
intersect: bool = True,
22472267
*,
22482268
q: float | list[float] | tuple[np.ndarray, pd.Index] | None = None,
2269+
zero_division: str = "warn",
22492270
component_reduction: Callable[[np.ndarray], float] | None = np.nanmean,
22502271
series_reduction: Callable[[np.ndarray], float | np.ndarray] | None = None,
22512272
n_jobs: int = 1,
@@ -2275,6 +2296,12 @@ def marre(
22752296
will consider the values only over their common time interval (intersection in time).
22762297
q
22772298
Optionally, the quantile (float [0, 1]) or list of quantiles of interest to compute the metric on.
2299+
zero_division
2300+
Controls behavior when the denominator :math:`\\max_t{y_t} - \\min_t{y_t}` is zero (i.e.
2301+
``actual_series`` is constant for a given component).
2302+
2303+
* ``"warn"`` (default) – returns ``np.nan`` for affected components and emits a warning.
2304+
* ``"raise"`` – raises a ``ValueError``.
22782305
component_reduction
22792306
Optionally, a function to aggregate the metrics over the component/column axis. It must reduce a `np.ndarray`
22802307
of shape `(t, c)` to a `np.ndarray` of shape `(t,)`. The function takes as input a ``np.ndarray`` and a
@@ -2298,7 +2325,7 @@ def marre(
22982325
Raises
22992326
------
23002327
ValueError
2301-
If :math:`\\max_t{y_t} = \\min_t{y_t}`.
2328+
If `zero_division="raise"` and :math:`\\max_t{y_t} = \\min_t{y_t}` for some component.
23022329
23032330
float
23042331
A single metric score for:
@@ -2322,6 +2349,7 @@ def marre(
23222349
pred_series,
23232350
intersect,
23242351
q=q,
2352+
zero_division=zero_division,
23252353
),
23262354
axis=TIME_AX,
23272355
)
@@ -2433,6 +2461,7 @@ def coefficient_of_variation(
24332461
intersect: bool = True,
24342462
*,
24352463
q: float | list[float] | tuple[np.ndarray, pd.Index] | None = None,
2464+
zero_division: str = "warn",
24362465
component_reduction: Callable[[np.ndarray], float] | None = np.nanmean,
24372466
series_reduction: Callable[[np.ndarray], float | np.ndarray] | None = None,
24382467
n_jobs: int = 1,
@@ -2464,6 +2493,11 @@ def coefficient_of_variation(
24642493
will consider the values only over their common time interval (intersection in time).
24652494
q
24662495
Optionally, the quantile (float [0, 1]) or list of quantiles of interest to compute the metric on.
2496+
zero_division
2497+
Controls behavior when the denominator :math:`\\bar{y}` (the mean of ``actual_series``) is zero.
2498+
2499+
* ``"warn"`` (default) – returns ``np.nan`` for affected components and emits a warning.
2500+
* ``"raise"`` – raises a ``ValueError``.
24672501
component_reduction
24682502
Optionally, a function to aggregate the metrics over the component/column axis. It must reduce a `np.ndarray`
24692503
of shape `(t, c)` to a `np.ndarray` of shape `(t,)`. The function takes as input a ``np.ndarray`` and a
@@ -2514,10 +2548,10 @@ def coefficient_of_variation(
25142548
q=q,
25152549
)
25162550
# not calling rmse as y_true and y_pred are np.ndarray
2517-
return (
2518-
100
2519-
* np.sqrt(np.nanmean((y_true - y_pred) ** 2, axis=TIME_AX))
2520-
/ np.nanmean(y_true, axis=TIME_AX)
2551+
return 100 * _safe_pct_divide(
2552+
np.sqrt(np.nanmean((y_true - y_pred) ** 2, axis=TIME_AX)),
2553+
np.nanmean(y_true, axis=TIME_AX),
2554+
zero_division=zero_division,
25212555
)
25222556

25232557

darts/metrics/utils.py

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -940,6 +940,73 @@ def _safe_scaled_divide(
940940
return result
941941

942942

943+
def _safe_pct_divide(
944+
errors: np.ndarray,
945+
scale: np.ndarray,
946+
zero_division: str = "warn",
947+
) -> np.ndarray:
948+
"""Divides ``errors`` by ``scale`` for percentage-style metrics, returning
949+
``np.nan`` where ``scale`` is zero.
950+
951+
Unlike :func:`_safe_scaled_divide` — which fills the ``0/0`` case with
952+
``1.0`` to express "on par with naive baseline" for scaled-error metrics
953+
— this helper always fills zero-scale entries with ``np.nan`` because
954+
percentage metrics multiply the ratio by ``100``; a fill of ``1.0`` would
955+
surface as a ``100 %`` error and be misleading.
956+
957+
Parameters
958+
----------
959+
errors
960+
Numerator array. Broadcasts against ``scale``.
961+
scale
962+
Denominator array (e.g. the sum, mean, or range of ``actual_series``).
963+
zero_division
964+
Controls behavior when ``scale`` is (near) zero.
965+
966+
* ``"warn"`` (default) – fill zero-scale entries with ``np.nan`` and
967+
emit a warning.
968+
* ``"raise"`` – raise a ``ValueError`` (the legacy behavior).
969+
970+
Returns
971+
-------
972+
np.ndarray
973+
The result of ``errors / scale`` with zero-scale entries replaced by
974+
``np.nan``.
975+
"""
976+
if zero_division not in ["warn", "raise"]:
977+
raise_log(
978+
ValueError(
979+
f"`zero_division` must be 'warn' or 'raise'. Received {zero_division}."
980+
),
981+
logger=logger,
982+
)
983+
984+
zero_mask = np.isclose(scale, 0.0)
985+
if not zero_mask.any():
986+
return errors / scale
987+
988+
if zero_division == "raise":
989+
raise_log(
990+
ValueError(
991+
"Cannot compute percentage metric: the denominator "
992+
"(e.g. sum, mean, or range of `actual_series`) is zero "
993+
"for some components."
994+
),
995+
logger=logger,
996+
)
997+
998+
# Avoid runtime warnings from the masked divide
999+
safe_scale = np.where(zero_mask, 1.0, scale)
1000+
result = np.where(zero_mask, np.nan, errors / safe_scale)
1001+
1002+
logger.warning(
1003+
"The denominator (e.g. sum, mean, or range of `actual_series`) is "
1004+
"zero for some components in the percentage metric. Those entries "
1005+
"are set to NaN."
1006+
)
1007+
return result
1008+
1009+
9431010
def _unique_labels(y_true: np.ndarray, y_pred: np.ndarray) -> list[np.ndarray]:
9441011
"""Returns unique labels for each component in the true and predicted labels."""
9451012
labels = []

0 commit comments

Comments
 (0)