Context
While adapting openfisca-paris-rh to the feat/transition-formula branch (PR #1368), two bugs were found when using as_of variables with a transition_formula that references period.last_month.
Both bugs surface in the same scenario: a transition_formula on variable X at period T calls individu('X', period.last_month) or individu('Y', period.last_month), where T-1 is before the base instant established by set_input.
Bug 1 — _reconstruct_at returns None before the base, causing a scalar comparison in _set_as_of
Root cause
_reconstruct_at returns None when target_instant < _as_of_base_instant (line 144 of holder.py).
When a formula result for a pre-base period is put in cache via put_in_cache → _set_as_of, the code does:
prev = self._reconstruct_at(instant) # returns None
changed = value != prev # EnumArray != None → scalar bool (0-d array)
idx = numpy.where(changed)[0] # ValueError: Calling nonzero on 0d arrays
This always crashes for Enum variables because EnumArray.__ne__(None) returns a scalar True rather than an element-wise boolean array.
Proposed fix — holder.py::_reconstruct_at
def _reconstruct_at(self, target_instant):
if self._as_of_base is None:
return None
if target_instant < self._as_of_base_instant:
# The initial state is the best estimate for any period before
# the base was established.
return self._as_of_base
# ... existing logic
This makes semantic sense: set_input establishes the earliest known state, which applies to all time before that point too.
Bug 2 — _calculate_transition runs the transition_formula at/before the base instant, causing infinite recursion
Root cause
_calculate_transition (simulation.py) checks _as_of_base is None to skip the formula when no input has been set, but does not check whether the requested period is at or before the base instant.
When a transition_formula for period T references period.last_month (T-1), and T-1 ≤ base_instant, the engine:
- Calls
calculate(var, T-1) → _calculate_transition
- Base exists, period <
start_computation_period guard not triggered
- Runs the
transition_formula for T-1 → which asks for T-2 → … → infinite recursion
SpiralError is not raised because each recursive call is for a different period.
Proposed fix — simulation.py::_calculate_transition
Insert a guard after the _as_of_base is None check:
# Period at or before the base instant: the set_input value is authoritative.
# Running the formula here would recurse infinitely into the past.
if (
holder._as_of_base is not None
and holder._as_of_base_instant is not None
and instant <= holder._as_of_base_instant
):
holder._as_of_transition_computed.add(instant)
result = holder.get_array(period)
return result if result is not None else holder.default_array()
Semantics: set_input for period T0 means "the state at T0 is known; compute transitions only for T > T0."
Minimal reproducer
from openfisca_core.model_api import *
from openfisca_core.simulations import SimulationBuilder
from openfisca_core.taxbenefitsystems import TaxBenefitSystem
from openfisca_core.entities import Entity
import numpy as np
person = Entity('person', 'persons', '', '')
class MyEnum(Enum):
a = "A"
b = "B"
class my_enum_var(Variable):
value_type = Enum
possible_values = MyEnum
default_value = MyEnum.a
entity = person
definition_period = MONTH
as_of = True
# No transition_formula: value persists
class my_int_var(Variable):
value_type = int
entity = person
definition_period = MONTH
as_of = True
def transition_formula(p, period, params):
prev = p('my_int_var', period.last_month) # triggers Bug 2
prev_enum = p('my_enum_var', period.last_month) # triggers Bug 1
return np.ones(p.count, dtype=bool), prev + 1
tbs = TaxBenefitSystem([person])
tbs.add_variables(my_enum_var, my_int_var)
sb = SimulationBuilder()
sb.create_entities(tbs)
sim = sb.build_default_simulation(tbs, 2)
sim.set_input('my_enum_var', '2024-01', np.array([MyEnum.a, MyEnum.b]))
sim.set_input('my_int_var', '2024-01', np.array([0, 10]))
# Bug 2: infinite recursion when computing '2024-02'
# (transition_formula for '2024-02' asks for '2024-01' which asks for '2023-12' …)
result = sim.calculate('my_int_var', '2024-02')
print(result) # expected: [1, 11]
Workaround (applied in openfisca-paris-rh)
Both bugs are currently patched via monkey-patches in the consumer project's __init__.py while waiting for upstream fixes:
from openfisca_core.holders.holder import Holder
from openfisca_core.simulations.simulation import Simulation
_orig_reconstruct_at = Holder._reconstruct_at
def _patched_reconstruct_at(self, target_instant):
if self._as_of_base is not None and target_instant < self._as_of_base_instant:
return self._as_of_base
return _orig_reconstruct_at(self, target_instant)
Holder._reconstruct_at = _patched_reconstruct_at
_orig_calculate_transition = Simulation._calculate_transition
def _patched_calculate_transition(self, variable, population, holder, period):
instant = period.start if variable.as_of == "start" else period.stop
if (
holder._as_of_base is not None
and holder._as_of_base_instant is not None
and instant <= holder._as_of_base_instant
):
holder._as_of_transition_computed.add(instant)
result = holder.get_array(period)
return result if result is not None else holder.default_array()
return _orig_calculate_transition(self, variable, population, holder, period)
Simulation._calculate_transition = _patched_calculate_transition
Both fixes are confirmed to pass the full test suite of openfisca-paris-rh (16 tests).
Context
While adapting openfisca-paris-rh to the
feat/transition-formulabranch (PR #1368), two bugs were found when usingas_ofvariables with atransition_formulathat referencesperiod.last_month.Both bugs surface in the same scenario: a
transition_formulaon variableXat periodTcallsindividu('X', period.last_month)orindividu('Y', period.last_month), whereT-1is before the base instant established byset_input.Bug 1 —
_reconstruct_atreturnsNonebefore the base, causing a scalar comparison in_set_as_ofRoot cause
_reconstruct_atreturnsNonewhentarget_instant < _as_of_base_instant(line 144 ofholder.py).When a formula result for a pre-base period is put in cache via
put_in_cache → _set_as_of, the code does:This always crashes for Enum variables because
EnumArray.__ne__(None)returns a scalarTruerather than an element-wise boolean array.Proposed fix —
holder.py::_reconstruct_atThis makes semantic sense:
set_inputestablishes the earliest known state, which applies to all time before that point too.Bug 2 —
_calculate_transitionruns thetransition_formulaat/before the base instant, causing infinite recursionRoot cause
_calculate_transition(simulation.py) checks_as_of_base is Noneto skip the formula when no input has been set, but does not check whether the requested period is at or before the base instant.When a
transition_formulafor periodTreferencesperiod.last_month(T-1), andT-1 ≤ base_instant, the engine:calculate(var, T-1)→_calculate_transitionstart_computation_periodguard not triggeredtransition_formulaforT-1→ which asks forT-2→ … → infinite recursionSpiralErroris not raised because each recursive call is for a different period.Proposed fix —
simulation.py::_calculate_transitionInsert a guard after the
_as_of_base is Nonecheck:Semantics:
set_inputfor periodT0means "the state atT0is known; compute transitions only forT > T0."Minimal reproducer
Workaround (applied in openfisca-paris-rh)
Both bugs are currently patched via monkey-patches in the consumer project's
__init__.pywhile waiting for upstream fixes:Both fixes are confirmed to pass the full test suite of openfisca-paris-rh (16 tests).