Skip to content

feat/transition-formula: two bugs with as_of variables before the base instant #1369

Description

@benjello

Context

While adapting openfisca-paris-rh to the feat/transition-formula branch (PR #1368), two bugs were found when using as_of variables with a transition_formula that references period.last_month.

Both bugs surface in the same scenario: a transition_formula on variable X at period T calls individu('X', period.last_month) or individu('Y', period.last_month), where T-1 is before the base instant established by set_input.


Bug 1 — _reconstruct_at returns None before the base, causing a scalar comparison in _set_as_of

Root cause

_reconstruct_at returns None when target_instant < _as_of_base_instant (line 144 of holder.py).

When a formula result for a pre-base period is put in cache via put_in_cache → _set_as_of, the code does:

prev = self._reconstruct_at(instant)   # returns None
changed = value != prev                 # EnumArray != None → scalar bool (0-d array)
idx = numpy.where(changed)[0]          # ValueError: Calling nonzero on 0d arrays

This always crashes for Enum variables because EnumArray.__ne__(None) returns a scalar True rather than an element-wise boolean array.

Proposed fix — holder.py::_reconstruct_at

def _reconstruct_at(self, target_instant):
    if self._as_of_base is None:
        return None
    if target_instant < self._as_of_base_instant:
        # The initial state is the best estimate for any period before
        # the base was established.
        return self._as_of_base
    # ... existing logic

This makes semantic sense: set_input establishes the earliest known state, which applies to all time before that point too.


Bug 2 — _calculate_transition runs the transition_formula at/before the base instant, causing infinite recursion

Root cause

_calculate_transition (simulation.py) checks _as_of_base is None to skip the formula when no input has been set, but does not check whether the requested period is at or before the base instant.

When a transition_formula for period T references period.last_month (T-1), and T-1 ≤ base_instant, the engine:

  1. Calls calculate(var, T-1)_calculate_transition
  2. Base exists, period < start_computation_period guard not triggered
  3. Runs the transition_formula for T-1 → which asks for T-2 → … → infinite recursion

SpiralError is not raised because each recursive call is for a different period.

Proposed fix — simulation.py::_calculate_transition

Insert a guard after the _as_of_base is None check:

# Period at or before the base instant: the set_input value is authoritative.
# Running the formula here would recurse infinitely into the past.
if (
    holder._as_of_base is not None
    and holder._as_of_base_instant is not None
    and instant <= holder._as_of_base_instant
):
    holder._as_of_transition_computed.add(instant)
    result = holder.get_array(period)
    return result if result is not None else holder.default_array()

Semantics: set_input for period T0 means "the state at T0 is known; compute transitions only for T > T0."


Minimal reproducer

from openfisca_core.model_api import *
from openfisca_core.simulations import SimulationBuilder
from openfisca_core.taxbenefitsystems import TaxBenefitSystem
from openfisca_core.entities import Entity
import numpy as np

person = Entity('person', 'persons', '', '')

class MyEnum(Enum):
    a = "A"
    b = "B"

class my_enum_var(Variable):
    value_type = Enum
    possible_values = MyEnum
    default_value = MyEnum.a
    entity = person
    definition_period = MONTH
    as_of = True
    # No transition_formula: value persists

class my_int_var(Variable):
    value_type = int
    entity = person
    definition_period = MONTH
    as_of = True

    def transition_formula(p, period, params):
        prev = p('my_int_var', period.last_month)   # triggers Bug 2
        prev_enum = p('my_enum_var', period.last_month)  # triggers Bug 1
        return np.ones(p.count, dtype=bool), prev + 1

tbs = TaxBenefitSystem([person])
tbs.add_variables(my_enum_var, my_int_var)

sb = SimulationBuilder()
sb.create_entities(tbs)
sim = sb.build_default_simulation(tbs, 2)
sim.set_input('my_enum_var', '2024-01', np.array([MyEnum.a, MyEnum.b]))
sim.set_input('my_int_var',  '2024-01', np.array([0, 10]))

# Bug 2: infinite recursion when computing '2024-02'
# (transition_formula for '2024-02' asks for '2024-01' which asks for '2023-12' …)
result = sim.calculate('my_int_var', '2024-02')
print(result)  # expected: [1, 11]

Workaround (applied in openfisca-paris-rh)

Both bugs are currently patched via monkey-patches in the consumer project's __init__.py while waiting for upstream fixes:

from openfisca_core.holders.holder import Holder
from openfisca_core.simulations.simulation import Simulation

_orig_reconstruct_at = Holder._reconstruct_at
def _patched_reconstruct_at(self, target_instant):
    if self._as_of_base is not None and target_instant < self._as_of_base_instant:
        return self._as_of_base
    return _orig_reconstruct_at(self, target_instant)
Holder._reconstruct_at = _patched_reconstruct_at

_orig_calculate_transition = Simulation._calculate_transition
def _patched_calculate_transition(self, variable, population, holder, period):
    instant = period.start if variable.as_of == "start" else period.stop
    if (
        holder._as_of_base is not None
        and holder._as_of_base_instant is not None
        and instant <= holder._as_of_base_instant
    ):
        holder._as_of_transition_computed.add(instant)
        result = holder.get_array(period)
        return result if result is not None else holder.default_array()
    return _orig_calculate_transition(self, variable, population, holder, period)
Simulation._calculate_transition = _patched_calculate_transition

Both fixes are confirmed to pass the full test suite of openfisca-paris-rh (16 tests).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions