feat: Generic Entity Links (LIAM2-inspired) (#1363)

Author: benjello

.benchmarks/Linux-CPython-3.11-64bit/0001_pr_vectorized.json

@@ -1,5 +1,19 @@
 # Changelog
 
+## 44.3.0
+
+#### New Features
+
+- **Generic Entity Links (Phase 1-6)**: Introduced a new Liam2-inspired generic entity linking system avoiding rigid hierarchies like `Person -> Household`.
+  - Added new `Many2OneLink` and `One2ManyLink` models to create powerful inter-entity networks (e.g., `Person -> Employer`).
+  - Added implicit links directly binding members arrays. This powers the new `population.links` property natively inside `TaxBenefitSystem.instantiate_entities()`.
+  - Full capability to chain relationships via python: `person.mother.household.get("rent", period)`.
+  - Powerful vectorized declarative aggregations out-of-the-box (e.g., `households.persons.sum("salary", period, condition=is_female)`).
+
+#### Technical Changes
+
+- Backward compatibility is 100% maintained. Existing syntax via Projectors natively redirects to implicit links via modified `__getattr__`.
+
 ## 44.2.2
 
 #### Bug fixes

.benchmarks/Linux-CPython-3.11-64bit/0002_master_loop.json

@@ -0,0 +1,57 @@
+# Feature: Generic Entity Links (LIAM2-inspired)
+
+## Context & Motivation
+
+OpenFisca's traditional entity model has historically been strictly hierarchical and bipartite: individuals belong to groups (households, families, tax units), and groups contain individuals. This rigid structure works well for static tax-benefit systems but struggles with complex, real-world socioeconomic models, such as:
+- **Intra-entity relationships**: Kinship graphs (person $\rightarrow$ mother, person $\rightarrow$ spouse).
+- **Arbitrary inter-entity networks**: Employment networks (person $\rightarrow$ employer), geographical mobility, or ad-hoc associations.
+- **Deep chaining**: Navigating multiple relationship hops (e.g., "the region of the household of the mother of the person").
+
+To solve this, we drew inspiration from [LIAM2's linking system](https://liam2.plan.be/) and adapted it to OpenFisca's unique architecture (specifically integrating with our `Role` semantics and vectorized execution).
+
+## What we did
+
+This PR introduces a generic, highly performant, and **100% backward-compatible** Entity Linking system.
+
+### 1. Core Link Classes (`openfisca_core/links`)
+- **`Many2OneLink`**: Resolves *N* source members to *1* target entity (e.g., `person.mother`, `person.employer`). Supports fetching values (`.get()`) and dynamic chaining (`.mother.household.rent`).
+- **`One2ManyLink`**: Aggregates from *N* target members back to *1* source entity. Supports a wide suite of vectorized aggregations (`sum`, `count`, `any`, `all`, `min`, `max`, `avg`) along with filtering by `role` or an arbitrary boolean `condition` mask.
+
+### 2. Implicit Links & Backward Compatibility
+A major design goal was to avoid breaking existing country packages (`openfisca-france`, `openfisca-tunisia`, etc.).
+- Links are strictly **additive**.
+- During `Simulation` initialization, OpenFisca now automatically reads the existing `GroupEntity` structure and injects **Implicit Links**:
+  - `ImplicitMany2OneLink`: Automatically adds `person.household`, mapping directly to the high-performance `GroupPopulation.members_entity_id` array.
+  - `ImplicitOne2ManyLink`: Automatically adds `household.persons`, replacing the need for verbose legacy aggregations.
+- `Population.__getattr__` was carefully patched to first check `self.links["..."]` before natively falling back to the legacy `get_projector_from_shortcut()` route. *Everything keeps working identically.*
+
+### 3. Syntax Sugar & Chaining
+The new API allows natural, pythonic data fetching:
+```python
+# Old projector way (still works!):
+rents = sim.persons.household("rent", "2024")
+
+# New explicit link definition (e.g., for arbitrary networks)
+mother_link = Many2OneLink(name="mother", link_field="mother_id", target_entity_key="person")
+person_entity.add_link(mother_link)
+
+# New chaining syntax:
+mother_household_rents = sim.persons.mother.household.get("rent", "2024")
+
+# New declarative aggregations:
+female_salaries = sim.households.persons.sum("salary", "2024", condition=is_female)
+```
+
+## Performance
+Performance is a critical constraint for OpenFisca simulations. We added `pytest-benchmark` tests validating the new mechanics.
+- `.get()` resolutions (Many-to-One) perform identically to legacy Projectors (~118μs on 15,000 entities).
+- Aggregations (`One2Many.sum()`) introduce a negligible setup overhead (< 1ms) but execute fully vectorized `numpy.bincount` and `numpy.maximum.at` operations under the hood.
+
+## Associated Documentation
+We've added guides to help framework users model new relationships:
+- `docs/implementation/links-api.md`: Reference for creating and querying `Many2OneLink` and `One2ManyLink`.
+- `docs/implementation/transition-guide.md`: Migration guide demonstrating how to gradually adopt Links over Legacy Projectors.
+
+## Testing
+- 12 new, comprehensive tests covering unit mechanics, system integrations, filtering, chaining, and OpenFisca core lifecycle (`_resolve_links`).
+- All 158 core tests and existing Country Template tests continue to pass locally (`make test-code`).

CHANGELOG.md

@@ -0,0 +1,39 @@
+# Benchmarks
+
+## How to run
+
+```bash
+# Run all benchmarks
+make benchmark
+
+# Run compute benchmarks only
+.venv/bin/python -m pytest benchmarks/test_bench_compute.py -v --benchmark-sort=name
+
+# Run memory benchmarks only
+.venv/bin/python -m pytest benchmarks/test_bench_memory.py -v -s
+
+# Save results for later comparison
+.venv/bin/python -m pytest benchmarks/ --benchmark-save=my_baseline
+
+# Compare with a saved baseline
+.venv/bin/python -m pytest benchmarks/ --benchmark-compare=0001_my_baseline
+```
+
+## Benchmarks included
+
+### Compute (`test_bench_compute.py`)
+
+| Benchmark | What it measures | Sizes |
+|---|---|---|
+| `members_position` | GroupPopulation position assignment | 100 → 1M |
+| `group_sum` | `household.sum(salary)` | 100 → 1M |
+| `disposable_income` | Full variable cascade (~15 vars) | 100 → 100K |
+| `tbs_loading` | TaxBenefitSystem initialization | 1 |
+
+### Memory (`test_bench_memory.py`)
+
+| Benchmark | What it measures | Sizes |
+|---|---|---|
+| `members_position_memory` | Peak memory for position calc | 10K → 1M |
+| `simulation_memory` | Peak memory for full simulation | 10K → 1M |
+| `per_variable_memory` | Memory per variable per person | 10K → 100K |

PR_DESCRIPTION.md

@@ -0,0 +1,61 @@
+"""Shared fixtures for OpenFisca benchmarks."""
+
+import numpy
+import pytest
+
+
+@pytest.fixture(params=[100, 10_000, 100_000, 1_000_000], ids=lambda n: f"N={n:_}")
+def population_size(request):
+    """Population sizes to benchmark."""
+    return request.param
+
+
+@pytest.fixture(params=[100, 10_000, 100_000], ids=lambda n: f"N={n:_}")
+def simulation_size(request):
+    """Population sizes for full simulation benchmarks (capped for speed)."""
+    return request.param
+
+
+@pytest.fixture
+def rng():
+    """Deterministic random number generator."""
+    return numpy.random.default_rng(42)
+
+
+@pytest.fixture
+def make_group_population():
+    """Factory to create a GroupPopulation with random entity assignment."""
+
+    def _make(nb_persons, nb_entities=None):
+        from openfisca_core.populations.group_population import GroupPopulation
+
+        if nb_entities is None:
+            nb_entities = max(1, nb_persons // 3)
+
+        rng = numpy.random.default_rng(42)
+        pop = GroupPopulation.__new__(GroupPopulation)
+        pop._members_entity_id = rng.integers(0, nb_entities, size=nb_persons)
+        pop._members_position = None
+        pop._ordered_members_map = None
+        return pop
+
+    return _make
+
+
+@pytest.fixture
+def make_simulation():
+    """Factory to create a Simulation with salary input."""
+
+    def _make(nb_persons):
+        from openfisca_country_template import CountryTaxBenefitSystem
+
+        from openfisca_core.simulations import SimulationBuilder
+
+        tbs = CountryTaxBenefitSystem()
+        sim = SimulationBuilder().build_default_simulation(tbs, count=nb_persons)
+
+        rng = numpy.random.default_rng(42)
+        sim.set_input("salary", "2024-01", rng.uniform(1000, 5000, nb_persons))
+        return sim
+
+    return _make

benchmarks/README.md

@@ -0,0 +1,141 @@
+"""Compute time benchmarks for OpenFisca-Core.
+
+Uses pytest-benchmark for statistically rigorous measurements.
+Run with: pytest benchmarks/test_bench_compute.py -v --benchmark-sort=name
+"""
+
+import pytest
+
+# ---------------------------------------------------------------------------
+# S1: members_position (the function we just vectorized)
+# ---------------------------------------------------------------------------
+
+
+class TestMembersPositionBench:
+    """Benchmark GroupPopulation.members_position."""
+
+    @pytest.mark.parametrize(
+        "nb_persons,nb_entities",
+        [
+            pytest.param(100, 40, id="N=100"),
+            pytest.param(10_000, 4_000, id="N=10K"),
+            pytest.param(100_000, 40_000, id="N=100K"),
+            pytest.param(1_000_000, 400_000, id="N=1M"),
+        ],
+    )
+    def test_members_position(
+        self, benchmark, nb_persons, nb_entities, make_group_population
+    ):
+        pop = make_group_population(nb_persons, nb_entities)
+
+        def run():
+            pop._members_position = None  # force recompute
+            return pop.members_position
+
+        result = benchmark.pedantic(run, iterations=3, rounds=5, warmup_rounds=1)
+        assert len(result) == nb_persons
+
+
+# ---------------------------------------------------------------------------
+# S2: GroupPopulation aggregations (sum, any)
+# ---------------------------------------------------------------------------
+
+
+class TestGroupAggregationBench:
+    """Benchmark household.sum() and household.any()."""
+
+    @pytest.mark.parametrize(
+        "nb_persons",
+        [
+            pytest.param(10_000, id="N=10K"),
+            pytest.param(100_000, id="N=100K"),
+        ],
+    )
+    def test_household_sum(self, benchmark, nb_persons, make_simulation):
+        sim = make_simulation(nb_persons)
+
+        def run():
+            household = sim.populations["household"]
+            salaries = household.members("salary", "2024-01")
+            return household.sum(salaries)
+
+        result = benchmark.pedantic(run, iterations=5, rounds=5, warmup_rounds=1)
+        assert len(result) > 0
+
+    @pytest.mark.parametrize(
+        "nb_persons",
+        [
+            pytest.param(10_000, id="N=10K"),
+            pytest.param(100_000, id="N=100K"),
+        ],
+    )
+    def test_household_any(self, benchmark, nb_persons, make_simulation):
+        sim = make_simulation(nb_persons)
+
+        def run():
+            household = sim.populations["household"]
+            salaries = household.members("salary", "2024-01")
+            return household.any(salaries > 3000)
+
+        result = benchmark.pedantic(run, iterations=5, rounds=5, warmup_rounds=1)
+        assert len(result) > 0
+
+
+# ---------------------------------------------------------------------------
+# S3: Full simulation (disposable_income)
+# ---------------------------------------------------------------------------
+
+
+class TestFullSimulationBench:
+    """Benchmark a full disposable_income calculation."""
+
+    @pytest.mark.parametrize(
+        "nb_persons",
+        [
+            pytest.param(100, id="N=100"),
+            pytest.param(10_000, id="N=10K"),
+            pytest.param(100_000, id="N=100K"),
+        ],
+    )
+    def test_disposable_income(self, benchmark, nb_persons, make_simulation):
+        sim = make_simulation(nb_persons)
+
+        def run():
+            return sim.calculate("disposable_income", "2024-01")
+
+        result = benchmark.pedantic(run, iterations=1, rounds=3, warmup_rounds=1)
+        assert len(result) > 0
+
+    @pytest.mark.parametrize(
+        "nb_persons",
+        [
+            pytest.param(100, id="N=100"),
+            pytest.param(10_000, id="N=10K"),
+        ],
+    )
+    def test_income_tax(self, benchmark, nb_persons, make_simulation):
+        sim = make_simulation(nb_persons)
+
+        def run():
+            return sim.calculate("income_tax", "2024-01")
+
+        result = benchmark.pedantic(run, iterations=3, rounds=5, warmup_rounds=1)
+        assert len(result) > 0
+
+
+# ---------------------------------------------------------------------------
+# S4: TBS loading
+# ---------------------------------------------------------------------------
+
+
+class TestTBSLoadingBench:
+    """Benchmark TaxBenefitSystem initialization."""
+
+    def test_tbs_loading(self, benchmark):
+        def run():
+            from openfisca_country_template import CountryTaxBenefitSystem
+
+            return CountryTaxBenefitSystem()
+
+        result = benchmark.pedantic(run, iterations=1, rounds=3, warmup_rounds=1)
+        assert result is not None