fix(fair2): idealised conc-driven PI ERF + empty-bundle splice

changelog/105.docs.md

@@ -0,0 +1 @@
+Added an "Idealised experiments and the non-conc-species baseline" documentation page describing how the MAGICC7, FaIR2 and CICEROSCMPY2 adapters treat non-concentration-driven species in concentration-driven runs, and recommending the explicit `esm-piControl` emissions overlay for reproducible idealised experiments.

changelog/105.fix.md

@@ -0,0 +1 @@
+Fixed two FaIR2 issues affecting idealised concentration-driven experiments (`abrupt-4xCO2`, `1pctCO2`, `esm-flat10*`): a `KeyError` in `_splice_bundle_with_user` when a user supplied `Emissions|*` rows for a scenario with no RCMIP3 emissions baseline, and a constant non-CO2 (aerosol-dominated) preindustrial ERF residual (~1.5 W/m²). The residual arose because idealised non-CO2 species were left at zero emissions, which is not the reference FaIR evaluates emissions-driven forcing against; they are now held at their `baseline_emissions` so the preindustrial forcing is ~0.

docs/source/idealised-experiments.md

@@ -0,0 +1,62 @@
+# Idealised experiments and the non-conc-species baseline
+
+Concentration-driven runs (`RunMode.CONCENTRATION_DRIVEN`) only drive the species
+you supply as `Atmospheric Concentrations|*`. **Every other species still needs an
+emissions input** — aerosol precursors (Sulfur, BC, OC, …), ozone precursors (NOx,
+CO, VOC, NH3), and any greenhouse gas you did not supply a concentration for. What
+each adapter does for those "non-conc" species differs, and it matters most for the
+**idealised CMIP experiments** (`abrupt-4xCO2`, `1pctCO2`, `esm-flat10*`), where the
+intent is "nothing but CO2 changes; everything else stays at pre-industrial (PI)".
+
+## Per-adapter behaviour
+
+| Adapter | Non-conc species (general conc-driven) | Idealised experiments |
+|---|---|---|
+| **MAGICC7** | Requires `Emissions\|*` for species it cannot concentration-drive (written to the SCEN7 file). Missing species fall back to MAGICC's built-in defaults. | No idealised auto-detection — you must supply the non-CO2 emissions you want (e.g. an `esm-piControl` overlay). |
+| **FaIR2** | Non-idealised scenarios inherit the canonical RCMIP3 emissions baseline (forward-filled from the last historical year). | Idealised = `protocol_natural_forcing="off"` **and** `protocol_land_use_forcing="constant_zero"` (or a name match like `abrupt-*` / `1pctCO2*`). Natural (solar/volcanic) and land-use forcing are zeroed, and non-CO2 emissions-mode species are **held at their `baseline_emissions`** so their PI forcing is ~0. |
+| **CICEROSCMPY2** | Non-supplied species inherit the `historical_em_file` (emissions) / `historical_conc_file` (concentrations) baseline. | Non-supplied emissions are zeroed (`zero_unsupplied`); non-supplied concentrations are held at PI (`hold_unsupplied_at_pi`). |
+
+```{note}
+FaIR2 holds idealised non-CO2 species at `baseline_emissions` rather than at zero
+emissions. FaIR evaluates emissions-driven forcing (aerosols especially) *relative
+to* `baseline_emissions`, so zeroing emissions would leave a constant spurious ERF
+(~1 W/m² of aerosol forcing) instead of the intended PI zero.
+```
+
+## Recommended pattern: supply the PI baseline explicitly
+
+The per-adapter idealised defaults differ, and MAGICC7 does no auto-detection at all.
+For reproducible cross-model idealised runs, **supply the non-CO2 baseline you want
+explicitly** rather than relying on per-adapter behaviour. The canonical choice is the
+`esm-piControl` emissions from the RCMIP3 bundle:
+
+```python
+import scmdata
+from openscm_runner import RunMode, run
+from openscm_runner.io import load_rcmip3_emissions, canonicalise_rcmip3_variable
+
+# 1. Idealised conc-driven scenario (e.g. abrupt-4xCO2 CO2 concentrations).
+conc_scen = ...  # ScmRun with Atmospheric Concentrations|* rows
+
+# 2. esm-piControl emissions for the non-CO2 species, relabelled onto the
+#    idealised scenario name. Drop CO2 sources (CO2 is concentration-driven).
+pi = load_rcmip3_emissions(RCMIP3_BUNDLE, scenarios=["esm-piControl"])
+pi["Variable"] = pi["Variable"].map(canonicalise_rcmip3_variable)
+pi = pi[~pi["Variable"].str.startswith("Emissions|CO2")]
+pi_overlay = relabel_scenario(pi, "abrupt-4xCO2")  # set scenario meta + protocol_* flags
+
+# 3. Merge and dispatch.
+scenarios = scmdata.run_append([conc_scen, pi_overlay])
+run(climate_models_cfgs=..., scenarios=scenarios,
+    output_variables=("Effective Radiative Forcing",),
+    mode=RunMode.CONCENTRATION_DRIVEN)
+```
+
+For FaIR2 and CICEROSCMPY2 the explicit overlay reproduces what their idealised
+auto-detection already does; for MAGICC7 it is the only way to get correct PI
+behaviour for the non-CO2 species.
+```{note}
+Supplying a user `Emissions|*` overlay for an idealised scenario also exercises the
+FaIR2 emissions splice; that path is fixed so an empty RCMIP3 baseline (which every
+idealised scenario has) no longer errors.
+```

docs/source/index.md

@@ -28,6 +28,7 @@
 ```{toctree}
 :caption: Contents
 :maxdepth: 2
+idealised-experiments
 notebooks
 development
 api/openscm_runner

src/openscm_runner/adapters/fair2_adapter/_emissions_translator.py

@@ -264,6 +264,19 @@ def _splice_bundle_with_user(
     overlay on the matching ``(scenario, variable)`` pair with unit
     scaling so the bundle's unit is preserved.
     """
+    # Idealised scenarios (abrupt-4xCO2, 1pctCO2, esm-flat10*) have no
+    # RCMIP3 emissions baseline, so _rcmip3_to_fair_emissions_df returns
+    # an empty (column-less) frame. Indexing ``bundle_df["scenario"]``
+    # below would then KeyError; short-circuit to the user's rows (the
+    # ``if spliced_df.empty`` branch further down is unreachable in this
+    # case because the KeyError fires first).
+    if bundle_df.empty:
+        return (
+            user_df.reset_index(drop=True)
+            if not user_df.empty
+            else pd.DataFrame()
+        )
+
     # Normalise bundle column names; FaIR is case-insensitive on them.
     bundle_df = bundle_df.copy()
     bundle_df.columns = [

src/openscm_runner/adapters/fair2_adapter/fair2_adapter.py

@@ -286,6 +286,43 @@ def _zero_fill_fair_arrays(f) -> None:
             values[mask] = 0.0
 
 
+# CO2 source species are concentration-driven (or back-calculated) in the
+# idealised conc experiments, so they must NOT be pinned to baseline.
+_CO2_SOURCE_SPECIES = frozenset({"CO2 FFI", "CO2 AFOLU"})
+
+
+def _hold_idealised_nonco2_at_baseline(f, idealised_scenarios) -> None:
+    """
+    Pin non-CO2 emissions-mode species to ``baseline_emissions`` for the
+    given idealised scenarios, so their PI forcing is ~0.
+
+    FaIR computes emissions-driven forcing relative to each species'
+    ``baseline_emissions`` (aerosol ERF in particular). Leaving idealised
+    non-CO2 species at zero emissions therefore produces a constant
+    non-zero ERF (anomaly ``0 - baseline_emissions``). Setting their
+    emissions to ``baseline_emissions`` makes the anomaly -- and hence the
+    forcing -- zero, which is the intended "everything but CO2 at PI"
+    state. Concentration-driven species (e.g. CH4 / N2O when supplied as
+    concentrations) are not emissions-mode and are left untouched.
+    """
+    if not idealised_scenarios:
+        return
+
+    run_scenarios = set(f.scenarios)
+    baseline = f.species_configs["baseline_emissions"]
+    for scenario in idealised_scenarios:
+        if scenario not in run_scenarios:
+            continue
+        for specie in f.species:
+            if specie in _CO2_SOURCE_SPECIES:
+                continue
+            if f.properties_df.loc[specie, "input_mode"] != "emissions":
+                continue
+            f.emissions.loc[
+                {"scenario": scenario, "specie": specie}
+            ] = baseline.sel(specie=specie)
+
+
 def _resolve_calibration(value: Any) -> NativeFairCalibration:
     """
     Accept either a path-like or an already-loaded
@@ -947,6 +984,15 @@ def _run_one_calibration(  # noqa: PLR0912, PLR0913, PLR0915
     # supplied as a concentration, etc.).
     _zero_fill_fair_arrays(f)
 
+    # Idealised experiments (abrupt-4xCO2, 1pctCO2, esm-flat10*) hold
+    # everything but CO2 at pre-industrial. Zero emissions (from the
+    # co2_only zeroing / the zero-fill above) is NOT the PI reference:
+    # FaIR evaluates emissions-driven forcing -- notably aerosols --
+    # relative to each species' baseline_emissions, so zero emissions
+    # leaves a constant spurious ERF. Hold the non-CO2 emissions-mode
+    # species at baseline_emissions instead, so their PI forcing is ~0.
+    _hold_idealised_nonco2_at_baseline(f, idealised_scenarios)
+
     f.run(progress=False, suppress_warnings=True)
 
     return extract_outputs(

tests/integration/test_idealised_pi_erf.py

@@ -0,0 +1,101 @@
+"""Idealised conc-driven runs hold non-CO2 forcing at pre-industrial.
+
+For an idealised experiment (abrupt-4xCO2) driven by CO2 concentration
+only, every non-CO2 species should sit at its PI baseline, so the
+non-CO2 ERF (total minus CO2) should be ~0 at the branch point. This
+guards the FaIR2 idealised baseline fix: non-CO2 emissions-mode species
+are pinned to ``baseline_emissions`` (not zero emissions), so their PI
+forcing is ~0 rather than a constant ~1 W/m^2 aerosol offset.
+
+Scope: FaIR2 only. The other adapters auto-handle idealised scenarios
+differently and do not expose a directly comparable total / CO2 ERF
+pair:
+  * MAGICC7 does not auto-detect idealised scenarios and uses non-PI
+    built-in defaults for non-conc species unless an esm-piControl
+    emissions overlay is supplied (see the "Idealised experiments"
+    docs page), so a conc-only assertion would not hold for it.
+  * CICEROSCMPY2 exposes only component-wise ERF (no plain total /
+    CO2 ERF), so a non-CO2 assertion needs adapter-specific
+    aggregation -- a follow-up.
+
+Env-gated: needs the full RCMIP3 bundle (the mini fixture lacks
+abrupt-4xCO2 concentrations). Point ``RCMIP3_BUNDLE`` at it.
+"""
+from __future__ import annotations
+
+import os
+from pathlib import Path
+
+import pandas as pd
+import pytest
+from scmdata import ScmRun
+
+import openscm_runner.run
+from openscm_runner import RunMode
+from openscm_runner.adapters import FAIR2
+from openscm_runner.adapters.fair2_adapter._compat import HAS_FAIR2
+from openscm_runner.io import load_rcmip3_concentrations
+
+pytestmark = pytest.mark.faircicero2_only
+
+FAIR2_BUNDLE = Path(__file__).parent.parent / "test-data" / "fair2-mini-bundle"
+RCMIP3_BUNDLE = os.environ.get(
+    "RCMIP3_BUNDLE", "/storage/no-backup-nac/users/bensan/rcmip3_protocol",
+)
+SCENARIO = "abrupt-4xCO2"
+PI_YEAR = 1850  # branch point: CO2 still ~PI, so total ERF ~ CO2 ERF
+ERF = "Effective Radiative Forcing"
+ERF_CO2 = "Effective Radiative Forcing|CO2"
+
+
+def _has_full_bundle() -> bool:
+    try:
+        return not load_rcmip3_concentrations(
+            RCMIP3_BUNDLE, scenarios=[SCENARIO],
+        ).empty
+    except Exception:
+        return False
+
+
+def _idealised_conc_scenario() -> ScmRun:
+    """abrupt-4xCO2 CO2 concentration trajectory as a conc-driven ScmRun."""
+    conc = load_rcmip3_concentrations(RCMIP3_BUNDLE, scenarios=[SCENARIO])
+    co2 = conc[
+        conc["Variable"].str.fullmatch("Atmospheric Concentrations.CO2")
+    ].iloc[0]
+    year_cols = [c for c in conc.columns if isinstance(c, str) and c.isdigit()]
+    return ScmRun(pd.DataFrame({
+        "model": ["protocol"], "scenario": [SCENARIO], "region": ["World"],
+        "variable": ["Atmospheric Concentrations|CO2"], "unit": ["ppm"],
+        **{int(c): [float(co2[c])] for c in year_cols},
+    }))
+
+
+@pytest.mark.skipif(not HAS_FAIR2, reason="fair>=2 not installed")
+@pytest.mark.skipif(
+    not _has_full_bundle(),
+    reason=f"full RCMIP3 bundle with {SCENARIO} concentrations not found "
+    f"(set RCMIP3_BUNDLE)",
+)
+def test_fair2_idealised_nonco2_erf_is_pi_zero():
+    adapter = FAIR2.from_native_distribution(
+        calibration_dir=FAIR2_BUNDLE, rcmip3_bundle_path=RCMIP3_BUNDLE,
+        mode=RunMode.CONCENTRATION_DRIVEN, member_indices=[0],
+        output_variables=(ERF, ERF_CO2),
+    )
+    res = openscm_runner.run.run([adapter], scenarios=_idealised_conc_scenario())
+    ts = res.timeseries(time_axis="year")
+
+    def _value(variable):
+        sub = ts[ts.index.get_level_values("variable") == variable]
+        assert not sub.empty, f"adapter did not output {variable!r}"
+        return float(sub.iloc[0].get(PI_YEAR))
+
+    total, co2 = _value(ERF), _value(ERF_CO2)
+    non_co2 = total - co2
+    # Everything but CO2 is held at PI -> non-CO2 ERF ~ 0 at the branch.
+    # (Before the fix this carried a constant ~+0.6 W/m^2 aerosol offset.)
+    assert abs(non_co2) < 0.1, (
+        f"non-CO2 ERF at {PI_YEAR} = {non_co2:+.3f} W/m^2 "
+        f"(total={total:+.3f}, CO2={co2:+.3f}); expected ~0"
+    )

tests/unit/io_tests/test_rcmip3.py

@@ -356,6 +356,76 @@ def test_fair2_emissions_canonical_path_translates_variables():
     assert 1750 in year_cols and 2100 in year_cols
 
 
+def test_fair2_splice_empty_bundle_returns_user_rows():
+    """Idealised scenarios (abrupt-4xCO2, 1pctCO2, esm-flat10*) have no
+    RCMIP3 emissions baseline, so the bundle frame is empty. The splice
+    must return the user's rows instead of raising ``KeyError`` on the
+    missing ``scenario`` column."""
+    from openscm_runner.adapters.fair2_adapter._emissions_translator import (
+        _splice_bundle_with_user,
+    )
+
+    user_df = pd.DataFrame([{
+        "scenario": "abrupt-4xCO2",
+        "variable": "CO2 FFI",
+        "region": "World",
+        "unit": "Gt CO2/yr",
+        1750: 0.0,
+        1850: 36.0,
+    }])
+
+    out = _splice_bundle_with_user(
+        pd.DataFrame(), user_df, scenario_names=["abrupt-4xCO2"],
+    )
+    assert not out.empty
+    assert set(out["variable"]) == {"CO2 FFI"}
+    assert out.loc[out["variable"] == "CO2 FFI", 1850].iloc[0] == 36.0
+
+    # Empty bundle + empty user -> empty frame (no crash).
+    assert _splice_bundle_with_user(
+        pd.DataFrame(), pd.DataFrame(), scenario_names=["abrupt-4xCO2"],
+    ).empty
+
+
+def test_fair2_build_emissions_df_idealised_scenario_with_user_emissions():
+    """End-to-end: an idealised scenario (no RCMIP3 baseline) with a
+    user emissions overlay must build without KeyError, keep CO2, and
+    zero the non-CO2 species via the ``co2_only_scenarios`` path."""
+    from scmdata import ScmRun
+
+    from openscm_runner.adapters.fair2_adapter._emissions_translator import (
+        build_emissions_df,
+    )
+
+    scmrun = ScmRun(pd.DataFrame({
+        "model": ["test", "test"],
+        "scenario": ["abrupt-4xCO2", "abrupt-4xCO2"],
+        "region": ["World", "World"],
+        "variable": [
+            "Emissions|CO2|MAGICC Fossil and Industrial",
+            "Emissions|CH4",
+        ],
+        "unit": ["Mt CO2/yr", "Mt CH4/yr"],
+        1750: [0.0, 100.0],
+        1850: [1000.0, 300.0],
+    }))
+
+    out = build_emissions_df(
+        scmrun, MINI_BUNDLE, scenario_names=["abrupt-4xCO2"],
+        co2_only_scenarios=("abrupt-4xCO2",),
+    )
+    assert not out.empty
+    variables = set(out["variable"])
+    assert "CO2 FFI" in variables and "CH4" in variables
+    # build_emissions_df stringifies year columns at the boundary.
+    year_cols = [c for c in out.columns if isinstance(c, str) and c.isdigit()]
+    co2 = out[out["variable"] == "CO2 FFI"]
+    ch4 = out[out["variable"] == "CH4"]
+    # CO2 survives; non-CO2 is zeroed for the idealised scenario.
+    assert (co2[year_cols].to_numpy() != 0).any()
+    assert (ch4[year_cols].fillna(0).to_numpy() == 0).all()
+
+
 def test_fair2_concentrations_canonical_path_translates_variables():
     from openscm_runner.adapters.fair2_adapter._concentrations_translator import (
         build_concentrations_df_from_rcmip3,