Adding FTT-H2 to the master version by pv-camecon · Pull Request #282 · cpmodel/FTT_StandAlone

pv-camecon · 2025-09-11T10:12:48Z

Pull request to include FTT-H2 into the master version.

There are a few oddities to sort out:

CSC approach works in selecting the right technologies, but it fails to produce sensible prices. So far, we have relied on domestic LCOH2 estimates.
Universal core FTT functions have been developed in parallel and need to be pulled into this version.
Techno-economic cost data and technology classification may also change to include bio-based options.
In the longer future, the CSC will be depreciated and replaced with another trade routine that is able to deal with bilateral information.

The CLean Energy Adoption in Fertiliser Sector (CLEAFS) module added to the FTT framework. The module currently uses a basic Bass diffusion model to estimate the diffusion of green fertiliser.

Add FTT:H2 to FTT framework, define few variables, add and populate masterfile, small adjustments to main codes to avoid confusion between FTT:H and FTT:H2

…sion

Committing the outer skeleton of the FTT-H2 model

Architectural stuff

Byproduction is not part of the technology scope due to lack of data

Calculate domestic and export production, and capacity factors

…into FTT-H2_dev

CSCs, CLOH2, etc.

The envisioned segmented pooling approach would not work as initially thought. Hence, why we switched to a classic CSC approach. However, we account for trade costs based on average distance from capacity heavy regions to demand heavy regions.

Small fix to the CSC routine to make sure production equals demand, and that prices are correct.

Information on endogenous green fertiliser demand now flows to FTT:H2 where it operates in a split market

Added a new function to determine the innovator effect. It still needs to be included. Also changed some of the parameters for capacity expansion, and capacity growth rates (as a running average).

Added new fuel variables from the baseline E3ME master run. This includes the gas price hike. Energy prices are now connected to the LCOH calculation. Also tweaked the innovator function as it needs to loop over all possible combinations.

New functions have been added to estimate: - green electricity cost for green H2 production - Added emission factor estimation (with hard-coded energy related emission factors) - Added energy consumption estimation - Plugged hydrogen demand calculation into a new script - Corrected energy cost calculation - Corrected cost components and changed units to (per kg H2)

Included static emission factors in the cost matrix. Added carbon price variable to calculate carbon costs.

Added curtailment effects to grid electrolysis. Added script to create graphs.

Fixed a mistake to off-grid electrolytic capacity factors. Hard-coded lifetime adjustments to the Aij matrix to reflect that low carbon capacity is new and probably won't phase out as quickly.

Changes to the market share equation for the default market. It now uses the combined shares of the green+default market to reflect uptake trajectory. Changed time scaling

Adding all input csvs to the master

Commented out code for graphs that are not used in the policy brief.

…into FTT-H2_dev

Add script to export tables for graphs

Adding os import and UTF decoding

- LCOH_range_single_scen values issues (wrong iterable called) - LCOH_range_single_scen labeling fix (avg SMR, electrolytic, min, max in Tech column instead of Region column) - Brazil vs world chart LCOH was mistakenly multiplied by 100 - Added a units column for every table

Previous version did not include O&M of VRE capacity for the off-grid electrolytic options. Corrections to run_file that produces graphs and the one that produces CSVs outputs for editors.

Pull request to include FTT-H2 into the master version. There are a few oddities to sort out: - CSC approach works in selecting the right technologies, but it fails to produce sensible prices. So far, we have relied on domestic LCOH2 estimates. - Universal core FTT functions have been developed in parallel and need to be pulled into this version. - Techno-economic cost data and technology classification may also change.

Copilot

Pull Request Overview

This pull request integrates the FTT-H2 (Hydrogen) module into the master version of the model. The changes primarily involve updating parameter files across multiple regions to replace uniform values with country-specific data, adding a new "Sector" column header, and providing differentiated values for specific sectors like road transport, households, agriculture, and marine/aviation bunkers.

Updates multiple regional PFRB (Parameter File Regional Breakdown) CSV files with differentiated sectoral data
Adds sector-specific values to replace previously uniform data across all sectors
Introduces new column structure with "Sector" header for better data organization

Reviewed Changes

Copilot reviewed 47 out of 422 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
PFRB_KE.csv	Adds sector column and updates Kenya-specific values with differentiated data for transport and household sectors
PFRB_JA.csv	Updates Japan data with sector-specific values, particularly for transport sectors
PFRB_IT.csv	Modifies Italy parameters with differentiated values for road transport and household sectors
PFRB_IS.csv	Updates Iceland data with sector-specific values, notably for road transport and households
PFRB_IN.csv	Replaces India's uniform sectoral data with country-specific differentiated values
PFRB_IE.csv	Updates Ireland parameters with sector-specific data for transport and household categories

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-09-11T13:55:31Z

+13 Other industry,1199.374,1393.5208,1457.0782,1397.6902,1332.7166,1162.3152,1075.1416,1154.0266,1286.6663,1265.0654,1123.9985,1282.4702,1302.2014,1322.5244,1343.4573,1365.0181,1387.2256,1410.0995,1433.6595,1457.9264,1482.9214,1508.666,1535.1831,1562.4956,1590.6276,1619.6034,1649.4485,1680.1892,1711.8518,1744.4644,1778.0554,1812.6541,1848.2908,1884.9966,1922.8035,1961.7446,2001.854,2043.1667,2085.7188,2129.5474,2174.6909,2221.1887,2269.0813,2318.4109,2369.2202,2421.554,2475.4575,2530.9783,2588.1646,2647.0664,2707.7356,2770.2249,2834.5886,2900.8833,2969.1667,3039.499,3111.9409,3186.5562,3263.4099,3342.5693,3424.1035
+14 Construction,1199.374,1393.5208,1457.0782,1397.6902,1332.7166,1162.3152,1075.1416,1154.0266,1286.6663,1265.0654,1123.9985,1282.4702,1302.2014,1322.5244,1343.4573,1365.0181,1387.2256,1410.0995,1433.6595,1457.9264,1482.9214,1508.666,1535.1831,1562.4956,1590.6276,1619.6034,1649.4485,1680.1892,1711.8518,1744.4644,1778.0554,1812.6541,1848.2908,1884.9966,1922.8035,1961.7446,2001.854,2043.1667,2085.7188,2129.5474,2174.6909,2221.1887,2269.0813,2318.4109,2369.2202,2421.554,2475.4575,2530.9783,2588.1646,2647.0664,2707.7356,2770.2249,2834.5886,2900.8833,2969.1667,3039.499,3111.9409,3186.5562,3263.4099,3342.5693,3424.1035
+15 Rail transport,1199.374,1393.5208,1457.0782,1397.6902,1332.7166,1162.3152,1075.1416,1154.0266,1286.6663,1265.0654,1123.9985,1282.4702,1302.2014,1322.5244,1343.4573,1365.0181,1387.2256,1410.0995,1433.6595,1457.9264,1482.9214,1508.666,1535.1831,1562.4956,1590.6276,1619.6034,1649.4485,1680.1892,1711.8518,1744.4644,1778.0554,1812.6541,1848.2908,1884.9966,1922.8035,1961.7446,2001.854,2043.1667,2085.7188,2129.5474,2174.6909,2221.1887,2269.0813,2318.4109,2369.2202,2421.554,2475.4575,2530.9783,2588.1646,2647.0664,2707.7356,2770.2249,2834.5886,2900.8833,2969.1667,3039.499,3111.9409,3186.5562,3263.4099,3342.5693,3424.1035
+16 Road transport,1502.4196,1684.5791,1865.6624,1703.1218,1631.3662,1646.8123,1714.7281,1925.1091,1832.7454,1832.7454,1832.7454,1832.7454,1852.4766,1872.7997,1893.7324,1915.2932,1937.5009,1960.3748,1983.9348,2008.2017,2033.1965,2058.9412,2085.4583,2112.7708,2140.9028,2169.8787,2199.7236,2230.4644,2262.127,2294.7397,2328.3306,2362.9292,2398.5659,2435.2717,2473.0786,2512.02,2552.1294,2593.4419,2635.9941,2679.8228,2724.9661,2771.4639,2819.3564,2868.686,2919.4954,2971.8291,3025.7329,3081.2534,3138.4399,3197.3418,3258.0107,3320.5,3384.8638,3451.1584,3519.4419,3589.7742,3662.2161,3736.8313,3813.6851,3892.8445,3974.3787


The data shows identical values for years 2018-2021 in the road transport sector (1832.7454). This pattern may indicate missing data interpolation or placeholder values that should be reviewed for accuracy, especially since other sectors show continuous year-over-year progression.

Suggested change

16 Road transport,1502.4196,1684.5791,1865.6624,1703.1218,1631.3662,1646.8123,1714.7281,1925.1091,1832.7454,1832.7454,1832.7454,1832.7454,1852.4766,1872.7997,1893.7324,1915.2932,1937.5009,1960.3748,1983.9348,2008.2017,2033.1965,2058.9412,2085.4583,2112.7708,2140.9028,2169.8787,2199.7236,2230.4644,2262.127,2294.7397,2328.3306,2362.9292,2398.5659,2435.2717,2473.0786,2512.02,2552.1294,2593.4419,2635.9941,2679.8228,2724.9661,2771.4639,2819.3564,2868.686,2919.4954,2971.8291,3025.7329,3081.2534,3138.4399,3197.3418,3258.0107,3320.5,3384.8638,3451.1584,3519.4419,3589.7742,3662.2161,3736.8313,3813.6851,3892.8445,3974.3787

16 Road transport,1502.4196,1684.5791,1865.6624,1703.1218,1631.3662,1646.8123,1714.7281,1925.1091,1906.0903,1887.0715,1868.0527,1849.0339,1852.4766,1872.7997,1893.7324,1915.2932,1937.5009,1960.3748,1983.9348,2008.2017,2033.1965,2058.9412,2085.4583,2112.7708,2140.9028,2169.8787,2199.7236,2230.4644,2262.127,2294.7397,2328.3306,2362.9292,2398.5659,2435.2717,2473.0786,2512.02,2552.1294,2593.4419,2635.9941,2679.8228,2724.9661,2771.4639,2819.3564,2868.686,2919.4954,2971.8291,3025.7329,3081.2534,3138.4399,3197.3418,3258.0107,3320.5,3384.8638,3451.1584,3519.4419,3589.7742,3662.2161,3736.8313,3813.6851,3892.8445,3974.3787

Femkemilene

Thanks Pim for this PR. Broadly speaking, there's 3 things I'd like to see

Having doc strings throughout. It's difficult to check the logic of the model when this is missing. A doc string should not only be a description of the variables, but also explain succintly what the functions do and why.
After we're happy with 1, it's time to check if we're happy with the logic of the code, and test the results are as expected.
Integrate the new core functions. Mostly, this will be adjusting this branch to align with the existing core functions, which are optimised and in some cases (sales) corrected. I can help here.

Femkemilene · 2025-09-11T13:56:05Z

 *.pickle
 Output/Scenarios.json
 settings.ini
+CostSupplyCurve/CSC_green_2023.csv


Use regex to avoid repeating yourself (DRY coding). That is CostSupplyCurve/CSC*.csv

Femkemilene · 2025-09-11T13:57:16Z

@@ -0,0 +1,8 @@
+[codestyle]


Can you explain what these config files are?

From Spyder project. Redundant back-up file that should not have been committed. Whole Spyder project object can be removed.

Femkemilene · 2025-09-11T13:58:53Z

@@ -0,0 +1,4 @@
+# Ignore everything in this directory


I don't think we need to create all these empty folders

Femkemilene · 2025-09-11T13:59:15Z

+"""
+Created on Wed Oct 18 21:11:06 2023
+
+@author: adh


Please add a human-readable author name

Femkemilene · 2025-09-11T13:59:40Z

+# os.chdir("C:/Users/adh/OneDrive - Cambridge Econometrics/ADH CE/Phd/ÚNKP_2023/data")
+
+
+def Bass_param_estimation(data, titles):


functions start with a lower-case. Please add a doc-string

Femkemilene · 2025-09-11T14:10:48Z

@@ -0,0 +1,225 @@
+# -*- coding: utf-8 -*-


Remove the first function, as the existing shares equation is optimized

Femkemilene · 2025-09-11T14:11:39Z

@@ -0,0 +1,87 @@
+# -*- coding: utf-8 -*-


Great to have a function for this! But if we do this, should we not use the original formulation of the substitution matrix from Mercure (2015). That is, computing the weighted average lead and life times (rather than redefining kappa to include this).

Femkemilene · 2025-09-11T14:18:27Z


+                        if var == 'PFRA': 


Remove debugging statements

Femkemilene · 2025-09-11T14:18:44Z

                    csv.columns = [int(year) for year in csv.columns]

                    #print(file)
+                    if var == 'WFTR':


Remove debugging statements

Femkemilene · 2025-09-11T14:19:04Z

Delete this file

Femkemilene · 2025-09-11T14:36:45Z

When I run the model, I'm getting:

ValueError: Error: The results contain NaN values.

It seems to come from this bit of code in the main h2 file:

if (np.any(np.isnan(data['HYG1'][:, :, 0])) or
518 np.any(np.isnan(data['HYWK'][:, :, 0])) or
519 np.any(np.isnan(data['HYCF'][:, :, 0]))
520 ):

This seems to be related to fuel costs: HYFC is not included in the Masterfile

Femkemilene · 2025-09-11T14:54:59Z

+    buildtime_adjust = data['BCHY'][:, :, c7ti['Buildtime']]*0
+
+    # Manual adjustment of lifetimes for new technologies
+    idx = [i for i in range(len(titles['HYTI'])) if i not in [0, 2]]


Is this a hack to compensate for the lack of age cohort tracking? Please explain (I know how to do this now!)

No, this is a placeholder for a variable that extends or contracts the lead time of technologies. This should be replaced with a variable. A similar one exists for lifetime extension or contraction. Both go into the substitution_frequencies function.

Not correct, and now all included in the default files

Remove unused functions, and switch to faster copying

pv-camecon · 2025-09-11T17:44:10Z

When I run the model, I'm getting:

ValueError: Error: The results contain NaN values.

It seems to come from this bit of code in the main h2 file:

if (np.any(np.isnan(data['HYG1'][:, :, 0])) or 518 np.any(np.isnan(data['HYWK'][:, :, 0])) or 519 np.any(np.isnan(data['HYCF'][:, :, 0])) 520 ):
This seems to be related to fuel costs: HYFC is not included in the Masterfile

HYFC is calculated using csv files in the "General" folder (i.e. PFR[x], PRSC, EX) and they need to be removed from the ignore list. Similar variables are also stored in the FTT-P folder and they should be removed, or stored somewhere else for access. In any case, any FTT model that requires outside energy price projections should use the ones from the "General" folder.

hartviga and others added 30 commits October 21, 2024 14:25

Basic Bass model for fertilisers

7942773

The CLean Energy Adoption in Fertiliser Sector (CLEAFS) module added to the FTT framework. The module currently uses a basic Bass diffusion model to estimate the diffusion of green fertiliser.

Setup FTT:H2

c6ef80b

Add FTT:H2 to FTT framework, define few variables, add and populate masterfile, small adjustments to main codes to avoid confusion between FTT:H and FTT:H2

Fertiliser module: update financial factors in green fertiliser diffu…

af1a44e

…sion

Basic FTT-H2 design

911e691

Committing the outer skeleton of the FTT-H2 model

Adding empty scripts

54a09ed

Architectural stuff

Updated excel files for cost-supply curve calculation

e0696ff

H2 cost-supply curves + draft pooledtrading

a916ebb

Removed byproduction

39ddadf

Byproduction is not part of the technology scope due to lack of data

Trade module

f22bbb4

Calculate domestic and export production, and capacity factors

Merge branch 'FTT-H2_dev' of https://github.com/cpmodel/FTT_StandAlone …

ebc89ee

…into FTT-H2_dev

Added functions

58cc349

CSCs, CLOH2, etc.

Changed trade routine

d5f2ec0

The envisioned segmented pooling approach would not work as initially thought. Hence, why we switched to a classic CSC approach. However, we account for trade costs based on average distance from capacity heavy regions to demand heavy regions.

Fixed CSC

a1af152

Small fix to the CSC routine to make sure production equals demand, and that prices are correct.

Connected with CLEAFS and included a split market

43f2ca4

Information on endogenous green fertiliser demand now flows to FTT:H2 where it operates in a split market

Added innovator effect, and small changes

ea23e1c

Added a new function to determine the innovator effect. It still needs to be included. Also changed some of the parameters for capacity expansion, and capacity growth rates (as a running average).

Added carbon costs and restraints on capacity growth

be57bb0

Included static emission factors in the cost matrix. Added carbon price variable to calculate carbon costs.

Version for 1st results

44dced7

Added curtailment effects to grid electrolysis. Added script to create graphs.

Version used for policy brief v1

6e8ed1b

Fixed a mistake to off-grid electrolytic capacity factors. Hard-coded lifetime adjustments to the Aij matrix to reflect that low carbon capacity is new and probably won't phase out as quickly.

Final version used for the policy draft

47cb6bd

Changes to the market share equation for the default market. It now uses the combined shares of the green+default market to reflect uptake trajectory. Changed time scaling

Master file and variables list update

0f88170

Adding all input csvs to the master

Update run_file_create_graphs.py

60d6501

Commented out code for graphs that are not used in the policy brief.

Merge branch 'FTT-H2_dev' of https://github.com/cpmodel/FTT_StandAlone …

f36e9c5

…into FTT-H2_dev

Tables export

4e64ab9

Add script to export tables for graphs

Correction to run_file_graphs_tables.py

52bc478

Adding os import and UTF decoding

Fix to OPEX and result printing scripts

988e5cb

Previous version did not include O&M of VRE capacity for the off-grid electrolytic options. Corrections to run_file that produces graphs and the one that produces CSVs outputs for editors.

Clean and update master file and Variables excel

b0d22e2

pv-camecon requested review from Femkemilene and jp-camecon September 11, 2025 10:12

pv-camecon added the enhancement New feature or request label Sep 11, 2025

Femkemilene requested a review from Copilot September 11, 2025 13:54

Copilot AI reviewed Sep 11, 2025

View reviewed changes

Femkemilene requested changes Sep 11, 2025

View reviewed changes

Femkemilene reviewed Sep 11, 2025

View reviewed changes

Femkemilene added 3 commits September 11, 2025 16:13

Remove unused sales function

f72a30b

Not correct, and now all included in the default files

Quick clean

8f89991

Remove unused functions, and switch to faster copying

Typo

0f6225e

		# os.chdir("C:/Users/adh/OneDrive - Cambridge Econometrics/ADH CE/Phd/ÚNKP_2023/data")


		def Bass_param_estimation(data, titles):

Conversation

pv-camecon commented Sep 11, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Copilot AI Sep 11, 2025

Choose a reason for hiding this comment

Uh oh!

Femkemilene left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Femkemilene commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pv-camecon commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Femkemilene commented Sep 11, 2025 •

edited

Loading

pv-camecon commented Sep 11, 2025 •

edited

Loading