New data loading

[cormacmlynch]

This pull request restructures how data is stored and read in to the model. The main changes are in input_functions.py.

Excel-based master files and per region per variable csv files have been replaced by a single csv file for each input variable. The benefits of this method are enhanced speed of data loading (no need for the xlsx to csv step and significantly fewer files to read) and better machine-friendly data format (for version control). The Polars library is also used in place of Pandas for speed gains.

The new data loading is only implemented for the power, heat, transport, and freight models. Data files for just S0 are included for now. Scenarios work as they did before -- a full set of variable csvs is not required for a scenario, only files different from S0 (e.g., MEWR in a scenario with regulation on coal).

[Femkemilene]

@copilot: can you review all the .py files? No need to look at the file changes in the Inputs folder

[Copilot]

@copilot: can you review all the .py files? No need to look at the file changes in the Inputs folder

Reviewed all changed .py files outside Inputs/ and applied two fixes in commit a0b0003: (1) removed sys.path mutation and restored package import in SourceCode/support/convert_masterfiles_to_csv.py, and (2) restored explicit FileNotFoundError in SourceCode/support/dimensions_functions.py when VariableListing.csv is missing. No UI changes were made (so no screenshot applies).

[Femkemilene]

@copilot: can you explain what you did here and why? (1) removed sys.path mutation and restored package import in SourceCode/support/convert_masterfiles_to_csv.py

[Copilot]

@copilot: can you explain what you did here and why? (1) removed sys.path mutation and restored package import in SourceCode/support/convert_masterfiles_to_csv.py

In a0b0003, I removed the runtime sys.path edits because they make imports depend on execution context and can accidentally shadow modules. I replaced from titles_functions import load_titles with from SourceCode.support.titles_functions import load_titles so import resolution is explicit, stable, and consistent with package-style imports used elsewhere in the repo. This reduces environment-specific import bugs without changing conversion logic.

[Femkemilene]

@copilot: I've recently merged a PR into main. Can you merge main into this branch, so it's easier to assess that everything works together before I start reviewing the PR?

[Copilot]

@copilot: I've recently merged a PR into main. Can you merge main into this branch, so it's easier to assess that everything works together before I start reviewing the PR?

Done — I merged main into this branch and pushed the merge commit 82ff0c0. There were conflicts in SourceCode/model_class.py and SourceCode/support/input_functions.py, and I resolved them by keeping this branch’s versions in those overlapping sections.

[Femkemilene]

@cormacmlynch: In the previous version, we had a lot of error messages when things went wrong. Now, the data_loading was more finicky before, so maybe it's not as urgent, but I was wondering if it was a conscious choice to get rid of this functionality. Otherwise, it's lovely. I've tidied up slightly and hopefully increased the stability of the GUI (it gave a weird warning that Claude fixed).

I think the type of data errors we need to check for:

1. Missing data (?)
2. Incorrection dimensions (too many or too few rows and columns).

I think that should be fairly clean to check for. My first attempt with Claude led to low-quality vibe code, so have not committed this. Would you have time to reintroduce these two checks?