Running GCAM core with broken out region

[dobrien13]

Hi GCAM team,

I've broken the UK out of the EU-15 region but am encountering an issue when the model runs. The scenario components update successfully to reference the new XMLs, they parse successfully, then on running a new scenario, I get 54 error messages reading "Could not delete node gdp-control as it does not exist." The model then begins a run but fails due to "No emissions coefficient set for SO2_1 in USA period 0. I'm happy to attach any relevant XMLs here, but have not debugged at this stage before, and am not sure where to start.

Thanks for your guidance.

[pkyle]

I've seen this error happen a few times, not sure why it's happening here though. Basically what it means is that there's some technology/NonCO2 that's being created, but there's no input-emissions or emiss-coeff assigned. This would make more sense if it happened in one of the regions impacted by the revised mappings; not sure why it's happening with SO2 in the USA. Could indicate a bigger problem.
Still one easy check that should address this specific error is just to comment out the file emission_factor_controls.xml from the configuration file, which is an add-on that revises some emissions coefficients according to the data read into the tables in inst/extdata/emissions/emission_controls/*.csv. That XML file is likely responsible for the Could not delete node gdp-control warning message, and looking at how it's implemented, the XML file emission_factor_controls.xml won't create new supplysectors, subsectors, or technologies, but it can create new NonCO2 objects if they don't already exist (see below, where the nocreate="1" flags are assigned). So that's probably the sole source of this message. But again there might be a bigger issue too; this is the sort of error that crashes the model the moment it happens, so we don't get more info, but I don't know why the specific NonCO2 wouldn't exist.
`

    <region name="USA">
        <supplysector name="trn_shipping_intl" nocreate="1">
            <subsector name="International Ship" nocreate="1">
                <stub-technology name="Liquids" nocreate="1">
                    <period year="1975">
                        <Non-CO2 name="SO2_1">
                            <gdp-control delete="1" name="GDP_control"/>
                        </Non-CO2>

`

[dobrien13]

Interesting. That all makes sense. On commenting out the file emission_factor_controls.xml from the configuration file, the "could not delete node..." errors go away, and the model still fails in Period 0. However, it now fails due to having "No emissions coefficient set for CH4 in USA in period 0."

[pkyle]

It's hard to say, but just in order to make sure everything in the XML files is consistent with all of the input files and code, I'd recommend running source("data-raw/generate_package_data.R") to re-build the system .rda files from scratch, and then once that's done (takes about an hour), re-build XML files using driver() rather than driver_drake().

[dobrien13]

Alright, we've failed in the source("data-raw/generate_package_data.R") command with ~50 chunks to go. Argh.

[1] "module_gcamusa_L2261.regional_biomass"
[1] "- make 0.36"
[1] "module_gcamusa_L270.limits"
[1] "- make 0.07"
Error in check_chunk_outputs(chunk, chunk_data, input_names, promised_outputs = po, :
Some precursors for 'L270.NegEmissBudget_USA' aren't inputs - chunk module_gcamusa_L270.limits
In addition: There were 14 warnings (use warnings() to see them)

[pkyle]

Haha OK that's a bug that we fixed shortly after the release, but it still isn't in a release version or the Github master branch. Instead I've just been recommending for people to retrieve it from my forked repo on github, as described here: #393 (comment)

[dobrien13]

Sorry, I must have only copied over the gcam_limits not the gcamusa_limits file when I originally rolled that bug fix in. Confirming that generate_package_data does run to completion now.

Driver(), however, did not complete its run and got stuck on modules in aglu, such as module_aglu_L152.ag_GTAP_R_C_GLU_irr. It simply freezes or gets caught in a loop and R crashes. Unsure why. Is this something worth trying to debug?

If useful, we are using a bug fix from gcamdata (#f6d240e8, May 31 2023) that includes files in aglu adapted to v7 that are in the vicinity of the aglu freezes. (fix explained here: JGCRI/gcamdata#1236 (comment)) Was a permanent fix implemented such that we should get rid of this patch?

[pkyle]

I don't know; if it was able to run generate_package_data.R then that means all of the code is working, but I guess when it's saving outputs etc. it runs out of memory? If you aren't already, I'd recommend using driver(write_outputs = FALSE, write_xml = TRUE) just so you don't have to wait around for it to write like a thousand CSV files to the disk. That's the command from the makefile, that's executed when you make xml from the root directory of the workspace.

[dobrien13]

That did it! Good call.

Unfortunately, we've still failed while running in period 0. No emissions coefficient set for CH4 in USA in period 0.

Unsure if related, Robbie pointed out that configuration.xml uses the same tag for multiple non-CO2 gas files. e.g.:

	<Value name = "nonco2_energy">../input/gcamdata/xml/all_energy_emissions.xml</Value>
	<Value name = "nonco2_energy">../input/gcamdata/xml/all_energy_emissions_MAC.xml</Value>

Other than that, not sure of next steps. Sorry to be less useful here

[pkyle]

As noted in a prior comment in this thread, this error message just means that a nonCO2 object was created but not assigned an emissions coefficient or an input-emissions quantity in the base year(s). Searching out internal forum I see that @jayfuhrman had this issue a few years ago, so Jay feel free to weigh in if you have any notes about the source of the issue. In his case the crash was also preceded by a warning about NA values; do you have that too? It would be in the main_log immediately after the XML file that causes the problem is read in, so it's easy to identify the source of the problem if you have that issue.
That it happened in the USA region suggests that it's unrelated to the UK breakout, but also it's probably just in the USA because the USA is the first region in the XML structure, and the model crashes immediately when it encounters that situation. I.e. there's no reason to expect that whatever technology is missing a coefficient is only missing it in the USA. That it was CH4 and not CH4_AGR or CH4_AWB means it was a technology in the energy system; past that we don't have any information.
Things you might try:
(1) You can try removing that all_energy_emissions_MAC.xml file from the configuration. The second one is just assigning MAC curves to nonCO2s. If there's any technology in all_energy_emissions_MAC.xml that wasn't in all_energy_emissions.xml, then that will cause this crash, and it's an easy test that might narrow it down.
(2) after saving/committing your fileset and structure, you can try undoing the regional breakout and running. That will tell you whether this issue is related to the additional regional breakout, as opposed to just in the base fileset.

[dobrien13]

#1 was a good test -- the calibration periods solved and the model failed instead in 2020 with a similar message.

No emissions coefficient set for SF6 in USA in period 5