kth-desa / osemosys_step

Scripts to conduct myopic runs with OSeMOSYS models

OSeMOSYS step

This repository contains a set of scripts to conduct model runs with limited foresight with OSeMOSYS models.

osemosys_step's Issues

Limits clashing with solver tolerance

When passing the installed capacity from one step to the next OSeMOSYS might crash if the installed capacity passed is slightly higher than implemented limits due to the tolerance of solvers.

sol-files taking storage space

The sol files created when solving with gurobi or cplex take significant storage space. They could be deleted after the conversion of the results to csv.

Didn't consider the subdirectories under step directories in case scenarios exist

In the merge of ResidualCapacity and NewCapacity, the read in of the ResCap doesn't consider that there are potentially subdirectories under the ../data/step_*/ directory.

Package OSeMOSYS_step

To ease the use of the scripts and facilitate easy integration in workflows it would be good to create a package of the scripts.

Function to create folder structure locally

To package the tool so that it can be installed locally, write a function which creates the desired folder structure in a user-defined location. For example, the command:

step create ./my_folder/my_myopic_model

would create the desired folder structure containing data scenarios, model, results and steps. A second command:

step run ./my_folder/my_myopic_model

would then run the workflow on the folder structure.

Growth rate indicated in scenario parameter is applied to wrong emission

When indicating a reduction rate for emissions the script takes as the starting value the last value of a different emission than intended.

No final results generated if no step has options

If no step has options the results of the steps are not merged in the results folder.
Issue is that main.py in line 431 searches for subdirectories in ../results but if no step has option no subdirectories exist.

Add logs folder

Snakemake bug

The current snakefile doesn't produce any output and gives the below bug.
(osemosys_step) KTH-dESA/OSeMOSYS_step - (snakemake) > snakemake --cores 1 Academic license - for non-commercial use only - expires 2021-10-07 Using license file /Users/cotti/gurobi.lic No parameters matching '_test' found SyntaxError in line 16 of /Users/cotti/GitHub/KTH-dESA/OSeMOSYS_step/Snakefile: invalid syntax (Snakefile, line 16)

Large capacity additions for no reason

Hi @trevorb1,
I notice large amounts of NewCapacity for IMPHCO, IMPURN, and RLu from step 1 and onwards. Always exactly 10 million. But I don't see why the model is doing so.

Add check if snakemake is installed

The error one get's if snakemake is not installed doesn't indicate that it is related to the missing installation. A upfront check for the installation might be useful

The final results accumulate by keeping them for every step.

The storage needed for running models with the scrip could be reduced by deleting the final results once they have been passed to the next level of scenarios.

Implement option for other solvers than glpk in the simple step function

The step function without multiple-scenarios does not contain the option yet to solve with other solvers than glpk.

Vertical bar in file name creates issue on Windows

In the names of the solv logs a vertical bar aka pipe is used.
This can't be used in file names on Windows.

Integrate lp generation into snakemake workflow

The generation of lp file can take quite some time for larger models. Hence it would be good to parallelize this part as well.
Do you think this would be very complicated @trevorb1?
I think in the original scripts I had it within the script called by snakemake.

Solver integration

In the current step function the models are run with GLPK.
Especially for large models other solvers like gurobi, cbc, or cplex are commonly used.
It would be good to write an integration for those. The desired solver could be provided as an argument.
To consider is that glpk is needed to generate the lp file.

In GLPK process preprocessing scripts not called

So OseMOSYS code needs to look different than when running with gurobi.
--> Alighn processes

Historic capacity is dropped in merge of residual and New capacity

I observe that the original ResidualCapacity is being dropped and only the NewCapacity is passed on.
Would you have time to have a look @trevorb1?

OS indpendet paths for simple step function

Currently the paths used in the simple step function use unix format. The scripts will therefore not run on Windows.

User determined foresight

Allow the user to determine the foresight the step models should have.
Currently the foresight in each step is the same length as the step, i.e. with a step length of 5 years, the foresight is five, which means the model covers 10 years.
However, it would be interesting to be able to investigate the effect of different foresight horizons. Considering that many power plants have a life time of approximately 25 years a model horizon of 25 years would be interesting. The step length could be 5 years and the foresight 20.
My suggestion would be to leave the current behaviour of doubling the step length to determine the model length as default, but allow to provide the desired foresight as an additional input.
What do you think @trevorb1?

Error when no folder called `solves` in `logs` directory

@trevorb1

ERROR:utils:Step must be of type int or List[int]

When running main.py with python main.py --step_length 5 I get an error saying that the there is no sol file produced from step 0. In the log file I see the below message:
ERROR:utils:Step must be of type int or List[int]. Recieved type <class 'tuple'> WARNING:otoole.read_strategies:Parameter ResultsPath could not be found in the configuration.

Stranded assets

In a myopic setting the likelihood for stranded assets is higher than under perfect-foresight.
It would be interesting to add the required calculations in this package or in otoole.

Add option for cbc

In the solv.py script option could be created to run the models with cbc.

Merge of ResidualCapacity and NewCapacity

When running a model that with a TotalAnnualMaxCapacity of 6GW on a certain technolgy the model installs in step 4 in 2035 3 GW and in 2039 again 3 GW. So far no step had options, but the next step, step 5 has one decision with two options. However, here the run crashes at the osemosys check of ResidualCapacity and TotalAnnualMaxCapacity. The two models for step 5 both show a ResidualCapacity of 9GW, even though it should be 6, not just because of the limit but also because in step 4 there were twice 3 GW installed.

I know you are currently very time constrained, but if you have 20 minutes to spare today could you have a look @trevorb1?
Otherwise, I will give it a go tomorrow. But it seems odd to me. Had we not already looked into the logic of adding ResidualCap. and NewCap.?

Add option for cplex

In the solv.py an option run modesl with cplex could be added.

Line for results path in osemosys code can change

the main_step script changes the results path in the osemosys file when running models with GLPK. This creates an error when the lines in the OSeMOSYS file have changed.

the change of input parameter over time depends on the decisions of previous steps

the current set-up of the step scripts works well for decisions that can be described as ON/OFF.
However, decisions like for example the pace of emission reductions can't be done. The issue is that previous decisions affect the starting point of the next step.
To represent such decions it would be good to be able to indicate the different paces of reduction e.g. 5% or 10% and then calculate the time series for the upcoming step using the last year of the previous modelling period as a starting point.

Half of scenarios don't get the scneario specific parameter data

Half of the scenarios in a step don't contain the paramter data specified in the scenario csv.

Use gurobipy

For running models with gurobi one could use gurobipy instead of calling gurobi via subprocess

Option to run with perfect foresight and scenarios

A option to run the model under perfect foresight while providing scenario options like in the myopic set up with limited foresight would facilitate to have one base case/reference scenario of a model and then run scenarios without creating each time by hand a modified new model file/set of csv files.

Residual Capacity + NewCapacity

Hi @trevorb1, I have gone through the results. I can't explain all differences, but I see that it seems like the NewCapacity of RHO, I don't know if others as well, from step 0 is not passed on to step1, this makes the model at the beginning of step1 install the amount that is missing. Not sure why this capacities are not passed on.

Rounding of solver tolerated values leaves too many values

In the merge of ResidualCapacity and NewCapacity the New capacity is currently rounded to 4 digits behind the dot. However this still creates errors with TotalAnnualMaxCapacity limits. Since these limits are often having one digit behind the point I suggest to reduce the rounding to one digit behind the point.
The alternative would be to use np.floor but that could potentially cause a significant loss of capacity between steps.

Passing of NewCapacity doesn't work as expected

Hi @trevorb1,
When digging into the input data and results, I noticed, that the calculation of the ResidualCapacity doesn't work as expected.
I notice that NewCapacity is also passed from years that are not in the step but in the foresight horizon, e.g. with a five year step length and model starting in 1990, NewCapacity from 1996 and onwards should not be added to the next step anymore.
Haven't looked into the scripts yet. Might not manage to do so today, so if you have time ;)

Use provided model file independent of it's name

Currently the scripts are using a model file called osemosys.txt. If one wants to use a modified or specific version of osemosys it might be nice to indicate that in the file name. this could be done by letting the script check the files in the model folder and throw a bug if there are more files than expected.

ilp logs are written into step directory, which then deleted

When a model is infeasible gurobi produces a ilp file which can be useful to understanding why a model is infeasible. The file is written into the corresponding step directory. However, if the model is infeasible, this directory is deleted, which makes it difficult to understand why a model is infeasible...

lp-files taking storage space

Delete lp-files after running models would reduce the required harddrive space needed

Subprocess on Windows

It seems like on Windows the calling of snakemake from the python script via subprocess doesn't work.
A solution that works on Unix and Windows would be desirable.

Errror when scenario doesn't return results for all previously reported parameter

Depending on scenario settings it can happen, that a parameter doesn't have results in all scenarios. For example when emissions are strongly reduced emission related parameter might not show anything in certain scenarios.
Currenty the scripts then throw an error when the step results are to be appended to the final results.
https://github.com/KTH-dESA/OSeMOSYS_step/blob/main/src/step_to_final.py#L43-L44

Last year skipped when foresight horizon reaches last year

When the foresight reaches the last year of the modelling period the last year is not made part of the model.

check of feasibility for gurobi missing

This kills the process when scenarios are infeasible.
It seems like when infeasible gurobi writes an ilp-file AND an empty .sol-file

@trevorb1, since you wrote the other tests and in case you have 15 minutes could you give it a go?
Otherwise, don't worry, I can do it tomorrow.

[Packaging] Error: Snakefile not found

When calling the package outside the cloned repo, the snakefile is not found.

Problem is that the path is relative.

Error: Snakefile "src/osemosys_step/snakefile" not found.