shengrenhou / optimal-energy-system-scheduling-combining-mixed-integer-programming-and-deep-reinforcement-learning

The Source code for paper "Optimal Energy System Scheduling Combining Mixed-Integer Programming and Deep Reinforcement Learning". Safe reinforcement learning, energy management

Optimal Energy System Scheduling Using A Constraint-Aware Reinforcement Learning Algorithm

• This code accompanies the paper Optimal Energy System Scheduling Using A Constraint-Aware Reinforcement Learning Algorithm, to appear in International Journal of Electrical Power & Energy Systems.

Abstract

• The massive integration of renewable-based distributed energy resources (DERs) inherently increases the energy system's complexity, especially when it comes to defining its operational schedule. Deep reinforcement learning (DRL) algorithms arise as a promising solution due to their data-driven and model-free features. However, current DRL algorithms fail to enforce rigorous operational constraints (e.g., power balance, ramping up or down constraints) limiting their implementation in real systems. To overcome this, in this paper, a DRL algorithm (namely MIP-DQN) is proposed, capable of \textit{strictly} enforcing all operational constraints in the action space, ensuring the feasibility of the defined schedule in real-time operation. This is done by leveraging recent optimization advances for deep neural networks (DNNs) that allow their representation as a MIP formulation, enabling further consideration of any action space constraints. Comprehensive numerical simulations show that the proposed algorithm outperforms existing state-of-the-art DRL algorithms, obtaining a lower error when compared with the optimal global solution (upper boundary) obtained after solving a mathematical programming formulation with perfect forecast information; while strictly enforcing all operational constraints (even in unseen test days).

Organization

• Folder "Data" -- Historical and processed data.
• script "Parameters"-- General parameters for batteries and other energy units.
• script "MIP_DQN"-- The implementation of the proposed MIP-DQN algorithm.
• script "random_generator_battery" -- The energy system environment
• script "random_generator_more_battery" -- The energy system environment with multi-batteries. Developing this because the reviewer asked, even I don't think it is essential.
• Run scripts after installing all packages. Please have a look for the code structure.

Dependencies

This code requires installation of the following libraries: PYOMO,pandas 1.1.4, numpy 1.20.1, matplotlib 3.3.4, pytorch 1.11.0, OMLT,wandb. I used wandb to monitor the changes during the training. you can find more information at this page.

Recommended citation

A preprint is available, and you can check this paper for more details Link of the paper.

• Paper authors: Hou Shengren, Pedro P. Vergara, Edgar Mauricio Salazar, Peter Palensky
• Accepted for publication at International Journal of Electrical Power & Energy Systems
• If you use (parts of) this code, please cite the preprint or published paper

Additional Information

• The clean code and data are now uploaded.

optimal-energy-system-scheduling-combining-mixed-integer-programming-and-deep-reinforcement-learning's Issues

库的版本问题

Some question about the MIP_DQN.py

Hello, I'd like to understand how to use the "Actor_MIP" class in the provided code. This part is mentioned as a highlight in your paper, but it seems that the class is not called or utilized in the code. I'm interested in learning how this class should be employed.

I wanna communicate some thing about RL in energy dispatch problem

Hello, thanks for your excellent work. I benefit a lot from it. I have some question about DRL as below:
Do you try to solve the hybrid action space reinforcement problem? such as unit comittment and dispatch problem.
Once i see you repo's title, i guess your work is about the problem that i told. So i am eager to exchange study with you.
Here is my wechat id: 15736716954 .

State space issue

Hi, Hou.

Thanks for sharing. Just one question regarding the state space limits. It seems that the net_load could be negative. In that case, shouldn't the normalized state space defined as [-1,1]?

Thanks.

Parameters

How you are taking battery capacity in Kw? (Capicity should be in Ah or Kwh)
For current_capicity you are taking a random number (0.2-0.8). What do you mean by current capacity, do you mean SOC?
I am confused in Battery class....BTW the over all code is 👌
@ShengrenHou

Congrats to Brother Hou!

Congrats to Brother Hou!