knowledgedefinednetworking / a-deep-rl-approach-for-sdn-routing-optimization

A Deep-Reinforcement Learning Approach for Software-Defined Networking Routing Optimization

A Deep-Reinforcement Learning Approach for Software-Defined Networking Routing Optimization

1709.07080: Giorgio Stampa, Marta Arias, David Sanchez-Charles, Victor Muntes-Mulero, Albert Cabellos

In this paper we design and evaluate a Deep-Reinforcement Learning agent that optimizes routing. Our agent adapts automatically to current traffic conditions and proposes tailored configurations that attempt to minimize the network delay. Experiments show very promising performance. Moreover, this approach provides important operational advantages with respect to traditional optimization algorithms.

---

Code and datasets here.

---

Keras and Deep Deterministic Policy Gradient to control an OMNeT++ network simulator

How to build?

1. Use a modern Linux or macOS system.
2. Install OMNeT++ version 4.6 in your system (please see instructions at http://omnetpp.org).
3. Run 'make' from inside 'omnet/router'. This will generate networkRL which is needed by the python script.
4. Install Python 3.6 in your system.
5. Install the packages listed in requirements.txt (please use exact versions). Virtualenv could be of help (https://pypi.python.org/pypi/virtualenv).

How to train?

Single run with fixed parameters

Reads configuration from DDPG.json

python3 ddpg.py

How to play?

Single run with fixed parameters

Reads configuration (ddpg, neural network weights, etc.) from folder

python3 ddpg.py play folder

---

EXAMPLE JSON CONFIG

{
    "ACTIVE_NODES": 3,                                          # number of active nodes in the network
    "ACTUM": "NEW",                                             # action: NEW or DELTA
    "BATCH_SIZE": 50,                                           # size of learning batch
    "BUFFER_SIZE": 2000,                                        # max size of replay buffer
    "ENV": "label",                                             # "label" or "balancing"
    "EPISODE_COUNT": 10,                                        # number of episodes
    "EXPLORE": 0.8,                                             # exploration: rate if <=1, number of steps otherwise
    "GAMMA": 0.99,                                              # discount factor
    "HACTI": "selu",                                            # non-linear activation function for hidden layers
    "HIDDEN1_UNITS": 300,                                       # neurons of layer 1
    "HIDDEN2_UNITS": 600,                                       # neurons of layer 2
    "LRA": 0.0001,                                              # learning rate of the actor network
    "LRC": 0.001,                                               # learning rate of the critic network
    "MAX_STEPS": 1000,                                          # number of steps per episode
    "MU": 0.0,                                                  # Ornstein-Uhlenbeck process' μ
    "PRAEMIUM": "MAX",                                          # reward function
    "PRINT": true,                                              # verbosity
    "ROUTING": "Linkweight",                                    # "Balancer", "Linkweight", "Pathchoice"
    "RSEED": null,                                              # random seed: null or number
    "SIGMA": 0.07,                                              # Ornstein-Uhlenbeck process' σ
    "STATUM": "T",                                              # state representation: T or RT
    "TAU": 0.001,                                               # soft target update
    "THETA": 0.03,                                              # Ornstein-Uhlenbeck process' θ
    "TRAFFIC": "EXP"                                            # traffic: static or changing (~randomly)
}

---

author: [email protected]

• Keras
• DDPG
• OMNeT++

a-deep-rl-approach-for-sdn-routing-optimization's Issues

can not get the paper result

have you anyone get the fig2 result in paper? my model doesn't convergence。

how to start train

I don't know how to generate traffic. txt and route. txt files in the code. How can I get these files? Can anyone teach me？

How long it will take to run the program.

I have a question. How long did it take you to run this program? I estimate that it will take about 100,000 seconds to finish it, which is too long.

Can't Run 'make' from inside 'omnet/router'.

Why cant I make in omnet/router?

How to deal with this, please!!!

(base) scnu-go@scnu-go:~/Projects_RL/a-deep-rl-approach-for-sdn-routing-optimization/omnet/router$ make
Application.cc
In file included from Application.h:24,
                 from Application.cc:16:
Statistic.h:35:17: error: ‘simtime_t’ has not been declared
   35 |     void infoTS(simtime_t time);
      |                 ^~~~~~~~~
Statistic.h:36:19: error: ‘simtime_t’ has not been declared
   36 |     void setDelay(simtime_t time, int i, int j, double d);
      |                   ^~~~~~~~~
Statistic.h:37:21: error: ‘simtime_t’ has not been declared
   37 |     void setTraffic(simtime_t time, int i, int j, double t);
      |                     ^~~~~~~~~
Statistic.h:41:18: error: ‘simtime_t’ has not been declared
   41 |     void setLost(simtime_t time, int n, int p);
      |                  ^~~~~~~~~
Statistic.h:42:18: error: ‘simtime_t’ has not been declared
   42 |     void setLost(simtime_t time);
      |                  ^~~~~~~~~
Statistic.h:62:5: error: ‘simtime_t’ does not name a type; did you mean ‘gmtime_r’?
   62 |     simtime_t INI;
      |     ^~~~~~~~~
      |     gmtime_r
Statistic.h:63:5: error: ‘simtime_t’ does not name a type; did you mean ‘gmtime_r’?
   63 |     simtime_t END;
      |     ^~~~~~~~~
      |     gmtime_r
In file included from Application.cc:16:
Application.h:33:1: error: expected class-name before ‘{’ token
   33 | {
      | ^
Application.h:54:32: error: ‘cMessage’ has not been declared
   54 |     virtual void handleMessage(cMessage *msg);
      |                                ^~~~~~~~
In file included from /home/scnu-go/SDN_RL_env/omnetpp-5.6.2-src-linux/omnetpp-5.6.2/include/omnetpp/cobject.h:26,
                 from /home/scnu-go/SDN_RL_env/omnetpp-5.6.2-src-linux/omnetpp-5.6.2/include/omnetpp/cnamedobject.h:23,
                 from /home/scnu-go/SDN_RL_env/omnetpp-5.6.2-src-linux/omnetpp-5.6.2/include/omnetpp/cownedobject.h:23,
                 from /home/scnu-go/SDN_RL_env/omnetpp-5.6.2-src-linux/omnetpp-5.6.2/include/omnetpp/carray.h:19,
                 from /home/scnu-go/SDN_RL_env/omnetpp-5.6.2-src-linux/omnetpp-5.6.2/include/omnetpp.h:28,
                 from Application.h:19,
                 from Application.cc:16:
Application.cc: In function ‘omnetpp::cObject* __factoryfunc_19()’:
Application.cc:19:15: error: cannot convert ‘Application*’ to ‘omnetpp::cModule*’ in initialization
   19 | Define_Module(Application);
      |               ^~~~~~~~~~~
Application.cc: In destructor ‘virtual Application::~Application()’:
Application.cc:29:6: error: ‘cancelAndDelete’ was not declared in this scope
   29 |      cancelAndDelete(interArrival);
      |      ^~~~~~~~~~~~~~~
Application.cc: In member function ‘virtual void Application::initialize()’:
Application.cc:36:10: error: ‘par’ was not declared in this scope
   36 |     id = par("id");
      |          ^~~
Application.cc: At global scope:
Application.cc:53:33: error: variable or field ‘handleMessage’ declared void
   53 | void Application::handleMessage(cMessage *msg)
      |                                 ^~~~~~~~
Application.cc:53:33: error: ‘cMessage’ was not declared in this scope; did you mean ‘omnetpp::cMessage’?
   53 | void Application::handleMessage(cMessage *msg)
      |                                 ^~~~~~~~
      |                                 omnetpp::cMessage
In file included from /home/scnu-go/SDN_RL_env/omnetpp-5.6.2-src-linux/omnetpp-5.6.2/include/omnetpp/cchannel.h:21,
                 from /home/scnu-go/SDN_RL_env/omnetpp-5.6.2-src-linux/omnetpp-5.6.2/include/omnetpp.h:35,
                 from Application.h:19,
                 from Application.cc:16:
/home/scnu-go/SDN_RL_env/omnetpp-5.6.2-src-linux/omnetpp-5.6.2/include/omnetpp/cmessage.h:95:15: note: ‘omnetpp::cMessage’ declared here
   95 | class SIM_API cMessage : public cEvent
      |               ^~~~~~~~
Application.cc:53:43: error: ‘msg’ was not declared in this scope; did you mean ‘osg’?
   53 | void Application::handleMessage(cMessage *msg)
      |                                           ^~~
      |                                           osg
make: *** [Makefile:107：out/gcc-release//Application.o] ERROR 1

Thanks a lot !!!!!!

Need Guideline

Hello Everyone,
I am very new to deep reinforcement learning. Would you mind showing me step-by-step how to run this code?
Best,
Imtiaz Ahmed

error during training

I am getting the following error while trying to run ddpg.py

TypeError: glorot_normal() got an unexpected keyword argument 'dtype'

How to assign the parameters during execute your code!!

I have tried to run your code on my computer, but I do not know how to set the networkAll.node().tcontroller.folderName when I run the code of router on omnett++?

How to generate traffic matrices?

Does anyone know how to generate traffic matrices with diffferent traffic intensity levels using
a gravity model? Is there any tool to generate?

Regarding tensorflow version change

Hello Sir,
The code has been written for tensorflow 1.0.1.But updated version does not support it.
Can you please guide me, to write the code for current tensonflow version..
@gissimo

Getting an error : No such file or directory

Now we build the Actor model
Now we build the Actor model
2021-02-10 15:19:50.560629: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2021-02-10 15:19:50.582805: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 2699905000 Hz
Now we build the Critic model
Now we build the Critic model
Now we load the weight
Cannot find the weight
OMNeT++ Experiment Start.
Episode : 0 Replay Buffer 0
/home/shivam/keras/lib/python3.8/site-packages/tensorflow/python/keras/engine/training.py:2325: UserWarning: Model.state_updates will be removed in a future version. This property should not be used in TensorFlow 2.0, as updates are applied automatically.
warnings.warn('Model.state_updates will be removed in a future version. '

<!> Error: Could not load NED sources from 'omnet/router/': Declared package 'inet.examples.adhoc.hostautoconf' does not match expected package 'networksimulator.inet.examples.adhoc.hostautoconf' in file Scenario.ned
Traceback (most recent call last):
File "ddpg.py", line 215, in
playGame(DDPG_config, train_indicator=1)
File "ddpg.py", line 123, in playGame
s_t1, r_t, done = env.step(a_t[0])
File "/home/shivam/Documents/DRL/Environment.py", line 452, in step
om_output = file_to_csv(self.folder + OMDELAY)
File "/home/shivam/Documents/DRL/Environment.py", line 45, in file_to_csv
with open(file_name, 'r') as file:
FileNotFoundError: [Errno 2] No such file or directory: 'runs//t1612950590335656/Delay.txt'

Can this code run under windows?

The author write it run under a modern Linux or MaxOS. But I only have windows computer. I have tried to install Linux in a virtual machine, but it is too slow. So, I run 'make' in windows, the result is OMNET++ only generate a 'Router.exe' instead of 'NetworkRL'. What can I do to make the code executable in windows?

FileNotFoundError: [Errno 2] No such file or directory: 'omnet/router/networkRL'

I have generated the network RL using OMNET4.6 and put it in /omnet/router/.
However, while running the code，it occurred an error:FileNotFoundError: [Errno 2] No such file or directory: 'omnet/router/networkRL'.

The code 👍
if env.CLUSTER == 'arvei':
# prefix = '/scratch/nas/1/giorgio/rlnet/'
prefix = '/home/zhen/桌面/sdn/'

simexe = prefix + 'omnet/' + sim + '/networkRL'

Cannot find the weight

When I run ddpg.py It says unable to find the weight. I have confusion where to place file "traffic" that contains 1000 TMs.

FileNotFoundError: [Errno 2] No such file or directory: 'runs//t1677080026736617/Delay.txt'

I have run and modified the program using Windows, but it still does not generate Delay.txt in the folder after running.

python ddpg.py
2023-02-22 23:33:46.740377: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-02-22 23:33:47.139438: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:357] MLIR V1 optimization pass is not enabled
C:\Users\bill8.conda\envs\fordemo\lib\site-packages\keras\optimizers\optimizer_v2\adam.py:114: UserWarning: The lr argument is deprecated, use learning_rate instead.
super().init(name, **kwargs)
2023-02-22 23:33:47.411766: W tensorflow/c/c_api.cc:291] Operation '{name:'Adam/a_V/bias/v/Assign' id:345 op device:{requested: '', assigned: ''} def:{{{node Adam/a_V/bias/v/Assign}} = AssignVariableOp[_has_manual_control_dependencies=true, dtype=DT_FLOAT, validate_shape=false](Adam/a_V/bias/v, Adam/a_V/bias/v/Initializer/zeros)}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
Cannot find the weight
OMNeT++ Experiment Start.
Episode : 0 Replay Buffer 0
C:\Users\bill8.conda\envs\fordemo\lib\site-packages\keras\engine\training_v1.py:2356: UserWarning: Model.state_updates will be removed in a future version. This property should not be used in TensorFlow 2.0, as updates are applied automatically.
updates=self.state_updates,
2023-02-22 23:33:47.570289: W tensorflow/c/c_api.cc:291] Operation '{name:'a_V/Sigmoid' id:85 op device:{requested: '', assigned: ''} def:{{{node a_V/Sigmoid}} = SigmoidT=DT_FLOAT, _has_manual_control_dependencies=true}}' was changed by setting attribute after it was run by a session. This mutation will have no effect, and will trigger an error in the future. Either don't modify nodes after running them or create a new session.
Traceback (most recent call last):
File "C:\omnetpp-4.6\ddpg.py", line 231, in
playGame(DDPG_config, train_indicator=1)
File "C:\omnetpp-4.6\ddpg.py", line 140, in playGame
s_t1, r_t, done = env.step(a_t[0])
File "C:\omnetpp-4.6\Environment.py", line 467, in step
om_output = file_to_csv(self.folder + OMDELAY)
File "C:\omnetpp-4.6\Environment.py", line 46, in file_to_csv
with open(file_name, 'r') as file:
FileNotFoundError: [Errno 2] No such file or directory: 'runs//t1677080026736617/Delay.txt'

The modified code is available at
https://github.com/bill8363/FIX_a-deep-rl-approach-for-sdn-routing-optimization/blob/main/Environment.py

How to run the code

Dear author:
can you explain how to run the code on my computer? since I do not know how to run omnet++ and python at the same time?

Cannot run the ddpg.py !!!

The program says "Cannot find weight". I had run the make file in the oment/router and perform the build operations. Pls help