Masked Deep Reinforcement Learning for Virtual Optical Network Embedding Over Elastic Optical Networks
Research project into the use of deep reinforcement learning (DRL) for optimisation of dynamic virtual optical network embedding (VONE) on an elastic optical network (EON) substrate.
Contains code for:
- Gym environments for VONE, with action spaces for node-and-path, node-only, and path-only selection.
- Scripts for:
- training
- evaluation
- data visualisation
Used to produce the results in this paper, submitted to ONDM 2023.
Here's a step-by-step guide on setting up a Conda environment for a project called vone-drl and using the install_requirements.py script:
- Install Miniconda or Anaconda: If you haven't already, install Miniconda or Anaconda on your system. You can download Miniconda from here or Anaconda from here.
- Create a new Conda environment: Open a terminal (or Anaconda Prompt on Windows), and run the following command to create a new Conda environment named vone-drl with Python installed:
conda create -n vone-drl python
- Activate the environment:
conda activate vone-drl
- Navigate to your project directory: In the terminal, navigate to the root directory of your vone-drl project.
- Run the install_requirements.py script: In the terminal, run the following command to execute the install_requirements.py script:
python install_requirements.py
This will install the required packages, as specified in the requirements.txt and requirements-gpu.txt files.
Now, you should have a Conda environment named vone-drl with all the necessary packages installed. You can use this environment to work on your project. When you want to deactivate the environment, run conda deactivate. To reactivate the environment later, use conda activate vone-drl.
The results shown in the paper can be recreated by:
- Obtain trained models:
- Option 1: Download pre-trained models
- Option 2: Train from scratch
- Evaluate model performance
- Plot results
It's recommended to download the pre-trained models in order to avoid any difficulties with reproducing training (due to stochastic action sampling during training).
poetry run python train.py --env_file ./config/agent_combined.yaml --log WARN --masking --multistep_masking --gamma=0.5499732330963527 --learning_rate=0.001048322384752267 --n_steps=46 --output_file ./data/agent_combined_train.csv --save_model --log_dir ./models --no_wandb --id combined
poetry run python train.py --env_file ./config/agent_nodes.yaml --log WARN --masking --gamma=0.5664841514329136 --learning_rate=0.001309728909201273 --n_steps=63 --output_file ./data/agent_nodes_train.csv --save_model --log_dir ./models --no_wandb --id nodes
poetry run python train.py --env_file ./config/agent_paths.yaml --log WARN --masking --gamma=0.6344458270083639 --learning_rate=0.00031675064580752364 --n_steps=60 --output_file ./data/agent_paths_train.csv --save_model --log_dir ./models --no_wandb --id paths
To evaluate trained-from-scratch model:
poetry run python eval.py --env_file ./config/agent_combined.yaml --log WARN --model_file ./models/combined/combined_model.zip --output_file ./eval/agent_combined.csv --eval_masking --multistep_masking --id combined
poetry run python eval.py --env_file ./config/agent_nodes.yaml --log WARN --model_file ./models/nodes/nodes_model.zip --output_file ./eval/agent_nodes.csv --eval_masking --id nodes
poetry run python eval.py --env_file ./config/agent_paths.yaml --log WARN --model_file ./models/paths/paths_model.zip --output_file ./eval/agent_paths.csv --eval_masking --id paths
To evaluate pre-trained model:
poetry run python eval.py --env_file ./config/agent_combined.yaml --log WARN --output_file ./eval/agent_combined.csv --masking --eval_masking --multistep_masking --artifact micdoh/VONE-DRL/4ampum3d_model:v0 --id combined
poetry run python eval.py --env_file ./config/agent_nodes.yaml --log WARN --output_file ./eval/agent_nodes.csv --masking --eval_masking --artifact micdoh/VONE-DRL/agent_nodes:v0 --id nodes
poetry run python eval.py --env_file ./config/agent_paths.yaml --log WARN --output_file ./eval/agent_paths.csv --masking --eval_masking --artifact micdoh/VONE-DRL/agent_routes:v0 --id paths
To generate heuristic evaluation data:
poetry run python eval_heuristic.py --output_file ./eval/heur.csv
For training curves:
poetry run python plot.py --training --node_train_file ./data/agent_nodes_train.csv --path_train_file ./data/agent_paths_train.csv --combined_train_file ./data/agent_combined_train.csv
For evaluation results:
poetry run python plot.py --eval --node_eval_file ./eval/agent_nodes.csv --path_eval_file ./eval/agent_paths.csv --combined_eval_file ./eval/agent_combined.csv --heur_eval_file ./eval/heur.csv