alirezashamsoshoara / reinforcement_learning_team_q_learnig_marl_multi_agent_uav_spectrum_task

A solution for Dynamic Spectrum Management in Mission-Critical UAV Networks using Team Q learning as a Multi-Agent Reinforcement Learning Approach

A solution for Dynamic Spectrum Management in Mission-Critical UAV Networks using Team Q learning as a Multi-Agent Reinforcement Learning Approach

Paper

You can find the article related to this code here at IEEE or You can find the preprint from the Arxiv website.

In the related project, we study the problem of spectrum scarcity in a network of unmanned aerial vehicles (UAVs) during mission-critical applications such as disaster monitoring and public safety missions, where the preallocated spectrum is not sufficient to offer a high data transmission rate for real-time video-streaming. In such scenarios, the UAV network can lease part of the spectrum of a terrestrial licensed network in exchange for providing relaying service. In order to optimize the performance of the UAV network and prolong its lifetime, some of the UAVs will function as a relay for the primary network while the rest of the UAVs carry out their sensing tasks. Here, we propose a team reinforcement learning algorithm performed by the UAV’s controller unit to determine the optimum allocation of sensing and relaying tasks among the UAVs as well as their relocation strategy at each time. We analyze the convergence of our algorithm and present simulation results to evaluate the system throughput in different scenarios.

• The system model of this paper is based on:

Code

This code is run and tested on Python 2.7 on both linux and windows machine with no issues. There is a config file in this directoy which shows all the configuration parameters such as transmit power, the grid size, number of steps, number of epochs, number of runs, etc. The number of UAVs in this study is assumed to be two. For more UAVs, the code should be altered. You can simply run the main.py file to run the code. It doesn't need any input argument, all you need to configure is available in the config.py. All dependency files are available in the root directory of this repository.

python main.py

Some sample data from the simulation are stored and located in the /data/ directory for a grid size of 4x4 and 2 UAVs.

Required Packages

• time
• copy
• Scipy
• match
• Numpy
• Random
• matplotlib.pyplot

Results

Citation

If you find the code or the article useful, please cite our paper using this BibTeX:

@inproceedings{shamsoshoara2019solution,
  title={A solution for dynamic spectrum management in mission-critical UAV networks},
  author={Shamsoshoara, Alireza and Khaledi, Mehrdad and Afghah, Fatemeh and Razi, Abolfazl and Ashdown, Jonathan and Turck, Kurt},
  booktitle={2019 16th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON)},
  pages={1--6},
  year={2019},
  organization={IEEE}
}

License

For academtic and non-commercial usage

reinforcement_learning_team_q_learnig_marl_multi_agent_uav_spectrum_task's Issues

i found some question

when i run it ,it have some problems
File "E:/python_code/Reinforcement_Learning_Team_Q_learnig_MARL_Multi_Agent_UAV_Spectrum_task-main/Reinforcement_Learning_Team_Q_learnig_MARL_Multi_Agent_UAV_Spectrum_task-main/main.py", line 174, in
csi_coef = get_csi(num_UAV, loc_dict, X_Mat[Step, Eps, :], Y_Mat[Step, Eps, :])
File "E:\python_code\Reinforcement_Learning_Team_Q_learnig_MARL_Multi_Agent_UAV_Spectrum_task-main\Reinforcement_Learning_Team_Q_learnig_MARL_Multi_Agent_UAV_Spectrum_task-main\csi.py", line 158, in get_csi
csi_h[:, source_uav] = np.squeeze(h_S_uav)
ValueError: could not broadcast input array from shape (2,2) into shape (2)

could you help me?