oleffa / leagueai

LeagueAI software framework for League of Legends that provides information about the state of the game based on Image Recognition using OpenCV and Pytorch.

LeagueAI

Implementation of an A.I. Player for the videogame League of Legends based on Image Recognition using PyTorch

Demo Video: https://youtu.be/iB4PoNJuXzc

Attention: The old version using Tensorflow can be found in the branch "LeagueAI_2017".

Attention: This project is still under development, the datasets I generate and the YOLOv3 object detection weights I trained will be made available soon!

OLD VERSION

This is a newer version of the old implementation and shall provide a framework that lets you build your own applications based on detecting objects in the game. As an example I will implement the old 2017 LeagueAI bot on this framework. Demo video of Tensorflow implementation from 2017: https://www.youtube.com/watch?v=KRWFCaXfOTk

TODO

1. Dataset generation, add these objects to the raw data

• Fix some bugs with the tower cropping of the raw data
• Towers in fog of war
• Turret plating
• Fog of war strucutres, maybe fog of war filter?
• Dead minions
• Add random particles to the screenshots, explosions and so on
• More different cursors
• More cursors
• health bars

2. Other TODOs:

• Extract object positions from the minimap to get an understanding of the global map.
• The mAP calculation needs a rework, sometimes objects are used twice for map computation.

Currently Detectable Objects

• Red Tower
• Red Canon Minion
• Red Caster Minion
• Red Melee Minion
• Vayne

Missing Objects

• More champions
• Add Red inhibitors, nexus, super minions
• Add Blue towers, nexus, inhibitors, minions

Abstract

The task is to create an agent that is able to play 3rd person massive multiplayer online battle arena games (MOBA) like League of Legends, Dota 2 and Heroes of the Storm with the same input as a human player, namely vision. Since League of Legends does not provide an interface to the game, object detection is used. In this project a python implementation of Yolo v3 object detector and a way of randomly generating an infinite amout of training data is introduced.

Object detection

For more information on the object detector refer to my technical report at: https://arxiv.org/abs/1905.13546 or directly to the YOLOv3 website [2]

Installation/Usage

TODO

The LeagueAI Dataset

Creating large datasets from scratch can be very work intensive. For the first implementation of the LeageAI about 700 hand labeled pictures were used. Labeling 700 pictures took about 4 days of work and only included 4 game objects (1 champion model, allied and enemy minions and enemy towers). Therefore, the new dataset was created by automatically generating training data based on 3D models extracted from the game.

1. Obtaining champion and minion models form 3D models To obtain the image data I used the online League of Legends model viewer from https://teemo.gg/model-viewer. For each ingame object and each animation I recorded a short video clip while rotating the 3D model. Next I used the pyFrameExporter.py script to extract individual pictures from the clips. For the minions I used Adobe After Effects to add a green background to the videos of the minions and towers (all objects where I could not find the 3D models). For each of the objects exported frames I used the pyExportTransparentPNG.py script. The script removes the green/purple background from the individual screenshots and leaves you with the masked png of an objects. Furthermore, the scrip crops the images to the content and removes excess seethrough space.

This leaves me with about 1000 masked images of each object that can be later used to generate labeled fake screenshots of the game.

2. Combining the masked and cropped images with game background To generate a large amount of training data that cover all regions of the game map, I generated a series of 200 screenshots from all over the map using the frame exporter script. Then the masked and cropped images from step 1 are randomly combined with the map screenshots using the bootstrap.py script. Since the images are placed using a script it is possible to obtain the objects position in the image and thus automatically generate a label for it.

To generate a large variety of screenshots the script can be adjusted to:

• change the random amount of champions, minions and other objects
• randomly add a number of cursers
• randomly add the game HUD to the screenshot
• randomly scale all the objects
• cluster the minions to create more realistic clumps of fighting minions
• apply gaussian blur and random noise to the image

Using this method a dataset of many thousands of different labeled fake screenshots can be generated in a matter of hours.

A full description of the dataset generation process and evaluation compared to hand labeled data can be found in the following publication: https://arxiv.org/pdf/1905.13546.pdf

To cite please use:

@article{leagueaidatasetgeneration,
  title={LeagueAI: Improving object detector performance and flexibility through automatically generated training data and domain randomization},
  author={Struckmeier, Oliver},
  journal = {arXiv},
  year={2019}
}

Extracting health information

TODO

Sources

[1] Implementing Yolov3 object detection from scratch: https://github.com/ayooshkathuria/YOLO_v3_tutorial_from_scratch

[2] For training: https://pjreddie.com/darknet/yolo/, Yolov3: An Incremental Improvement, J. Redmond and A. Farhadi, 2018

leagueai's Issues

3D model Viewer Recommendation

Training the object detector

Hi,

This project is pretty interesting. I was wondering how you obtained the labeled image data set and perform the training to get the object detector model. As there are multiple objects that could be found on any given screen capture throughout the game play, I don't have much idea on how to accomplish it. The riot developer API also doesn't seem to have the functionality for you to query useful information on the map at any given time. Would you mind sharing some insights?

can you provide your python virtualenv？

• I run the LeagueAI , and then it throw an exception as flow.
• I don't know which version of MSS is correct.

2018-10-14 13:08:48.478820: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
Traceback (most recent call last):
  File "C:\Users\root\AppData\Local\JetBrains\Toolbox\apps\PyCharm-P\ch-0\182.4505.26\helpers\pydev\pydevd.py", line 1664, in <module>
    main()
  File "C:\Users\root\AppData\Local\JetBrains\Toolbox\apps\PyCharm-P\ch-0\182.4505.26\helpers\pydev\pydevd.py", line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Users\root\AppData\Local\JetBrains\Toolbox\apps\PyCharm-P\ch-0\182.4505.26\helpers\pydev\pydevd.py", line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Users\root\AppData\Local\JetBrains\Toolbox\apps\PyCharm-P\ch-0\182.4505.26\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/root/studyProject/LeagueAI/LeagueAI.py", line 806, in <module>
    main_routine()
  File "C:/Users/root/studyProject/LeagueAI/LeagueAI.py", line 485, in main_routine
    sct.get_pixels(mon)
AttributeError: 'MSS' object has no attribute 'get_pixels'
```

Singular matrix error

I don't know if you are still working on it but I keep having this error raised. The only thing I modified was the window and monitor resolution to 1920x1080. I've also been running the py file instead of the ipynb, so I don't know if it's a specific problem with the py file itself. Thanks

---

Traceback (most recent call last):
File "C:/Users/hudso/Desktop/Machine learning/object detection test/models/research/object_detection/LeagueAI-Oleffa/LeagueAI.py", line 655, in
estimate_policy_gradient_FD()
File "C:/Users/hudso/Desktop/Machine learning/object detection test/models/research/object_detection/LeagueAI-Oleffa/LeagueAI.py", line 309, in estimate_policy_gradient_FD
policy_grad = ((inv((delta_theta.reshape((-1,4)).T).dot(delta_theta.reshape((-1,4))))
File "C:\Users\hudso\Anaconda3\lib\site-packages\numpy\linalg\linalg.py", line 551, in inv
ainv = _umath_linalg.inv(a, signature=signature, extobj=extobj)
File "C:\Users\hudso\Anaconda3\lib\site-packages\numpy\linalg\linalg.py", line 97, in _raise_linalgerror_singular
raise LinAlgError("Singular matrix")
numpy.linalg.LinAlgError: Singular matrix

Getting the weights

Really love this project, and fine it really fascinating. I pulled it and wanted to try and get simple examples working. Unfortunately the weights are not on the repo (at least that I can find). Is there a way I can get them from you?

2022 error - unable to draw

This happens when installing a fresh clone of this repo and entering a practice game as vayne
I am using one of the yolo default weights as the LeagueAI weights

Traceback (most recent call last):
  File "C:\Users\Sylar6\PycharmProjects\LeagueAI\LeagueAI_minimal_example.py", line 28, in <module>
    objects = LeagueAI.get_objects(frame)
  File "C:\Users\Sylar6\PycharmProjects\LeagueAI\LeagueAI_helper.py", line 136, in get_objects
    list(map(lambda x: self.draw_results(x, input_frame, colors), output))
  File "C:\Users\Sylar6\PycharmProjects\LeagueAI\LeagueAI_helper.py", line 136, in <lambda>
    list(map(lambda x: self.draw_results(x, input_frame, colors), output))
  File "C:\Users\Sylar6\PycharmProjects\LeagueAI\LeagueAI_helper.py", line 154, in draw_results
    cv2.rectangle(img, c1, c2,color, 2)
cv2.error: OpenCV(4.5.5) :-1: error: (-5:Bad argument) in function 'rectangle'
> Overload resolution failed:
>  - Can't parse 'pt1'. Sequence item with index 0 has a wrong type
>  - Can't parse 'pt1'. Sequence item with index 0 has a wrong type
>  - Can't parse 'rec'. Expected sequence length 4, got 2
>  - Can't parse 'rec'. Expected sequence length 4, got 2


Process finished with exit code 1

Errors

I got these errors when I ran the Module

`Warning (from warnings module):
File "..\object_detection\utils\visualization_utils.py", line 25
import matplotlib; matplotlib.use('Agg') # pylint: disable=multiple-statements
UserWarning:
This call to matplotlib.use() has no effect because the backend has already
been chosen; matplotlib.use() must be called before pylab, matplotlib.pyplot,
or matplotlib.backends is imported for the first time.

The backend was originally set to 'TkAgg' by the following code:
File "", line 1, in
File "C:\Program Files\Python36\lib\idlelib\run.py", line 144, in main
ret = method(*args, **kwargs)
File "C:\Program Files\Python36\lib\idlelib\run.py", line 474, in runcode
exec(code, self.locals)
File "C:\tensorflow1\models\research\LeagueAI\LeagueAI.py", line 20, in
from matplotlib import pyplot as plt
File "C:\Users<name>\AppData\Roaming\Python\Python36\site-packages\matplotlib\pyplot.py", line 71, in
from matplotlib.backends import pylab_setup
File "C:\Users<name>\AppData\Roaming\Python\Python36\site-packages\matplotlib\backends_init_.py", line 16, in
line for line in traceback.format_stack()

Traceback (most recent call last):
File "C:\tensorflow1\models\research\LeagueAI\LeagueAI.py", line 73, in
serialized_graph = fid.read()
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 119, in read
self._preread_check()
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 79, in _preread_check
compat.as_bytes(self.__name), 1024 * 512, status)
File "C:\Program Files\Python36\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.NotFoundError: NewRandomAccessFile failed to Create/Open: LeagueAI_v3/frozen_inference_graph.pb : The system cannot find the path specified.

; No such process`

Can someone help me fix this?

training\LeagueAI_v2.pbtxt : The system cannot find the path specified.

I spent a day getting everything set up. Now I'm blocked with this error message.

It seems like the github repo is missing the training data. Can you please provide it?

(FYI, the updated LeagueAI_v3.zip link expired so I'm not able to access it)

Thanks!

Missing 'utils'

It appears that a library that you wrote called 'utils' is not included in the GitHub repo, and I cannot build the LegueAI.py file

hello

how can I test this? would like to test it and give you some feedback. seen like interesting project.

Regards.-