krumiaa / mindmaker

MindMaker UE4 Machine Learning Toolkit

MindMaker AI Plugin for Unreal Engine 4 & 5

Create Machine Learning AI Agents in Unreal Engine 4 & 5

Videos

Introduction to MindMaker: https://www.youtube.com/watch?v=ERm_pZhAPIA

Blueprints Overview: https://youtu.be/Tuo423NujEk

Discord Group: https://discord.gg/shxFwtmsHa

The MindMaker AI Plugin is an open-source plugin that enables games and simulations within UE4 and UE5 to function as OpenAI Gym environments for training autonomous machine learning agents. The plugin facilitates a network connection between an Unreal Engine Project containing the learning environment, and a python ML library that receives data from Unreal Engine and parses into a custom OpenAI Gym environment for training the agent. The standalone machine learning library can either be a custom python script in the event you are creating your own ML tool using MindMaker’s Remote ML Server, or it could be a precompiled learning engine such as MindMaker’s DRL Engine(Stable-Baselines3 Algorithms). Regardless of which option you choose, using the MindMaker AI Plugin developers and researchers can easily train machine learning agents for 2D, 3D and VR projects.

Possible applications extend beyond game design to a variety of scientific and technical endeavors. These include robotic simulation, autonomous driving, generative architecture, procedural graphics and much more. This API provides a central platform from which advances in machine learning can reach many of these fields. For game developers, the use cases for self-optimizing agents include controlling NPC behavior (in a variety of settings such as multi-agent and adversarial), prototyping game design decisions, and automated testing of game builds.

The DRL Learning Engine is included in the link to the example project as well as the python source code for modifying it, found in the Content\MindMaker\Source directory. Algorithms presently supported by the DRL Learning Engine include : Actor Critic ( A2C ), Sample Efficient Actor-Critic with Experience Replay (ACER), Actor Critic using Kronecker-Factored Trust Region ( ACKTR ), Deep Q Network ( DQN ), Proximal Policy Optimization ( PPO ), Soft Actor Critic ( SAC ), Twin Delayed DDPG ( TD3 ), Trust Region Policy Optimization ( TRPO ), Deep Deterministic Policy Gradient ( DDPG ). The plugin is functionally similar to Unity’s ML agents, with some advantages - rather than needing to create custom OpenAI Gym environment for every application, one uses a single environment and simply choose which of the Agent’s observations and actions to expose to the ML algorithm. Voila, let the learning begin!

Examples & Tutorials

UE4 Releases

• Download Complete Examples Project File and Python Source Code
  • CartPole Task: Creating A Custom Reinforcement Learning Environment [YouTube Demonstration Video]
  • Automated Content Creation & A/B Testing: Automated content creation with a player generated reinforcement learning signal [YouTube Demonstration Video]
  • Automated Stock Trading: Build a Bitcoin Bot In Unreal Engine 4 using Deep Reinforcement Learning
  • Match To Sample: Solving A Memory Puzzle with a Non Player Character
  • Architectural Reinforcement Learning

UE5.1 Stable-Baselines3 Algs

• Plugin Download
• Examples Download (includes python source code for DRL algorithms)

Videos

• Intro. To The MindMaker Plugin
• Automated Content Creation / AB Testing
• Creating a Custom Environment with Cart Pole Example

Documentation

• Overview Of MindMaker Blueprint Functions

Features

• Implement multiple python based ML libraries directly in Unreal Engine with the MindMaker Client Server files – See RemoteML Example and Documentation
• Precompiled Deep Reinforcement Learning Package for production use cases – auto-launches at start of game / simulation
• Ready to use example project demonstrating how to use the API
• Modify the full range of learning parameters including the number of networks layers, batch size, learning rate, gamma, exploration/exploitation trade off, etc
• Self-play mechanism for training agents in adversarial scenarios
• Supported off the shelf Deep RL Algorithms include*:
  • A2C
  • ACER
  • ACKTR
  • DDPG
  • DQN
  • PPO
  • SAC
  • TD3
  • TRPO *UE4 and UE5 versions of the DRL Learning engine contain different subset of above algorithms

Components

There are two primary components you will use while working with the MindMaker Plugin, an Unreal Enginge Project containing the learning environment, and a standalone machine learning library used by the agent to optimize whatever it is attempting to learn. The standalone machine learning library can either be a custom python script in the event you are creating your own ML tool using MindMaker’s Remote ML Server, or it could be a precompiled learning engine such as the MindMaker DRL Engine(Stable Baselines Algorithms).

MindMaker Client Server Remote ML

To experiment using different ML libraries in conjunction with MindMaker, use the MindMaker Remote ML Server. With it you can customize your own python learning engine, rather than use the pre-compiled MindMaker DRL Engine. To use the Remote ML Server, follow these steps:

• Download and install the free MindMaker AI Plugin for UE. This provides the Socket IO connections you will need to connect with the remote python learning engine. With the MindMaker Plugin enabled in your project, you will want to add a SocketIO component to the game object that will function as your AI. You will then need to add the relevant LaunchMinder blueprint functions for sending and receiving data from the Learning Engine and your AI game component. Examples of these can be found in DRL UE project in the next step.
• Download the DRL Example project for UE – this includes the MindMaker Remote Server Application in a Windows .exe format along with the python source code for this remote server located in the Content\MindMaker\Source directory. You can place this remote server executable(or the python script) on any machine which you would like to serve as your remote server where the actual machine learning will take place. This separation of client and server allows you to have a designated cloud based server for housing and training your models. This is convenient when you have large models with potentially many clients connecting, or if the client computer is not as well equipped for machine learning.
• Start the UE example project which contains the MindMaker plugin. Now locate and start the mindmaker executable or python source file on your remote server. Configure the SocketIO component in your UE project to connect with the remote host. The Mindmaker server you have loaded on your remote machine will default to port 3000, you will also need to know that computers remote IP address as well and configure your SocketIO component in UE accordingly. Once you launch the UE project, it will automatically attempt to connect to the remote server and begin training, assuming you have all necessary functions given within the examples files.
• Modify or replace the python client to use machine learning library of your choice. The python source included with the DRL examples uses an OpenAI Gym wrapper for UE that allows any OpenAI compatible machine learning library to interface with Unreal Engine.

MARL (Multi Agent Reinforcement Learning & Working with Multiple Clients)

If you would like to employ multiple ML clients connected to a single learning environment, for instance in a multi agent scenario, this can be done using MindMaker server and plugin.

To create multiple learning agents, first setup your learning agents as shown in one of the example blueprints. For each new learning agent, you will need to increment the socketio port settings in the new AI controller by 1. At time of launching the server, new server port numbers are automatically created for each new instance of mindmaker.exe that you launch, starting with 3000 and going up from there for a total of 100. If you require more than 100 learning agents, request this in the github repo.

For example, If you add a second learning agent to your map, you will need all the same functions that are in the first learning agent, the launch mindmaker blueprint node etc, but instead of assigning this one to port 3000 you will assign it port 3001 in blueprints. Besides changing the socketio port setting in blueprints, you will also need to also change to change the Connect SocketIO blueprint function, modifying the In Address and Port to the new number you have created “http://localhost:3001” for instance.

Once this is done, you will just need to create a second instance of your mindmaker_client.py file that will connect to your second learning agent. Training can be done simultaneously, in parallel. The only modification that you need to make to mindmaker_client.py is changing sio.connect('http://localhost:3000') at the bottom of the file to sio.connect('http://localhost:3001') or whatever is the number of new learning agents you are working with. If you have five learning agents, than you will have five instances of the client running and each will have a new port number all the way up to 3005

Imitation Learning (Experimental)

Imitation Learning with Stablebaselines 3 The Mindmaker python source can be modified to support Imitation Learning using the supported Stablebaselines Imitation Learning Algorithms

Quick Install & Setup Using The MindMaker DRL Engine Starter Content

1. Download Latest Plugin Release from GitHub or UE Marketplace
2. Download a compatible MindMaker Learning Engine or use the one included with the example project.
3. Move the learning engine and its accompanying files into the Content directory of your UE Project. The exact location of the learning engine should be "Content\MindMaker\dist\mindmaker\mindmaker.exe" if the location isnt as specified the plugin will not work to automaticaly launch the learning engine at the start of play and you will have to manually launch mindmaker.exe before begining training.
4. Place the MindMaker AI Plugin in the Plugins directory of your UE Project.
5. If you have downloaded the MindMaker DRL Starter Content & Example Project than simply open MindMakerActorBP blueprint or MindMakerAIControlerBP blueprint from the Content\MindMakerStarterContent\Assets\MindMakerStarterContent\MindMakerActorBP directory and begin creating your custom learning agent using the functions supplied. Be sure that the Socket IO address and port your are using is set to http://localhost:3000 .

Quick Install & Setup For Creating a Custom Learning AI from Scratch

1. Download Latest Plugin Release
2. Download a compatible MindMaker Learning Engine or use the one included with the example project.
3. Move the learning engine and its accompanying files into the Content directory of your UE Project. The exact location of the learning engine should be "Content\MindMaker\dist\mindmaker\mindmaker.exe" if the location isnt as specified the plugin will not work to automaticaly launch the learning engine at the start of play and you will have to manually launch mindmaker.exe before begining training. 4.Place the MindMaker AI Plugin in the Plugins directory of your UE Project. 5.Add a socket IO component to the blueprint you have chosen to work with. A socketIO client component is included with Mindmaker AI plugin. Ensure that the Socket IO address and port your are using is set to http://localhost:3000
4. Connect an Event begin play node to a MindMaker Windows node (One of the Plugins Assets) within your blueprint. The MindMaker Windows node can be found under the MindMaker AI blueprints class once the plugin is installed. Currently only MS windows is supported. Once you have MindMaker Windows node connected to an event begin play node, the MindMaker AI Learning Engine will automatically launch at the beginning of play assuming you have placed it in the correct location of your Projects Content Directory.
5. Create the Reward, Action, Obersvation and Launch MindMaker Functions to use with the learning engine. For examples of how to create these, see the /examples directory which includes two maps CartPole and MatchToSample, which can be downloaded with the starter content.

Saving and Loading Models:

To save a trained model, set the “Save Model after Training” checkbox in the Launch MindMaker Function to True. You will need to ensure your number of training episodes is a non zero number. The model will save after training completes. To load the trained models, uncheck the “Save Model after Training” checkbox and instead set the “Load Pre Trained Model” checkbox in the Launch MindMaker Function to True. You will also need to set the number of training episodes to zero, since no training is to occur. Ensure the number of evaluations episodes is a non-zero integer, since this will be how the pre-trained model demonstrates learning. Models are saved locally in the “Appdata roaming” folder of your computer, for instance c:\Users\LeoN\Appdata\Roaming

Logging with Tensorboard:

By default MindMaker only saves to the AppData/Roaming directory on windows machines. To enable Tensorboard logging, follow these steps.

1. Make sure the Use Custom Parameters Boolean checkbox is set to true within the Launch MindMaker Blueprint function. 2. Open the custom parameters for the algorithm you have selected to work with. Ensure the value for the full_tensorboard_log is set to "True". Next go to the tensorboard_log parameter and you will need to specify the FULL path to the directory within app data roaming where you want to save the Tensorboard log file to, for example “C:/Users/aaron/Appdata/Roaming/MindMaker/a2c_cartpole_tensorboard/” Substitute the name of the user you are working under for aaron within the path. Do not use quotation marks. After training, go to the directory you specified and your log files will be there.

Understand what problem your agent is trying to solve:

This is a three step process, you need to decide what actions the agent can take, what its reward criteria will be, and what observations the agent will need to make about its environment to successfully learn to receive a reward.

Diagram of the Learning Processfor use with MindMaker

Launch MindMaker ---------> Receive Action --------> Make Obs -----> Check Rewards --------> Send Obs and Rwrd to MindMaker ------ Return To Recieve Action

In the learning process, MindMaker Learning Engine must first be configured with the observation space the agent is using and the total number of actions available to the agent. You don’t need to provide any reward information when it is initialized, this will only be encountered during training.

The overall process for learning is that once launched and connected to connected to Unreal Engine, the MindMaker Learning Engine will begin supplying random actions for the Unreal Engine agent to take, and in response, the agent with UE will send back a list of observations it made once the action was taken, in addition to any reward it received in process. See above diagram. Over many episodes, the algorithm being employed by MindMaker will optimize the agents actions in response to the observations and rewards received from UE. This process is the same regardless of what machine learning algorithm one chooses to employ with MindMaker. With this information the learning algorithm being used MindMaker will begin to optimize the Agents action decisions, ideally discovering the sequence necessary to consistently receive rewards. The tradeoff between random actions and intentional ones is controlled in the exploration/exploitation parameters of ML library you have selected for use with MindMaker, for example Stable Baselines. This process repeats for each episode of training. After a fixed number of training episodes you can switch entirely to using the algorithm to predict “best” actions instead of taking random ones.

MindMaker and the Environment Wrapper

MindMaker functions by wrapping an unreal environment in a OpenAI Gym compatible format so that any ML library that has been designed to work with OpenAI Gym can be deployed on your Unreal Engine environment. The purpose of using Open AI Gym is to standardize the relevant factors for learning, namely, the format for receiving the agents observations, rewards and actions, so that any ML algorityhm can have access to the relevant variables for learning without needing to be retrofitted for each specific task. Algorithms that work with OpenAI Gym can than work with any environment and agent which is using the standardized OpenAI protocol.

Configuring MindMaker Learning Engine At the outset you will need to configure the Launch Mindmaker function within Unreal Engine for your learning agent. This is done by setting the action_space variable within MindMaker to equal the total number of actions available to your agent. You will also need to configure the observation_space variable to match the number and type of observations your agent will be using in regard to the reward it is attempting to receive. By default, observations are passed in from Unreal as an array, see the example project. Depending on the number of observations your agent will find necessary to use, the size of observation_space will change.

Key Variables to add in Unreal Engine

Reward – A reward is a variable that is set according to the specific criterion you have chosen for the agent to learn or optimize around. In the UE4 blueprint you will use a branch node to determine what environmental conditions and agent action must be fulfilled for the reward to be activated. This is than passed to MindMaker by the socketIO connection. See Project example. Action – This is a variable that contains an integer value representing whatever action the agent has taken. You will also need to decide the total number of actions available to the agent and set the maxctions in MindMaker to equal this number. Observations – Perhapse the trickiest variables you will be dealing with. The key to setting this correctly is to understand that the agents actions themselves must be included in the observations variable, in addition to any other environmental criterion referenced in the reward function. The agent needs to know what action or actions it took that influenced the reward and any environment variables that changed as well. These are passed to the MindMaker learning engine as an array and updated in the observations variable therein.

MindMaker Blueprint Functions

LaunchMindMaker Blueprint Function

Here we will discuss the individual parameters of the LaunchMindMaker blueprint node, which is the main component of the MindMaker Blueprints Functions.

RL Algorithm — This is where one can select the flavor of RL algorithm one wants to train the agent with. There are ten options in the drop down menu, with each algorithm having its own pros and cons. A detailed discussion of the available of the relevant algorithms and their use cases can be found here. https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html

Num Train EP –this is an integer input representing the number of training episodes one wishes the agent to undertake. The larger the number of training episodes, the more exploration the agent does before transitioning to the strategic behavior it acquires during training. The complexity of the actions the agent is attempting to learn typically determines the number of training episodes required — more complex strategies and behaviors require more training episodes.

Num Eval EP — This is also an integer input and represents the number of evaluation episodes the agent will undergo after training. These are the episodes in which the agent demonstrates its learned behavior.

Continuous Action Space — This is a Boolean input which determines if the agent is using a continuous action space. A continuous action space is one in which there are an infinite number of actions the agent can take, for example if it is learning to steer a car, and range of angles over which the steering column can change is a decimal value between 0 and 180, than there is an infinite number of values within that range such as .12 and 145.774454. You will want to identify at the outset of using if your agent has an infinite number of actions or finite number actions they can take. The action space must either be continuous or discrete, it cannot be both.

Discrete Action Space — This is a Boolean input which determines if the agent is using a discrete action space. A discrete action space is one in which there are a finite number of actions the agent can take, such as if the AI can only move right one space or left one space. In which case it only has two actions available to it and the action space is discrete. The user determines which kind of action space the agent will be using before using MindMaker and set these values accordingly.

Action Space Shape — This defines the lower and upper boundaries of the actions available to the agent. If you are using a discrete action space, than this is simply the total number of actions available to the agent, for instance 2 or 8. If you are using a continuous action space, things are more complicated and you must define the low and high boundaries of the action space seperatly. The format for doing so is as follows: low= lowboundary, high= highboundary,shape=(1,) In this case, lowboundary is an value such as -100.4 and highboundary is a values such as 298.46. All decimal values between these bounds will then represent actions available to the agent. If you had an array of such actions, you could change the shape portion to reflect this.

Observation Space Shape — Properly speaking this input is a python derivative of the OPEN AI custom environment class and defines the lower and upper boundaries of observations available to the agent after it takes an action. The format for doing so is as follows: low=np.array([lowboundary]), high=np.array([highboundary]),dtype=np.float32. Imagine an agent that needed to take three specific action in a row to receive a reward, then its observation space would need to include access to those three actions, which would each be represented by a unique observation. Therefore the array of observations would have to include three different values, each one with own unique boundaries. For example, such an action space might be defined as such: low=np.array([0,0,0]), high=np.array([100,100,100]),dtype=np.float32 if each of its own actions that agent needed to observe was a value between 0 and 100. A rule of thumb is that if a value is part of the reward function for the agent, ie their behavior is only rewarded if some condition being met, than the observation space must include a reference to that value. If five conditions must be met for an agent to rewarded, than each of these five conditions must be part of the agents observation space.

Load Pre Trained Model — This is a Boolean value that determines if you want to the agent to load some pre trained behavior that was previously saved. If you set this to true, you will want to specify the name of the file in the Save /Load Model name input box. All models are saved by default to the app data roaming directory of the computer for instance C:\Users\username\AppData\Roaming

Save Model After Training — This is a Boolean value that determines if you want to the agent to save the behavior it has learned after training. If you set this to true, you will want to specify the name of the file in the Save/Load Model name input box. All models are saved by default to the app data roaming directory of the computer for instance C:\Users\username\AppData\Roaming

Save/Load Model Name — This is a string representing the name of the model you wish to save or load. Files are saved to the app data roaming directory of the computer for instance C:\Users\username\AppData\Roaming

Use Custom Params — This is Boolean value that determines if you want to use the stock version of the algorithm you have selected or wish to modify its parameters. If you wish to use custom parameters these can be accessed via the custom parameters structure variables. If you click on the them, for instance A2Cparams, you will see all the values that can be set within these structures. A detailed breakdown of the parameters for each algorithm can be found here: https://stable-baselines.readthedocs.io/en/master/

Other Blueprint Functions

A sample list of functions from the example project are presented below to understand how information is passed between MindMaker and Unreal Engine All of the UE assets relevant to the toy problem are contained in the Assets/DeeplearningNPC folder. Of particular importance is the blueprint called AI_Character_Controler_BP In the AI_Character_Controler_BP blueprint, all of the environment variables are configured for passing to the MindMaker standalone application. These include the following essential functions

Load Sensory Input function - Imports the objects to which the AI will have access to for sensing or manipulation of its environment Environmental Controls function - This controls the logic for parts of the environment that change such switching lights on and off etc

Define Action Space function - Encode all possible agent actions into a single numeric value that can be passed to the standalone application for evaluation by the RL algorithm

LaunchMindMaker function – this calls the standalone application at the commencing of play so that it can begin evaluation data from the UE environment. After this is initiated, the RL application begins probing the environment with random actions it generates itself, like a blind person searching in the dark for a light. The light is the reward,which is specified in UE function Check Reward function. LaunchLearningEngine also passes in some basic UE environment information to the standalone application, like the number of actions the agent can take, the total number of episodes to train for, and the number of episodes to display the agents acquired strategy after training. Displaying all the agents random training would take far too long.

ReceiveAction function – after the launch learning engine function has begun, the next function to fire is recieveaction. This receives the action that is chosen by the standalone application, and does a number of follow up procedures with it, such as updating the agents location in the environment, checking if the new action satisfies the reward condition, displaying the agents actions if we are through with training, and updated the agents observations about its environment so that they can be passed back to the standalone application in the next episode.

Make Observations function – The purpose of this is to update the agents observations about its environment following the action it has just taken. These will include, for instance, the agents location with the environment and any other environmental data that has changed since it last took an action. These are stored in a custom structure variable.

CheckReward – this specifies the reward condition for the agent in the environment. If this reward condition is met following the agent taking an action, this information is passed to the standalone application in the send observations function that follows. Send Observations Function – takes the new observations made by the agent as well as any reward information and passes them to the standalone application. This is how the RL algorithm will be able to evaluate whether the action it has just taken was a good one, and update its strategy accordingly. After this function fires, the one iteration or episode of the game is complete, and the process repeats ad infinitum.

FAQ

Q: In the exploration stage it seems that the agent does not move, only stands still.

Certain tasks may require extended periods of training where visualizing the agent movements would prove prohibitively time consuming. As such in certain examples visualizing the agent's movements has been disabled, but training is happening in the background once the example is run and upon completion, the agent will demonstrate the acquired strategy.

Q: How is exploration used, what strategy is used to explore? It is not clear to me what this "random" means, can you give an example?

Random in this case means that the agent is using a random number generator to choose between the actions available to it during training. The RL Algorithm then observes the results of these random actions as well as any rewards received and uses this information to choose better actions during the “exploitation” phase. This is how a learned strategy is developed.

Q: What is the information gathered in the learning stage, what does it look like?

The information gathering during learning takes the form of an array of observations that are generated after each of the agent’s random actions. If using the MindMaker plugin, the exact shape of the array is defined in the Observation Size property of the Launch Mindmaker Blueprint function and will depend on what variables are necessary for the agent to observe in that particular game or learning task. It will change depending on the learning task or game.

Q: Can the agent perceive the whole environment, or just a small area around him?

The agent perceives only the part of the environment that is exposed to them by the game designer. When using the Mindmaker plugin, these observations are populated in the Make Observations blueprint function call within Unreal Engine. This will generate an array of numbers in the shape defined by Observation Size property of the Launch Mindmaker Blueprint function. Observations should be selected so that they only comprise the data that is necessary for the agent to learn from, otherwise training could become prohibitively time consuming.

Q: What neural network is used? ANN/CNN/RNN?

In vanilla Q Learning – no neural network is required and learning is stored in a tabular format. When using the MindMaker Deep Reinforcement Learning one can choose between a variety of neural network architectures including RNN, CNN etc. One can set these within each algorithm's custom properties of the Launch Mindmaker Blueprint function call.

Further Resources

Creating a Custom Deep Reinforcement Learning Environment

Intro. to Reinforcement Learning for Video Game AI

Reinforcement Learning – It’s Promise and Peril

Stable Baselines Documentation

mindmaker's Issues

Deep RL Plugin Available for 5.2 but cannot create project and run due to base plugin incompatible

I downloaded both plugins with UE 5.2 on my system however when I tried to create project from the Deep RL the editor could not load the base plugin necessary since it is only compatible with up to 5.1

Can the low and high ends of the observation shape vary?

Hey, it's me again.

I've gotten a whole lot further since the last time I posted :)
This time I wanted to ask if passing in parameters with different ranges is allowed or detrimental to the Learning Engine working?
Apart from all of my unsigned floating point variables I also need to pass in a few unit vectors which I've split into float values.
The issue with this is, that obviously these can contain values down to -1.0

Is this problematic or can I just use them like that as long as I define the right range?

UE5 Plugin does not load learning engine

Out of the box the UE4 example projects all work, however in ue5 the learning engine doesn't seem to launch, and when launched manually doesn't seem to connect. Upgrading the ue4 project to 5 blows up on compile for a number of obvious reasons. Copying the dist folder from a working copy into the ue5 project also doesn't seem to affect this issue.

problem "local variable 'policy_kwargsval' referenced before assignment"

I am creating platformer game with RL agent. It is mario-like game, where AI player gets array of nearby tiles. This is 1D array with 90 elements (in my game i am transforming it into 2D (9,10) Array ).

I started really simple, but when got to moment when i starts MindMaker i got issue:
This is log from mindmaker console

(censored path)...Project\Content\MindMaker\dist\mindmaker\tensorboard\compat\tensorflow_stub\dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
MindMaker running, waiting for Unreal Engine to connect
(14476) wsgi starting up on http://0.0.0.0:3000
(14476) accepted ('127.0.0.1', 2927)
Connected To Unreal Engine
False
6
discrete action space
(censored path)...Project\Content\MindMaker\dist\mindmaker\gym\logger.py:30: UserWarning: ←[33mWARN: Box bound precision lowered by casting to float32←[0m
save model value: false
load model value: false
alg selected: PPO2
use custom: true
Exception in thread Thread-4:
Traceback (most recent call last):
  File "threading.py", line 926, in _bootstrap_inner
  File "threading.py", line 870, in run
  File "site-packages\socketio\server.py", line 679, in _handle_event_internal
  File "site-packages\socketio\server.py", line 708, in _trigger_event
  File "mindmaker.py", line 716, in recieve
UnboundLocalError: local variable 'policy_kwargsval' referenced before assignment

I also catch .json that is being sent to mindmaker server:

{
    "observations": "[0,0]",
    "trainepisodes": 1000,
    "evalepisodes": 100,
    "maxactions": 6,
    "loadmodel": "false",
    "savemodel": "false",
    "algselected": "PPO2",
    "customparams": "true",
    "modelname": "platformer_game_ai_model",
    "conactionspace": false,
    "done": false,
    "actionspace": "6",
    "observationspace": "low=np.zeros(90), high=np.full(90,10), dtype=np.float32",
    "disactionspace": true,
    "a2cparams": {
        "gamma": 0.9900000095367432,
        "n_steps": 5,
        "vf_coef": 0.25,
        "ent_coef": 0.009999999776482582,
        "max_grad_norm": 0.05000000074505806,
        "learning_rate": 0.000699999975040555,
        "alpha": 0.9900000095367432,
        "epsilon": 9.999999747378752e-06,
        "lr_schedule": "constant",
        "verbose": 0,
        "tensorboard_log": "None",
        "_init_setup_model": "True",
        "full_tensorboard_log": "False",
        "seed": "None",
        "n_cpu_tf_sess": "None",
        "network_arch": "[8, 'lstm', 64, 32]",
        "act_func": 0,
        "policy": 0
    },
    "acerparams": {
        "gamma": 0.9900000095367432,
        "n_steps": 5,
        "ent_coef": 0.009999999776482582,
        "max_grad_norm": 10,
        "learning_rate": 0.000699999975040555,
        "alpha": 0.9900000095367432,
        "lr_schedule": "linear",
        "verbose": 1,
        "num_procs": "None",
        "tensorboard_log": "None",
        "_init_setup_model": "True",
        "full_tensorboard_log": "False",
        "seed": "None",
        "n_cpu_tf_sess": "1",
        "network_arch": "[64, 64, 32]",
        "rprop_alpha": 0.9900000095367432,
        "q_coef": 0.5,
        "rprop_epsilon": 4.999999873689376e-05,
        "buffer_size": 5000,
        "replay_ratio": 4,
        "replay_start": 1000,
        "correction_term": 10,
        "trust_region": "True",
        "delta": 1,
        "act_func": 0,
        "policy": 0
    },
    "acktrparams": {
        "gamma": 0.9900000095367432,
        "n_steps": 5,
        "ent_coef": 0.009999999776482582,
        "max_grad_norm": 0.5,
        "learning_rate": 0.25,
        "lr_schedule": "linear",
        "verbose": 1,
        "tensorboard_log": "None",
        "_init_setup_model": "True",
        "full_tensorboard_log": "False",
        "seed": "None",
        "n_cpu_tf_sess": "1",
        "network_arch": "[64, 64, 32]",
        "nprocs": "None",
        "vf_coef": 0.25,
        "vf_fisher_coef": 1,
        "kfac_clip": 0.0010000000474974513,
        "async_eigen_decomp": "False",
        "kfac_update": 0,
        "gae_lambda": "None",
        "act_func": 0,
        "policy": 0
    },
    "dqnparams": {
        "learning_rate": 0.0005000000237487257,
        "verbose": 1,
        "tensorboard_log": "None",
        "_init_setup_model": "True",
        "full_tensorboard_log": "False",
        "seed": "None",
        "n_cpu_tf_sess": "1",
        "layers": "[64, 64]",
        "buffer_size": "50000",
        "exploration_fraction": 0.10000000149011612,
        "exploration_final_eps": 0.019999999552965164,
        "exploration_initial_eps": 1,
        "train_freq": 1,
        "batch_size": 32,
        "double_q": "True",
        "learning_starts": 100,
        "target_network_update_freq": 500,
        "prioritized_replay": "False",
        "prioritized_replay_alpha": 0.6000000238418579,
        "prioritized_replay_beta0": 0.4000000059604645,
        "prioritized_replay_beta_iters": "None",
        "prioritized_replay_eps": 6.000000212225132e-06,
        "param_noise": "False",
        "act_func": 0,
        "policy": 0,
        "gamma": 0.9900000095367432
    },
    "ppo2params": {
        "gamma": 0.9900000095367432,
        "verbose": 1,
        "tensorboard_log": "None",
        "_init_setup_model": "True",
        "full_tensorboard_log": "False",
        "seed": "None",
        "n_cpu_tf_sess": "None",
        "layers": "[64, 64]",
        "lam": 0.949999988079071,
        "n_steps": 128,
        "ent_coef": 0.009999999776482582,
        "learning_rate": 0.0002500000118743628,
        "vf_coef": 0.5,
        "max_grad_norm": 0.5,
        "nminibatches": 4,
        "noptepochs": 4,
        "cliprange": 0.20000000298023224,
        "cliprange_vf": "None",
        "act_func": 0,
        "policy": 0
    },
    "sacparams": {
        "gamma": 0.9900000095367432,
        "learning_rate": 0.0003000000142492354,
        "verbose": 1,
        "tensorboard_log": "None",
        "_init_setup_model": "True",
        "full_tensorboard_log": "False",
        "seed": "None",
        "n_cpu_tf_sess": "None",
        "layers": "[64, 64]",
        "buffer_size": "50000",
        "train_freq": 1,
        "batch_size": 64,
        "learning_starts": 100,
        "tau": 0.004999999888241291,
        "ent_coef": "auto",
        "target_update_interval": 1,
        "gradient_steps": 1,
        "target_entropy": "auto",
        "action_noise": "None",
        "random_exploration": 0,
        "act_func": 0,
        "policy": 0
    },
    "td3params": {
        "gamma": 0.9900000095367432,
        "learning_rate": 0.0003000000142492354,
        "verbose": 1,
        "tensorboard_log": "None",
        "_init_setup_model": "True",
        "full_tensorboard_log": "False",
        "seed": "None",
        "n_cpu_tf_sess": "None",
        "layers": "[64, 64]",
        "buffer_size": 50000,
        "train_freq": 100,
        "batch_size": 128,
        "learning_starts": 100,
        "tau": 0.004999999888241291,
        "gradient_steps": 100,
        "action_noise": "None",
        "random_exploration": 0,
        "policy_delay": 2,
        "target_policy_noise": 0.20000000298023224,
        "target_noise_clip": 0.5,
        "act_func": 0,
        "policy": 0
    },
    "reward": 0
}

tbh i need this to graduate my studies, so help would be welcome

How to include learned .pth files in a build instead of a PIE run

I am currently facing difficulty in using a saved model file (pth file) in the Unreal Engine 5 MindMaker AI framework. Specifically,

I am unable to build a Windows standalone game using the saved model file. Can someone please guide me on how to effectively use a saved model file in the MindMaker framework to build a Windows standalone game?

Any help or pointers would be greatly appreciated. Thank you!

Increasing the length of episodes

Hey @krumiaa ,

I am currently working with MindMaker to build a game AI (using DQN). This AI should search and catch an opponent. To do this it utilizes basic FPS inputs (moving forward, backward, left and right, turning left, right, up and down and so on). The first challenge here is to get the Agent to properly use these controls to navigate the environment. After some trial and error this actually works. The problem now is, that most actions are not actually helpful in achieving the goal of catching the opponent.

Usually in machine learning an episode would be one game session. So for example in a game of Chess it would be one whole match. Now the problem is, that with the current UE4 implementation the agent only takes one action and then immediatly emits a reward. I would like to change that, but don't know how I would go about doing that.
Currently rewards and observations are emitted at the same time, signaling the end of an episode. I could technically only emit these two every 20 actions, but then the network would not reviece the new game state after every action. So what I would like to do is take let's say 20 actions and emit the new observations after each action. And at the end of the 20 actions the cummulative rewards of all 20 actions are emitted.

In theory this should enable the agent to act with intend and planning over multiple actions. I hope this makes sense to you. If you could point me in the right directions this would be great.

Cheers

problem building Socketio for UE 5.1

Abort signal received

ucrtbase
ucrtbase
ucrtbase
VCRUNTIME140_1
VCRUNTIME140_1
VCRUNTIME140_1
ntdll
ntdll
ntdll
KERNELBASE
VCRUNTIME140
UnrealEditor_SocketIOLib!std::invoke<<lambda_ca95016d73eb338498511f9be7c424bb> &,bool,std::shared_ptr<std::basic_string<char,std::char_traits,std::allocator > const > const &>() [d:\build\AutoSDK\Sync\HostWin64\Win64\VS2019\14.29.30147\INCLUDE\type_traits:1534]
UnrealEditor_SocketIOLib!sio::packet_manager::encode() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\SocketIOLib\Private\internal\sio_packet.cpp:485]
UnrealEditor_SocketIOLib!sio::client_impl<websocketpp::clientwebsocketpp::config::asio_client >::on_ping() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\SocketIOLib\Private\internal\sio_client_impl.cpp:607]
UnrealEditor_SocketIOLib!sio::client_impl<websocketpp::clientwebsocketpp::config::asio_client >::on_decode() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\SocketIOLib\Private\internal\sio_client_impl.cpp:639]
UnrealEditor_SocketIOLib!sio::packet_manager::put_payload() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\SocketIOLib\Private\internal\sio_packet.cpp:529]
UnrealEditor_SocketIOLib!sio::client_impl<websocketpp::clientwebsocketpp::config::asio_client >::on_message() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\SocketIOLib\Private\internal\sio_client_impl.cpp:560]
UnrealEditor_SocketIOLib!std::_Func_impl_no_alloc<std::_Binder<std::_Unforced,void (__cdecl sio::client_impl<websocketpp::clientwebsocketpp::config::asio_client >::*)(std::weak_ptr,std::shared_ptr<websocketpp::message_buffer::message<websocketpp::mes() [d:\build\AutoSDK\Sync\HostWin64\Win64\VS2019\14.29.30147\INCLUDE\functional:823]
UnrealEditor_SocketIOLib!std::_Func_class<void,std::weak_ptr,std::shared_ptr<websocketpp::message_buffer::messagewebsocketpp::message_buffer::alloc::con_msg_manager > >::operator()() [d:\build\AutoSDK\Sync\HostWin64\Win64\VS2019\14.29.30147\INCLUDE\functional:869]
UnrealEditor_SocketIOLib!websocketpp::connectionwebsocketpp::config::asio_client::handle_read_frame() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\ThirdParty\websocketpp\websocketpp\impl\connection_impl.hpp:1099]
UnrealEditor_SocketIOLib!websocketpp::transport::asio::connectionwebsocketpp::config::asio_client::transport_config::handle_async_read() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\ThirdParty\websocketpp\websocketpp\transport\asio\connection.hpp:897]
UnrealEditor_SocketIOLib!asio::asio_handler_invoke<asio::detail::binder2<websocketpp::transport::asio::custom_alloc_handler<std::_Binder<std::_Unforced,void (__cdecl websocketpp::transport::asio::connectionwebsocketpp::config::asio_client::transport_config() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\ThirdParty\asio\asio\include\asio\handler_invoke_hook.hpp:69]
UnrealEditor_SocketIOLib!asio::detail::wrapped_handler<asio::io_context::strand,websocketpp::transport::asio::custom_alloc_handler<std::_Binder<std::_Unforced,void (_cdecl websocketpp::transport::asio::connection<websocketpp::config::asio_client::transport() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\ThirdParty\asio\asio\include\asio\detail\wrapped_handler.hpp:98]
UnrealEditor_SocketIOLib!asio::detail::read_op<asio::basic_stream_socketasio::ip::tcp,asio::mutable_buffers_1,asio::mutable_buffer const *,asio::detail::transfer_at_least_t,asio::detail::wrapped_handler<asio::io_context::strand,websocketpp::transport::asio() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\ThirdParty\asio\asio\include\asio\impl\read.hpp:284]
UnrealEditor_SocketIOLib!asio::detail::completion_handler<asio::detail::rewrapped_handler<asio::detail::binder2<asio::detail::read_op<asio::basic_stream_socketasio::ip::tcp,asio::mutable_buffers_1,asio::mutable_buffer const *,asio::detail::transfer_at_leas() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\ThirdParty\asio\asio\include\asio\detail\completion_handler.hpp:69]
UnrealEditor_SocketIOLib!asio::detail::asio_handler_invoke<asio::detail::binder2<asio::detail::read_op<asio::basic_stream_socketasio::ip::tcp,asio::mutable_buffers_1,asio::mutable_buffer const *,asio::detail::transfer_at_least_t,asio::detail::wrapped_handl() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\ThirdParty\asio\asio\include\asio\detail\wrapped_handler.hpp:231]
UnrealEditor_SocketIOLib!asio::detail::win_iocp_socket_recv_op<asio::mutable_buffers_1,asio::detail::read_op<asio::basic_stream_socketasio::ip::tcp,asio::mutable_buffers_1,asio::mutable_buffer const ,asio::detail::transfer_at_least_t,asio::detail::wrapped() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\ThirdParty\asio\asio\include\asio\detail\win_iocp_socket_recv_op.hpp:98]
UnrealEditor_SocketIOLib!asio::detail::win_iocp_io_context::do_one() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\ThirdParty\asio\asio\include\asio\detail\impl\win_iocp_io_context.ipp:420]
UnrealEditor_SocketIOLib!asio::io_context::run() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\ThirdParty\asio\asio\include\asio\impl\io_context.ipp:61]
UnrealEditor_SocketIOLib!sio::client_impl<websocketpp::clientwebsocketpp::config::asio_client >::run_loop() [D:\build\U5M-Marketplace\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\SocketIOLib\Private\internal\sio_client_impl.cpp:261]
UnrealEditor_SocketIOLib!std::thread::_Invoke<std::tuple<std::_Binder<std::_Unforced,void (__cdecl sio::client_impl<websocketpp::clientwebsocketpp::config::asio_client >::)(void),sio::client_impl<websocketpp::clientwebsocketpp::config::asio_client > *> () [d:\build\AutoSDK\Sync\HostWin64\Win64\VS2019\14.29.30147\INCLUDE\thread:56]
ucrtbase
kernel32
ntdll

PermissionError: [Errno 13] Permission denied: 'C:\\Users\\user\\AppData\\Roaming\\PPOCart'

Trying to save and load the training model but getting this error. Where would I change the location it is trying to use? Already tried running MindMaker as an administrator.

How to properly reset my Agent when he does something fatal?

Hello,

in your cartpole example, whenever the Cylinder falls below a certain pitch, the cylinder and cart are reset to their original position and the training commences. How did you implement this, because I couldn't find the code for it in the example.

In my case, if my car drives into a wall, I'd like to give it a negative reward to punish it for that behavior and reset its position back to the start of the race track. When that happens, do I pause sending observations until the car is back at its location and ready to go or do I have to reset the training episode counter in some way?

It would be extremely helpful if you could shed some light on this for me :)

best regards,

FLOROID

TD3 Action noise

I'm having problems with the action noise syntax. I've looked at the documentation for stable baselines, as well as through some of the source python code, but to no avail. What is the allowed syntax that I can use within this parameter? Thank you.

Error when launching the example maps

Hello, I'm trying to test the plugin on the examples map but the training doesn't start. I'm using UE5

when I press the play button nothing happens. This is what I read in the output log:

LogBlueprintUserMessages: [cart_2] MindMaker Launched
LogBlueprintUserMessages: [cart_2] Waiting for Connection from Learning Engine
PIE: Server logged in
PIE: Play in editor total start time 1.347 seconds.
SocketIO: SocketIO Invalid appears to have lost connection, reconnecting attempt 0 with delay 5000
SocketIO: SocketIO Invalid appears to have lost connection, reconnecting attempt 1 with delay 7500
SocketIO: SocketIO Connected with session: 353587e91bee425d8acada58836da487
SocketIO: SocketIO 353587e91bee425d8acada58836da487 connected to namespace: /
SocketIO: Warning: Attempted to connect to http://localhost:3000 when we're already connected. Request ignored.
LogOutputDevice: Warning: Script Stack (3 frames):
cart_BP_C.LaunchMindMaker
cart_BP_C.ExecuteUbergraph_cart_BP
cart_BP_C.BndEvt__SocketIOClient_K2Node_ComponentBoundEvent_0_SIOCOpenEventSignature__DelegateSignature
LogOutputDevice: Error: Ensure condition failed: Lhs.Array.Num() == Lhs.InitialNum [File:C:\Users\epilo\Documents\GitHub\MindMaker_UE5\Engine\Source\Runtime\Core\Public\Containers\SparseArray.h] [Line: 972]
Container has changed during ranged-for iteration!
LogStats: FDebug::EnsureFailed - 0.000 s
LogSlate: Updating window title bar state: overlay mode, drag disabled, window buttons hidden, title bar hidden
LogWorld: BeginTearingDown for /Game/MindMaker/Examples/Assets/Maps/UEDPIE_0_Cartpole
LogWorld: UWorld::CleanupWorld for Cartpole, bSessionEnded=true, bCleanupResources=true
SocketIO: SocketIO Disconnected 353587e91bee425d8acada58836da487 reason: CLOSE_REASON_NORMAL

Debugging the code it seems the function emit gets called but, after that I'm kind of lost.

if I look at netstat in cmd the process seems connected to localhost port 3000 correctly.

Maybe I'm missing something simple. Are the examples supposed to work out of the box or do I have to change any parameter?

Training the agent is not working

Hi,

I am testing MindMaker example project with UE5.
I would like to know that default setting(e.g. PPO2, number of training episode, ..) of this example project is enough to train the cart pole RL agent? I have tried training about over 30 times but never succeeded.

the mindmaker server

I run the code of mindmaker server in the pycharm, but it can't build connection between mindmaker and ue4. UE4 suggests the following error " SocketIO: SocketIO Invalid appears to have lost connection, reconnecting attempt 0 with delay, 5000". I want to know when using mindmaker server, how to set up the blueprint nodes in ue4.

python-socketio version support

python-socketio==5.x
python-engineio==4.x
in this version, ue4 can’t connent to server, can you list your pip list, It will help me a lot !

Mindmaker is returning the same action for every receiveAction call during Evaluation phase.

I've set everything up in my project now and the agent is training without any errors, however when the Evaluation phase starts, he will continue executing the same action he executed in the first evaluation episode for every evaluation episode until the evaluation is done.
What could be causing this and how could I go about fixing this issue? Until I can actually properly evaluate what the agent is learning I can't really make use of the training, so this would be very crucial to fix for me.

MARL

I want to do multi-agent reinforcement learning, But the Mindmaker Remote ML Server can't be downloaded, and there's no repository on GitHub. Please can you re-upload, I need to use this software urgently.

dont understand how the agent visualization works.

i have bought the DRL engine and im messing around with the match to sample example. the only problem is that i dont understand how the agent visualization works, to be more specific, the display agent actions function. ill be completely honest that i know only the basics of neural networks. if anyone can give me even a brief explanation on how the visualization works ill be grateful.

Terminating episode early

I'm setting the done flag when I reach an early termination event with the expectation that the episode will end. I track the number of steps and episodes within UE but it seems even after setting the done flag to true and passing it back through mindmaker, the episode continues until the defined number of timesteps is complete - how can I terminate early? I'm using PPO2.

Multi-Agent Setup for one network

Hello,

I've been enjoying the plugin a lot, however I was trying to figure out if it's possible to have a non-MARL multi-agent setup, specifically with PPO.

Buy this I mean having multiple agents collect experience for one neural network, which is a very common technique for networks like PPO. I found a section in the wiki talking about setting up multiple clients in order to train multiple networks, but I am trying to use multiple agents to train one network.

Is this possible with the plugin, and how if so?

Missing connector in Cartpole example?

Hi @krumiaa

2 questions:

Just recently purchased MindMaker: Deep Reinforcement Learning (SingleNode) from UE marketplace and had a look at your Cartpole example, it seems a connector is missing in cart_BP?

Also, from my understanding, the "Done" boolean is used to notify the RL engine that the current episode has ended, is that correct?

Thanks.

How to integrate DRL into an already existing unreal 4.27 project

Hi, im trying to add DRL to a game i made, however i have found no resourcces anywhere that outline how DRL can be used on a custom game, any help will be greatly appreciated

autonomousduck.com is inaccessible

Do Mindmakers RL Algorithms support float Inputs?

Hello there,

I'm currently working on a project for my bachelors paper and
want to train an AI agent using MindMaker.
When looking at the example project I noticed that in the "Make Observations" function
two Integer values are turned into strings appended in a [Integer, Integer] structure and passed into MindMaker.
Could I also pass float values to MindMaker here or are these values supposed to be either 0 or 1?

Thanks in advance.

Support for Ports other than 3000

I need to be able to run 2+ separate instances of MindMaker on a single project at the same time. How do I accomplish this?

Unreal Engine crashes upon compiling or running anything involving the SocketIO client

My program ran perfectly fine and without editing any blueprint that was using the SocketIO client suddenly my game started crashing within a few seconds of launching it. The MindMaker executable does open and prints out until

but even before those two lines are printed out the engine already crashes.

If I don't place the blueprint using the SocketIO client in the scene, everything runs fine, but as soon as code from that blueprint is either compiled OR executed the engine crashes. I can't even remove the SocketIO client from the blueprint without the engine crashing immediately.

Any idea what might be causing this and how I could fix it?

This is what Unreal Engine shows me as a crash report.:

Fatal error!

Unhandled Exception: EXCEPTION_ACCESS_VIOLATION writing address 0x00007ffba15cea48

0x00007ffba6e076e0 UE4Editor-SocketIOLib.dll!sio::client_impl::close() [D:\build++Portal\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\SocketIOLib\Private\internal\sio_client_impl.cpp:169]
0x00007ffba6df4617 UE4Editor-SocketIOLib.dll!sio::client_impl::~client_impl() [D:\build++Portal\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\SocketIOLib\Private\internal\sio_client_impl.cpp:87]
0x00007ffba6df458d UE4Editor-SocketIOLib.dll!sio::client::~client() [D:\build++Portal\Sync\LocalBuilds\PluginTemp\HostProject\Plugins\MindMaker\Source\SocketIOLib\Private\sio_client.cpp:42]
0x00007ffba15b1a98 UE4Editor-SocketIOClient.dll!SharedPointerInternals::TReferenceControllerWithDeleter<sio::client,SharedPointerInternals::DefaultDeletersio::client >::DestroyObject() [D:\RocketSync\4.27.0-17155196+++UE4+Release-4.27\Working\Engine\Source\Runtime\Core\Public\Templates\SharedPointerInternals.h:116]
0x00007ffba15a8177 UE4Editor-SocketIOClient.dll!FSocketIONative::~FSocketIONative() []
0x00007ffba15b1a67 UE4Editor-SocketIOClient.dll!SharedPointerInternals::TReferenceControllerWithDeleter<FSocketIONative,SharedPointerInternals::DefaultDeleter >::DestroyObject() [D:\RocketSync\4.27.0-17155196+++UE4+Release-4.27\Working\Engine\Source\Runtime\Core\Public\Templates\SharedPointerInternals.h:116]
0x00007ffba15acb0b UE4Editor-SocketIOClient.dll!UE4Function_Private::TFunction_CopyableOwnedObject<<lambda_97e8ba7979b1b36926b54dc4cade2d91>,0>::scalar deleting destructor'() [] 0x00007ffba6f19875 UE4Editor-CoreUtility.dll!TAsyncQueuedWork<void>::scalar deleting destructor'() []
0x00007ffbe9de5298 UE4Editor-Core.dll!UnknownFunction []
0x00007ffbea26ff0b UE4Editor-Core.dll!UnknownFunction []
0x00007ffbea267050 UE4Editor-Core.dll!UnknownFunction []
0x00007ffca5fb7034 KERNEL32.DLL!UnknownFunction []
0x00007ffca6dc2651 ntdll.dll!UnknownFunction []

Crash in runnable thread PoolThread 9
Crash in runnable thread PoolThread 4
Crash in runnable thread PoolThread 0

ImportError: DLL load failed

Hiya,

I'm using UE version 5.0.3 and when I load up the cart example (with no changes, created from the UE marketplace), MindMaker.exe starts but has this ImportError straight away and closes.

Any ideas or suggestions?

plan to update to ue5 preview?

Now that ue 5 preview is released, its api should be fixed by now. Can you please update this plugin to ue5?

I tried using ue5 plugin you provided, but it can't find an entry called IsSelectedInEditor.

I also tried rebuilding the plugin using UnrealBuildTool.exe, but it failed on SocketIOLib. I think it is caused by ue5 preview has different impl from ue5 ea.

Error training with PPO2 Full tensorbord logs = True

Hi,

I got an error exclusively when using PPO2 with full tensorboard logs = true. It happens after around 300 steps:

Here are my settings:

It trains successfully while full tensorboard logs = false.
The agent also trains successfully using other algorithms with full tensorboard logs = true. Not sure why PPO2 only have this problem.

Note that I still get tensorboard logs just not the full data, so it's not a major issue.

Tensorboard logs do not generate

Hi, I bought and modified the MindMaker Single Node Project. I am trying to output the training statistics to tensorboard. However, regardless of my input to the tensorboard_log parameter, no files are generated. Would you have any ideas as to why?

Thank you.

Training stopping short

I'm trying to run my training for 10k iterations (and that is set in "Trainepisodes" in the structure) but the training seems to stop short at ~391. Is there any reason it might quit early?

would like to know about plan for supporting unreal engine 5

please could you share your plans for supporting unreal engine 5 ?

What does mindmaker showing the error while launching it?

I am trying to build a basic RL project with mindmaker plugin. The project contains an agent and a target, agent goal is to reach the target by taking (up, down, left, right) action and the agent will get a reward (+10) when it reaches the target otherwise -1 reward on each action. Ball is an agent and cube is the target in the project.

I am sharing the same project, if someone wants to download it to check the error. The structure of blueprint is very much similar to the inbuilt cartpole project. Kindly have a look.

I am using UE5.1.

https://drive.google.com/file/d/1SSVsYukVMrxQ076r3BdY0aI6KCM92pfI/view?usp=share_link

While launching the mindmaker this error is coming.

MindMaker running, waiting for Unreal Engine to connect
(6564) wsgi starting up on http://0.0.0.0:3000
(6564) accepted ('127.0.0.1', 51323)
Connected To Unreal Engine
False
4
discrete action space
C:\Users\aiina\Documents\Unreal Projects\Rl_test\Content\MindMaker\dist\mindmaker\gym\spaces\box.py:74: UserWarning: WARN: Box bound precision lowered by casting to float32
save model value: true
load model value: false
Total Training Episodes: 1000
alg selected: PPO
use custom: true
Exception in thread Thread-5:
Traceback (most recent call last):
  File "threading.py", line 917, in _bootstrap_inner
  File "threading.py", line 865, in run
  File "site-packages\socketio\server.py", line 720, in _handle_event_internal
  File "site-packages\socketio\server.py", line 745, in _trigger_event
  File "MindMakerDRLStableBv3.py", line 606, in recieve
  File "ast.py", line 46, in literal_eval
  File "ast.py", line 35, in parse
  File "<unknown>", line 0

    ^
SyntaxError: unexpected EOF while parsing

What is going wrong in the project?
How to solve it?

MARL disconnection from Unreal Engine after a while

Hi!

I setup the MARL as described in the github. However, after a while (10 seconds) the second agent training stops working with the print "Disconnected from Unreal Engine. Exiting Mindmaker". The first agent trains fine.

Do you have any idea why this might happen?

I did the following steps:

• Second agent mindmaker blueprint with the new address & port "http://localhost:3001"
  
  
• Second agent SocketIO address & port changed to "http://localhost:3001"
  
• In the python source code, changed the "3000" to "3001".
  
• Launched the second duplicate python source code (with 3001), then Unreal. The mindmaker.exe window also appears. Both agents start to train.
• After around 10-15 seconds the second agent python training stops due to a disconnection from Unreal Engine.

UE5: MindMaker (DL single node) gives (KeyError: '_init_setup_model') on play

After clicking create project for UE5 (previously it would allow project creation for other versions, and the free plugin was also available for other versions, but now it shows support for only UE5 on the marketplace), it shows the following attached error.

However, when I download and run the example project (cartpole), it runs as it should but on UE4.25, as Ai plugin is manually present in the plugins folder.
I have tried one too many things on different systems, but now I am more confused than ever if the error is environment on my system related, related to the changes in the 'supported versions' of the Mindmaker, or something else.

When are the chosen actions supposed to be executed?

Hello there ^^

I am currently trying to set up my AI Agent for Autonomous Driving.
I've set up my Make Observations Functio and the Check Reward function and I am now wanting to bind the actions chosen by the Algorithm to their respective actions in unreal engine. For example having the action with the iterator 1 increase the throttle of the car.
Now I'm trying to figure out when to do that. Do I execute those actions inside the Receive Action function before Observations are made using the "Action Selected" Integer Value? Also, does the range of values for Action Selected start at 0?

And as a final question - does the Mindmaker executable automatically call the Receive Action function or do I have to call it anywhere because I couldn't find any place in the example project where the function was being called?

Best regards,

Flo

Receiving "KeyError: 'done'"

I'm trying to integrate this project into UE5, Preview 1. It's all going well so far and thank you for releasing this! I am running into an issue, though, after calling the Launch Mind Maker node but before the receive actions or receive observations are called:

Traceback (most recent call last):
  File "threading.py", line 926, in _bootstrap_inner
  File "threading.py", line 870, in run
  File "site-packages\socketio\server.py", line 679, in _handle_event_internal
  File "site-packages\socketio\server.py", line 708, in _trigger_event
  File "mindmaker.py", line 271, in recieve
KeyError: 'done'

It seems to have something to do with how I package my parameter structure. I've tried to keep this as close to the cartpole example as possible but I may have missed something vital. I tried to look for the mindmaker.py source code but as it seems to be unavailable I'm posting here.

Any help is much appreciated!