unrealcv / unrealcv

UnrealCV: Connecting Computer Vision to Unreal Engine

UnrealCV

UnrealCV is a project to help computer vision researchers build virtual worlds using Unreal Engine 4 (UE4). It extends UE4 with a plugin by providing:

1. A set of UnrealCV commands to interact with the virtual world.
2. Communication between UE4 and an external program, such as Caffe.

UnrealCV can be used in two ways. The first one is using a compiled game binary with UnrealCV embedded. This is as simple as running a game, no knowledge of Unreal Engine is required. The second is installing UnrealCV plugin to Unreal Engine 4 (UE4) and use the editor of UE4 to build a new virtual world.

Please read Tutorial: Getting Started to learn using UnrealCV.

Images generated from the technical demo RealisticRendering

New Features

• Call any Blueprint function from Python by vbp [obj_name] [func_name] [arg1] [arg2] ... command.
• Support RPC communication between Server and Client in Linux, higher FPS and more reliable.
• A set of new commands for camera control and object manipulation, please refer to command system for more details.

How to install UnrealCV

To install the UnrealCV Server, you need:

1. Download the source code and place it on the Plugin folder of a C++ UE4 project.
2. launch the C++ project with Visual Studio 2019, UnrealCV will be compiled at the same time.
3. To check the success installation of UnrealCV, you can run vget /unrealcv/status in the console (Press ` to display the console).

To install the UnrealCV Client, just run: pip install unrealcv

Citation

If you found this project useful, please consider citing our paper

@article{qiu2017unrealcv,
  Author = {Weichao Qiu, Fangwei Zhong, Yi Zhang, Siyuan Qiao,Zihao Xiao, Tae Soo Kim, Yizhou Wang, Alan Yuille},
  Journal = {ACM Multimedia Open Source Software Competition},
  Title = {UnrealCV: Virtual Worlds for Computer Vision},
  Year = {2017}
}

Contact

If you have any suggestion or interested in using UnrealCV, please contact us.

unrealcv's Issues

UnrealCV crashed when the number of actors was greater than about 1000.

Returned depth images for certain scenes contain no data

Following the unrealcv generate-images tutorial, I tried to get the depth images for the Infinity Blades: Grass Lands scene. However, the image I got back contained no data.

The depth image is supposed to be in the top righthand corner.

However, when I inspect the depth image inside Unreal, it is clear that there should be data here.

The depth image is in the bottom left corner.

The tutorial example worked as expected for me, as did a few other scenes. I'm not sure what's different about this one.

object_mask in standalone mode

Hi,

First of all, thank you very much for the plugin. I'm new to unreal and I'm learning a lot going through the code.

I have a question about object_mask in standalone mode. I downloaded the RealisticRendering example and tried the plugin and it works well. I'm attaching the screen shot of running it against the unreal starter content.

However, when I run the vset /viewmode object_mask mode in standalone mode, it changes the lighting but doesn't change colors. It kind of looks like it's running the FViewMode::VertexColor, but not PaintObject routine in FPlayerViewMode::Object

I'm attaching the lit mode for comparison

Is that anything I need to do during packaging process to make this work? or do you have any suggestion on what might be the problem please?

I would appreciate any insight you might have on this.

thanks, :-)
alex

LOD support for maps needs to be improved

Current implementation here assumes the map is only with one LOD (Level of detail), which is not true for many complex maps.

This needs to be solved.

Object pose command

First off let me say that this is a very interesting plugin and I will definitely see if I can try to use it at some point to create a synthetic dataset.

Something I've been wanting to try in the near future was to directly learn object poses (w.r.t. the camera) from 2D (possibly combined with 3D) data. Judging by the list of commands I cannot get a pose of an object w.r.t. the camera? What about a 3D pointcloud? Can this data be made available?

UnrealCV module not found (with UE 4.15.2)

When unzipped and copied under Engine/Plugins/, with UE restarted, it pops out a messages says the UnrealCV module not found and there is a loading error. P.s. I am on macOS sierra 10.12.4

Build error and crash

I'm using the current version from master. First of all it doesn't build on Linux (Missing #include <string> in Source/UnrealCV/Private/libs/cnpy.h). With that fixed, unrealcv works fine within the unreal editor console, but if I try to connect from the python client the system just crashes. I'm not really familiar with unreal, do you have any idea what is causing this?

unreal console output

it would be better to use hdf5 files to export depth images, rather than exr

Hello,

In my opinion, it would be better to use hdf5 (http://www.h5py.org/) to export depth images, rather than exr. My rationale here is as follows:

1. As a data exchange format, hdf5 is more widely supported than exr.
2. It is easier and requires fewer external dependencies to read hdf5 files in Python and Matlab.
   • Reading exr files requires the OpenCV Python module, which is a heavyweight dependency, and attempting to read exr actually crashes using the OpenCV Python module obtained using mainstream package managers on OSX. Indeed, this failure mode means that the unrealcv/client/examples/generate-images.ipynb example notebook distributed with UnrealCV crashes when calling cv2.imread(frame['depth'], cv2.IMREAD_ANYDEPTH) by default on OSX.
   • Alternatively, one could use the OpenEXR Python module directly, but this is also a heavyweight dependency, and doesn't work out of the box using mainstream package managers on OSX (see AcademySoftwareFoundation/openexr#207).
   • As a last resort, one could also build the OpenEXR C++ library and corresponding Python bindings manually, but OpenEXR has build issues on OSX that have been acknowledged by OpenEXR's own authors (see https://github.com/openexr/openexr/blob/develop/IlmBase/README.OSX and AcademySoftwareFoundation/openexr#208).
   • In contrast, the h5py library comes installed by default on most Python distributions.
3. The HDF5 C++ library for writing hdf5 files is widely used, easy to build, and cross-platform.

Change TCP port at game start

I would like to launch multi thread UE4 for simulation and
change tcp port(default 9000) at game start.
I know that I should manually type

vset /unrealcv/port PORT_NUMBER

in console comand line.
But I would like to launch UE4 simulation automatically.
Please Help me.

Cannot open console using Brazilian keyboard

Cannot open the console on the RealisticRendering example using Brazilian keyboard:

https://en.wikipedia.org/wiki/Portuguese_keyboard_layout

vget /camera/[id]/object_mask resets the object mask colors

Hey,

I'm using the plugin v0.3 with UE 4.13. I'm trying to define my own colors for the object mask of different objects in the scene using 'vset /object/[object_name]/color [r] [g] [b]'. However, when I use 'vget /camera/[id]/object_mask', to capture the image, the object mask colors are reset. Is there something wrong I'm doing here?

Thanks,
Ayush

Error when packaging a UE4 project with EXR lib in Linux

Hi,
I've been following the instructions in https://docs.unrealcv.org/en/latest/plugin/install/, i.e. applied the provided "OpenEXR patch for Linux" during UE4 installation.
The UnrealCV plugin loads perfectly well in the editor, but when I'm packaging my project (using File->Package Project->Linux) I get numerous link error, all related to the OpenEXR module.
For example (the full list is in error log.txt) :

• [2017.01.26-07.22.12:611][ 1]UATHelper: Packaging (Linux): UnrealBuildTool: [1/1] Link MyProject
• /ssd/dannyr/UnrealEngine-4.14.3-release/Engine/Source/ThirdParty/openexr/Deploy/lib//Linux/StaticRelease/libIlmImf.a(ImfHeader.o): In function `basic_ios':
• UnrealBuildTool: /usr/include/c++/4.8/bits/basic_ios.h:456: undefined reference to `std::ios_base::ios_base()'
• UnrealBuildTool: /ssd/dannyr/UnrealEngine-4.14.3-release/Engine/Source/ThirdParty/openexr/Deploy/lib//Linux/StaticRelease/libIlmImf.a(ImfHeader.o): In function `basic_istream':
• UnrealBuildTool: /usr/include/c++/4.8/istream:607: undefined reference to `VTT for std::basic_stringstream<char, std::char_traits, std::allocator >'
• UnrealBuildTool: /ssd/dannyr/UnrealEngine-4.14.3-release/Engine/Source/ThirdParty/openexr/Deploy/lib//Linux/StaticRelease/libIlmImf.a(ImfHeader.o): In function `basic_ios':
• UnrealBuildTool: /usr/include/c++/4.8/bits/basic_ios.h:456: undefined reference to `vtable for std::basic_ios<char, std::char_traits >'
• UnrealBuildTool: /ssd/dannyr/UnrealEngine-4.14.3-release/Engine/Source/ThirdParty/openexr/Deploy/lib//Linux/StaticRelease/libIlmImf.a(ImfHeader.o): In function `basic_istream':
• UnrealBuildTool: /usr/include/c++/4.8/istream:608: undefined reference to `std::basic_ios<char, std::char_traits >::init(std::basic_streambuf<char, std::char_traits >*)'
• UnrealBuildTool: /ssd/dannyr/UnrealEngine-4.14.3-release/Engine/Source/ThirdParty/openexr/Deploy/lib//Linux/StaticRelease/libIlmImf.a(ImfHeader.o): In function `Imf::Header::operator[](char const*)':
• UnrealBuildTool: /home/qiuwch/workspace/openexr/openexr-1.7.1/IlmImf/ImfHeader.cpp:337: undefined reference to `std::basic_stringstream<char, std::char_traits, std::allocator >::basic_stringstream(std::_Ios_Openmode)'
• Packaging (Linux): UnrealBuildTool: /home/qiuwch/workspace/openexr/openexr-1.7.1/IlmImf/ImfHeader.cpp:337: undefined reference to `std::basic_ostream<char, std::char_traits >& std::operator<< <std::char_traits >(std::basic_ostream<char, std::char_traits >&, char const*)'
• Packaging (Linux): UnrealBuildTool: /home/qiuwch/workspace/openexr/openexr-1.7.1/IlmImf/ImfHeader.cpp:337: undefined reference to `std::basic_ostream<char, std::char_traits >& std::operator<< <std::char_traits >(std::basic_ostream<char, std::char_traits >&, char const*)'

I also tried to disable the EXR build target and then the packaging (with UnrealCV) succeeds, but since I'm using UnrealCV to extract accurate depth information the EXR feature is a must.

I'm using Unreal Engine 4.14 with Ubuntu 16.04.

Any help would be much appreciated!

Under depth mode, other characters cannot be seen?

Is it enabled? I believe this is needed for many purposes. Thanks.

4.13 support

UnrealCV is developed and tested in Unreal Engine 4.12. A few users experienced compatible issues with 4.13. #29 #20. We are working on fixing these issues and new updates will be posted here.

Currently, it is recommended to use 4.12.

The plugin is not compilable

I've attempted compiling the plugin for both Linux and Windows, and it doesn't seem able to do that. Unfortunately, the errors are opaque, in that I don't know the next one until I resolve the one I'm looking at, but the first one is:
LogCompile: /where/I/have/UE4/Engine/Plugins/unrealcv/Source/UnrealCV/Public/TcpServer.h(50): Error: Property is exposed to the editor or blueprints but has no Category specified.

Has anyone run into these issues?

Mac build issue, UE 4.16, Xcode: Expected *.h to be first header included

I'm building the UnrealEngine 4.16.1 itself and the UnrealCV master source code together from XCode on Mac OS 10.12.2. It seems that all of the UnrealCV headers get a compile error in this environment:

Parsing headers for UE4Editor
  Running UnrealHeaderTool UE4Editor "/Users/bcbrock/UnrealEngine/Engine/Intermediate/Build/Mac/UE4Editor/Development/UE4Editor.uhtmanifest" -LogCmds="loginit warning, logexit warning, logdatabase error" -Unattended -WarningsAsErrors
Reflection code generated for UE4Editor in 10.8587898 seconds
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/CaptureManager.cpp(1): error: Expected CaptureManager.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/CommandDispatcher.cpp(1): error: Expected CommandDispatcher.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/ConsoleHelper.cpp(1): error: Expected ConsoleHelper.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/ExecStatus.cpp(1): error: Expected ExecStatus.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/GTCaptureComponent.cpp(1): error: Expected GTCaptureComponent.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/ObjectPainter.cpp(1): error: Expected ObjectPainter.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/PlayerViewMode.cpp(1): error: Expected PlayerViewMode.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/ScreenCapture.cpp(1): error: Expected ScreenCapture.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/ServerConfig.cpp(1): error: Expected ServerConfig.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/TcpServer.cpp(1): error: Expected TcpServer.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/UE4CVGameMode.cpp(1): error: Expected UE4CVGameMode.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/UE4CVServer.cpp(1): error: Expected UE4CVServer.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/ViewMode.cpp(1): error: Expected ViewMode.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/Commands/ActionHandler.cpp(1): error: Expected ActionHandler.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/Commands/AliasHandler.cpp(1): error: Expected AliasHandler.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/Commands/CameraHandler.cpp(1): error: Expected CameraHandler.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/Commands/ObjectHandler.cpp(1): error: Expected ObjectHandler.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/Commands/PluginHandler.cpp(1): error: Expected PluginHandler.h to be first header included.
/Users/bcbrock/UnrealEngine/Engine/Plugins/UnrealCV/Source/UnrealCV/Private/libs/cnpy.cpp(1): error: Expected cnpy.h to be first header included.
Build canceled.
Command /Users/bcbrock/UnrealEngine/Engine/Build/BatchFiles/Mac/Build.sh failed with exit code 1

This error is apparently due to a recent UnrealEngine feature called "Insert What You Use" (IWYU). A quick fix is to modify UnrealCV.Build.cs:

diff --git a/Source/UnrealCV/UnrealCV.Build.cs b/Source/UnrealCV/UnrealCV.Build.cs
index 1a395ee..ebf01f2 100644
--- a/Source/UnrealCV/UnrealCV.Build.cs
+++ b/Source/UnrealCV/UnrealCV.Build.cs
@@ -7,6 +7,8 @@ namespace UnrealBuildTool.Rules
 	{
 		public UnrealCV(TargetInfo Target)
 		{
+			bEnforceIWYU = false;
+
 			// This trick is from https://answers.unrealengine.com/questions/258689/how-to-include-private-header-files-of-other-modul.html
 			string EnginePath = Path.GetFullPath(BuildConfiguration.RelativeEnginePath);

Interestingly, this error does not occur when I build the UnrealCV plugin with build.sh, only when built under XCode.

How to embed a new game?

Hi Weichao,

How to embed my own game with this beautiful unrealcv plugin? I built some maps and want to use this program to get some screenshots for research. How can I do this?

Thanks!

Optimize unrealcv performance

Currently, the speed of unrealcv is roughly 10 - 20fps on my workstation, it is enough for generating image and video dataset, but not enough for interesting tasks such as reinforcement learning. Optimizing its speed will be the next focus.

Errors with bytes as strings in python3

The current client version installed with pip does not work with python 3.
It's close, the only issue is the use of strings as byte arrays.

This can be solved by changing __init__.py as follows, which work for both python3.4 and python 2.7 (I tested on 3.4.3 and 2.7.6):
line 73:
payload = b""
line 87:
return payload.decode('UTF-8')
line 109:
wfile.write(payload.encode('UTF-8'))

This has the added benefit of ensuring the character encoding is the same in both client and server code.

Thanks.

errors when trying to work with UnrealCV plugin: "Plugin missing or incompatible"

Hello there,

I'm attempting to install the UnrealCV plugin into the Unreal Engine. I'm using Unreal Engine 4.12.5 on OSX. I've followed the instructions on this page:

https://unrealcv.github.io/tutorial/plugin.html

But when I try to do the "Open Menu -> Edit -> Plugins, make sure UnrealCV is installed and enabled" step, I get the following error when I try to load an Unreal project. For what it's worth, I'm attempting to load the EpicZenGarden uproject from the Epic Games Launcher app, after I've installed the UnrealCV plugin, both in the project Plugins folder and in the Engine/Plugins folder, following the UnrealCV tutorial instructions.

If I click "No", then I get more errors:

When I click "OK", I get another error:

I realize I'm on charting new ground here, trying to get this running on OSX. Is there an easy hack I can do to work around these errors? :)

Object masks show up in UE4Editor but not in saved png file

Hello,

I am using the UnrealCV plugin with Unreal Engine 4.12. I've imported several bicycle .fbx files as type "StaticMeshActor" for a machine learning application and am having issues with the object_mask viewmode. Changing the viewmode to object_mask (vset /viewmode object_mask) in the UE4Editor results in what I am looking for - each bicycle instance is a different color (see screenshot of UE4Editor below)

however when I save this image using the Python API (client.request('vget /camera/0/object_mask')) or directly from the UE4 command line (vget /camera/0/object_mask) the returned png file is blank and doesn't show the bicycles.

Any thoughts about what could be the issue?

Any help would be greatly appreciated.

UnrealCV matlab client 'error: Malformat raw message 'vset''

I'm trying to use matlab to execute commands in UE4. When I hit 'play' in the editor, followed by running demo.m in matlab, I get the following output in matlab's command window:

Try connecting to localhost:9000
Connected to server
connected to RealisticRendering

However, the response variable reads: 'error: Malformat raw message 'vset''.

Any suggestions are appreciated!

changing resolution

I found after searching online that it's possible to change the resolution of the environment by using the 'setres WxH' command in the game console.

This works for display on screen, but when using the generate-images code, the results are still at 640x480, apart from the object mask, which is at the desired resolution. How would the render size (and perhaps other render settings) be passed on correctly for export to disk?

vget /objects command returns 'ok' instead of a list of object names.

Instaling Plugin

vget /objects return empty result

vset /viewmode/object_mask needs to be called before using vget /objects, otherwise it will return an empty string instead of returning object name in the scene.

Load unrealcv plugin failed

I download the zip on your tutorial page https://unrealcv.github.io/tutorial/plugin.html, and copied to the Plugins folder of Unreal Engine (Engine/Plugins).
And the error is I can't open the unreal because failed to load the UnrealCV module.
I try to put it in my project and failed, either. tip is missing or incompatible modules in UnrealCV plugin
Thanks for your help～

How to use UnrealCV with UE4 games?

Can you plug UnrealCV into existing UE4 games, or does it only work with games that were packaged with it embedded inside, i.e. the games currently available on Model Zoo?

Thanks! Sorry if this has been answered elsewhere, I did try and look for the answer but wasn't too sure. I've tried it successfully with Realistic Rendering.

Which version of UE4 should be supported in compiled binary

UnrealCV is originally developed for 4.12 and has been updated to support up to 4.14 while also maintaining the backward compatibility. So compiling the source code can support version from 4.12 - 4.14.

I will provide compiled unrealcv binaries in release page, but I only have the resource to provide compiled binary for only one UE4 version, which version to support is a problem. My thought is I will provide the version documented in this page https://wiki.unrealengine.com/Building_On_Linux, making sure it can support Linux.

If you are using a different version of UE4, compiling from source code is always an option.

Parsing object instance mask

So, I was the section about the object mask at reading generate-images.ipynb, and I was wondering which is the inefficient part, is it getting the mask or getting the colors?

Also, are the colors going to be constant through time for each object?

Is it possible to send action command from the client to UE?

It would be useful for reinforcement training in UE. Thanks.

Getting the projection matrix of camera

I would like to get the projective camera matrix of the rendering engine. But the code is commented out in the plugin.

FExecStatus FCameraCommandHandler::GetCameraProjMatrix(const TArray& Args)
{
// FMatrix& ProjMatrix = FSceneView::ViewProjectionMatrix;
// this->Character->GetWorld()->GetGameViewport()->Viewport->
// this->Character
return FExecStatus::InvalidArgument;
}

Is it possible to add this feature??

If the camera matrix is fixed can you send me the matrix information for FOV 90.

The RealisticRendering app for Mac is unavailable

I get a message

Not Found
The requested URL /~qiuwch/unrealcv/binaries//RealisticRendering-mac.zip was not found on this server.

in the browser window.

Get various images directly as numpy arrays or similar

Hi,

Is it possible to get the depth image (and other images) directly in python as numpy arrays instead of saving to a file and then reading them back to Python? I could not figure out how to do so from the commands list or examples.

Thanks in advance!

Is it possible to change the position of a particular object?

Since we can change the position and orientation of the camera, is it possible to do something similar with the objects in the world?

Set Object Position Command

Hi Weichao,

When you get a chance could you add a command to set object locations? It would be helpful to be able to dynamically alter the scene.

Objects other than StaticMeshActors do not show up in modes other than lit

I'm relatively new at using Unreal Engine. I've created a simple map with a StaticMeshActor floor as well as a Landscape object using the UE4Editor. When I hit "Play" in the editor, the entire scene is visible. However, when I switch viewmodes to depth (or normal, or object_mask), then the non-static-mesh objects are no longer visible. I've found that this occurs with other objects as well. If I export the Landscape as an object and then re-import it as a static mesh, then it is visible, but this workaround would not scale to a large map with many objects.

Blueprint objects are not shown in the object instance annotation

According to the code in https://github.com/unrealcv/unrealcv/blob/ff8da15/Source/UnrealCV/Private/ObjectPainter.cpp#L191. Only StaticActor is considered as an object, so when calling vget /objects. Blueprint objects used in a game will not be considered and will be just ignored.
This will be fixed in the next release.

unrealcv plugin fails to load

LogLinux:Warning: dlopen failed: /home/sung/Documents/Unreal Projects/simple 4.14/Plugins/Binaries/Linux/libUE4Editor-UnrealCV.so: symbol _ZN10ACharacter16ApplyWorldOffsetERK7FVectorb, version UE4 not defined in file libUE4Editor-Engine.so with link time reference

Hi, I get this error when I try to load the plugin. I tried installing in both editor and project folders, same thing happens. Tried version 0.3.0, using UE 4.14.3.

No objects listing in custom scenes

Hi all!

I am trying to create my own scene with UE4 (4.12, compiled for Linux following the instructions) and UnrealCV (0.3) rather than just using the RealisticRendering cooked binary. I am just creating a blank project in which UE4 inserts a couple of chairs, a table, a statue, and two floor tiles.

I just choose the correct GameMode Override and run the scene. Then when sending the command to retrieve the list of objects (more specifically):

object_list_ = client.request('vget /objects').split(' ')
print("There are " + str(len(object_list_)) + " objects in this scene...")
print(object_list_)

I just get a list with a single 'ok' element like this:

There are 1 objects in this scene...
['ok']

The scene contains various static meshes so I don't know what could be causing this problem. That makes impossible to obtain object segmentation masks. I am able to get good RGB images, depth maps, and normal information.

I tried to find the RealisticRendering UE4 project to see if objects must be arranged in any special way but I was not able to find anything else than already compiled binaries.

If any of you knows what could be going on, I would be really grateful. I would also like to say thanks to the creators for such an awesome tool!

Problem with UnrealCV plugin for Unreal Engine 4.12

Hey, I'm trying to compile a game binary using unrealcv plugin but am unable to do so.

MainFrameActions: Packaging (Linux): mono: LogCompile: /home/sudarshan/UnrealEngine/Engine/Plugins/unrealcv/Source/UnrealCV/Public/TcpServer.h(50): Error: Property is exposed to the editor or blueprints but has no Category specified.

This is the error I'm getting.
The complete error log is here

UnrealCV does not build on Mac with Unreal 4.16

For the record, I'm going to ask and answer my own question. This is related to #55.

When trying to execute build.sh on my Mac, I received errors very similar to those shown in the 3rd comment in #55. This is what is happening: By default, when you install the Unreal engine on a Mac, it installs the engine in /Users/Shared/Epic Games/.... The build issue is caused by the space in the file path, as I learned here:

https://answers.unrealengine.com/questions/579232/cannot-package-on-macos-no-automationtoolexe.html

To solve the issue, uninstall the engine and reinstall it, but edit the installation file path to remove the space. Instructions for removing the Unreal engine can be found here:

https://answers.unrealengine.com/questions/32845/how-to-uninstall-ue4.html

These instructions are a bit dated - you'll want to remove /Users/Shared/Epic Games/ instead of /Users/Shared/UnrealEngine/, but everything else is the same I believe.

The master branch plugin appears to work with Unreal 4.16.1, subject to the caveat of issue #46, that is, none of the commands return values when called from a Python client.

Compiling the plugin error

Currently i am trying to compile the unrealCV plugin into Unreal engine 4.12. I get this error during make

UnrealEngine/Engine/Plugins/unrealcv/Source/UnrealCV/Public/TcpServer.h(50): Error: Property is exposed to the editor or blueprints but has no Category specified.

Property is exposed to the editor or blueprints but has no Category specified.

which version of unreal engine are you working with??

Incorrect image generation when ViewMode is changed

vget /camera/0/lit will return depth information when ViewMode is set to depth.

How to create a game binary of a virtual world for UnrealCV ?

Hey,
I want to create a short game binary, similar to the one provided on the unrealcv website. How would I go about it ?
Thanks.

Object mask

Hi Weichao,

Awesome work!! I'm new to Unreal, I would like to use unrealCV for a deep learning project. I have issues to assign a color ID to objects. After I packaged my unreal (4.12) project, I can't access to the command line. The only way to connect to the package is through a command console. Then I can switch from depth to normal to lit. Using the "vset /object/[object_name]/color 255 0 0" command doesn't work. It returns this error : "IndentationError: unexpected indent".
My goal is to create a script that will randomly assign color ID to all the objects in my scene.
(I tried to connect to the unreal editor with my scene loaded and I have this error "can not connect to ('localhost', 9000) | Error [Errno 10061] No connection could be made because the target machine actively refused it")
Did I miss something? Any help will be much appreciated :)
Thank you!
Noel.

Is it possible to iterate through cameras, or return the 3D scene geometry?

Hello there,

This system looks amazing! Great work. I have a couple of quick silly questions.

1. Is it possible to iterate through all the cameras in the scene somehow? There seems to be a command to iterate through all the objects, but not the cameras.
2. Is it possible to return the 3D scene geometry somehow? Globally or per object? Presumably the Unreal Engine internally has access to the 3D scene geometry. Is this access exposed to plugins? It would be extremely helpful if I could say "vget /object/my_couch/geometry" and it would return a triangle mesh or something.

Cheers,
Mike

is it possible to detect objects inside UE4 using faster-rcnn?

I was wondering if it's possible to use faster-rcnn inside UE4 with unrealCV instead of separately rendering the images using the packaged game

"Can not receive a response from server, timeout after 5.00 seconds" after client.connect()

What could be the possible reason? Thanks.

Support for Linux Unreal 4.13?

Wondering which versions of Unreal this plugin is supported for. Tried to install the plugin via your instructions after the build for Linux and got this error:

/home/daeil/Programs/UnrealEngine4.13/Engine/Plugins/unrealcv/Binaries/Linux/libUE4Editor-UnrealCV.so: symbol _ZN7UObject7PreSaveEv, version UE4 not defined in file libUE4Editor-CoreUObject.so with link time reference
[2016.10.15-19.55.00:799][ 0]LogModuleManager:Warning: ModuleManager: Unable to load module '/home/daeil/Programs/UnrealEngine4.13/Engine/Plugins/unrealcv/Binaries/Linux/libUE4Editor-UnrealCV.so' because the file couldn't be loaded by the OS.

Any tips on solving this would be much appreciated. Thanks!