offchan42 / unity3d-python-communication Goto Github PK

Hello, how can I connect SUMO to Unity3D? I have cloned your projects and simulation is running but unfortunately connection between Unity3D with SUMO is not working.

I tried this code and it is working well for unity-python communication when I ran both on my local machine. But since i need to run python in browser and send audio files to unity. Does this code work for that?

This is a good solution to my project.
But I have another question. I have multiple clients(in unity) and How to write a broadcast message from one client to all other clients.I look forward to your reply.

Hello!
1、Should messages be sent only in child threads in order not to block the Unity main thread？
2、I need to send pictures,so I used client.SendFrame(Convert.ToBase64String(CameraHelper.CaptureFrame(Camera)));
But I got an error from unity:UnityException: get_targetTexture can only be called from the main thread

So How to send pictures by your plugin?Thank you!

When I opened your project in Unity3D, some error happened:

First of all, thanks for sharing this with the community.
Having a problem simply sending a message after the initial handshake, say from some UI input like a button press.
Most likely because of my lack of understanding on how threads work in Unity.

Any chance you might post a quick example?
If not a button then a simple message sent from HelloClient a few seconds after launch?

Thanks in advance.

First, Thank you for your amazing repository. UnfortunateIy, I have a difficult time extracting the data from your RunAbleThread to my monobehaviour. This might be a simple question but could you please elaborate the solution?

thank you

Hello,

I have an object classification model on Python which is trained using Keras. I want to get frames from Unity and analyze the frames on Python. Is it possible? If it is, can you explain?

I am working on a screen capture using a bridge between Python and Unity with this library.

This is my client code:

public class HelloRequester : RunAbleThread
{
    /// <summary>
    ///     Request Hello message to server and receive message back. Do it 10 times.
    ///     Stop requesting when Running=false.
    /// </summary>
    protected override void Run()
    {
        ForceDotNet.Force(); // this line is needed to prevent unity freeze after one use, not sure why yet
        using (RequestSocket client = new RequestSocket())
        {
            client.Connect("tcp://localhost:6789");

            for (int i = 0; i < 100000 && Running; i++)
            {
                client.SendFrame("Hello");
                // ReceiveFrameString() blocks the thread until you receive the string, but TryReceiveFrameString()
                // do not block the thread, you can try commenting one and see what the other does, try to reason why
                // unity freezes when you use ReceiveFrameString() and play and stop the scene without running the server
//                string message = client.ReceiveFrameString();
//                Debug.Log("Received: " + message);
                byte[] bytes = null;
                bool gotMessage = false;
                while (Running)
                {
                    gotMessage = client.TryReceiveFrameBytes(out bytes); // this returns true if it's successful
                    if (gotMessage) break;
                }

                if (gotMessage)
                {
                    var colors = new List<Color>();
                    for (var p = 0; p < bytes.Length; p += 4)
                        colors.Add(new Color(bytes[p + 2] / 255f, bytes[p + 1] / 255f, bytes[p] / 255f));

                    HelloClient.pixels = colors;
                    HelloClient.update = true;
                }
            }
        }

        NetMQConfig.Cleanup(); // this line is needed to prevent unity freeze after one use, not sure why yet
    }
}

This is my server code:

import time
import zmq
from mss import mss

context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:6789")

while True:
    #  Wait for next request from client
    message = socket.recv()

    #  Do some 'work'.
    #  Try reducing sleep time to 0.01 to see how blazingly fast it communicates
    #  In the real world usage, you just need to replace time.sleep() with
    #  whatever work you want python to do, maybe a machine learning task?
    time.sleep(0.1)

    #  Send reply back to client
    #  In the real world usage, after you finish your work, send your output here
    with mss() as sct:
        # Use the 1st monitor
        monitor = sct.monitors[0]

        # Grab the picture
        im = sct.grab(monitor)

        # Get the entire PNG raw bytes
        socket.send(im.raw)

(I tried to keep it as similar as possible as the hello world example)

If I set the time.sleep() to 2 seconds, I can get exactly 14 requests through until the client just stops getting any messages from the python server (it just gets stuck in the while gotMessage loop with no errors). If I set the time.sleep() to lower I can get more requests through but it seems like the connection just stops working after around ~30 seconds. What am I doing wrong?

Thanks!

Hi @off99555 thanks for this cool stuff.
I just test your example and it is working fine. I want to ask that, I'm running this github project which is in Python. My teacher asked me to transfer all the python code to C++ to run it in Unity.
So my question is that using your stuff can I directly run python code to Unity and how we can build dll files directly from Python?

Can you please share a short example of sending images from Unity to Python over the socket? The type of image is not of great importance, all that is required is for each image to come in sequentially with latency as low as possible. Thanks.

We are looking for a in-game Python interpreter, so player can code Python in game.

We came across in this great repo and would like to know if this can be used in game, instead of using an external terminal to communicate...?

Or in this case IronPython is enough for me? (I havn't dig deep into IronPython yet...)

Hi,
Thank you, again for the repo. I am trying to learn from your codes. I see that you created a new thread and commented here that we need a new thread so that unity can do other stuff. I assume the new thread will be running the client in "parallel".
Then in here you said that if we used ReceiveFrameString() we would block unity from doing other stuff.

My first question is that, if we have already started a new thread to deal with blocking receive function, then why do we need to use TryReceiveFrameString(). Even if this new thread gets blocked for using ReceiveFrameString(), why would it hamper unity?

If we used ReceiveFrameString() and the new thread gets blocked until it received a response from the server isn't that good for the processing power? I see in your ToDo (in the readme.md) that, the current implementation is calling TryReceiveFrameString() in a loop without any delay which is bad for processing power.

Again, thank you for your time.

Hi,
Thank you for this great repo.
I am a beginner so my question might seem very trivial.

My project requires me to run a Unity project on one computer and run a python script in another computer. The two computers will be on the same network. We need to send data from the python script (pc-2) to Unity (pc-1). Can this be done using your repo?

The python script will output a matrix of 6 numbers. So we are thinking of using UDP. It is a PyTorch inferencing script. As the Unity pc's GPU will already be under stress we decided to run the deep learning inferencing on another pc.

The troubleshooting section says that killing the process using CTRL + C will break the server, and the example file "server.py" has a while True statement making the only way to stop the scrip (CTRL + C) crash the server.

Thanks for the response to #16 ! I think the question was not clearly deliver to you (#16). The issue was not about type of data. The issue is that i need to communicate between web browser (the python server is running in the web browser (using Jupyter Notebook)) and unity (running on local machine). But i am not able to send data from browser to unity. Please give some guidance on that.

I have such scenario, in the python server, it uses class to update and calculate the instance' s state, then sends such state to corresponding client; in the client side, more than one clients need to communication with the server to get its one result and not interfere by others. Can your tool distinguish with different clients?

Thank you for the inspiring code pieces. It has more info about one-way communications, passing data to Unity event loop, and sending JSON and images. You can read the article here.

Hi
Trying to implement the 'Lazy Pirate' Reliability pattern in the Unity client.
Any chance this has been implemented somewhere?
http://zguide.zeromq.org/cs:lpclient

Hi, I've been looking for something like this (I really felt the second to last question on your readme lol) I am trying to run a modified version of R-CNN mask from python to unity, the script is from this repository and just gets the image saved inside the unity project so I can display it later with opencv for unity.

so on server.py, I added some stuff (I am pretty new to python so it may be ugly)

import time
import zmq
import os
import threading

context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")

#run on thread
def runMask(image):
    os.chdir("path/to/rcnn/mask")
    #I know there must be a better way to just call that function, I'm open to suggestions 
    cmd = "python ColorPopAndPotraitMode.py ColorPop --image "+image.decode()
    os.system(cmd)

while True:
    #  Wait for next request from client
    message = socket.recv()

    #message will be the name of the picture (path)
    print("Received request: %s" % message)
    print(os.getcwd())
    print("starting mask script")

    #starting thread to run mask (remember to put the ',' on args, else it would be treated as a tuple)
    thread = threading.Thread(target=runMask, args=(message,))
    thread.start()
    print("running mask")
    socket.send(b"... running mask ...")
    thread.join()

    #  Send reply back to client
    #  In the real world usage, after you finish your work, send your output here
    response = "done, "+os.getcwd()
    socket.send(response.encode())

I wrapped in a thread because is quite heavy and I wanted to make sure it would wait for the process to finish, however it gets stuck on this part forever until I close the window

2020-08-01 21:42:51.503401: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_10.dll
2020-08-01 21:42:51.757888: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudnn64_7.dll
2020-08-01 21:42:52.802719: W tensorflow/stream_executor/gpu/redzone_allocator.cc:314] Internal: Invoking GPU asm compilation is supported on Cuda non-Windows platforms only
Relying on driver to perform ptx compilation.
Modify $PATH to customize ptxas location.
This message will be only logged once.

I think it may have to do with tensorflow but if I run the exact same code in a different file without the server part, it works all right.
So, is there like a timeout or something that may provoke that from server.py?

btw the reason I am not using the Mask rcnn integrated on Opencv for unity is because it is using just a checkpoint (that much lighter .pb file) from the original model and the accuracy is not enough for our project, where we need to take body measurements, although we need that to run on android, and we are using GPU here, but that's for another day.

I running on windows 10, Nvidia GeForce GTX 1650 SUPER, python 3.8.3, tensorflow 2.2, keras 2.4, unity 2019.3.15

I know it is kind of unrelated, but if you have any questions, please let me know.

Thank you

I have downloaded and opened the Unity project in Unity 2020.3.27f1 and run server.py. It works as is and stops at 10 message sent and received. When I modify the code HelloRequester.cs to run continuously the process stops at 30. Every time. It stops here:

gotMessage = client.TryReceiveFrameString(out message);
if (gotMessage) break; // gotMessage is always false after 30 messages.

The strange thing is I can stop and play Unity again and the messages start sending/receiving again - unit 30. I have tired reducing the delay in the python code and the number of messages goes up but they still stop consistently at the same time each time.

Any ideas on how to get around this?