Comments (29)
It's relatively straightforward. Depending on the kinds of constraints or guarantees you need it can be as simple as adding the following to a MonoBehaviour
that manages the communication with the external process:
public void Start() {
MjScene.Instance.preUpdateEvent += UpdateWithRemoteQpos;
}
public unsafe void UpdateWithRemoteQpos(object sender, MjStepArgs e) {
double[] qposExternal = GetMostRecentQpos();
for (int i=0; i<e.model->nq; i++)
{
e.data->qpos[i] = qposExternal[i];
}
for (int i=0; i<e.model->nv; i++)
{
e.data->qvel[i] = 0;
}
}
I did not test this myself (just quickly typed it up, sorry for the potential typo or syntax error!), as I've not yet needed to pipe in simulation data from an external process to the Unity plugin. The expectation is that this should work. If it does, we could add an example component or scene somewhere that demonstrates it. If it doesn't, let me know and I'll try to fix it/troubleshoot.
You need to implement the "GetMostRecentQpos()
" method yourself, which could just reuse values from the last communication if new ones have not arrived since then.
This does mean that you might preform more updates than needed (busy polling). This is unlikely to be the case, since you are using Unity for visualization only, and can set a high timestep without detriment to simulation quality, and achieve a stable smooth framerate (and you'll miss some qpos frames instead, which is no problem).
If you need to render every single frame reliably and therefore need precise manual control over stepping the physics, let me know, it is a reasonably simple addition.
You could also set some disable flags in Global options component in your Unity scene to make sure you are not wasting operations on dynamics/contacts etc as that is managed outside.
from mujoco.
Hi @Balint-H , Thanks for your quick resonse. I have a function GetMostRecentQpos() within my MonoBehavior which gets the qpos from the simulation through ZMQ. But when I try to add these lines of code I get an error because of MjStepEventArgs mentioning error CS0246: The type or namespace name 'MjStepEventArgs' could not be found (are you missing a using directive or an assembly reference?)
. I tried to look out in MjScene.cs file generated at runtime as well as MjActuator.cs file, I don't see this Event Handler. Could you let me know how I can include within my Monobehavior?
from mujoco.
My bad, it is simply MjStepArgs. I'll update the original answer too.
from mujoco.
Thanks @Balint-H. I updated my changes you mentioned. But still I cannot render the assets realtime. I added my Monobehavior script to the asset game object after importing using Mujoco plugin. Attached image for reference.Here the clientobject subscribes to a port using NetMQ and updates the qpos and qvel. Am I doing something wrong ?
from mujoco.
Could you drop a Debug.Log($"Cur qpos is {string.Join(", ", qposExternal)}")
into UpdateWithRemoteQpos() to see if its being called and that the correct values are retrieved?
from mujoco.
Hi @Balint-H , thanks for your reply. I broke down the problem into two modules to debug the issue. I removed the communication and loaded a static asset using mujoco unity plugin and try to send different positions to it based on the code snippet you gave me. I am able to set positions after I pause the unity for sometime. I have set the Scene recreation at late update requested to be True. But when I render this in VR it still renders static asset in the first step. Can I not step it in the VR? Looking forward to hearing from you.
Screen.Recording.2024-05-10.at.7.07.45.PM.mov
from mujoco.
I did a test of the sample script from the previous message above, with the following placeholder kinematic motion source (a sine varying over time on all DoFs):
private unsafe double[] GetMostRecentQpos() {
return Enumerable.Repeat((double)Mathf.Sin(Time.time * 0.4f)/2, MjScene.Instance.Model->nq).ToArray();
}
Movie_002.mp4
https://github.com/google-deepmind/mujoco/assets/41113387/9e5961f7-3a15-482d-b3fe-a5e5b3187795
It seems to work as intended in an example scene with a humanoid, bypassing all joint constraints and contacts.
I also disabled all non-relevant simulation steps (e.g. contacts) in the GlobalOptions component. No need to do scene recreation on late update, unless you are adding/removing things from the scene, and should work without pausing.
Could you confirm in a minimum working example scene that only has mujoco components (e.g. no VR components that cause you exceptions) and your Qpos setter whether you are still having issues?
from mujoco.
Hi @Balint-H , thanks for your reply. I tried that before itself. I was able to get my Qpos setter working ( with no VR components) and was able to play the scene in unity. I wanted to know if I can get this working with the VR rendering. Has anyone tried this before?
Screen.Recording.2024-05-13.at.11.40.29.AM.mov
from mujoco.
I have used the MuJoCo plugin in Unity successfully without issues in VR. That scene was running the physics as well, so simple kinematic control should be supported too.
What system are you running on? The playmode seems to be running on a very low framerate.
Have you set up a minimal scene only with VR/unity components (no MuJoCo) and confirmed you can have the desired features you need from the VR side working?
from mujoco.
I am running on a Mac Pro M2. My plugin version is 3.1.5. Unity version 2021.3.5f1
Yes I have my scene only with VR components and it is all working , it is publicly released as well(https://open-teach.github.io/).
Also one more thing , how do you set the MJGlobalOptions. I dont't have that set right now. That could be a reason it is running on a very low framerate
from mujoco.
Add the MjGlobalSettings
component to your scene if its not in it already (or search for it). In the inspector with it selected you can edit Global Options > Flag
to disable simulation elements. That will speed things up but it is likely not the bottleneck unless you have a tiny timestep (<0.001).
You can also try to downgrade to 3.1.4 for a while, unfortunately you do need to downgrade at the moment if you want to use MjGlobalSettings.
Does your Unity-only scene run smoothly in VR? If yes, but not with MuJoCo, a troubleshoot step you can try is importing e.g. the humanoid.xml example scene and running it in VR, without any script, just raw mujoco physics and VR.
from mujoco.
Yes unity only scene runs smoothly in VR. My timestep is 0.02 so that is fine I think. I will test the humanoid.xml in VR and get back to you.
from mujoco.
Hi @Balint-H , I tested out Humanoid.xml example, it renders as a static scene in VR. Without any script if I try to play the unity file with humanoid, unity crashes immediately. Do you know why this could be happening?
from mujoco.
And it runs no problem without VR? Any error messages when using VR?
from mujoco.
Ah yes I think I know the current issue, its about using 3.1.5, the plugin on main is not stable due to a small bug, could you downgrade to MuJoCo 3.1.4 (plugin and DLL) or check out the Pull request fixing the issue with the "sensor noise" feature?
from mujoco.
Even without the VR it crashes. How do I downgrade the plugin ? Does altering package.json and loading it again works?
from mujoco.
You can either download it at the 3.1.4 version (https://github.com/google-deepmind/mujoco/tree/3.1.4) or update it in the manifest.json file inside your Unity project folder (Packages > manifest.json) and add this line:
"org.mujoco": "https://github.com/google-deepmind/mujoco.git?path=unity#3.1.4",
Make sure that the mujoco.so library you are using is also 3.1.4 (https://github.com/google-deepmind/mujoco/releases/tag/3.1.4).
If you reference it with the git url like above Unity will download it for you, but you might have to add the library manually to your asset folder.
You likely also have to close and reopen the project to unload the old library.
from mujoco.
I get this error IOException: Error loading the model: XML Error: Schema violation: unrecognized attribute: 'sensornoise' on 3.1.4 when I play the unity.
from mujoco.
I was able to solve that problem. But I still have the issue when I play it in unity it works but I when I load the APK in the VR it just renders a static image.
from mujoco.
Thanks for getting it to work (I apologize for directing you to 3.1.4, as the version without the bug was 3.1.3... Thanks for figuring it out)!
Could you click on the mujoco.so file and see what platforms are selected for it? Its possible you have to enable android there.
In the past when I used MuJoCo with VR it was for the Oculus Quest. A quick search online showed that it is possible to use shared libraries with it, but it is possible you need to recompile MuJoCo specifically for it. However, I expected that to be only a potential issue when making a build, not in the Editor.
@MyelinsheathXD did you end up succeeding in compiling MuJoCo for android? #1638
from mujoco.
I am using Oculus Quest as well. In the editor things work smoothly, only while making a build in the VR, the scene rendered is static. I will try to enable android on the mujoco.so and see if that works
from mujoco.
In my example I used Quest Link to stream the app to the headset. Is that an option for you?
from mujoco.
yes I use Quest Link with a cable.
from mujoco.
@Balint-H hey! Unfutunately, I am not able to build mujoco library for android yet.
Currently I am working on another project .
However , here the link for the engine’s 3rd party libraries that was build for android, if you like to build the mujoco itself for android using Android studio’s CMake
https://github.com/MyelinsheathXD/modelsLib
from mujoco.
I tried to reinstall everything from scratch on 3.1.3 and even mujoco so 3.1.3. I found my mujoco so was 3.1.4 and plugin was 3.1.3 which could be causing issues. But when I reinstall I get this error which I never got before. Do you know the fix for it? @Balint-H
DllNotFoundException: mujoco assembly: unknown assembly type:unknown type member:(null)
Mujoco.MjVfs..ctor () (at /Users/aadhithyaiyer/Downloads/mujoco-3.1.3/unity/Runtime/Tools/MjVfs.cs:88)
from mujoco.
Hi @Balint-H , do you know the fix to this issue?
from mujoco.
My guess is the following issue:
Make sure the mujoco library is named libmujoco.so and not "libmujoco.so.3.1.3" . Annoyingly Unity will display it as "mujoco.so" even if its extension is longer, but will ignore it unless you truncate the extension to just so.
Can't guarantee that's the issue as I only used the plugin on Windows and Linux in the past (where this is a common cause of unrecognized .so files), but I suspect it could be related.
from mujoco.
Hi , I modified that already. I also tested things in windows, it plays in unity editor but the humanoid asset is static when build in VR. I feel this is a bug. Let me know if you care able to find a fix for it. Thanks.
from mujoco.
The time I had MuJoCo running on the Quest, I was using Quest link via USB, but ran the app from the Editor (and streamed the VR scene in play mode to the headset). If you have not tried that, and only tried to make a stand-alone build, this approach could be a temporary solution for you. I'd expect an android compatible build of MuJoCo is needed for the stand-alone build to work, which could be a nice feature, but is not yet supported.
from mujoco.
Related Issues (20)
- Support for Rendering in MJX for Simulated Camera Reinforcement Learning HOT 5
- `.skn` file not working with replicated bodies (dmcontrol rodent model) HOT 1
- Domain randomization for MJX HOT 7
- Resource not found (ValueError), but only on consecutive runs. HOT 12
- Equation for the Scalar Length l_i in the Actuator Transmission for Hinge Joints HOT 2
- Estimate qvel from sequence of rotations HOT 5
- How to simulate glass window/door in mujoco? HOT 8
- Assistance Required with Renderer Updates for Multiple XML Models in Reinforcement Learning HOT 2
- Modelling Shallow Water Correctly
- GLFW crashes on aarch64 wayland HOT 3
- [Unity plugin] Add kv property for position actuators HOT 2
- MJX False Collision in 3.1.4 and 3.1.5 only (no collision in MuJoCo) HOT 4
- [<Actuator>] Custom Control Range Mapping for Actuators in MJCF XML HOT 2
- MuJoCo ROS Integration - segmentation faults in model instance (from version 3.1.1 -> ^3.1.2) HOT 1
- Camera Rendering Incomplete After Modifying znear and zfar
- D4RL mujoco datasets compatibility between versions HOT 1
- MuJoCo cup model penetrating floor/table HOT 3
- Issues with Point Cloud Visibility in Camera's Field of View Using MuJoCo
- Document MJX "broadphase" fields
- Simulating tension only cables with variable lengths
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mujoco.