nvidia / dataset_synthesizer Goto Github PK
View Code? Open in Web Editor NEWNVIDIA Deep learning Dataset Synthesizer (NDDS)
License: Other
NVIDIA Deep learning Dataset Synthesizer (NDDS)
License: Other
I used the SceneCapture_Simple actor and set the Time Between Scene Capture = 2.0, then it waited for 2 seconds and started capturing. But it kept capturing the rest of the images without any 2 seconds delay between two images, just one frame after another.
So how to solve this problem?
Hey,
I have a problem with the visibility score in the annotation files.
The visibility score of my hands will always be 1 even if they are occluded by the person.
Scores of the two hands:
"visibility": 1
"visibility": 1
However if the hand is partly occluded by the other hand nearer to the camera the visibility score will be reduced:
"visibility": 1
"visibility": 0.81819999217987061
So always when there is a different object than the person between my camera and the hand, the visibility score works:
"visibility": 1
"visibility": 0
This is the how my blueprint class is constructed:
Can someone explain me why it isn't working like it is supposed to?
after reading the readme I still dont know how to use it,is there any describing more detail ?
According to the tutorial video the object's location should in camera coordinate system like "location": [ 10.816499710083008, 39.035198211669922, 235]", but i got "location": [ 0, 0, 0 ]", and the visibility should be 0 according to the video, but i got 1. I'm confused, can somebody help me to find out what happens. Thanks a lot!
I think it would be great if NDDS supported domain randomization via direct JSON manipulation similar to the functionality built into the Isaac platform.
Unfortunately, the Isaac simulator uses an outdated version of Unreal Engine, and this repo seems to be updated more frequently.
Hi
after i put cube into example scene, get wrong annotation.
why?
or how to get right annotation? "projected_cuboid_centroid": [ 256.00009155273438, 287.43289184570312 ], "bounding_box": { "top_left": [ -116833.7734375, -140902.03125 ], "bottom_right": [ 117898.078125, 141403.25 ] }, "cuboid": [ [ 21.611099243164062, -33.871700286865234, -97.358100891113281 ], [ 21.611099243164062, -33.871700286865234, -2.6419000625610352 ], [ 21.611099243164062, 21.59320068359375, -2.6419000625610352 ], [ 21.611099243164062, 21.59320068359375, -97.358100891113281 ], [ -21.611099243164062, -33.871700286865234, -97.358100891113281 ], [ -21.611200332641602, -33.871700286865234, -2.6419000625610352 ], [ -21.611200332641602, 21.59320068359375, -2.6419000625610352 ], [ -21.611099243164062, 21.59320068359375, -97.358100891113281 ] ], "projected_cuboid": [ [ 199.17430114746094, 345.06460571289062 ], [ -1838.110595703125, 3538.15771484375 ], [ -1838.110595703125, -1836.37744140625 ], [ 199.17430114746094, 199.22129821777344 ], [ 312.8258056640625, 345.06460571289062 ], [ 2350.114501953125, 3538.15771484375 ], [ 2350.114501953125, -1836.37744140625 ], [ 312.8258056640625, 199.22129821777344 ] ]
as you can see, bounding box value is invalid, projected_cuboid is invalid. projected_cuboid_centroid is not at the center of cube in rgb image.
NDDS is supported on Windows and Linux (Ubuntu 16.04.4 LTS).
When trying to load NDDS into Unreal Engine I'm getting the error
Failed to load map!
../../../../Dataset_Synthesizer/Source/Content/Map/TestCapturer.umap
appears to be an asset file.
steps I followed:
I installed Unreal Engine on Arch Linux, following the instructions on this page. Note that I downloaded the Unreal Engine source code using the command
git clone -b 4.22 https://github.com/EpicGames/UnrealEngine.git
which installed version 4.22.3 on my system.
I then download NDDS using the command
git lfs clone https://github.com/NVIDIA/Dataset_Synthesizer.git
which resulted in a directory of size 406MB.
I then ran Unreal Engine:
cd UnrealEngine
./Engine/Binaries/Linux/UE4Editor
In the Unreal Project Browser dialog box, I clicked Browse..., selected DatasetSynthesizer/Source/NDDS.uproject and clicked Open. I got the Convert Project dialog box so (as the NDDS documentation suggests) I chose More Options... then Skip conversion. This generated the Missing NDDS Modules dialog box asking if I'd like to rebuild some modules, so (as the NDDS documentation suggests) I chose Yes.
If click OK, Unreal Editor still opens, but NDDS obviously didn't load correctly
I don't have too much experience with Unreal Engine, so I'm not sure what this means or what to do. Any help would be appreciated.
I tried to export class segmentation image with scene has object with translucent material. It looks like the segmentation cannot gave any output of those objects. I think it may caused by translucent material doesn't write to depth buffer and it won't work with depth grabber and segmentation? If it is true, anyone has any workaround idea?
I'm getting incorrect bounding_box
dimensions on simple cube-like actors. I've reproduced this issue on UE 4.21 with NDDS v1.1 as well as UE 4.22 with the latest NDDS commit.
Below is an example. Notice that the bounding box is very far off with respect to the location and cuboid.
{
"class": "cube2",
"instance_id": 14,
"segmentation_class_id": 5,
"segmentation_instance_id": 14,
"visibility": 0,
"location": [ 112.7113037109375, 75.990898132324219, 225.98260498046875 ],
"quaternion_xyzw": [ -0.71619999408721924, 0.69760000705718994, 0.0013000000035390258, -0.021600000560283661 ],
"pose_transform": [
[ -0.15369999408721924, -0.14000000059604645, -4.7954998016357422, 0 ],
[ 0.10050000250339508, -3.7471001148223877, 0.10620000213384628, 0 ],
[ 3.5968999862670898, 0.093199998140335083, -0.11800000071525574, 0 ],
[ 112.7113037109375, 75.990898132324219, 225.98260498046875, 1 ]
],
"cuboid_centroid": [ 112.70860290527344, 76.033599853515625, 225.97419738769531 ],
"projected_cuboid_centroid": [ 624.32220458984375, 444.7001953125 ],
"bounding_box":
{
"top_left": [ -21585.841796875, -196730.453125 ],
"bottom_right": [ 194923.109375, 1012.8798828125 ]
},
"cuboid": [
[ 95.781303405761719, 56.772098541259766, 254.08050537109375 ],
[ 94.0115966796875, 98.204399108886719, 250.59910583496094 ],
[ 131.16709899902344, 99.687896728515625, 249.36790466308594 ],
[ 132.93670654296875, 58.255599975585938, 252.84930419921875 ],
[ 94.250099182128906, 52.379299163818359, 202.58059692382812 ],
[ 92.480400085449219, 93.811599731445312, 199.09910583496094 ],
[ 129.63589477539062, 95.295196533203125, 197.86790466308594 ],
[ 131.40559387207031, 53.862899780273438, 201.34930419921875 ]
],
"projected_cuboid": [
[ 549.10882568359375, 374.91390991210938 ],
[ 547.98187255859375, 478.91018676757812 ],
[ 641.1383056640625, 483.77789306640625 ],
[ 640.98797607421875, 379.20819091796875 ],
[ 603.622314453125, 396.59719848632812 ],
[ 603.1572265625, 527.8726806640625 ],
[ 720.9031982421875, 534.31207275390625 ],
[ 719.3353271484375, 402.12249755859375 ]
]
}
Note, each actor is scaled using their Construction Script. This scale does not change and seems to be accurately reflected in the exported cuboid dimensions.
The collision boxes also seem to be accurate, which I know affects the calculated bounding box.
There are nine different classes of cube in total that I'm generating using Spatial Layout Generator Grid 2d
.
Does the NDDS plugin generate the ground truth belief map of synthetic images as metadata as I'm unable to find it in the documentation?
If not, are you following the given formula in image below to calculate it as given in this paper?
I tried to use the tool and draw a bounding box (yellow line) over it using bounding_box data in json file.
So far, it seems so good. But sometimes the value I take from NDDS seems to be not normal.
Any idea why this happens?
How does NDDS calculate a bounding box?
Hi, thanks for providing this tool!
I found many components which can change the light or the movement of objects randomly.
But how to change the projective matrix of virtual camera randomly like in FAT dataset?
Hi there, I am trying to create dataset with translucent object but had no luck. In class segmentation case, I was able to bring translucent objects in image thanks to this step.
Question is what do i have to do in order to capture depth image of translucent objects in NDDS?
Here are my results so far. (Cup)
Hello, thank you so much for this powerful tool!
If I am using a doll, is there any way I can extract every hinge location?
Position of neck, hands, legs, etc
Hey,
in which way did you import the YCB objects into Unreal Engine such that you can use the meshes to render the images?
On the YCB Website (http://ycb-benchmarks.s3-website-us-east-1.amazonaws.com/) it is possible to download the processed data or the laserscan. However, they do not provide a .fbx file for the unreal engine.
How did you create these .fbx files?
Thank you really much in advance!
Best regards,
Björn
Hi,
related to another issue (last comments) in the DOPE repo, I fail to make any of the 16bit depth extractors work.
On the following image (left) the 8-bit map (middle) works fine, whereas the 16-bit map (right) appears to be too flat:
Minimum and Maximum values of the depth map are also strange on the 16-bit image:
min: 10502 max: 13311, so only a depth distance of 30cm and the minimum is too far away.
I got the same results with all three SceneCaptures and with all depth extractors (quantized and absolute) I did a depth_image * MAX_DEPTH / 2^16
conversation to get the real depth values.
Am I misunderstanding something or is the extractor broken?
Working with the current version under linux.
Thanks!
hey folks, the assets in your 4.22 do not seem to be ready for that version. They are still formatted for 4.21. Any idea about how to fix this? My CAD imports do not have this issue.
I don't expect anyone to troubleshoot this on my behalf because it's an unusual use-case. I'm just reporting it for reference.
If you try to set the Image Capture Size to anything with a width exceeding 1024, it the output images look like an old TV that has lost sync.
I was trying to create a capturer that matched the intrinsics and output of a camera I am using when I ran into the problem.
I will dig into it, and if I find anything useful, I will report back.
I would like to randomly move several objects in one room. This works fine so far, but now I try to figure out how these figures don't overlap. I already tried lots of collision options but I didn't managed to finde the right configuration.
Hey,
I am currently creating my own dataset in the same way than the FAT dataset with help of the NDDS. For this purpose, I am using the Simple_Scene_Capturer, which has the options of overwriting, removing or creating a new directory with new timestamps to export pictures.
How did you export all pictures enumerated into one folder, when you were dropping the objects five times in the same location for the FAT dataset?
By creating a new directory with the timestamps, I would have to add all pictures together manually afterwards.
Thank you really much for your help!
Best regards,
Björn
I tried to calculate the 2d projection from 3d location using matrix that is provided on object data and camera data.
The matrix that I used is from https://en.wikipedia.org/wiki/Camera_resectioning .
2D = K [R|T] 3D
intrinsic parameters are taken from camera data, then extrinsic parameters from object data.
sample:
{ "camera_settings": [ { "name": "Viewpoint", "horizontal_fov": 90, "intrinsic_settings": { "resX": 640, "resY": 480, "fx": 320, "fy": 320, "cx": 320, "cy": 240, "s": 0 }, "captured_image_size": { "width": 640, "height": 480 } } ] }
{ "camera_data": { "location_worldframe": [ -75.283897399902344, -618.38958740234375, 0 ], "quaternion_xyzw_worldframe": [ 0, 0, 0.66299998760223389, 0.74860000610351562 ] }, "objects": [ { "class": "", "instance_id": 15578836, "visibility": 1, "location": [ 9.9271001815795898, 40, 621.746826171875 ], "quaternion_xyzw": [ 0, -0.66299998760223389, 0, 0.74860000610351562 ], "pose_transform": [ [ -0.99269998073577881, 0, 0.120899997651577, 0 ], [ 0.120899997651577, 0, 0.99269998073577881, 0 ], [ 0, -1, 0, 0 ], [ 9.9271001815795898, 40, 621.746826171875, 1 ] ], "cuboid_centroid": [ 10.677499771118164, -9.4661998748779297, 621.6553955078125 ], "projected_cuboid_centroid": [ 260.3970947265625, 250.80239868164062 ], "bounding_box": { "top_left": [ 222.77049255371094, 248.47850036621094 ], "bottom_right": [ 279.02569580078125, 271.57791137695312 ] }, "cuboid": [ [ -11.595800399780273, -58.982799530029297, 650.97088623046875 ], [ -17.978900909423828, -58.982799530029297, 598.54022216796875 ], [ -17.978900909423828, 40.050399780273438, 598.54022216796875 ], [ -11.595800399780273, 40.050399780273438, 650.97088623046875 ], [ 39.333900451660156, -58.982799530029297, 644.7706298828125 ], [ 32.950901031494141, -58.982799530029297, 592.34002685546875 ], [ 32.950901031494141, 40.050399780273438, 592.34002685546875 ], [ 39.333900451660156, 40.050399780273438, 644.7706298828125 ] ], "projected_cuboid": [ [ 251.43980407714844, 225.07260131835938 ], [ 248.310302734375, 222.36349487304688 ], [ 248.310302734375, 278.83981323242188 ], [ 251.43980407714844, 277.00021362304688 ], [ 271.6171875, 224.77520751953125 ], [ 270.24090576171875, 222.01139831542969 ], [ 270.24090576171875, 279.07888793945312 ], [ 271.6171875, 277.20208740234375 ] ] } ] }
so K matrix is [[320,0,320], [0,320,240], [0,0,1]]
suppose a is rotation matrix from quaternion (xyzw) ([ 0, 0, 0.66299998760223389, 0.74860000610351562 ])
R matrix is a transposed
T matrix is -a transposed multiplied by translation matrix ([[-75.283897399902344, -618.38958740234375, 0]]) transposed
Then end result would be K [R|T] multiplied by [ 10.677499771118164, -9.4661998748779297, 621.6553955078125 ].
This method gives me the wrong location. Where did I go wrong?
Hi,
In the Synthetically Trained Neural Networks for LearningHuman-Readable Plans from Real-World Demonstrations paper, how did you guys exactly generate ground truth for relationship inference?
Does NDDS provide an algorithm where when producing the synthetic image of multiple cubes, it gives ground truth telling which cube is above or left of another cube? Or do we have to write our own algorithm for ground truth?
Hey,Great work!
I would like to ask if it is possible to export the coordinates of any point on the object model? For example, mustard_bottle in the YCB model. Can I select the top point of the bottle and export its coordinates? Thanks in advance!
Hey,
I am currently trying to get the bounding box of an item that a person is wearing. This person's skeleton is animated and has the item attached to its head (or hand, ...) with the help of a socket so that the item moves with the animation. My problem is that I can only attach the NVCapturableActorTag (which produces the bounding box) to the whole actor that is the person. But I am not able to attach the NVCapturableActorTag to a subpart of the skeletal mesh. When I try to move it there it tells me: "The selected component is not a scene component and cannot be attached to other components"
I have already tried to generate a moving actor that moves in a similar way like the person does and then place at the right location but this won't give me sufficiently good results.
Is there a solution or a better workaround for this problem?
Thank you for your help!
I'm wondering if there's a Unity version of the dataset synthesizer anywhere or if there's any plans to port this one to Unity.
Hey folks, so I had been using 4.21, which works well. I wanted to migrate to 4.22 to use ray tracing. That's when I started to encounter problems. So, I recently started from scratch in 4.22, and for whatever reason, the "testscapturer" cannot capture any data. Please help!
Hey guys, thanks for making this public. I was wondering about visibility in meta data (for example: 0000.json file). In this video , you guys explained that visibility = 0 is fully visible, and 1 is completely occluded. But in FAT dataset, it is kinda opposite.
Which one is right? All the data I am exporting have visibility set to 0. How can I change this value?
I have a question about the Random Background Manager. Instead of using the default materials (wood and so on) and textures as background I would like to use my own pictures. I managed to change the texture folder under RandomMaterialParam_Texture and now my Pictures are in the mix of backgrounds. But the default materials are also visible, which I would like to change. I tried deleting all of them, but apperently this does not seem to work since only one default material is showing and nothing else. How do I fix this?
Thanks in advance and thanks for your great work!
when i compared with fat data set and my custom made data set i figured out following differences
@swarren @aaronp24 @lukeyeager @drossetti @narendasan
coud you please help in solving this problem!
i encountered with following error in dope train.py code
[(https://github.com/NVlabs/Deep_Object_Pose/issues/97#issuecomment-583229903)]
I have multiple boxes in my scene that move and rotate randomly. Every box has a CapturableActorTag, but the class segmentation doesn't contain all of the visible boxes.
I attached some example pictures below. The box in the upper left corner can be seen in the depth and the instance segmentation but not in class segmentation
Great work for sythesize dataset, but i just have several confusion in mind.
(1) would it be possible to provide a scprits for automation synthesize ?
(2) if it possile to qutomation synthesize image and annotation, should we place several rule for image sythesis ?
Thanks, very much. Looking forward your reply !
Hey,
I have a problem with the visibility score in the annotation files. I changed the size of the objects (hand_cubes) I want to track but the projected_cuboid size won't change as in the picture below.
Even tho the 2D bbox will be displayed right I believe that the visibility score gets calculated from the cuboid parameters and because of that I can't filter out my 2D bbox below a certain threshold (0.5 or sth.). This results in bad images I which I can't use for training.
Is there a way to get the visibility score or the cuboid parameters right?
Thanks in advance!
I try to create a dataset for my deep learning neural network. For this reason I need a bunch of photos of different perspectives of my object. What I want to do is a complete 360 degree rotation around the object with the camera. How is this possible? I already tried the components RandomLookAt and RandomMovement and RandomRotation, but it is not working.
Hi,
I am trying to create a simple synthetic cube/cuboid placed in an environment(level) and generate metadata which will give coordinates of the vertices of the cuboid. I am following the steps mentioned in the pdf which can be found on this Ndds.pdf.
The problem is I'm unable to find the NVSceneCapturerGameMode_BP game logic through the UE4(Unreal Engine) but I can find it locally through the ubuntu file explorer. I've tried both the steps mentioned in last section of page 8 of the pdf but am unable to find it. The same problem with ScreenCapturer_Simple(mentioned on page 10 of pdf) which I can find through ubuntu file explorer but not through UE4.
Has anyone else faced the same problem? I'm not able to understand why UE4 can't recognise it.
I've been trying to figure out how the color for the background is chosen in the rendering of the Instance Segmentation output, but I find myself going in circles. I can see how they are chosen for the actors based on the MaskID, but I have been unable to find where the color for the background is specified. Any info you could provide would be much appreciated.
Is the axis in the object model transformed before the ground truth pose is printed in the annotation file? I observed that in some cases when the object model had a coordinate frame with z pointing up, the ground truth pose annotation has a different orientation.
Thank you really much for your fast reply!
Is it also possible to download objects that are not in your list? Or how would you proceed with a object that is not in your list?
I specifically need the 012_strawberry object.
Originally posted by @bjbraun in #24 (comment)
This is probably an easy question, but I'm not an expert with Unreal and I've spent a bunch of time trying to figure this out.
I'm trying to create synthetic images for my own object. So I followed the Creating a new Domain Randomization map tutorial section of the NDDS Documentation pdf. Everything seemed to work fine, but my training object always renders as a green color. I can't figure out how to change the color.
things I've tried:
Content/Blueprint/TrainingObjects/Test_AnnotatedActor_BP
and changing Static Mesh to a different objectI've added another level to the demo but it's unloaded when I hit play. Is there a way to keep the level I've added? I'm trying to replicate the realistic image production instead of the random background and I'm unsure how to proceed.
Hi!
How can I prevent this class "SMRailrig_Mount" from being tracked? I have no idea what this is...
This is a part of the output generated by my SceneCapturer.
Thank you.
{ "camera_data": { "location_worldframe": [ -720, 10, 150 ], "quaternion_xyzw_worldframe": [ 0, 0, 0, 1 ] }, "objects": [ { "class": "SM_RailRig_Mount", "instance_id": 16268800, "visibility": 1, "location": [ -203.99249267578125, -199.49519348144531, 182.39869689941406 ], ... }
I am currently doing a Master Project on Bin Picking of 3D objects, using an Aubo collaborative Robot.
Currently i am trying Nvidia-DOPE library for Object Detection and Pose Esimation for custom domains. I make use of UE4 Engine for data set generation.
can you share your approach that done in; that the 3d model created can randomly rotated in 3d inside the workspace.
i loaded my 3d model inside UE4 , but failed to give random 360 movement for the model that is done in blender(fbx format).
Hi ,
Thank you for providing this tool to us,
I would like to use MaskRCNN and train the network with the data I generate from NDDS.
MaskRCNN needs the polygon like labels for the training and validation. when I create a dataset using NDDS, I cannot see polygon labels for my training objects, is there any feature that can help me to also write polygon labels of objects to .json files ?
Thank you very much
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.