normanderwan / arucounity Goto Github PK

Hi! I'm trying to follow this Calibrate Camera documentation here: https://normanderwan.github.io/ArucoUnity/manual/calibrate-a-camera.html
and underneath fig.2 it says "Configure the calibration board object by measuring the one you just printed : units are in meters this time. The ArucoObjectDisplayer helps you visualize the configured board to see if it's match with the printed one. Don't switch x and y values." but i can't change any of the values on the ArucoObjectDisplayerImagePlane object at all. I tried changing the ArucoCharucoBoard object as well but it doesn't change as well. I might be completely misunderstanding so if someone could clear it up for me it'd be really appreciated!

The following error happens when trying to create my own marker following the tutorial, opening the scene CreateMarkers and clicking play:

Exception: OpenCV(3.4.2) c:\projects\arucounityplugin\build\opencv\opencvcontrib-prefix\src\opencvcontrib\modules\aruco\src\dictionary.cpp:169: error: (-215:Assertion failed) id < bytesList.rows in function 'cv::aruco::Dictionary::drawMarker'

ArucoUnity.Plugin.Cv+Exception.Check () (at Assets/ArucoUnity/Scripts/Plugin/Cv/Exception.cs:64)
ArucoUnity.Plugin.Aruco+Dictionary.DrawMarker (System.Int32 id, System.Int32 sidePixels, ArucoUnity.Plugin.Cv+Mat& img, System.Int32 borderBits) (at Assets/ArucoUnity/Scripts/Plugin/Aruco/Dictionary.cs:126)
ArucoUnity.Objects.ArucoMarker.Draw () (at Assets/ArucoUnity/Scripts/Objects/ArucoMarker.cs:44)
ArucoUnity.Objects.Displayers.ArucoObjectDisplayer.CreateImage () (at Assets/ArucoUnity/Scripts/Objects/Displayers/ArucoObjectDisplayer.cs:155)
ArucoUnity.Objects.Displayers.ArucoObjectDisplayer.UpdateImage () (at Assets/ArucoUnity/Scripts/Objects/Displayers/ArucoObjectDisplayer.cs:207)
ArucoUnity.Objects.Displayers.ArucoObjectCreator.UpdateImage () (at Assets/ArucoUnity/Scripts/Objects/Displayers/ArucoObjectCreator.cs:63)
ArucoUnity.Objects.Displayers.ArucoObjectDisplayer.Update () (at Assets/ArucoUnity/Scripts/Objects/Displayers/ArucoObjectDisplayer.cs:112)

Thanks for awesome plugin. Can it be used for mobile phone (android)?

Hi,
Thanks for this code allowing quick implementation and testing of ArUco.
Do you have any plan to support ARM64 or a workaround ? When generating the projet using "Visual Studio 16 2019" -A ARM64, I'm annoyed with the inclusion of emmintrin.h which is dedicated to X86/X64, being able to get the NEON alternative directly would be great :)

First off, i'd like to say: wow it works. I've spent weeks toiling at this issue and this plugins saves me. However i have one little issue.

in this image you can see that i don't recieve a great camera image. it works. but i'd like something more... normal.
i used the pinhole calibration prefab for my integrated webcam. Should i have used something else?
after this i'll try to use the plugin on a HoloLens2. i think stereoscopic would be my best bet for that?

Actually, only Windows x64 is supported as a C wrapper (the "plugin") is necessary to call the algorithms used from OpenCV.

The plugin should be build also for Windows x86, Linux, Mac, Android, iOS, UWP.

An alternative would be to replace to library with one of the following open-source project:

• EmguCV
• OpenCvSharp
• OpenCV.NET

Regarding the Method Aruco.GridBoard.Create, it seems that we should be able to start the dictionnary from a given marker index to create different grid boards.

However, when using it with something different than 0, the grid board always begins with the marker 0.

Maybe is it just an oversight inside the native dll.

I have a project that requires developing an AR application. Right now, I am trying to setup that AR environment in Unity to detect a marker that is 10mm x 10mm in real-life size. I have tried Vuforia, EasyAR and AR foundation to detect the marker, but it was not possible. However, if I changed the size to be 20 x 20 mm then they will start detecting it easily, but I am required to use 10x10 mm. I was wondering if I can use OpenCV and ArUco with Unity to detect markers of this size.

First of all, thanks for this library. I'm very excited to give it a try. I'm trying to build on windows 10 following the direction from here. The first step completes successfully, but it fails on the second step. Below are the commands I've used and the output.

$ cd ArucoUnity\ArucoUnityPlugin\build\opencv
$ cmake -DCMAKE_INSTALL_PREFIX=install/ -G "Visual Studio 15 2017 Win64" ..\..\3rdparty\opencv_contrib
$ cmake --build . --config Release

$ cd ArucoUnity\ArucoUnityPlugin\build
$ cmake -DCMAKE_INSTALL_PREFIX=.. -DOpenCV_DIR=../../build/opencv/install/ - "Visual Studio 15 2017 Win64" ..

Output

-- OpenCV ARCH: x86
-- OpenCV RUNTIME: vc14
-- OpenCV STATIC: OFF
CMake Warning at build/opencv/install/OpenCVConfig.cmake:156 (message):
  Found OpenCV Windows Pack but it has no binaries compatible with your
  configuration.

  You should manually point CMake variable OpenCV_DIR to your build of OpenCV
  library.
Call Stack (most recent call first):
  CMakeLists.txt:31 (find_package)


CMake Error at CMakeLists.txt:31 (find_package):
  Found package configuration file:

    ArucoUnity/ArucoUnityPlugin/build/opencv/install/OpenCVConfig.cmake

  but it set OpenCV_FOUND to FALSE so package "OpenCV" is considered to be
  NOT FOUND.


-- Configuring incomplete, errors occurred!
See also "/ArucoUnity/ArucoUnityPlugin/build/CMakeFiles/CMakeOutput.log".

Hi, I'm starting to use ArucoUnity to try and create a basic AR app but I have noticed an issue during the calibration process.
I have been using a plugin USB webcam attached to my PC initially to begin to understand how the system works and I have run through the calibration process and marker detection using it with no problems. However my app will need to run as a standalone build on a windows tablet so I have started to use a laptop that has a built in webcam to check that the same process works on hardware with built in cameras and I have found an issue. The laptop's camera is able to work at a maximum resolution of 1920 x 1080 (the same as the plugin webcam) but the calibration process is running at 640 x 480 when I read through the calibration output XML file. Is there a setting I'm missing to set the calibration resolution in ArucoUnity? I'm using a HP ZBook running Windows 10 and I don't see any Windows hardware settings for the camera. Thank you in advance for any help you maybe able to provide.

Thanks for integrating this tool into UNITY3D!
Recently I was on the ARrUko website.
They have released a very cool update.
Now markers will be recognized even if the camera twitches and the image is blurry.

Could you add this new feature to your unit plugin?
Thank!

This feature named as DCF.
LINK TO ArUco DCF Example

News: Aruco Nano version 2 released (17/07/2020). A single file for 95% of the users
News: New Aruco Version of the library available. Tracking with Correlation Filters!

Hi,

I'm running the tracking scene and the marker detection part goes fine. The aruco axis are shown in a right way. However, the game object has problems with translation and rotation. When I move the webcam, the marker is tracked right but the game object move is wrong. I think the problem is in the method PlaceArucoObject from the class ArucoObjectTracker. Has anybody found similar problem? How did you solve it?

Thank in advacen,

I have not reproduced this error on a PC but the problem should still apply.

I conjecture this is due to EstimateTransforms running async to UpdateTransforms, thus changing the native data while UpdateTransforms is running.

good morning, I was wondering if with unity it is possible to create apps for android or ios platforms? Maybe by updating the plugin?

Hello so im currently playing with ArucoUnity. However for some unknown reason I dont have same resolt as in tutorial. My resolt is wierdly fisheye or a some high FOV Value. I tryed to google but i founded only how to calibrate with fisheye lens (whitch i dont have.) Image here Im really not soure what is going on here; at calibration the image was normal. Thanks for your advise!

Hello!

I'm a graphic designer and I like to support open source projects. I would like to design a logo for your project if you interested, I will be happy to work with you! :)

Best Regards

Baran Pirinçal

Graphic Designer

Does it support ios platform? Thanks!

https://docs.unity3d.com/Packages/[email protected]/manual/index.html

When I build the app (platform:windows), i only get a blackscreen and no connecting to my webcam

How can I get the real distance between two Aruco Markers?

when I use for kinect v2, only two mark work,and calibrate fail

Hi,
I'm experiencing an issue while compiling a project with VSCode, where the required ArucoUnity.dll is not found when building for the ARM64 architecture. The package is located in a folder named "x86". I need ARM64 to execute on Hololens application to complete a assignement of Univerity course. Please help me

Anyone tried to build the samples on tablets or phones? I tried *exe on windows Surface and *apk on Android, but they both failed to activate the camera. My xml file paths should be correct, so there must be some other reasons.

Any help would be appreciated.

Cheers,
Johnny

I get this error when I press the calibrate button in the calibrate scene.

I am using the 2.0 alpha of the build

Exception: OpenCV(4.0.0-pre) C:\Users\user\Documents\p4p\ArucoUnity\ArucoUnityPlugin\build\opencv\OpenCVAruco-prefix\src\OpenCVAruco\modules\calib3d\src\calibration.cpp:3139: error: (-210:Unsupported format or combination of formats) objectPoints should contain vector of vectors of points of type Point3f in function 'cv::collectCalibrationData'


ArucoUnity.Calibration.ArucoCameraCalibration.LateUpdate () (at Assets/third-party/ArucoUnity/Scripts/Calibration/ArucoCameraCalibration.cs:145)

The error is below:

DllNotFoundException: ArucoUnity
ArucoUnity.Plugin.Aruco.GetPredefinedDictionary (PredefinedDictionaryName name) (at Assets/ArucoUnity/Scripts/Plugin/Aruco/ArucoUnity.cs:397)
ArucoUnity.Objects.ArucoObject.Awake () (at Assets/ArucoUnity/Scripts/Objects/ArucoObject.cs:102)

I used visual studio 2015 to build opencv library with contrib successfully. My problem is that I don't understand this part:

$ cd <aruco_unity_build_directory>
$ cmake -DCMAKE_INSTALL_PREFIX=<aruco_unity_install_directory> -DOpenCV_DIR=<opencv_install_directory> -G <generator-name> <aruco_unity_directory>
$ cmake --build . --config Release --target INSTALL

I can build INSTALL through visual studio and have all the library ready in <opencv>/bin and <opencv>/lib. But I don't understand where to put inside your ArucoUnity project.

Even more, based on the error message from unity, I think there should be a dll file called ArucoUnity.dll, which is not generated by opencv library as I think.

Looking forward to your reply. Thanks

Following the steps of the example scenes, I moved on for the most basic test.

Sadly I get this error playing the trackmarkers scene after 2 seconds every time:

Runtime Error!

Program:

This application has requested the Runtime to terminate it in a n unusual way. Please contact the applications support team for more information

An image:
https://i.imgur.com/S8W6LnV.jpg

Let me know if my project files or any log can be of help. All I changed in the scene is the calibration path to the one generated in the calibration scene, and the aruco marker ID.

Edit: Using Unity 2019.1.1 (which might be a reason).

Hi! awesome work here

I've been using it but it seems that when you have an aruco board being detected and an aruco marker at the same time, if you move the markers quickly or have one occlude the other. it will cause an out of bounds exception in the c++ code and hard crash Unity.

I tested this by building your source as well and the issue is presented there as well, and in the Tracking Example I set up 2 markers ( id 49 and id 1) to be tracked. and I set up a 7x5 Aruco board. I then quickly shook the board and markers to make them tracked and not tracked as well as put one in front of the other etc...

I tried my best to look into it but I wasn't entirely sure about how the project was laid out. I did find some things that might help you figure out the issue.

More specifically it looks like in vector_vec3d.cpp you have

try {
    element = &(vector->at(pos));
} catch (cv::Exception& e){...}

but the position pos is usually greater than vector.size(); when the exception occurs... Since its a std exception and not an openCV exception it will not get caught. However, it still crashes when adding a std::exception catch statement there because it might be returning a null pointer to wherever this result gets used.

Additionally

the functions in vector_vec3d.cpp get called in VectorVec3d.cs I tried looking into it but I couldn't figure out why it was breaking since it also crashes unity as soon as it hits here. However; I think I narrowed it down to the function public Cv.Vec3d At(uint pos) being the entry point into the error. I think when it is used in the UpdateTransforms cycle you have it tried to access a point in a vector that doesn't exist.

Also: I'm on windows using Unity 2018.3.0f2 and OpenCV from your CMake file
Let me know if someone already brought this issue up!
Thanks.

Instead of using the slow remap function on CPU, it would be better to undistort the camera image with a shader on GPU as Steptoe did with its AR-Rift : http://willsteptoe.com/post/67401705548/ar-rift-aligning-tracking-and-video-spaces-part for pinhole cameras

To shaders are needed : one for pinhole undistortion, and one for omnidir undistortion.

Hi,

Hope you are all well !

I am actually working on some C++ libraries, like Dlib or OpenCV, and I wanted to auto generate unsafe bindings for some computer vision libraries to build native plugins for Unity3D.

After a couple of researches, I found your great work and some questions popped out of my brain :-)

Context:
I wanted to test this example, dnn_mmod_dog_hipsterizer.cpp, from dlib, in order unity3d to leverage the WebCamTexture class. I saw a module in the asset store using dlib but I wanted to test ccv and some other CMake driven projects. So, I am looking for something scriptable and more flexible for my studies.

So, in a nutshell, I would like to auto-generate bindings from some C++ libraries and allow interoperability and build these projects with CMake, with polly, a collection of toolchains to cross-platform build and test these projects..

I found CppSharp and still trying to find out the best way to get something, like FFmpeg.AutoGen, also using CppSharp.

Ultimately, I would like to create a boilerplate like this project testing cross-language interactions (link). To make it more flexible, I was thinking to wrap all these commands inside a docker container with onbuild instructions. Even if you cannot compile xcode projects from a docker container, it would be great for Android, Linux or Windows builds.

Questions:

bindings related:

1. How did you proceed for aruco-unity like Calib3d.cs ? Did you write all the bindings yourself ? (if yes, that's a lot of work :-) )

2. Did you try to use Swig or CppSharp for the aruco-unity project ?

3. How would you see a much more structured and semi-automated flow to build native modules, based on Cmake driven projects, for Unity3d ?

cmake related:

3. Would aruco-unity work for iOS or Android if we provide a cmake toolchain ?

Thanks in advance for your input on my quest :-)

Wish you a great week !

Cheers,
Richard

Hi, thanks for this great work!
I was wondering whether is it possible to "extend" the current methods for image acquisition.
In particular, I have a camera placed on a robot, which images are sent trough the network. Thus, they are not directly acquired from a camera connected to the the pc where unity is running. Is it possible to customize the way images are provided to the software?

I went trough the code, and I found the
Camera folder, which contains classes I might take as example for creating my own "camera" class. However, I was wondering whether there is an official documentation or standard method for achieving this.

Thanks in advance for your help!

In Mat's DataByte accessor the native uchar* object is being directly assigned/returned as a byte[] without marshalling it as a managed object first. This can cause issues such as this: https://issuetracker.unity3d.com/issues/crash-while-inspecting-byteslist-values-of-a-dictionary-in-vs-debugger-when-using-arucounity-plugin

I was able to work around the crash by doing this:

[DllImport("ArucoUnityPlugin")]
      static extern IntPtr au_cv_Mat_getData_uchar(IntPtr mat);

      public byte[] DataByte
      {
        get
        {
          byte[] bytes = new byte[Total()];
          IntPtr inputBuffer = au_cv_Mat_getData_uchar(CppPtr);
          Marshal.Copy(inputBuffer, bytes, 0, (int)Total());
          return bytes;
        }
        set { au_cv_Mat_setData_uchar(CppPtr, value); }
      }

Hope this helps! Thanks!

When I try to open the "TrackMarkers" Scene, add an Empty and add the ArucoCharucoBoard Script plus Aruco Object Displayer and also add it to the ArucoObjects in the ArucoObjectTracker, then I get the following two error messages:

ApplicationException: Mutex is not owned
System.Threading.Mutex.ReleaseMutex () (at <695d1cc93cca45069c528c15c9fdd749>:0)
(wrapper remoting-invoke-with-check) System.Threading.Mutex.ReleaseMutex()
ArucoUnity.Utilities.ArucoCameraSeparateThread.b__25_0 () (at Assets/ArucoUnity/Scripts/Utilities/ArucoCameraSeparateThread.cs:93)
System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) (at <695d1cc93cca45069c528c15c9fdd749>:0)
System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <695d1cc93cca45069c528c15c9fdd749>:0)
System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) (at <695d1cc93cca45069c528c15c9fdd749>:0)
System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) (at <695d1cc93cca45069c528c15c9fdd749>:0)
System.Threading.ThreadHelper.ThreadStart () (at <695d1cc93cca45069c528c15c9fdd749>:0)
UnityEngine.<>c:b__0_0(Object, UnhandledExceptionEventArgs)

NullReferenceException: Object reference not set to an instance of an object
ArucoUnity.Objects.Trackers.ArucoObjectsTracker.ArucoCamera_ImagesUpdated () (at Assets/ArucoUnity/Scripts/Objects/Trackers/ArucoObjectsTracker.cs:315)
ArucoUnity.Cameras.ArucoCamera.OnImagesUpdated () (at Assets/ArucoUnity/Scripts/Cameras/ArucoCamera.cs:177)
ArucoUnity.Cameras.ArucoCamera.Update () (at Assets/ArucoUnity/Scripts/Cameras/ArucoCamera.cs:67)

Is tracking a Charuco Board not supported?

Thanks!

when building and deploying to another device i get these errors:

can anyone help?

Hello, I'm starting in C# and on Unity. After importing the unitypackage and the asset I try to launch the calibration and I get this as error...

Do you have an idea how to solve it?
Thank you in advance
Paul

Hey there, I've tried to take a look at your current project and unfortunately ended up with this:

Another Question:
Have you tried to deploy it on a hololens yet?

regards dustin

Is this the correct address actually?:
https://normanderwan.github.io/ArucoUnity/manualbuild-from-sources.html

In calibration scene, unity project cannot find the xml file to open camera. And it is not optional. I checked all the commits in 2017-10-27 and failed to find the xml file. Can you commit it again or give an example? Thanks