jarvis-mocap / jarvis-acquisitiontool

AcquisitionTool to record multi-camera recordings for the JARVIS 3D Markerless Pose Estimation Toolbox

JARVIS - 3D markerless motion capture made easy

JARVIS (Joint Acquisition, Recording and Voxel based Inference System) aims to make neural network based 3D markerless pose estimation easy. From calibrating your camera system to final 3D predictions, we to provide easy to use software for every step along the way.

• Our AcquisitionTool allows you to record synchronised videos from multiple views at high FPS thanks to GPU accelerated online JPEG compression.
• The AnnotationTool is a convenient way to extract and annotate training sets for our state of the art 3D pose estimation network.
• HybridNet is our hybrid 2D- and 3D-CNN based network architecture that enables highly precise markerless motion capture - even in scenarios with heavy occlusion and complex natural environments.

Check out https://jarvis-mocap.github.io/jarvis-docs/ for more information about designing a 3D motion capture setup and our Getting Started Guide.

Contact

The toolbox was developed at the Neurobiology Lab of the German Primate Center (DPZ). If you have any questions or other inquiries related to JARVIS please contact:

Timo Hüser - @hueser_timo - [email protected]

jarvis-acquisitiontool's Issues

Error during cmake of Qt6

CMake Warning at /home/pruszynskilab/Desktop/JARVIS-AcquisitionTool/libs/Qt/build/qtbase/lib/cmake/Qt6/Qt6Config.cmake:213 (find_package):
Found package configuration file:

/home/pruszynskilab/Desktop/JARVIS-AcquisitionTool/libs/Qt/build/qtbase/lib/cmake/Qt6Core/Qt6CoreConfig.cmake

but it set Qt6Core_FOUND to FALSE so package "Qt6Core" is considered to be
NOT FOUND.
Call Stack (most recent call first):
qtimageformats/CMakeLists.txt:13 (find_package)

-- Configuring incomplete, errors occurred!
See also "/home/pruszynskilab/Desktop/JARVIS-AcquisitionTool/libs/Qt/build/CMakeFiles/CMakeOutput.log".
See also "/home/pruszynskilab/Desktop/JARVIS-AcquisitionTool/libs/Qt/build/CMakeFiles/CMakeError.log".
CMake Error at /home/pruszynskilab/Desktop/JARVIS-AcquisitionTool/libs/Qt/qt/qtbase/cmake/QtProcessConfigureArgs.cmake:963 (message):
CMake exited with code 1.

make: Makefile: No such file or directory
make: *** No rule to make target 'Makefile'. Stop.
CMake Error: Error processing file: /home/pruszynskilab/Desktop/JARVIS-AcquisitionTool/libs/Qt/build/cmake_install.cmake

Trigger arduino not working

Hi,

I am trying to set up JARVIS!
After following the current installation procedure including the installation of arduino firmare using PlatformIO, and wiring up the arduino upon start of the recording the arduino line6 does not emit any trigger pulses. I tried to upload myself on the arduino an old version of the trigger firmware and the recording starts but automatically stops after a very short time (around 1-2 minutes).
I am using a Blackfly S camera and changed acquisition settings as mentioned in the instructions!

Thanks for your help and this very nice repo!

Digitial input on FLIR cameras with AcquisitionTool

Is there any functionality built in to AcquisitionTool to read/save signals from the digital input on FLIR cameras? e.g. for syncing data with other acquisition devices? If so, where is that information saved?

DIO documentation for FLIR cameras here. and here

Missing triggerdata.csv

Hi Timo,
We have been using the JARVIS Acquisition tool for some time now (6x FLIR Blackfly as we mentioned in issues in the past) and were able to use it successfully. However, with some recent recordings, we have an aberrant behaviour of the tool. It appears that the number of frames in the video does not match the number of frames from the metdata.csv and neither do either of these match the number of frames that we measure using exposure active pulses from the cameras. The main thing that appears to be predictive of this behaviour was a missing triggerdata.csv file in the recordings that had that and they all are from a single day of recordings. The other strange thing we have not observed before is that in the metadata file, the cameras do not start from frame 0, rather they all start from a different number and all end at a different number (using frame ID).
Running it again now does not reproduce this issue.
We have attached below an example metadata file. We were wondering if given this behavior you could help us diagnose the problem, help us avoid it in the future and if there was some possibility to salvage the current recordings.

metadata.csv

Thanks a lot.

Triggering new recordings?

If I want to record for an extended period of time, is there any way to trigger a new recording at a given time interval (e.g. every n hours)?

arduino acquisition sometimes captures no frames

It's a bit hard to describe this bug because I don't know exactly what's going on.

Sometimes when starting AcquisitionTool up, I am unable to get the arduino acquisition working. Terminal says it is starting acquisition, but no frames are captured. When I switch over to non-arduino acquisition, the cameras work fine. To fix the problem, I unplug all of the USBs as well as the arduino, reboot the computer, and re-create the camera settings defaults. I haven't been able to localize the problem to any one component so I just reinitialize them all.

I also have had trouble with the cameras continuing to acquire after pressing pause or stop, which freezes the GUI. Maybe these two are related, and the arduino gets stuck in a loop and needs to be rebooted to communicate with the GUI again.

Difficulty installing Jarvis Toolbox

Hello,

I'm following the installation guide on a Windows 11 PC, and I'm sure I'm missing something simple, however I'm getting stuck. May I please ask for help?

I'm following this guide: https://jarvis-mocap.github.io/jarvis-docs/getting_started/2_exploring_example/

Step 1: The git clone command in Anaconda Powershell is breaking for me.

(base) PS C:\Users\soline> git clone https://github.com/JARVIS-MoCap/JARVIS-HybridNet.git && cd JARVIS-HybridNet
At line:1 char:64

• ... clone https://github.com/JARVIS-MoCap/JARVIS-HybridNet.git && cd JARV ...

•                                                            ~~
  

The token '&&' is not a valid statement separator in this version.
+ CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : InvalidEndOfLine

(base) PS C:\Users\soline>

Step 1 (Alternate): I downloaded the .zip file in the link into my Downloads folder, and unzipped the file.

Step 2a: Create the jarvis Anaconda environment by running:
conda create -n jarvis python=3.9 pytorch=1.10.1 torchvision cudatoolkit=11.3 notebook -c pytorch
[completed]
Step 2b: Activate the environment (you will need to do this every time you open a terminal to use JARVIS):
[completed]
Step 2c: conda activate jarvis
[completed]
Step2d: Install the required version of the setuptools package:
pip install -U setuptools==59.5.0
[completed]
Step2e: Install JARVIS:
pip install -e .
[Breaks here]

(jarvis) PS C:\Users\soline> pip install -e .
Obtaining file:///C:/Users/soline
ERROR: file:///C:/Users/soline does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.
(jarvis) PS C:\Users\soline>

I have a feeling that this may have something to do with attaching the downloaded, unzipped folder to Anaconda, but I don't have the foundation to proceed. May I please ask for help?

Thanks so much!

Multicamera acquisition

Hi Timo,

We are still working to get the system up to speed for six cameras.
Just to recap: we have 6 BFS-U3-32S4M cameras and a computer with an i7-9800X, an RTX 3080, an M2 SSD, 2PCiE Fresco 4xUSB cards.
All below is doe with
We did some progressive testing and were able to record 4 cameras at fullres, JPEG95 compression and we got synchronized videos of the same n of frames. Here we always get the error below error in the command window upon start. Additionally, sometimes upon stop of recording the acquisition tool crashes.

Could not create/open file:
Metadata filepath: C:/ProgramData/JARVIS-Mocap-Acquisition/Data/Test4cam/Test_2/test6_bin2/metadata.csv

With 5 or 6 cameras we are not able to record at full res without the tool crashing immediately at start. Upon binning we can occasionally record with 5 or 6 cameras but either the tool crashes at start time or when we stop the recordings.

Finally and probably most imporantly, we tried to compress the data more (JPEG50) and the acquisition starts with 6 cameras at full resolution (same error as above) and works (tested up to 10 minutes more or less) but upon stopping the AcqTool freezes and the command window shows:

Trying to stop Acquisition
stopped Worker
After 30mins of frozen screen we stopped the AcqTool and were able to open the video files that have insanely different number of frames (50k one camera, 14k other camera)

Sorry for the multilayered issue! Let us know what you think about these issues and thanks a lot for all the help and the nice repo and software!

Acquisition tool not starting with plugged camera

Hi,
We have this camera BFSU3-16S2M-cs from FLIR and were trying to set up the jarvis acquisition tool.
When we don't have any camera connected the tool starts normally but when the camera is plugged the software does not start anymore.
Of note if we launch spinnaker and try to start recording the software fails to grab images or stream.
Let us know if you have any clue on how to troubleshoot this.

Basler camera support?

Hi,
Are you by any chance planning to implement support for Basler Ace2 cameras ?
( https://www.baslerweb.com/en/products/cameras/area-scan-cameras/ace2/a2a1920-160umpro/ )
If so what would be the expected timeline?
Otherwise do you think it would be feasible and or very time-consuming ?
Thanks a lot in advance!

Strategy for calibration?

Hello,

I'm having difficulty running calibration with my camera setup. I have 7 cameras, and two of them are pointed inward in such a way that they rarely / never see the charuco board at the same time. It seems like this is causing problems, but maybe it's related to how I'm setting up the pairs?

In my diagram, the two red circled cameras are the ones highlighted in the error. I used the blue circled camera as the primary, and so every pair is the blue circled cam and one of the other cameras.

Should I use a different strategy when setting up my inputs? I see that in the guide, one of the pairs is actually a set of three, but it's not clear to me which cameras should be selected for that, or if it's even necessary.

Thanks for your help!
Stefan

CMake Error: Error: generator : Ninja

if you try to install the Acquisition tool without deleting this file it won't work
CMakeLists.txt on the root

Error :
CMake Error: Error: generator : Ninja
Does not match the generator used previously: NMake Makefiles
Either remove the CMakeCache.txt file and CMakeFiles directory or choose a different binary directory.

Support for Ubuntu 22.04

Hi,
we have a linux workstation that we would like to try for acquisition, but it has an Ubuntu 22.04 installation and we cannot currently downgrade it given some other restricitons of software on the computer. Is there any plan for compatability with Ubuntu 22.04 for the acquistion tool?

Thanks.

Error on 'qdbusmessage.h' when building under Ubuntu 20.04

I tried several times with 'master' branch and 'v1.1' tag, but failed to build under my Ubuntu machine,

Errors occurred :

[ 97%] Building CXX object CMakeFiles/AcquisitionTool.dir/main.cpp.o
[ 98%] Building CXX object CMakeFiles/AcquisitionTool.dir/AcquisitionTool_autogen/E47P2AXCJR/qrc_darkstyle.cpp.o
In file included from /opt/spinnaker/include/SpinGenApi/Node.h:22,
from /opt/spinnaker/include/SpinGenApi/SpinnakerGenApi.h:21,
from /home/master_node/src/_Git/JARVIS-AcquisitionTool/src/flirconfigbackend.hpp:15,
from /home/master_node/src/_Git/JARVIS-AcquisitionTool/main.cpp:12:
/home/master_node/src/_Git/JARVIS-AcquisitionTool/libs/Qt/qt_install/include/QtDBus/qdbusmessage.h:76:74: error: expected ‘,’ or ‘...’ before ‘struct’
76 | static QDBusMessage createSignal(const QString &path, const QString &interface,
| ^~~~~~~~~
/home/master_node/src/_Git/JARVIS-AcquisitionTool/libs/Qt/qt_install/include/QtDBus/qdbusmessage.h:79:61: error: expected ‘,’ or ‘...’ before ‘struct’
79 | const QString &interface, const QString &name);
| ^~~~~~~~~
/home/master_node/src/_Git/JARVIS-AcquisitionTool/libs/Qt/qt_install/include/QtDBus/qdbusmessage.h:81:57: error: expected ‘,’ or ‘...’ before ‘struct’
81 | const QString &interface, const QString &method);
| ^~~~~~~~~
In file included from /home/master_node/src/_Git/JARVIS-AcquisitionTool/libs/Qt/qt_install/include/QtDBus/qdbusabstractinterface.h:49,
from /home/master_node/src/_Git/JARVIS-AcquisitionTool/libs/Qt/qt_install/include/QtDBus/QtDBus:6,
from /home/master_node/src/_Git/JARVIS-AcquisitionTool/libs/Qt/qt_install/include/QtGui/QtGuiDepends:4,
from /home/master_node/src/_Git/JARVIS-AcquisitionTool/libs/Qt/qt_install/include/QtGui/QtGui:3,
from /home/master_node/src/_Git/JARVIS-AcquisitionTool/libs/Qt/qt_install/include/QtCharts/QtChartsDepends:4,
from /home/master_node/src/_Git/JARVIS-AcquisitionTool/libs/Qt/qt_install/include/QtCharts/QtCharts:3,
from /home/master_node/src/_Git/JARVIS-AcquisitionTool/gui/acquisitionMode/latencychartview.hpp:17,
from /home/master_node/src/_Git/JARVIS-AcquisitionTool/gui/acquisitionMode/monitoringwindow.hpp:15,
from /home/master_node/src/_Git/JARVIS-AcquisitionTool/gui/acquisitionMode/acquisitionmode.hpp:18,
from /home/master_node/src/_Git/JARVIS-AcquisitionTool/gui/acquisitionMode/../mainwindow.hpp:13,
from /home/master_node/src/_Git/JARVIS-AcquisitionTool/main.cpp:14:
/home/master_node/src/_Git/JARVIS-AcquisitionTool/libs/Qt/qt_install/include/QtDBus/qdbusmessage.h:99:22: error: expected identifier before ‘(’ token
99 | QString interface() const;
| ^
/home/master_node/src/_Git/JARVIS-AcquisitionTool/libs/Qt/qt_install/include/QtDBus/qdbusmessage.h:99:23: error: expected unqualified-id before ‘)’ token
99 | QString interface() const;
| ^
make[2]: *** [CMakeFiles/AcquisitionTool.dir/build.make:120: CMakeFiles/AcquisitionTool.dir/main.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:201: CMakeFiles/AcquisitionTool.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

How can I fix this ?

• My build environment:

• OS : Ubuntu 20.04.5 LTS (kernel 5.15.0-67)
• CMake version: 3.24.4

Thanks,

small bug with ffmpeg videoio lib

there is a small bug (current upstream opencv ffmpeg issue) where compilation of acquisition tool gets hung up.

all that needs to be changed is add
#include <libavcodec/version.h> to JARVIS-Acquisitiontool/libs/OpenCV/opencv/modules/videoio/src/ffmpeg_codecs.hpp

Thanks

Jan

AcquisitionTool: error while loading shared libraries: libopencv_videoio.so.4.5: cannot open shared object file

I am installing the prebuilt version of acquisitiontool on a new version of ubuntu 20.04. I run into this error:

AcquisitionTool: error while loading shared libraries: libopencv_videoio.so.4.5: cannot open shared object file: No such file or directory

Bulding on freshly installed ubuntu

Hey,

I added this to the beginning my cmake file:

set(UBUNTU_VERSION "2004")

since it was not set by my system, and I'm currently trying to build the project.

I also installed this version of spinnaker: https://flir.app.boxcn.net/v/SpinnakerSDK/file/941078493472
-> spinnaker-2.6.0.157-Ubuntu20.04-amd64-pkg.tar.gz

And it builds, but when I try to run it, I get this shared library error.

/home/smg/git/JARVIS-AcquisitionTool/build/AcquisitionTool: error while loading shared libraries: libGenApi_gcc540_v3_0.so: cannot open shared object file: No such file or directory

From what I cat tell its a file that gets installed alongside Spinnaker, and it is present in /opt/spinnaker/lib/libGenApi_gcc9_v3_0.so which is added to the system path by /etc/ld.so.conf.d/libspinnaker.conf

Did you had this kind of error before?

Additionally, is there a particular reason why spinnaker is included like this in the repo?