mobileroboticsskoltech / opencamera-sensors

Android app for synchronized recording of video and IMU data with advanced camera options, useful for 3D reconstruction, SLAM, AR, video stabilization. Supports remote control over network.

OpenCamera Sensors is an Android application for synchronized recording of video and IMU data. It records sensor data (accelerometer, gyroscope, magnetometer) and video with frame timestamps synced to the same clock.

Install

Or get the latest APK from GitHub releases

Description

OpenCamera Sensors is an Android application for synchronized recording of video and IMU data. It records IMU data and video with frame timestamps synced to the same clock.

This project is based on Open Camera — a popular open-source camera application with flexibility in camera parameters settings, actively supported by the community. By regular merging of Open Camera updates our app will adapt to new smartphones and APIs — this is an advantage over the other video + IMU recording applications built from scratch for Camera2API.

Usage

• Go to preferences, enable Camera2API and press the “Enable sync video IMU recording” switch in "IMU settings..."
• (Optional) Disable video stabilization in video preferences of OpenCamera Sensors to minimize preprocessing effects
• (Optional) Enable save frames option if you want to verify recorded data correctness
• (Optional) Enable flash strobe and specify its frequency in additional sensor settings
• Switch to video, setup ISO and exposure time
• Record video
• Get data from DCIM/OpenCamera:
  • Video file
  • Sensor data and frame timestamps in the directory {VIDEO_DATE}: -{VIDEO_NAME}_gyro.csv, data format: X-data, Y-data, Z-data, timestamp (ns)
    • {VIDEO_NAME}_accel.csv, data format: X-data, Y-data, Z-data, timestamp (ns)
    • {VIDEO_NAME}_magnetic.csv, data format: X-data, Y-data, Z-data, timestamp (ns)
    • {VIDEO_NAME}_timestamps.csv, data format: timestamp (ns)
    • {VIDEO_NAME}_flash.csv, data format: timestamp (ns) (timestamps of when the flash fired)

Remote recording

• Connect smartphone to the same network as PC
• Use scripts provided in ./api_client/ directory to send requests for the application.
  • Note: phase, which is returned by start_recording method, can be used to perform synchronization with external devices

Synchronized recording on multiple smartphones (RecSync)

Important: smartphones are required to support real-time timestamping to be correctly synchronized. This can be checked on the preview message when RecSync is enabled ("Timestamp source" should be "realtime").

Leader smartphone setup:

• Start a Wi-Fi hotspot
• Open OpenCamera Sensors, go to preferences - "RecSync settings..." and enable the "Use RecSync" switch
• (Optional) Enable phase alignment option if synchronization precision better than half of a frame duration is required
• (Optional) Choose which camera settings will be broadcasted to client smartphones in the "Sync settings" section
• Switch to video, adjust the camera settings as needed and press the settings synchronization button
• Wait for client smartphones to connect if needed
• (Optional) If phase alignment was enabled, press the phase alignment button to start the alignment and wait for it to finish ("Phase error" on the preview indicates how much the current phase differs from the targeted one -- when it becomes green, the phase is considered aligned)
• Start a video recording

Client smartphones setup:

• Connect to the leader's Wi-Fi hotspot
• Open OpenCamera Sensors, go to preferences - "RecSync settings..." and enable the "Use RecSync" switch
• Adjust the camera settings as needed (the ones that will not be broadcast by the leader) and wait for the leader to start the recording

Note: the phase needs to be re-aligned before every recording.

Good practices for data recording

• When recording video with audio recording enabled, MediaRecorder adds extra frames to the video to match the sound. Due to this problem, the audio recording feature is disabled in our app by default.

• To minimize the amount of preprocessing done by the smartphone, we also disable video stabilization and OIS options.

Restrictions

One important restriction is that our app requires full Camera2API support.

Another restriction of our application is that synchronized timestamping for camera and IMU data isn’t available on all the devices with Camera2API support. You can check whether your device supports this feature in preferences.

Contribution

The project follows AOSP Java Code Style, main principles:

• Non-public fields should start with m, constants are ALL_CAPS_UNDERSCORES
• Standard brace style:

if () {
    //...
} else {
    //...
}

• Limit line length

opencamera-sensors's Issues

Get timestamps for each frame of video

Hello,

I am trying to record a video with synchronized IMU. I can get the IMU data with all timestamps but the application does not provide timestamps for each video frame.
Is it currently possible with the application ?

Thanks in advance !

Rename app to differ it from original Open Camera

failed to synchronized recording

I have downloaded the latest apk and set "Use RecSync" successfully. However, when I press the settings synchronization button, the leader phone try to boardcast the settings, then the leader camera goes black and it prompts that the video recording failed and that camera has a severe error. The only sync setting that is boardcasted is exposure time. The type of the leader phone and client phone is different. Does it matter?

not fully synchronized recording

I am able to perform synchronized recording after using another type phone. While I have found that the recorded videos are not fully synchronized. I have recorded two videos with two phones while the total frames of these two videos are different. One of the videos has a total frame count of 472 and the other has 469. I upload the video to the computer and use cv2 to read the total number of frames. Is this normal ? The two phones are different types. Will the results be more synchronized using two identical phones?

Asyncs in ExtendedAppInterface for integration with RecSync

Carefully revisit the current implementation of the interaction between sync and camera controller that uses async

Change remote sensor recording logic

• the app should restore all sensors as enabled on every restart
• the remote module shouldn't change the app settings state when asked for imu files specifically

F-Droid inclusion

Currently the app is available on izzy repo.
Would you consider putting this app on F-Droid, please?

RFP : https://gitlab.com/fdroid/rfp/-/issues/2129

Add option to write only gyroscope or accelerometer

In some applications, accelerometer writing can be excessive and create additional load on the OS.

Not an issue, but advice needed

Great application of an already great program. What application can I use to stabilize recorded video using the dumped gyro data?

Thanks

Move new string values to resources

Move strings with TODOs to resources

Add video frame timestamping

• This answer has some helpful info and links
• Find out if it's possible to record timestamp of every frame using this method

Add Android 11 feautures support, extend firebase test devices

• Update app to support scoped storage

Avoid usage of hide API in AP detection

Current implementation could be found here 430daca

Local IP address detection in remote control

In it's current state RemoteRpcServer.getIPAddress() finds the first IPv4 address that is not localhost and is not on a dummy interface. But if there are several addresses that meet these requirements picking the first one may not be the right choice. For example, the one from usbnet0 may be picked when we need the one from wlan0.

Integrate basic network communication logic from RecSync/libsoftwaresync

As a starting point, integration could be done like this:

• Add code for leader/client setup to OpenCamera Sensors app launch (OnCreate)

• Create simple textbox which displays connected clients and IP address information after successful connection

Synchronized multi-device video recording

Global task - integration of libsoftwaresync method into OpenCamera Sensors

1. Investigate the possibility of video recording with this method (as oppposed to synchronized snapshots)

Crash on startup when requesting permissions

The app may crash on startup if it lacks some permissions with the following exception:

2022-08-12 20:47:29.340 3992-3992/com.opencamera_sensors.app E/AndroidRuntime: FATAL EXCEPTION: main
    Process: com.opencamera_sensors.app, PID: 3992
    java.lang.RuntimeException: Unable to start activity ComponentInfo{com.opencamera_sensors.app/net.sourceforge.opencamera.MainActivity}: java.lang.NullPointerException: Attempt to invoke virtual method 'void net.sourceforge.opencamera.PermissionHandler.onRequestPermissionsResult(int, int[])' on a null object reference
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:4031)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:4197)
        at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:103)
        at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
        at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2434)
        at android.os.Handler.dispatchMessage(Handler.java:106)
        at android.os.Looper.loopOnce(Looper.java:226)
        at android.os.Looper.loop(Looper.java:313)
        at android.app.ActivityThread.main(ActivityThread.java:8633)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:567)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1133)
     Caused by: java.lang.NullPointerException: Attempt to invoke virtual method 'void net.sourceforge.opencamera.PermissionHandler.onRequestPermissionsResult(int, int[])' on a null object reference
        at net.sourceforge.opencamera.MainActivity.onRequestPermissionsResult(MainActivity.java:5647)
        at android.app.Activity.requestPermissions(Activity.java:5355)
        at androidx.core.app.ActivityCompat.requestPermissions(ActivityCompat.java:502)
        at net.sourceforge.opencamera.MainActivity.onCreate(MainActivity.java:234)
        at android.app.Activity.performCreate(Activity.java:8282)
        at android.app.Activity.performCreate(Activity.java:8262)
        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1329)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:4005)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:4197) 
        at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:103) 
        at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135) 
        at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95) 
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2434) 
        at android.os.Handler.dispatchMessage(Handler.java:106) 
        at android.os.Looper.loopOnce(Looper.java:226) 
        at android.os.Looper.loop(Looper.java:313) 
        at android.app.ActivityThread.main(ActivityThread.java:8633) 
        at java.lang.reflect.Method.invoke(Native Method) 
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:567) 
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1133)

Tis probably happens because some permissions are requested before the permission handler is initialized.

Add IMU data recording with different frequencies

• Record files with raw sensor data and timestamps

• Create an option for custom sampling frequency (see documentation for details)

Report smartphone hostname

• Use a reliable way of reporting smartphone hostname when connecting remote recording control (so if, for instance, USB tethering is used instead of Wi-Fi, the hostname gets reported accordingly)

Overloading of onSensorChanged handler

Set up some default options for typical usage of our app

• Disable optical/digital stabilization
• Sound recording

Erroneous Rotation vector

I am trying to get the Rotation vector data using the add_sensors branch. I have observed that the Rotation vector data has two modes and the values keep switching between these two modes. The data from other sensors are fine. The "x","y","z","t" plot for the rotation vector looks like this

And the CSV files looks like this

For this data, I have kept the phone static lying on a table (not horizontal, slightly inclined).

Upload by tag TODOs

Unfinished branch with the implementation of tag unloading from multiple devices.

TODO:

• d0580f1 python script returns error
• do not use serial phone numbers

Phase parameters in remote control using algorithm from RecSync

Add support of periodical flash during the video

It should be an option to turn it on/off and configure period.

Update tests according to the changed video recording logic

• ````testTakeVideoSettings``` relies on the synchronous recording stop and sets a short wait time.

• Some other tests not included in instrumentation might have been affected too.

Linear Acceleration Data

Hi, is there any way to get the linear acceleration data without the influence of gravity? The current acceleration data logged has gravity included, which I do not want.

A possible way for me to remove it on my end is to get the average gravity value by placing my phone on a table and getting its average magnitude. Afterwards, I can use the gyroscope data of the timestamp to calculate the gravitational acceleration in each axis and minusing it from the acceleration data logged. However, I am hoping for a simpler solution.

Thanks in advance!

Taking pictures while recording a video

Sometimes taking a picture while recording a video fails: no error is displayed, no app crash, we just don't get a photo and GUI stays in "taking-a-photo" mode.
It seems like this happens because this callback never gets invoked in such cases.

Add file with timestamps when flash was raised

Record additional IMU parameters

• getResolution()
• getMaximumRange()
• getMinDelay()
• getMaxDelay()
• getVendor() and getVersion()
• TYPE_SAMPLING
• TYPE_SENSOR_PLACEMENT
• TYPE_UNTRACKED_DELAY
• TYPE_VEC3_CALIBRATION

F-Droid tag

Add fdroid and f-droid tags to the repository description

Add instrumented testing to GitHub Actions CI

CI added in #62, in contrast to the old Travis CI, does not run unstrumented tests.

Implement remote recording

• Choose communcation protocol
• Remotely record video
• Remotely record IMU
• Specify duration in a message

Display and collect smartphone properties

• Add menu with important smartphone properties (sensor timestamp source, sensor support)
• Create an option to collect them and organize this data into a useful table of models and their properties

tags & versions do not match

Looks like the APKs to the latest 3 releases are all identifying with the very same version: 1.48.3, versionCode 80. That means users having a previous version installed won't get the newer ones offered as updates (as versionCode is identical). And also means you cannot tell apart what version was used with errors being reported. Would it be possible to fix this? Thanks!

No RecSync settings

Thanks for sharing this great work! In order to synchronized recording on multiple smartphones, the "Use RecSync" switch needs to be enabled. However, I can not find "RecSync settings" in the preferences page. My perferences page is similar to
. And there are only 4 buttons when I open this app. The synchronization button is missing. Does this releated to the type of phones?

Stop video with IMU recording

Thanks for your great application. There is a strange problem. The error message “Stop video with IMU recording” displays in the screen, while I record video with imu recording. Why this happens?

Update README description with new features

• Remote control: description
• Remote control: diagram
• List of projects where our app is used

Image IMU data

I have OpenCamera-Sensors installed on a Moto Z3. I can take a picture with the camera and read the EXIF and XMP tags but I do not see the IMU values stored there. I see from README.MD that IMU for video is stored in the DCIM folder as a CSV. This appears to not be the case for images. Are image IMU values not supported?

question about phase alignment

I have used the script to align the video. But I have found that only a few frames (less than ten) can match. I changed the THRESHOLD_NS to 10000000 and more frames can match while still some frames could not match. So I enabled the phase alignment to get a more synchronized result. But I found that the result are similar. I found some problems in using phase alignment. When I press the phase alignment button, the results showed on two phones are different. One shows that the sync was successful and the other shows that it failed. And I tried to press phase alignment button some times, phase error did not become green and is greater than 10 ms. How can I align the two phase and get synchronized videos? Is it a problem with the Wi-Fi hotspot or different type phones?

Failed to record video

While recording video with IMU recording, the error message "Failed to record video" appears on the screen. What causes this to happen?

Change destination dir to OpenCamera Sensors instead of OpenCamera

Now, all data is saved to standard OpenCamera path.

Leader IP address detection

NetworkHelpers is only capable of determining IPs for clients but not for a leader: a leader cannot detect the IP of itself.

create csv file compatible for gyroflow app

add gravity vector and Rotation vector

can you add gravity vector and Rotation vector?

Add instrumented testing to CI

• Filter out tests that require ./testdata being copied to the device
• Configure smaller test run for regular checks (use gradle documentation to filter tests)
• Configure full test run for merge into master

Frame image saving for debug

• Make sure current frame saving setup is enough not to cause frame rate drop / buffer allocation problems on devices other than already tested

Split release and debug/test workflows to run in parallel

Currently, all the scripts are in one travis job