hhy5277 / drishti

Real time eye tracking for embedded and mobile devices.

drishti

Real time eye tracking for embedded and mobile devices in C++11.

NEWS (2018/08/10)

Native iOS, Android, and "desktop" variants of the real-time facefilter application have been added here: src/examples/facefilter. These applications link against the installed public drishti::drishti package interface, which is designed without external types in the API definition. The facefilter demos are enabled by the DRISHTI_BUILD_EXAMPLES CMake option, and the entire src/examples tree is designed to be relocatable, you can cp -r src/examples ${HOME}/drishti_examples, customize, and build, by simply updating the drishti package details.

iOS

The iOS facefilter target requires Xcode 9 (beta 4) or above (Swift language requirements) and will be generated directly as a standard CMake add_executable() target as part of the usual top level project build -- if you are using an appropriate CMake iOS toolchain for cross compilation from your macOS + Xcode host for your iOS device. Please see Polly Based Build and iOS Build below for more details.

Android Studio

UPDATE: Android NDK r19 is not currently supported due to significant structural changes in the android-ndk toolchain that conflict with CMake's internal Android support. See this _discussion for more details.

You can download standalone NDK r18x toolchains or earlier and specify those directly in your gradle local.properties.

drishti/android-studio/local.properties example:

ndk.dir=/Users/username/android/android-ndk-r18b
sdk.dir=/Users/username/Library/Android/sdk
cmake.dir=/usr/local

The top level Android Studio application is located in the android-studio directory. This target will build and manage repository C++ sources directly as part of the project. Android Studio/Gradle is required to build the application layer, and the CMake build is managed directly by gradle. There are a few platform specific configurations that must be addressed before building. Please see Android Studio Build below for more details.

Overview

Goal: SDK size <= 1 MB and combined resources (object detection + regression models) <= 4 MB.

• Hunter package management and CMake build system by Ruslan Baratov, as well as CI and much of the real time facefilter mobile application(s) layer: "Organize Freedom!" :)
• A C++ and OpenGL ES 2.0 implementation of Fast Feature Pyramids for Object Detection (see Piotr's Matlab Toolbox) for face and eye detection -- the ACF library is available as a standalone Hunter package here
• Iris ellipse fitting via Cascaded Pose Regression (Piotr Dollar, et al) + XGBoost regression (Tianqi Chen, et al)
• Face landmarks and global eye models provided by "One Millisecond Face Alignment with an Ensemble of Regression Trees (Kazemi, et al) using a modified implementation from Dlib (Davis King) (normalized pixel differences, line indexed features, PCA size reductions)
• OpenGL ES friendly GPGPU shader processing and efficient iOS + Android texture handling using a modified version of ogles_gpgpu (Markus Kondrad) with a number of shader implementations taken directly from GPUImage (Brad Larson)

iPhone @ 30 FPS (VIDEO)

Drishti Right Eye Annotation Scheme

FEATURE	SPECIFICATION
eyelids	2D points 0-15
crease	2D points 16-24
iris center	2D point 25
outer limbus	limbus intersection with ray from outer corner to iris center
inner limbus	limbus intersection with ray from inner corner to iris center
iris ellipse	2D center, minor axis, major axis, angle (radians)
pupil ellipse	2D center, minor axis, major axis, angle (radians)

• the left eye is obtained by Y axis mirroring
• total (27*2)+(2*5) = 64 parameters
• the eye crease is useful for pose indexing, but better guidelines are needed
• the 2D limbus points are slightly redundant (given the ellipse iris model) but the intersection points are stable with respect to squinting and provide an efficient anchor for posed indexed features (accurate point-to-ellipse distances are non-trivial and are fairly computationally intensive)
• currently 2D only (gaze angle ground truth would be beneficial)

Quick Start (i.e., How do I make this library work?)

General

Drishti is a CMake based project that uses the Hunter package manager to download and build project dependencies from source as needed. Hunter contains detailed documentation, but a few high level notes and documentation links are provided here to help orient first time users. In practice, some working knowledge of CMake may also be required. Hunter itself is written in CMake, and is installed as part of the build process from a single HunterGate() macro at the top of the root CMakeLists.txt file (typically cmake/Hunter/HunterGate.cmake) (you don't have to build or install it). Each CMake dependency's find_package(FOO) call that is paired with a hunter_add_package(FOO CONFIG REQUIRED) will be managed by Hunter. In most cases, the only system requirement for building a Hunter project is a recent CMake , a working compiler corresponding to the operative toolchain and native build tool. If you're not familiar with CMake, you can try to build this minimal example to get a basic understanding.

Hunter will maintain all dependencies in a versioned local cache by default (typically ${HOME}/.hunter) where they can be reused in subsequent builds and shared between different projects. They can also be stored in a server side binary cache -- select toolchains will be backed by a server side binary cache (https://github.com/elucideye/hunter-cache) and will produce faster first time builds (use them if you can!).

Get Latest Sources

Clone this repository and initialize all submodules:

> git clone https://github.com/elucideye/drishti
> cd drishti
[drishti]> git submodule update --init .

or

> git clone --recursive https://github.com/elucideye/drishti

Generate and Build

Desktop platforms usually don't require a toolchain (a default toolchain with C++11 support will be set by Drishti) and you can generate and build Drishti as a regular CMake project.

Linux + GCC + Makefile with Drishti examples, Release:

cmake -H. -B_builds -DHUNTER_STATUS_DEBUG=ON -DDRISHTI_BUILD_EXAMPLES=ON -DCMAKE_BUILD_TYPE=Release
cmake --build _builds

macOS + Xcode with Drishti examples, Release:

cmake -H. -B_builds -GXcode -DHUNTER_STATUS_DEBUG=ON -DDRISHTI_BUILD_EXAMPLES=ON
cmake --build _builds --config Release

Windows + Visual Studio 15 2017 with Drishti examples, Release:

cmake -H. -B_builds -G "Visual Studio 15 2017" -DHUNTER_STATUS_DEBUG=ON -DDRISHTI_BUILD_EXAMPLES=ON
cmake --build _builds --config Release

To run the install procedure add the CMAKE_INSTALL_PREFIX variable and use --target install:

cmake -H. -B_builds -G "Visual Studio 15 2017" -DHUNTER_STATUS_DEBUG=ON -DCMAKE_INSTALL_PREFIX=_install
cmake --build _builds --config Release --target install

Polly Based Build

To support cross platform builds and testing, the CI scripts make use of Polly: a set of common CMake toolchains paired with a simple polly.py CMake build script. Polly is a Python script, make sure Python 3 is installed:

> which python3
/usr/bin/python3

Clone Polly and add bin folder to PATH:

> git clone https://github.com/ruslo/polly
> export PATH=`pwd`/polly/bin:$PATH

Check it:

> which polly.py
/.../polly/bin/polly.py

> polly.py --help
Python version: 3.5
usage: polly.py [-h]
    [--toolchain ...

Note: Polly is not a build requirement, CMake can always be used directly, but it is used here for convenience.

After the environment is configured, you can build for any supported Polly toolchain (below you can find some toolchains used in CI) with a command like this:

polly.py --toolchain ${TOOLCHAIN} --config-all ${CONFIG} --install --verbose

Building examples:

polly.py --toolchain ${TOOLCHAIN} --config-all ${CONFIG} --install --verbose --reconfig --fwd DRISHTI_BUILD_EXAMPLES=ON

Note: The --reconfig flag is included in the example above, which will re-run the CMake configure step (to incorporate CMake changes) for you. It is a reasonable step to add in cases where you aren't sure if it is needed.

iOS Build

Since CMake contains an Xcode generator, building for iOS is fairly straightforward. In practice, it is no different than the other polly.py toolchain builds. As always, you will need to have an Apple Developer Account to build and run on iOS devices. There are a few setup steps associated with Apple code signing requirements. Since iOS 10.0, Xcode projects require a valid Team ID entry, which can be set through CMake using the CMAKE_XCODE_ATTRIBUTE_DEVELOPMENT_TEAM CMake variable. If you generate an Xcode project through a polly.py command (described below), it will initialize the field for you if the POLLY_IOS_DEVELOPMENT_TEAM environment variable is set with your Team ID, which can be found in your Apple Developer Account. If you are using an Apple Enterprise Developer Account, the CMAKE_TRY_COMPILE step can fail with an error beginning with No profiles for 'com.example' were found: .... You can fix this with a one time Xcode initialization described in POLLY_IOS_DEVELOPMENT_TEAM.

Android Studio Build

For Android Studio, there are additional requirements:

• CMake 3.9.2+
• Ninja
• Android Studio 3.2.1

Note: Polly will not be used here, because CMake is launched by Android Studio itself.

Note: Host compiler is required for some parts of the build. E.g. on Windows you have to install Visual Studio. Please check that minimal C++ example is working.

The path to the CMake executable should be added to the local.properties file before opening drishti/android-studio in Android Studio, or before invoking the Gradle build script. If you do not have a local.properties file, it will be generated automatically by Android Studio in the top level android-studio folder (usually drishti/android-studio/local.properites), when it is launched, at which point you can add the cmake.dir=/path/to/native/cmake CMake entry and rerun. The local.properties file will look something like this:

ndk.dir=/home/username/Android/Sdk/ndk-bundle
sdk.dir=/home/username/Android/Sdk
cmake.dir=/opt/cmake

The cmake.dir entry should be set such that <cmake.dir>/bin/cmake points to a valid cmake executable file.

Please check these instructions for details and useful notes:

• https://docs.hunter.sh/en/latest/faq/android-studio.html

There is another entry point for Android Studio - src/examples/facefilter/android-studio. It should be used only for testing or as a template for starting your own project based on Drishti.

Android Studio Workarounds

The following factors may contribute to some instability in the Android Studio managed build.

• Using custom CMake 3.7+ in Android Studio is a relatively new feature
• Some issues are hard to track or confirm, some issues are already reported but still not fixed

With support for official CMake binaries now in the Android Studio 3.2.1 release, the Android build stability is very likely improved compared to earlier beta versions.

From experience, the weakest part in the build has been communication between Gradle and CMake. To minimize it, the following trick can be used:

• Open the top-level CMakeLists.txt file
• Find if(DRISHTI_DEBUG_STOP) condition
• Substitute if(DRISHTI_DEBUG_STOP) with if(TRUE)
• Run Gradle build:

[drishti]> cd android-studio
[drishti/android-studio]> ./gradlew assembleDebug

If you're running it a first time there will be a high chance to hit this Gradle issue:

* What went wrong:
Execution failed for task '...'.
> Conversion = c, Flags =

In this case, just wait for few seconds and run Gradle again:

[drishti/android-studio]> ./gradlew assembleDebug

• Revert CMakeLists.txt file, i.e. substitute if(TRUE) with if(DRISHTI_DEBUG_STOP).
• Run the CMake build without Gradle:

[drishti/android-studio]> cmake --build ../src/examples/facefilter/android-studio/app/.externalNativeBuild/cmake/debug/arm64-v8a

Once the CMake build is ready, you can use ./gradlew assembleDebug or open Android Studio IDE.

Applications

Please see the README for the drishti-hci console application to see an example of a full eye tracking pipeline with the GPGPU optimizations.

Integration

Drishti is also available as a Hunter package. If you would like to integrate Drishti in your project, please see the Hunter Drishti package documentation.

Steps (check https://docs.hunter.sh/en/latest/quick-start.html):

Add cmake/HunterGate.cmake and a minimal cmake/Hunter/config.cmake to your project:

mkdir -p cmake/Hunter
wget https://raw.githubusercontent.com/hunter-packages/gate/master/cmake/HunterGate.cmake -O cmake/HunterGate.cmake
wget https://raw.githubusercontent.com/ruslo/hunter/master/examples/drishti/config.cmake -O cmake/Hunter/config.cmake

Add HunterGate(URL <url> SHA1 <sha1>) to the top of your CMakeLists.txt (You can find updated release information here).

include("cmake/HunterGate.cmake")
HunterGate(
    URL "https://github.com/ruslo/hunter/archive/v0.19.140.tar.gz"
    SHA1 "f2c30348c05d0d424976648ce3560044e007496c"
    LOCAL # use cmake/Hunter/config.cmake
)

Finally, add the Drishti package to your CMakeLists.txt and link it to your target:

hunter_add_package(drishti)
find_package(drishti CONFIG REQUIRED)
target_link_libraries(your_app_or_lib PUBLIC drishti::drishti)

You can customize the drishti package (and dependencies) by specifying a VERSION and/or CMAKE_ARGS (options) list for each package in cmake/Hunter/config.cmake.

Please see https://github.com/elucideye/drishti_hunter_test for a minimal working example using the drishti hunter package.

Toolchains

The configurations listed below have all been tested. In general, most C++11 toolchains should work with minimal effort. A CI comment indicates that the configuration is part of the Travis or Appveyor CI tests, so all Hunter packages will be available in the server side binary cache.

Linux (Ubunty Trusty 14.04):

• TOOLCHAIN=clang-fpic-hid-sections CONFIG=Release # CI
• TOOLCHAIN=gcc-5-pic-hid-sections-lto CONFIG=Release # CI
• TOOLCHAIN=libcxx CONFIG=Release # w/ clang 3.8

OSX:

• TOOLCHAIN=osx-10-13 CONFIG=Release # CI
• TOOLCHAIN=osx-10-12-sanitize-address-hid-sections CONFIG=Release # CI
• TOOLCHAIN=xcode-hid-sections CONFIG=Release # generic

iOS:

• TOOLCHAIN=ios-nocodesign-11-3-dep-9-3-arm64 CONFIG=Release # CI
• TOOLCHAIN=ios-10-1-arm64-dep-8-0-hid-sections CONFIG=Release

Android:

• TOOLCHAIN=android-ndk-r17-api-19-armeabi-v7a-neon-clang-libcxx CONFIG=MinSizeRel # CI
• TOOLCHAIN=android-ndk-r17-api-24-arm64-v8a-clang-libcxx14 CONFIG=Release # CI
• TOOLCHAIN=android-ndk-r10e-api-19-armeabi-v7a-neon-hid-sections-lto CONFIG=MinSizeRel

Windows:

• TOOLCHAIN=vs-15-2017 CONFIG=Release # CI
• TOOLCHAIN=vs-14-2015-sdk-8-1 CONFIG=Release # CI
• TOOLCHAIN=vs-14-2015-win64-sdk-8-1 CONFIG=Release # CI
• TOOLCHAIN=vs-14-2015-win64-sdk-8-1 CONFIG=Debug # CI

The polly out of source build trees are located in _builds/${TOOLCHAIN}, the final build products (the stuff you want) are installed in _install/${TOOLCHAIN}, and the build logs are dumped in _logs/${TOOLCHAIN}. The iOS frameworks are installed in _frameworks/${TOOLCHAIN}.

Choosing simplest toolchain

On Linux you will usually want --toolchain gcc-pic (GCC based toolchain with position independent code).

On Windows, the preferred toolchain will depend on the generator you want, e.g., if you want "Visual Studio 15 2017", then use --toolchain vs-15-2017, if you want the 64 bit version use --toolchain vs-15-2017-win64.

On macOS, the choice of toolchain depends on Xcode version you have installed. Please check this table for Xcode versions and corresponding iOS/macOS SDK versions:

• https://polly.readthedocs.io/en/latest/toolchains/ios.html

E.g., if you have Xcode 8.3.1 installed, then the default SDK will be macOS 10.12 SDK, hence you can use --toolchain osx-10-12. Instead of the Xcode generator, you can use a Makefile toolchain - --toolchain osx-10-12-make.

In the same table, you can find iOS SDK version. E.g., if you have installed Xcode 9.4 with default iOS SDK 11.4, and you want to set the deployment SDK to version 9.3, you can use --toolchain ios-11-4-dep-9-3-arm64 to build the ARM64 architecture. If you have several versions of Xcode installed, you can use IOS_X_Y_DEVELOPER_DIR/OSX_X_Y_DEVELOPER_DIR environment variables for switching. E.g., if OSX_10_13_DEVELOPER_DIR will be set to Xcode 9.0 location, then Xcode 9.0 will be used with --toolchain osx-10-13, even if Xcode 9.3 is installed and set as the default.

You can use Polly toolchains to build Android if you don't want to rely on Android Studio. The only requirement is an environment variable with the Android NDK location. Set the ANDROID_NDK_r17 environment variable with the path to the Android NDK r17, and you can use any --toolchain android-ndk-r17-* variants.