andreped / dss

:vibration_mode: From training of transformers to real-time development in cross-platform mobile apps!

DSS: Deep Sensor Systems

📳 From training of transformers to real-time development in cross-platform mobile apps!

DSS was developed by SINTEF Medical Image Analysis with aim to integrate AIs into sensor systems.

This project serves as a demonstration on how to do it, and does not claim to be a generic framework.

Below there are described some of the key features of this project, but to see what else is possible, please, see the wiki.

Continuous integration

Build Type	Status
Test Training
Test Flutter
Build APK

How to your train own model?

Setup

When using this framework, it is a good idea to setup a virtual environment:

virtualenv -ppython3 venv --clear
source venv/bin/activate
pip install -r requirements.txt

The following dependencies will be installed:

• pandas<=1.5.3
• tensorflow<=2.12.0
• tensorflow-addons<=0.19.0
• tensorflow-datasets<=4.8.3

Tested with Python 3.7.9 on Win10, macOS, and Ubuntu Linux operating systems. Also tested with Python 3.10.4 on GitHub Codespaces.

Note that to activate the virtual environment on Windows instead run ./venv/Scripts/activate.

Usage

To train a model, simply run:

python main.py

The script supports multiple arguments. To see supported arguments, run python main.py -h.

Training history

To visualize training history, use TensorBoard (with example):

tensorboard --logdir ./output/logs/gesture_classifier_arch_rnn/

Example of training history for a Recurrent Neural Network (RNN) can be seen underneath:

The figure shows macro-averaged F1-score for each step during training, with black curve for training and blue curve for validation sets. Best model reached a macro-averaged F1 score of 99.66 % on the validation set, across all 20 classes.

Disclaimer: This model was only trained for testing purposes. The input features were stratified on sample-level and not patient-level, and thus validation performance will likely not represent true performance on new data. However, having a trained model enables us to test it in a Mobile app.

Available datasets

SmartWatch Gestures

The current data used to train the AI model is the SmartWatch Gestures dataset, which is available in tensorflow-datasets. The dataset has the following structure:

FeaturesDict({
    'attempt': tf.uint8,
    'features': Sequence({
        'accel_x': tf.float64,
        'accel_y': tf.float64,
        'accel_z': tf.float64,
        'time_event': tf.uint64,
        'time_millis': tf.uint64,
        'time_nanos': tf.uint64,
    }),
    'gesture': ClassLabel(shape=(), dtype=tf.int64, num_classes=20),
    'participant': tf.uint8,
})

How to test the model in a mobile app?

Converting model to TF-Lite

In order to be able to use the trained model in a mobile app, it is necessary to convert the model to a compatible format. TensorFlow Lite is an inference engine tailored for mobile devices. To convert the model to TF-Lite, simply run this command:

python dss/keras2tflite.py -m /path/to/pretrained/saved_model/ -o /path/to/save/converted/model.tflite

Model integration and testing in app

A simple Mobile app was developed in Flutter, which demonstrates the AI in action using the accelerometer data from the mobile phone in real time. The data can also be stored and deleted locally.

To use the app, you need an Android phone and have developer mode enabled (see here for how to enable it). Then simply download the APK from here, double-click to install, and use the app as you normally would.

Info on how the mobile app was developed (and how to make your own app), can be found in the wiki.

Acknowledgements

The training framework was mainly developed using Keras with TensorFlow backend.

The mobile app was developed using Flutter, which is a framework developed by Google. For the app, the following open packages were used (either MIT, BSD-2, or BSD-3 licensed):

• flutter_sensors
• tflite_flutter
• wakelock
• sqflite
• intl
• csv
• path_provider

How to cite?

If you found this project useful, please, consider citing it in your research article:

@software{andre_pedersen_2023_7701510,
  author       = {André Pedersen and Ute Spiske and Javier Pérez de Frutos},
  title        = {andreped/DSS: v0.2.0},
  month        = mar,
  year         = 2023,
  publisher    = {Zenodo},
  version      = {v0.2.0},
  doi          = {10.5281/zenodo.7701510},
  url          = {https://doi.org/10.5281/zenodo.7701510}
}

dss's Issues

Add more sensors

It would be great if there was a feature in the mobile app to choose which sensor to use for streaming. For instance Accelerometer vs Gyroscope. Or perhaps even stream all using different services, but then we need to have the interpreter logic as its own separate service, disconnected from the widget logic.

Step counter

As we do not have a dataset (yet) for training an AI to detect steps, it would be great to have a basic method running, such that we have a baseline and integrate this feature into the app.

Strange crash

After running the app for a long while as a physical device in Android Studio, it suddenly crashed and prompted:

The following assertion was thrown while finalizing the widget tree:
Duplicate GlobalKey detected in widget tree.

Bug when switching tabs

When switching tabs, we get this error:

E/flutter ( 1905): [ERROR:flutter/lib/ui/ui_dart_state.cc(209)] Unhandled Exception: setState() called after dispose(): _DataStreamState#29d1b(lifecycle state: defunct, not mounted)
E/flutter ( 1905): This error happens if you call setState() on a State object for a widget that no longer appears in the widget tree (e.g., whose parent widget no longer includes the widget in its build). This error can occur when code calls setState() from a timer or an animation callback.
E/flutter ( 1905): The preferred solution is to cancel the timer or stop listening to the animation in the dispose() callback. Another solution is to check the "mounted" property of this object before calling setState() to ensure the object is still in the tree.
E/flutter ( 1905): This error might indicate a memory leak if setState() is being called because another object is retaining a reference to this State object after it has been removed from the tree. To avoid memory leaks, consider breaking the reference to this object during dispose().

This is likely due to that we have not implemented a dispose() method for the class.

Settings tab?

For visualizing the sensor output, it would be great to be able to adjust some settings, to have a setup that is better suited for the end users' preferences.

For instance, if jittering is not optimal, it should be possible to disable the smoothing performed which contribute to this jittering effect when rendering.

Feature: Store recordings as CSVs and store path to CV in database

Currently, the recordings are stored as rows in the database.

However, it would be better if we stored the raw recordings in separate CSVs and stored the path of the CSVs in the database.

App icon?

Would be great if we could replace the default Flutter icon with a proper app icon.

Bug: Demo app occasionally freezes when using emulator

When the app is run on an Virtual device, it sometimes freezes around 20 - 30 seconds after the start of the app.
After a short time of frozen screen that doesn't respond any more the app closes.
Mostly happens on second screen.

Only error message is printed after closing of the app : Lost connection to device.

Python >= 3.10 support

Only TF >= 2.8 has precompiled wheels for Python >= 3.10. Hence, we should likely deprecate TensorFlow 2.6.

Might be that on Windows for Python 3.7.9, TF 2.6 is the newest version available. Will have to check that out.

However, could easily just deprecate Python 3.7.x support altogether.

Bug: Update app during APK installation sometimes fails

This was just observed when installing the latest artifact APK.

I believe it has to do with when dependencies have changed, between releases.

There should be a mechanism to handle this, but right now this does not work.

In this scenario, the current working solution is to delete the app first, and then reinstall it using the latest APK.

Fix: Update app snapshots to present the latest release

A database feature was just added. Hence, we should wait to update these snapshots until the next release is out.

Info home page

It is probably a good idea to have a tab with some info about the application. Perhaps that could be the home page? However, I also think its nice that the user gets direct demonstration on the app, which is the current behaviour now.

Feature request: ability to record sensor data

It would be convenient to be able to make recordings of sensor data. For instance for research purposes.

For this, it would be relevant to have a start/stop mechanism, as well as as a database storage.

This could be a on the mobile device itself, or on the cloud.

App launch fails occationally

When launching the emulator in Android Studio, the app may fail to launch.

This is the prompt:

ProcessException: Process exited abnormally:
cmd: Failure calling service activity: Broken pipe (32)
  Command: C:\Users\X\AppData\Local\Android\Sdk\platform-tools\adb.exe -s emulator-5554 shell am start -a android.intent.action.RUN -f 0x20000000 --ez enable-dart-profiling true --ez enable-checked-mode true --ez verify-entry-points true --ez start-paused true com.something.sw_app/com.something.sw_app.MainActivity

I just updated lots of stuff in Android Studio. Might be that the emulator is outdated and needs to be updated to work with the new setup.

Bug: Plot jitters due to too rapid dynamic ylim updates

This can be solved by having a more calm ylim updater, that is not updated for every single new measurement.

FPS measure numerically unstable

When testing the app on a different phone, it was observed that the FPS exploded and did not manage to recover. Might be that the current moving average filter needs some regularization to be more stable.

Feature: Enable recordings to be stored in a database

The database can be stored on the phone itself. It should also be possible to delete individual recordings or all of them at once.

In future work, it could be relevant to sync this with some server, but this requires more overhead, hence, outside the scope of this feature request.

Bug when launching application

When starting the application, we observe that we get some issues.

This is likely due to the maps of the points relevant for plotting are empty at the beginning, which results in an issue when drawing the widget.

Feature: Enable streaming of both accelerometer and gyroscope data

Add the Sensors.GYROSCOPE as an aditional sensor when retrieving the data.

Logo?

Would be great to have a logo or a schematic which somewhat describes this framework.

FPS is dependent on hardware

Currently, when using the pretrained RNN model with the app, the FPS is greatly dependent on how fast the accelerometer is streaming data. This means that if we are able to generate 50 samples in a second, and the AI only takes 50 samples, the AI will only use 1 second of history in the predictions.

This is of course suboptimal. Quite often models are also trained using a setup where data is streamed with a specific FPS. Hence, we should be able to cap the maximum FPS to a specified limit such that the AI is given sufficient history to solve the task.

This can be easily solved by adding a mechanism to only fetch data after a specific duration.