st0nedb / rooms

With "Rooms" mobile devices can perform indoor self-localization using an app and low-cost BLE beacons.

Update

Currently I do not have time to keep rooms updated. Please consider this project should be considered a proof-of-concept. If you would like to contribute to further development, please contact me or open an issue.

rooms

TL;DR Another way to perform room presence detection based on low-cost BLE beacons for mobile devices. MQTT support for smart-home integration. iOS only.

With rooms mobile devices can perform indoor self-localization using an app and low-cost BLE beacons. Its focus is the localization of mobile devices using low-cost COTS (commercial-off-the-shelf) hardware, e.g., the ESP32. By configuring and deploying multiple ESP32's as iBeacon transmitters into a region (e.g. an apartment, house, etc.), a mobile app on the device can detect in which room/area of the region the device currently is. It then uses MQTT to publish the determined region to an MQTT broker. The current room is estimated from the combination of RSSI values obtained from all beacons in range (RSSI fingerprinting) using a Machine Learning model.

Here is a neat little demo with two rooms. The device is moved from the Badezimmer (bathroom in german) to the Flur (hallway in german) and back. As you can see, rooms is able to quickly detect the change of the device location.

The prediction runs entirely on the device, giving full control to the user about if and how the data is shared. It is not possible for the iBeacons to detect if a device is performing self-localization with rooms or not.

To perform the RSSI sensing and ML inference, the app must run in the background. As Apple App Store guidelines restrict this, the app can not be published on the official App Store. Therefore, to use rooms you need to download and compile it yourself. For more inforation on power consumption and system utilization, please refer to the "Additional Information" section below.

Getting Started

Setting up your Beacon environment

It should be possible (not tested yet!) to use all kinds of different Beacon hardware. To use iBeacons with rooms you need to setup your beacons to all use the same UUID and Major value. Give each beacon a unique Minor value to differ between Beacons. To use ESP32 with iBeacon, I recommend to check out ESPHome. They have a tutorial how to set everything up and configure ESP32 to transmit as iBeacons here.

Setting up the app

On the first launch, the app won't do anything until it is properly configured. The process to get it running requires the following steps.

1. Setup your beacons
2. Setup your rooms
3. Setup MQTT (optional)
4. Collect training data from each room/area (~ two-three minute per room)
5. Export your data to train the AI specific to your setup (currently done with Python script)
6. Import the AI model back to the app
7. Done :) Below all steps are described in more detail.

1. Setup your iBeacons

Go to Settings > Beacons and configure your global UUID and Major value. Don't forget to type Set to confirm the values. Then add all Minor values of your Beacons to the list below. Swipe down once you are finished.

2. Setup your Rooms/Areas

Go to Settings > Rooms and add all your rooms/areas you want your device to detect to the list.

3. Setup MQTT (optional)

If you want the app to report changes of location to your MQTT broker, you need to configure it. Go to Settings > MQTT and configure fill out all required fields. Tap Set to make sure the value is properly configured! To test the connection, press Test at the very bottom of the display. If everything works out, the red X changes to a checkmark and you should find a short MQTT message on the specified topic.

4. Collect training data

Go to Data and choose a room using the Choose Room from List button. You should be able to see all rooms previously configured in step 2. Once selected, move your device to the room and press Start Recording. The device will then start to collect readings from your beacons. To get better results, you should move in the room and open/close doors. Also put the device in your pocket and move around (don't just stand still). Don't leave the room during this process! The time required per room depends on whats configured for the Number of Training Samples per Room in the Settings tab. The app will record one sample/second. Larger numbers of training samples might give you better results later, but also take more time to accuire. Repeat the process for every room.

5. Export the data and train your own AI

Once the data is collected for all rooms, you must export the data. The app generates on JSON file for each room with the collected samples. Pressing the Export button will open the default iOS share dialogue and allow you to export your files. Export your files to where you can access them with your computer. To train your model, copy the exported JSON files to the machine-learning-python folder in the repository and run the script with the appropriate options (--num-beacon is the only required option). To setup the python environment, you can use "pipenv" with the provided pipfiles. Use --help to learn about the other options.

6. Import the AI model back to the app

To import your model back into the app, you must provide it at a valid URL link. You can use the python scripts integrated option (pass the argument --http to run a server on your machine) or use NGINX to setup your own server. Go to Settings > Model and paste the link to the model there. Press Set and the Import. If everything works, the app will import your own model from the URL and compiles it.

7. Start the prediction

Finally, go to the initial Prediction tab. Wait for a few seconds to allow the app to compile your model and start collecting readings from your BLE beacons. After a few seconds it should show in which room you are currently in. If you configured MQTT, a new message will be published to the configured topic every time the room changes.

Additional Information

Here are some results of the resource impact of the approach on your device (e.g., CPU usage, battery impact).

To reduce power consumption, beacon measurements are only collected if the device is moved. For movement detection movement, the app monitors the devices’ gyroscope. To avoid accidentally draining the device battery, rooms suspends all activities when the device battery goes below 30% and the device is not charging.

More discussions of the approach are available on this Home-Assistant forum post.

Future

RSSI is not a real high-value wireless channel feature, just the best we can get with cheap hardware and reasonable effort at the moment. I think with better hardware, more advanced wireless channel features for the learning, e.g., multipath-parameters, or time-difference of arrival.

rooms's Issues

unable to train model

Describe the bug

WARNING:root:scikit-learn version 0.24.0 is not supported. Minimum required version: 0.17. Maximum required version: 0.19.2. Disabling scikit-learn conversion API.
WARNING:root:TensorFlow version 1.15.2 detected. Last version known to be fully compatible is 1.15.0 .
Traceback (most recent call last):
  File "roomsModelMaker.py", line 179, in <module>
    X, Y = parseData(jsondata=data, rooms=rooms, nsamples=nsamples, lb=lbinarizer)
  File "roomsModelMaker.py", line 84, in parseData
    data[ii:ii+N] = dd
ValueError: could not broadcast input array from shape (120,2) into shape (120,7)

To Reproduce
Folder has the various rooms .json files in
pipenv working with dependencies listed

Beacon ranging does not work when re entering zone

Describe the bug
If one reenters the beacon zone, after leaving it, the function locationManager(_ manager: CLLocationManager, didEnterRegion region: CLRegion) gets called, but predictions are not started.

To Reproduce
Leave zone,
re enter zone
Expected behavior
Ranging should start when re entering zone

Background task is not working probably

Describe the bug
Sometimes the background task doesn't behave like it should.
If the unexpected behavior occurs, the task is dead. Launching the app is needed for it to work probably again. It happens when "Making Fake Move" is performed multiple times instead of only once.
If the expected behavior occurs, it makes a fake move and restarts the background task.

Unexpected behavior

Background task registered
Background task ended.
Making Fake Move.
Making Fake Move.
Making Fake Move.
Room Buero, Likelihood 90.4
Making Fake Move.

OR

Background task registered
Background task ended.
Making Fake Move.
Making Fake Move.
Making Fake Move.
Room Buero, Likelihood 99.6
Making Fake Move.
Making Fake Move.
Room Buero, Likelihood 99.6

Expected behavior

Background task registered
Background task ended.
Making Fake Move.
Room Buero, Likelihood 97.2
Background task registered

Smartphone (please complete the following information):

• Tested on iOS 14.3, 14.4 and 14.5 Beta 3
• on iPhone X and iPhone 12 Pro

Add more detailed compile instructions

Is your feature request related to a problem? Please describe.
New users without Xcode experience might have trouble downloading and compiling the app.

Describe the solution you'd like
Step-by-Step instructions to download, compile and install the app. Possibly including external tutorials on how to install a self-developed app.

Describe alternatives you've considered
This could get obsolete, once #10 is resolved and altstore.io install is available.

Additional context

None

Add more descriptive instructions

In issue#3 it becomes obvious, that current error descriptions in the app are not very clear. They must be improved with clearer instructions to avoid frustration for users who encounter them.

cloud web Service to train data

Is your feature request related to a problem? Please describe.
everyone has to spin up a pipenv to train their data (when its essentially the same process)
Add Json files, crunch model, spit out result - import to phone.

Describe the solution you'd like
could we spin up a cloud service where the user uploads their models and it provides the results back?
I have a dev Azure subscription I could leverage? not sure how to make a web front end for it though?

Describe alternatives you've considered
not sure if its of value, but thought i'd suggest it

Prediction always at 90.00% and shows infinite start loading indicator, if screen is switched

Describe the bug
The likelihood percentage is always at 90.00%, no matter in which room I am and if I switch between the screens (i.e. to Settings and back to Prediction), it shows Starting ...
I am using 10 ESP32s running ESPHome with iBeacon enabled with the same uuid and major, but different minors.

Expected behavior
The right likelihood percentage and not the starting loading indicator.
Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: iOS 14.5 beta 4

Smartphone (please complete the following information):

• Device: iPhone 12 Pro

Issue with coreml (i think)

Hi,

After following all the required steps I have no joy I'm afraid.
The only non-standard setup I have is that I'm not using ESP iBeacons.

The app eventually times out with an "invalid model!" message.

I had used;

MacOS Catalina - 10.15.5 (19F101)
Xcode 11.5 (11E608c) (iv kept all project format to "Xcode 9.3-compatible"
Iphone 7 - Software Version 13.5.1

pipenv install

Creating a virtualenv for this project… Pipfile: /Users/ross/Documents/rooms3/machine-learning-python/Pipfile
Using /usr/local/bin/python3.7m (3.7.8) to create virtualenv…
⠇ Creating virtual environment...created virtual environment CPython3.7.8.final.0-64 in 1178ms creator CPython3Posix(dest=/Users/ross/.local/share/virtualenvs/machine-learning-python-nexj2ZoU, clear=False, global=False) seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/ross/Library/Application Support/virtualenv/seed-app-data/v1.0.1) activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

I have attached a few screenshots, the .json files and a log of the following output

./roomsModelMaker.py --num-beacon 2 --http

(the sample rate is low as, well it was quite frustrating doing all the moving about again and again!)

screenshots.zip
rooms-json.zip
machine-learning.zip

Please let me know if you need further info.
I'd be more than happy to try out any new suggestions.

Join the discord to chat about room presence

https://discord.gg/AEsfpPxp

Really value your input here!

Evaluate altstore.io

Is your feature request related to a problem? Please describe.
Users can't install the App without compiling it for themselves using Xcode.

Describe the solution you'd like
A easier way to install the app.

Describe alternatives you've considered
None.

Additional context
Test out altstore.io and see if it can be used to install and run the app.

MQTT not working

Hi,

I was able to get the app installed and the predictions working but I am not able to get it to send the room updates to MQTT.

The MQTT connection test on the settings page works and I can see the test message. Also sometimes it seems to get stuck on the "Starting..." message and won't start predictions until I force close the app.

I am running the app on an iphone Xr with ios 14 using the master branch.

Any suggestions would be appreciated.

Thanks