jo-jstrm / smarthomebuddy

Hi 👋, I'm Johannes

Your Friendly Neighborhood Data Scientist!

Languages and Tools ⚒️

Projects 💻

If you are really interested in my work, below are all projects I did while studying. In chronological order, so please don't be too judgy on my earlier code. We all had to start somewhere 😜. Will be cool memories in the future. PEP8 is my friend, I swear.

• Learn SQL and MapReduce (click).
• Implement an Extended Kalman Filter for pose estimation of a car in Apache Flink (click).
• Bachelor Thesis: Research and benchmark a decentral way of organizing nodes in a data streaming engine (click). Published at VLIoT@VLDB 2021 (paper).
• Find new clusters of Customers in a real-world dataset from Robert Bosch GmbH. Visualize Results and pitch them to management. Closed source.
• Predict the filling level of glass containers based on a real dataset from a German recycling company. Mostly time series data. Visualize Results and pitch them to management. Closed source.
• Pandemic Package: automatically detect damaged packages in a warehouse based on video data in real-time. Lots of CNNs and bounding boxes.
• SciQuack: Answer free-text questions about scientific papers by highlighting specific passages in the papers. Uses database and knowledge graph. Lots of deep learning and NLP. Includes website as frontend. Closed source.
• Compress 2d and 3d videos from autonomous vehicles for remote assistance using deep learning (click). Using GANs and VAEs on camera and lidar video data for real-time processing. Published in ML4AD@NeurIPS 2021 (paper).
• Optimize energy usage and storage, predict energy production in smart grids (click). Mostly time series data.
• Learn Vulkan & GLSL basics, build custom general-purpose shaders (click). Out of my comfort zone (I was the only non-computer-graphics guy) and painful at times, but totally worth it.
• SmartHomeBuddy: identify smart home devices in your home network using machine learning. Including PoC of a complete app. Three-month scholarship by the German Bundesnachrichtendienst.
• Cofvefe Scale: Fun little arduino project where the goal was to design an information appliance while applying the design thinking process. My appliance turned out to be a coffee scale. Very innovation, much wow.
• Master Thesis: Data-efficient Representation Learning for Cell Organelle Segmentation in Electron Microscopy Images. Using self-supervised learning and transformers. Lots of optimization for low-powered hardware. Currently closed source for reasons, but working on open-sourcing.

smarthomebuddy's Issues

Integrate random forest classifier into SHB

Set up CI/CD

• Code formating
  • https://github.com/marketplace/actions/prettier-action
• Testing
  • Future TODO
• Build executable -> Next week

Move SQL queries to separate files

Move the SQL queries that are currently stored as in-code strings to dedicated files.

• Queries in db.py
• #34

[GUI] improve Home page

Implement database for network packages

Error when log folder does not exist

Describe the bug
When no log folder is present, the app throws an error. You have to create it manually.

To Reproduce

1. Delete log folder (device-identifier/shbdeviceidentifier/logs)
2. Run SHB CLI

Expected behavior
I do not have to create a log folder manually.

Desktop (please complete the following information):

• PopOS 22.04

Additional context

15:12:02 | resolve_file_path:31           | [  ERROR  ] InfluxDB log file /home/jo/git/SmartHomeBuddy/device-identifier/shbdeviceidentifier/logs/influx.log does not exist. Please check your log directory. Default log directory is 'device-identifier/shbdeviceidentifier/logs/influx.log' Current working directory: /home/jo/git/SmartHomeBuddy

Fix release action trigger

Currently, the release action is either triggered by a push to main OR by a tag that matches v*. We would like to have an AND relation. See discussion here.

[Documentation] Add FAQ

• Crypto Warning is showing up
• Server won't start
• GUI Buttons not responsive

Documentation

• #78

Improve database ingestion for pcap files

Currently, capture files are read into a generator and processed sequentially, which takes a lot of time for bigger pcap files.

Do initial setup of InfluxDB from Python

Set up packaging/distribution/installation

• OSs
  • Linux
    • .deb
    • Flatpak
      • currently requires the runtime org.freedesktop.Platform/x86_64/19.08 which might be overkill and not required
      • Maybe we should leave that for now. Seems quite complex and currently unnecessary
  • Windows
  • Mac
• System parts included in build
  • Electron App
  • Python Code
  • InfluxDB
  • SQLite: Check if needed: not needed.
• Automate release process: Guide
  • Discussion about different ways to handle releases
  • I prefer creating a separate workflow for a release, which contains a lot of redundant code from the build workflow. The advantage is, that we have both workflows clearly distinguished. This allows for more flexibility regarding the workflow triggers. Also, because the release workflow will be executed comparatively infrequently, the release runs would be hard to recognize in the when being part of the build workflow.
  • Important: in the publisher config in package.json, the draft parameter should be left true (see Guide link above).

Info

• General description of packaging changes when having multiple languages that might be helpful
• Creating Python executable
• Using Pyinstaller

Add FAQ to docs

• Crypto Warning is showing up
• Server won't start
• GUI Buttons not responsive

Implement CLI interface

Extend DataLoader

• Implement from_influx() function for packet data
• get_labels_from_json() -> Contains mapping from IP addresses to device names
• Implement train functionality for database data according to app.py:train()
  • Keep given functionality to train directly from pcap data

Cache dependencies for GitHub Actions

We should cache the python pip installation and possibly the database setup to significantly reduce action execution time. See https://github.com/marketplace/actions/cache.

[Testing] db.py

Implement device query from frontend

Translate user stories to software architecture

Data and Control Flow

Update Device Component

• https://stackoverflow.com/questions/43445592/what-is-the-meaning-of-xs-md-lg-in-css-flexbox-system
• https://stackoverflow.com/questions/50610049/how-to-organize-material-ui-grid-into-rows

Improve desktop application

• implement uninstaller
• add icon
• remove dev tools sidebar on startup

Add all IP addresses from InfluDB to SQLite devices table

Current Distributable does not start InfluxDB

Describe the bug
The current distributable does not start the Device Identifier and therefore also InfluxDB.

To Reproduce

1. Install SHB and run it
2. pgrep influx -> empty

Expected behavior
Device Identifier and InfluxDB start.

Desktop (please complete the following information):
PopOS 22.04

Connect Python backend to Typescript frontend

• Fix Python autogen import paths
  • GRPC autogen import issue in Python
  • TODO Continue here. Also link to my search
• Make electron-app talk to Python

Set up documentation with Sphinx

Initial shbdeviceidentifier setup does not create table entries for influx user

Describe the bug
When cloning the repo and setting up the project for development, shbdeviceidentifier start does create the sqlite tables users and influxdb but does not populate them with the necessary entries.

To Reproduce
Steps to reproduce the behavior:

1. Remove all installed files from the SHB project
2. Clone Repo and setup the dev environment (python, influxdb)
3. start the identifier, no error is displayed
4. Check the SQLite db file, which should be empty now

Expected behavior
SQLite db should contain at least one user

Desktop (please complete the following information):

• OS: Ubuntu

Update Documentation for v0.1.0

Refactor electron-app

• Add "view" folder for views
• Merge app.tsx and index.tsx
• Move index.html from static to top-level and replace dummy index.html
• Resolve warning about missing key in device

Testing

• Integration / Unit Tests
  • app.py
  • #92
  • ml_models.py
  • #79
  • #88
• Add workflow
• System Tests with User Stories

Add extra info to release workflow

Such as:

with:
          tag_name: ${{ github.ref }}
          release_name: Release ${{ github.ref }}
          body: |
            Changes in this Release
            - First Change
            - Second Change
          draft: false
          prerelease: false

Implement device identification from GUI

• #76
• #87
• #97
• #103
• Add train button to GUI
• Add classify button to GUI

Add action for distribution

Basic setup

Add usage description to release v0.1.0 docs

[Testing] commands.py

Define user stories for SHB usage

Set up folders and project structure

• Typescript
• Python
• Electron
• Testing
• WebPack

Move data away from python code

I think we should move the data, namely ML models, and (potential future) SQL queries away from the python package. They could reside in device-identifier/data/ or similar. We would only have to adapt DATA_DIR accordingly.

Reasoning

Packaging these files using PyInstaller, which would speak for keeping them as-is, is a pain anyways. However, adding them via extraResources using npm would be pretty easy. So why not move all these extraResources for the device-identifier into one folder?

Some refactoring (Database, scapy)

Is this in the correct order? Initial DB setup reads as if it should be before starting influx.

SmartHomeBuddy/device-identifier/shbdeviceidentifier/db.py

Lines 122 to 125 in ced0ca0

	self.influx_process = self.start_InfluxDB()
	self._create_SQLite_tables()
	if not self._is_InfluxDB_setup():
	self._do_initial_db_setup()

---

This warning pops up constantly, should it be a trace instead? Or is this a bug?

SmartHomeBuddy/device-identifier/shbdeviceidentifier/db.py

Lines 189 to 192 in ced0ca0

	logger.warning(
	"The InfluxDB setup has already been run. No new token was received. "
	"Check the SQLite DB, if there is an admin token"
	)

---

Any ideas in how to get rid of the CryptographyDeprecationWarning? It is not relevant for our use case and is already fixed in the upcoming scapy release, but would be nice to filter it out until then. Unfortunately a warnings.filter() call does not suffice, since it is raised by scapy internally. Calling the filter before importing scapy does not work either, since the warning is raised as soon as CryptoDeprecationWarning class is imported from cryptography.

Suppressing all scapy messages, as suggested on SO, seems overkill.

[GUI] improve Frame

[Testing] DataLoader.py

Update Readme and add tag

Implement basic UI

• setup MUI

Fix frontend on Windows

Fixing

• grpc errors,
• sqlite problems
• dev-setup

Adapt basic UI for SHB

Git branch cleanup

Wrong path to query files when running cli tool from subfolder

Describe the bug
When running the cli tool from the device-identifier folder, it encounters an error, because the path to a query file is wrong. The program looks for SmartHomeBuddy/device-identifier/device-identifier/shbdeviceidentifier/utilities/queries/get_all_devices.flux, which should be SmartHomeBuddy/device-identifier/shbdeviceidentifier/utilities/queries/get_all_devices.flux instead.

To Reproduce

1. cd device-identifier
2. shbdeviceidentifier

Expected behavior
Device identifier starts without error.

Desktop (please complete the following information):

• PopOS 22.04

Additional context

Traceback (most recent call last):
  File "/home/jo/git/SmartHomeBuddy/device-identifier/.venv/bin/shbdeviceidentifier", line 5, in <module>
    from shbdeviceidentifier import __main__
  File "/home/jo/git/SmartHomeBuddy/device-identifier/.venv/lib/python3.8/site-packages/shbdeviceidentifier/__init__.py", line 1, in <module>
    from shbdeviceidentifier import __main__
  File "/home/jo/git/SmartHomeBuddy/device-identifier/.venv/lib/python3.8/site-packages/shbdeviceidentifier/__main__.py", line 1, in <module>
    from .app import app
  File "/home/jo/git/SmartHomeBuddy/device-identifier/.venv/lib/python3.8/site-packages/shbdeviceidentifier/app.py", line 15, in <module>
    from .db import Database, DataLoader
  File "/home/jo/git/SmartHomeBuddy/device-identifier/.venv/lib/python3.8/site-packages/shbdeviceidentifier/db.py", line 21, in <module>
    from .utilities.app_utilities import resolve_file_path
  File "/home/jo/git/SmartHomeBuddy/device-identifier/.venv/lib/python3.8/site-packages/shbdeviceidentifier/utilities/__init__.py", line 3, in <module>
    from .queries import INFLUX_QUERIES, SQLITE_QUERIES, QUERIES
  File "/home/jo/git/SmartHomeBuddy/device-identifier/.venv/lib/python3.8/site-packages/shbdeviceidentifier/utilities/queries.py", line 12, in <module>
    'get_all_devices': query_file_to_string(q_path / 'get_all_devices.flux'),
  File "/home/jo/git/SmartHomeBuddy/device-identifier/.venv/lib/python3.8/site-packages/shbdeviceidentifier/utilities/queries.py", line 7, in query_file_to_string
    with open(file_path, 'r') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/jo/git/SmartHomeBuddy/device-identifier/device-identifier/shbdeviceidentifier/utilities/queries/get_all_devices.flux'

GUI

Various tasks related to the electron app and its GUI.

• #84
• #94
• improve Devices page
• create Network Data page
• create Alert page
• create Counter measures page
• create Insights page
• create Settings page
• improve logo and color scheme

Select further classifier(s)

• XGBoost

Separate CLI commands from actual logic

In order to use the functionality that the CLI interface provides within the grpc server, we must separate the logic of the CLI functions from the click-wrappers. See the start command: we use app.py:read() as a wrapper for commands.py:read().

We should perform the same refactoring for the other CLI commands.

Handle Flux Queries in separate folder / files