project-owl / dms-lite

DMS LITE is the lightweight version of Project OWL's Data Management System. Build to run local on the PaPi

Deprecation Notice

This project is no longer actively worked on in favor of the new version made for Docker. NEW DOCKER VERSION AVAILABLE HERE

About

The PaPi and DMS LITE is a collaborative effort to collect all the data from a ClusterDuck Protocol network locally. It provides a local interface to see network activity and data traffic. The PaPi was brought to life for areas when there is no internet connection available at all. The PaPi can store data and act as a local MQTT broker to communicate with the PaPa Duck.

How it Works

There are two different ways to get your data into the PaPi: using a USB Serial connection or WiFi. Using the serial connection, the Raspberry Pi reads the incoming messages from the serial monitor by a wired connection from the papaduck and writes the data into the database. Using the Wifi connection the Raspberry pi turns into an acces point. This allows the Papa Ducks to send data over WiFi to a MQTT broker that writes the data into the database.

DMS LITE

The DMS LITE is the open source version of the cloud DMS. DMS LITE gives you simple network activity and data visuals.

The PaPa Duck is running a different Firmware than the regular ClusterDuck Protocol PaPa example.

Example of ClusterDuck Protocol Network

Local ClusterDuck Protocol Network

Parts required

• Raspberry Pi (3B+ preffered)

  • for using a pi zero you need a different setup for node.js

• Micro SD-Card 16GB or more

  • Raspbian OS installed

• Mouse and Keyboard

• Monitor

• Wired Ethernet Connection

• Latest Copy of ClusterDuck Protocol

Setup

How To Install

The PaPi and DMS LITE both need some setup before you can start seeing your data. To turn the Raspberry Pi into the PaPi, you will need to install multiple modules.

Prepare Heltec or TTgo Board for PaPi

1. Goto PAPI DMS Lite Examples
2. Select which example you need based off of your particular install
3. Upload the .INO and you are set to move on

Script Install

Both Raspbian images have Python 2 preinstalled but Raspbian Lite does not have python 3 preinstalled. But you will need to set you Pi to use python 3 as default. We recommend using the latest release of Rasbian Desktop with recommened software.

1. Check your python version by running: python --version *** if you don't see python 3 continue on to 2.**
2. nano ~/.bashrc
3. Add this to the very bottom: alias python='/usr/bin/python3'
4. Close terminal and open a new one
5. Try python --version and you should see your default is python 3

For Usb Serial

1. Connect the LoRa board to the Raspberry Pi
2. chmod u+x serial-install.sh will make the code excutable
3. ./Serial-install.sh will make the code run
4. Sit back and kick up your feet because install will take a bit

The DMS Lite will automatically open in a full sized window.

After Install Run

The next time you boot the Pi CD into the DMS-LITE folder and run the following script.

1. chmod u+x run.sh
2. Now that you made the script executable run it by typing: ./run.sh *** This will launch the scripts and bring you into kiosk mode. Which is full screen full resolution with no bars or anything. to exit out of this either reboot the pi or press fn f4 key to get back to desktop. ***

In your Browser go to Localhost:3000 to see the dms-lite

NOTE: You need to run the two scripts in two separate terminals.

For Wifi Acces Point

1. chmod u+x Wifi-install.sh will make the code excutable
2. ./Wifi-install.sh will make the code run
3. sit back and kick up your feet because install will take a bit

NOTE: You need an Internet Connection to install the packages and the DMS LITE application. After the Raspberry Pi is turned into an Access Point it doesn't have WiFi capabilities anymore and you will need a LAN connection for final setup.

After you have installed the required dependencies you can the two scripts.

python3 Wifi_sqlwriter.py

npm run start

In your Browser go to Localhost:3000 to see the dms-lite

NOTE: You need to run the two scripts in two separate terminals.

dms-lite's Issues

Data Visualization

Issue: Currently you can only see your data in table format. While this is useful to see a log of all your data it would also be helpful to some users to be able to generate some simple graphs about their networks. See Examples

Ex: DMS-Lite can be used to inform a user about how their network is performing, the health of devices, sensor data, etc... Anything really that could enhance network management.

Ex: Some of the Clusterduck devices have the ability to send GPS coordinates so maybe an interactive maps page to visualize where your ducks may be deployed.

Create Disk Image for v1.0

When ready for release, create an easy to install disk image of a working v1.0 release

Make install more configurable

Currently, the bash install includes all packages. While this is fine, it would make more sense to have an option to select how the PaPi is interfacing with PaPiDuck (WiFi vs serial vs other).

Dockerize the application

The use of docker containers would simplify the deployment of the software and other operating systems (e.g. Ubuntu Linux, Hypriot OS, Balena OS) could be supported. Furthermore, dependencies can be better managed and the installation can be done via script.

Here is a link to an example that shows how a Node.js application can be dockerized: https://nodejs.org/fr/docs/guides/nodejs-docker-webapp/

https://project--owl.slack.com no longer active

Probably not the right space to leave this issue ... but what replaced your "no longer active" https://project--owl.slack.com?
I would like to motivate at least one of my student to try this superb project idea.
Best regards
Fred
https://people.hes-so.ch/en/profile/jeanfred.wagen

Serial Connection To PaPi

Right now the PaPiDuck sends data to the PaPi using the PaPi's WiFi AP. While this works pretty well, there are openings for issues (such as problems connecting, security, using too much energy) and adds more complexity than necessary.

A more reliable solution would be to have a serial connection between the PaPi and PaPiDuck. This can be done using Tx and Rx pins or using a USB connection.

Typo in serial_sqlwriter.py leading to wrong Date/Time format

Issue: Typo in serial_sqlwriter.py leads to wrong Date/Time format preventing the "online" detection of the ducks-view from working.

Solution: Change line nr. 26 in serial_sqlwriter.py from

theTime = strftime("%Y-%m-%d %Hs:%M:%S", gmtime())

to

theTime = strftime("%Y-%m-%d %H:%M:%S", gmtime())

Can you post some screenshots in readme?

Is this similar to the dashboard shown in Linus Tech Tips video?

serial_sqlwriter.py never creates the clusterData table even when it does not exist

Issue: The try-block executing the "CREATE TABLE IF NOT EXISTS" query is unreachable.

Potential Solution: Move the try-block above the while-loop.

Ability to Edit Data with Buttons

Issue: Currently there is no easy way to select data to export to CSV, Delete data, or Filter out Data

Potential Solution: Some buttons on the front-end that would allow users to easily manipulate the data and export to a CSV if wanted.

Issues in Wifi-install.sh

I noticed some quirks in Wifi-install.sh which are as follows:

• sqlite3 is contained in the Python 3 standard library and doesn't need to be installed, PyPi has no package with that name, hence the pip3 install sqlite3 line should be removed
• doubles dashes for curl paramters in the last line are wrong, it needs to be one dash
• the -y parameter for bash on Raspbian yields this error: bash: -y: invalid option, hence it should be removed from the script
• nodejs 10 installer gives big deprecation warnings, maybe update to 12 ? (I'm no expert on that matter)