srishti-j18 / dirtviz

DirtViz is a project to visualize data collected from sensors deployed in sensor networks.

DirtViz

DirtViz is a project to visualize data collected from sensors deployed in sensor networks. The project involves developing web based plotting scripts to create a fully-fledged DataViz tool tailored to the data collected from embedded systems sensor networks. A live version of the website is available at https://dirtviz.jlab.ucsc.edu/.

Dependencies

Dependency
Python
Docker

Getting Started

Starting Services

A local version of Dirtviz can be started using docker-compose.yml. This will build the local images and start the required services in the background, including the database.

docker compose up --build -d

At this point the portal is accessible at http://localhost:3000/, but will likely show a blank page in your web browser and throw an error. This is due to the database being empty, therefore there is no data to display.

Migrate Database

Alembic is used to manage database migrations. Alembic is a python package and acts as a complement to sqlalchemy which is used to query the database. The following will upgrade the database to the most recent version and should be done any time the database schema changes.

NOTE: It is recommended that a virtual environment is setup and ALL dependencies are installed via pip install -r requirements.txt. If you are unsure what this means, read this.

A migration script is provided in this repository that abstracts the migration process.

# To check out usage run
./migrate.sh -h

Import Example Data

Real life example data is provided and can be imported with the following. The data was collected at UC Santa Cruz by jlab.

python ./import_example_data.py

Now some graphs should appear on the website and look like the following.

FAQ

How do I create database migrations?

This projects makes use of alembic to handle database migrations and flask-migrate as an extension to make alembic operations avaliable through the Flask cli. It is recommended to have a understanding of the package first before attempting to modify the database schema. Due to the way that alembic handles package imports, the config file needs to be specified while running from the root project folder. For example the following will autogenerate new migrations from the latest revision of the database.

The script migrate.sh takes in a "-m <msg>" for generating a new migration and by itself runs "alembic upgrade head".

NOTE: Autogeneration of migrations requires a running version of the database. Refer above to see how to create and connect to a local version of the database

./migrate.sh -m "migration message here"

How do I reset the local database?

Sometimes the database breaks and causes errors. Usually deleting the docker volume postgresqldata causing the database to be recreated fixes the issue. The following does exactly that and reapplies the migrations to the cleaned database.

docker compose down
docker volume rm dirtviz_postgresqldata
docker compose up --build -d
./migrate.sh -u

[Flask-migrate] Error: Can't locate revision identified by 'e5dbb2a59f94'

For this error, it either means that you've deleted a revision corresponding to the id located in ./backend/api/migrations/versions or that if it's during the deployment process, the alembic version in the db (under the alembic version table) is mismatched. Double check to see if the revision history is the same for both deployment and locally.

How do I import my own TEROS and Rocketlogger data previously collected?

There exists csv importers that can be used to populate the database. Python utilities currently exist to import RocketLogger and TEROS data. These are available as modules under dirtviz. More information on used can be found by running the modules with the --help flag.

python -m backend.api.database.utils.import_cell_data