codeforafrica / sensors.africa-api

The sensors.AFRICA API receiving, storing, and processing data received from sensors across the continentent. Accessible at https://api.sensors.africa/

sensors.AFRICA API

API to save and access data from deployed sensors in cities all around Africa.

Documentation

The API is documented here.

Development

Gitignore is standardized for this project using gitignore.io to support various development platforms. To get the project up and running:

• Clone this repo

Virtual environment

• Use virtualenv to create your virtual environment; virtualenv venv
• Activate the virtual environment; source venv/bin/activate
• Install feinstaub; pip install git+https://github.com/opendata-stuttgart/feinstaub-api
• Install the requirements; pip install .
• Create a sensorsafrica database with the following sql script:

CREATE DATABASE sensorsafrica;
CREATE USER sensorsafrica WITH ENCRYPTED PASSWORD 'sensorsafrica';
GRANT ALL PRIVILEGES ON DATABASE sensorsafrica TO sensorsafrica;

• Migrate the database; python manage.py migrate
• Run the server; python manage.py runserver

Docker

Using docker compose:

• Build the project; docker-compose build or make build
• Run the project; docker-compose up -d or make up

Docker compose make commands:

• make build
• make up - run docker and detach
• make log - tail logs
• make test - run test
• make migrate - migrate database
• make createsuperuser - create a super user for admin
• make compilescss
• make enter - enter docker shell
• make django - enter docker django shell

NOTE: docker-compose is strictly for development and testing purposes. The Dockerfile is written for production since dokku is being used and it will look for Dockerfile.

Tests

• Virtual Environment; pytest --pylama
• Docker; docker-compose run api pytest --pylama

NOTE: If entrypoint and start scripts are changed, make sure they have correct/required permissions since we don't grant permissions to the files using the Dockerfile. Run the commands:

chmod +x contrib/entrypoint.sh
chmod +x contrib/start.sh

Deployment

Dokku

On your local machine run:

git remote add dokku [email protected]:sensorsafrica-api
git push dokku master

For more information read Deploying to Dokku.

Cronjob

This project uses celery to create cronjobs and flower to monitor the cron jobs as a web admin. To create your jobs, add the task to the tasks.py and CELERY_BEAT_SCHEDULE in settings.py.

Everything starts automatically as setup in the contrib/start.sh:

celery -A sensorsafrica beat -l info &> /src/logs/celery.log  &
celery -A sensorsafrica worker -l info &> /src/logs/celery.log  &
celery -A sensorsafrica flower --basic_auth=$SENSORSAFRICA_FLOWER_ADMIN_USERNAME:$SENSORSAFRICA_FLOWER_ADMIN_PASSWORD &> /src/logs/celery.log  &

Note: If you run the project in the virtualenv you will have to start rabbitmq and pass that link to settings by the env variable SENSORSAFRICA_RABBITMQ_URL

Monitoring

Flower

It starts up in in the contrib/start.sh:

...
celery -A sensorsafrica flower --basic_auth=$SENSORSAFRICA_FLOWER_ADMIN_USERNAME:$SENSORSAFRICA_FLOWER_ADMIN_PASSWORD &> /src/logs/celery.log  &

Slack

Provide channel webhook as an enivronment variable SENSORSAFRICA_CELERY_SLACK_WEBHOOK. The default options are used:

DEFAULT_OPTIONS = {
    "slack_beat_init_color": "#FFCC2B",
    "slack_broker_connect_color": "#36A64F",
    "slack_broker_disconnect_color": "#D00001",
    "slack_celery_startup_color": "#FFCC2B",
    "slack_celery_shutdown_color": "#660033",
    "slack_task_prerun_color": "#D3D3D3",
    "slack_task_success_color": "#36A64F",
    "slack_task_failure_color": "#D00001",
    "slack_request_timeout": 1,
    "flower_base_url": None,
    "show_celery_hostname": False,
    "show_task_id": True,
    "show_task_execution_time": True,
    "show_task_args": True,
    "show_task_kwargs": True,
    "show_task_exception_info": True,
    "show_task_return_value": True,
    "show_task_prerun": False,
    "show_startup": True,
    "show_shutdown": True,
    "show_beat": True,
    "show_broker": False,
    "use_fixed_width": True,
    "include_tasks": None,
    "exclude_tasks": None,
    "failures_only": False,
    "webhook": None,
    "beat_schedule": None,
    "beat_show_full_task_path": False,
}

Sentry

Set the enivronment variable SENSORSAFRICA_SENTRY_DSN.

Archiving

Archives are sent to CKAN and require environment configuration:

- CKAN_ARCHIVE_API_KEY=..
- CKAN_ARCHIVE_OWNER_ID=...
- CKAN_ARCHIVE_URL=<url that supports bulk uploads>

Contributing

opendata-stuttgart/feinstaub-api prefer generating and applying migration to the database at the point of deployment (probably to reduce the number of changes to be applied). We, on the other hand, prefer the Django recommended approach of creating and reviewing migration files at the development time, and then applying the same migration files to different environments; dev, staging and eventually production.

Hence, with any contribution, include both sensors.AFRICA-api and opendata-stuttgart/feinstaub-api migration files by running python manage.py makemigrations command before creating a PR.

License

GNU GPLv3

Copyright (C) 2018 Code for Africa

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.

sensors.africa-api's Issues

CSRF validation failed in Django 4

A list of CSRF_TRUSTED_ORIGINS is required for unsafe requests.

Raised error: Origin checking failed - <custom_URL> does not match any trusted origins
A wildcard to support all subdomains would be suitable. @thepsalmist

Lint test failing on master

The fail started with this commit c6463a2

Sensors app views issues

Accessing views throws "object not callable" error on Django v4 upgrade from v1.

E.g. accessing this v1/data endpoint on a development instance will through this error; raised when calling a bool as a function is_authenticated() . Of course, removing the brackets works as expected. What baffles me is that this works in the production instance running on Django v1. @thepsalmist would you know anything about this?

Make route return sensor location

The/v1/sensors/<sensor_id> should return location.

Current request :

• http://api.sensors.africa/v1/sensors/93/

[
    {
        "timestamp": "...",
        "sensordatavalues": [...]
    }
]

Requirement:

[
    {
        "location": { ... },
        "timestamp": "...",
        "sensordatavalues": [...]
    }
]

TODO:

• Add location
• Add test

[#170920229] Make archive task CKAN URL configurable from env variable

To improve ease of use, we should make the CKAN URL for archive script configurable from env variable.

Move cache_lastactive_nodes back to celery tasks

• ssh into dokku-2.sensors.africa
• run command crontab -l
• You will a task that should be in celery but it is here because of issues with tasks taking too long

# Edit this file to introduce tasks to be run by cron.
# 
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
# 
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').# 
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
# 
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
# 
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
# 
# For more information see the manual pages of crontab(5) and cron(8)
# 
# m h  dom mon dow   command
*/5 * * * * dokku enter sensorsafrica-production web python manage.py cache_lastactive_nodes

cc @kilemensi

Use TimeSeries DB

Convert API to a time series database in this PR #63

Confirm the priority level of this with @chegejames and if we still want to move to this direction.

The app will still keep track of sensors, owners and locations but the data will exist in the TimeSeries database.

cc @kilemensi

---

All management/commands except upload to CKAN should not be needed after this including the caching JSON files since InfluxDB is super fast. However, make sure the routes don't change to ensure no breaking.

ValueError: Cannot assign "<SensorData: esp8266-5926214 7 [2]>": the current database router prevents this re...

Sentry Issue: SENSORSAFRICA-API-JS

ValueError: Cannot assign "<SensorData: esp8266-5926214 7 [2]>": the current database router prevents this relation.
(14 additional frame(s) were not displayed)
...
  File "rest_framework/serializers.py", line 686, in to_representation
    self.child.to_representation(item) for item in iterable
  File "django/db/models/query.py", line 250, in __iter__
    self._fetch_all()
  File "django/db/models/query.py", line 1121, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "django/db/models/query.py", line 83, in __iter__
    setattr(obj, field.name, rel_obj)
  File "django/db/models/fields/related_descriptors.py", line 226, in __set__
    raise ValueError('Cannot assign "%r": the current database router prevents this relation.' % value)

ValueError: Cannot assign "<...>": the current database router prevents this relation

Operation on DB models are prevented due to DB relation issues; possibly due to an additional read replica instance. Issue on bug_fix/module_dependencies and sensorsafrica-staging branches used for development.