rpj / rpi

RPJiOS: RPJ's RPi OS, a sensor data platform for the Raspberry Pi built with python2.7 and redis.

RPJiOS

A pub/sub-based implementation of a Raspberry Pi data pipeline built on redis and centered around sensors.

The general philosophy is that a "sensor" is any entity (physical or not) that operates primarily in an output-only mode (configuration doesn't necessarily count as an "input" so is allowable).

These outputs are treated ephemerally, in a "fire and forget" manner, creating a data stream. The intent is for interested entities to subscribe to the data stream and transform, interpret and/or persist it according to their requirements.

Caveat emptor

This is still very much an active work-in-progress! However, as it is functional and actively deployed I figured it was worth making public in the event it might help others in their projects.

Those current deployments consist of my atmospheric particulate matter sensor and my garden monitoring bots, with the entire system handling (on average) about a half-million units of sensor data per day.

Requirements

• Hardware:
  • a Raspberry Pi running a recent Raspbian build (for sensor nodes)
  • any Debian-like system running apt (for managment/non-sensor nodes)
• some sensors, configured appropriately (most easily done with raspi-config):
  • depending on your chosen sensors: I2C enabled, SPI enabled, 1-wire enabled
  • other sensors (LM335, Soil, TEPT5700) require an external ADC which itself will require SPI

Setup

• Clone the repo
• cd into repo dir
• ./setup.sh (you might need to enter your sudo password to install requirements)
• source env/bin/activate and go!

Tools

• sensors-src: source daemon that manages all specified sensors and publishes their data as configured
• downsample: a very flexible data stream downsampler/forwarder/transformer. example uses:
  • an at-frequency forwarder (set -r 1)
  • a loopback downsampler (set -o to the same as -i)
  • a many-to-one reducer (set -t to key)
  • a one-to-many exploder (set -m to flatten:[options])
  • a bounded ephemeral cache (set -t to list:[options], where the limit=X option sets the bound)
  • ...
  • profit!
• sqlite-sink: an SQLite sink for data streams. examples:
  • sink to an SQLite database on a different host a downsampled data stream:
    1. on the source device:
       • downsample -i redis://localhost -o redis://sql-db-host -r ... -p ...
    2. on the sink host "sql-db-host":
       • sqlite-sink path-to-db.sqlite3
         • (lots of "TODOs" here, obviously)
• oled-display: an OLED display driver for consuming & display some sensor data, among other things
• thingspeak: a simple example of a ThingSpeak data forwarder for the SPS30 particulate matter sensor data. Example resulting data set.

Library

Sensors

The following are currently supported (with the required drivers / interfaces setup of course):

• SPS30 I2C particulate matter air quality sensor. Source.
• BME680 temperature / humidity / barometeric pressure / volatile organic compound I2C sensor. My setup uses AdaFruit's awesome breakout board for ease-of-integration. Source.
• BME280 temperature / humidity / barometeric pressure I2C sensor. Source.
• DHTXX(DHT11/DHT22) temperature / humidity sensors. I use DFRobot's breakouts but you can find similar breakout's from many (re)sellers online. Source.
• DS18S20 high-precision 1-wire temperature sensor. Source.
• LM335 precision analog temperature sensor. Source.
• Capacative soil moisture sensors such as the DFRobot SEN0114 or any simple-to-build capactive analog moisture sensor. Source.
• TEPT5700 ambient light sensor. Source.
• "Virtual" sensors such as SysInfo and NetInfo that do not require any additional hardware.

Variants of these sensors could likely be made to work with this system via simple modifications if any are required at all.

Devices

• A simple Python wrapper driver for the SPS30 sensor is included here, wrapping the included embedded-sps I2C driver from Sensiron implemented in C (forked to include a shared object build step for Python consumption).
• A simple Python driver for the 74HC595 8-bit shift register is included here.
• The venerable MCP3008 10-bit 8-channel analog-to-digital converter is directly supported by the base analog sensor implementation. Other ADCs would be quite simple to adapt (even more so if I implemented a HAL... #TODO).
  • channel (zero-indexed) is specified in config.json

Related

• The old now-archived repository from whence this all came
  • might still contain some useful stuff: D3-based "live" analog data plotting code in rpjctrl, and I'm still using ledCount.py on some of my units because I'm too lazy to re-write it properly

rpi's Issues

sensors-src should wait for updated time fix before publishing

On a reboot, a device may have a localtime value that is stale from the last boot, and before the device updates time (from NTP) it can start publishing sensor data with stale timestamps. This isn't a huge deal, but it does cause dirty data.

Re-implement using streams

Redis 5.0 landed the feature that this code base half-implements! The examples in the docs are even sensor-based! Re-implement using streams, obviously. (Will need relatively wide adoption of v5, though)

REDIS_URL should be a CL arg for sensors-src (or something)

REDIS_URL handling needs to be fixed, is really the point.

Implement gpsd client support (for NEO-6M modules)

Will require a re-thinking of sensor runlooping (and therefore is related to #3), as ideally any NMEA stream-producing sensor should act in a push-only manner rather than the current sensor-src-tick-based-pull mechanism (confusingly named _runloop, hence #3).

Redis 5 streams would come in really handy here, but an implementation based on lists and using BLPOP to block consumers until a new datum is available could work well too.

thingspeak hangs

The thingspeak tool seems to hang very occasionally after running fine for long periods (~days), and seems to do so somewhere in the PSubscriber code for listening to redis. That's about all I know now.

Python 3

It's going to have to happen someday...

Fix _runloop inconsistency

_runloop is really _tick: fix this, and improve the implementation in general (to allow both!)

Is SPS30 self-cleaning running every so often?

Need to make this configurable and be sure it's setup to happen every so often (~once/week)!

Versioning!

Doit. Doit.

Discovery needs to set an expiry for HKeys it sets

So the discover information auto-cleans itself up after a bit of time (could be awhile too, 15 minutes or even an hour).

downsample & sqlite-sink leaking!

On charlie, the forwarding invocations of this seem to be leaking quite badly over long periods (these have been running for about 9 days):

ryan      3038  0.1 17.1 2233072 2090796 pts/1 S+   Jan06  20:54 python ./bin/downsample -i redis://u:XXX@localhost -o redis://h:[email protected]:15979 -p * -r 10 -t list:suffix=.list,limit=5000 -m flatten:mapping=mapping.json,include-raw=1
ryan      4796  0.1 16.5 2166996 2024552 pts/3 S+   Jan06  20:10 python ./bin/downsample -i redis://u:XXX@localhost -o redis://h:[email protected]:15979 -p * -r 10 -m flatten:mapping=mapping.json,include-raw=1 -t key

The cause isn't immediately obvious, though I'm suspecting the internal windowed cache isn't windowing or something.

Refactor "downsample"

The downsample tool has grown far too ungainly as it has gained features (and still needs to gain another feature with the integration of the thingspeak forwarder, #8). It needs to re-factored, maybe re-thought. There's also that send-outside-the-send-thread problem with lists. That's bad, mmkay.

URLError: <urlopen error [Errno -3] Temporary failure in name resolution>

Hello,
I am a new to these types of systems and following the guild I managed to get my system up and running and data feeding into Thingspeak, Channel ID: 802605

Issue is the code is failing to a bad internet connection I have, as I run your system through a VPN.
Thought I set up the device in tehran for a laugh as I am on Holiday here.

Are you able to just keep requesting the URL in a constant loop until it connects again?

More code here:
Success: entry #792 added
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/pi/rpi/env/local/lib/python2.7/site-packages/rpjios/SubscriberBase.py", line 54, in _sub_thread
map(lambda sf: sf(m), self._subs)
File "/home/pi/rpi/env/local/lib/python2.7/site-packages/rpjios/SubscriberBase.py", line 54, in
map(lambda sf: sf(m), self._subs)
File "./bin/thingspeak", line 39, in msg_rx
_resp = urllib2.urlopen(_url)
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
context=self._context)
File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
raise URLError(err)
URLError: <urlopen error [Errno -3] Temporary failure in name resolution>

^CTraceback (most recent call last):
File "./bin/thingspeak", line 57, in
time.sleep(0.5)
KeyboardInterrupt

Plus great job on the code even for a total nobe I got it working.

Cheers.
Mehdi

"thingspeak" really needs to just be a "downsample" output option

Could even implicitly enable via a URL scheme for -o such as thingspeak://WRITE_API_KEY@CHANNEL_ID. Also, downsample really needs to be re-factored. Should probably file a separate bug for that...: #10

Produce AQI numbers from raw PM data

Per this calculator from the EPA, concentrate-to-AQI JS source here.

setup.sh's SPS30 build is broken

A couple folks on Hackster.io have gotten in touch with trouble building the SPS30 submodule via setup.sh, and I've been able to reproduce it on stock current Raspbian (2018-11-13):

pi@raspberrypi:~/rpi $ ./setup.sh 
*** RPi platform detected: building sensor drivers and including hardware interface modules.

* Looking for virtualenv: NOT FOUND
* Trying to install virtualenv...
* Looking for redis-server: NOT FOUND
* Trying to install redis-server...
* Looking for zip: NOT FOUND
* Trying to install zip...
* Looking for python-dev: NOT FOUND
* Trying to install python-dev...
* Looking for libjpeg9-dev: NOT FOUND
* Trying to install libjpeg9-dev...
* Looking for python-smbus: NOT FOUND
* Trying to install python-smbus...
* Fetching embedded-sps submodule: The authenticity of host 'github.com (192.30.253.113)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
Are you sure you want to continue connecting (yes/no)? yes
./setup.sh: line 45: pushd: embedded-sps: No such file or directory
done
* Building embedded-sps submodule: ./setup.sh: line 51: pushd: release/sps30-i2c: No such file or directory
./setup.sh: line 52: pushd: hw_i2c: No such file or directory
mv: cannot stat 'sensirion_hw_i2c_implementation.c': No such file or directory
./setup.sh: line 55: popd: directory stack empty
failure!
./setup.sh: line 64: popd: directory stack empty
./setup.sh: line 65: popd: directory stack empty
* Initializing virtualenv: ^Cfailed! Cannot continue.

Fix shared_object_path configuration for SPS30

Fix ThingSpeak timestamps!

Ugh.