dojot / iotagent-json

IoT Agent for a JSON based protocol (with MQTT transport)

dojot

dojot was conceived with the goal of developing and demonstrating technologies for smart cities. Initially focusing on the pillars of public safety, urban mobility and health. It intends to build a multidisciplinary ecosystem in these areas.

dojot is a Brazilian IoT Platform that came up with an open source proposal in order to facilitate the development of solutions and the IoT ecosystem with local content focused on Brazilian needs.

The platform is one of the outcomes of the project "Plataforma Aberta para IoT e suas Aplicações", which is supported by FUNTTEL via Finep, and led by CPqD, its main executor, in partnership with other science and technology institutions: Instituto Atlântico, Centro de Tecnologia da Informação Renato Archer - CTI, and Fundação de Apoio à Capacitação em Tecnologia da Informação - FACTI.

dojot Manifesto

dojot is a derivation of dojo (or dôjo), a Japanese word that, basically, means "the place where Japanese martial arts are trained".

Dojo is not just a simple training area, it is also a place to be respected as the practitioners' home. Therefore, culturally, it is very common to see the practitioner bowing before entering this area, just as they do before entering their homes.

Linking this concept to the digital world and applying it to software development, dojo represents the place of a developer meeting, seeking for good practices and absorbing the vision of software development as an art.

By combining this concept with the Internet of Things, we pursue IoT software development best practices, bringing together multidisciplinary experts to accelerate solution delivery and reduce the risks inherent in innovation.

dojot = dojô, IoT, and connectivity.

Repository Overview

When the project started, it was decided to create a repository for each service that made up the dojot IoT Platform. However, over time this approach has been shown to have a high maintenance cost. Therefore, all new services are being created in this monorepository (https://github.com/dojot/dojot) and our goal is to migrate the other services to this one in the future.

Getting Help

More documentation can be found here.

If you cannot find an answer for your doubts or need some clarification, don't hesitate, open an issue at https://github.com/dojot/dojot/issues and our team will reply promptly.

If you are interested in taking a course in the dojot Platform, please contact our partner IWF.

Contributing

We welcome contributions, but there are just a few small guidelines you need to follow.

Bug Reports

Before open a bug report, please search the issues list to see if it has already been raised.

If the bug is a new one, go ahead and open an issue at https://github.com/dojot/dojot/issues.

If you find a bug and know how to fix it, please don't hesitate to contribute, raise a pull-request with the bug fix!

Feature Requests

If you are demanding a specific feature, you may open an issue at https://github.com/dojot/dojot/issues describing it. Our team will analyse your needs, and it might be added to our Roadmap.

You may also develop this feature and raise a pull-request, but before doing this, please open an issue to discuss with the dojot team the best solution. Without a previous discussion, your pull-request might be rejected.

Pull-Requests

To submit a pull-request, you must follow some steps:

1. Fork the github repository at https://github.com/dojot/dojot if you haven't already.
2. Clone your fork, create a new branch, push changes to the branch.
3. Add documentation and unit tests as part of the change if it makes sense.
4. Check whether your change doesn't break any existing unit tests.
5. Open a pull request against the development branch.

After raising a pull-request, it will be reviewed by dojot developers and some changes might be requested before merging it.

Copyright and license

The dojot IoT Platform is based on well-known open source software such as Apache Kafka, RabbitMQ, PostgreSQL, MongoDB, Redis, Kong Gateway and VerneMQ, which have their own licenses.

The services developed by the dojot team to integrate these open-source software components and implement the business logics are copyrighted by CPQD <www.cpqd.com.br> and till release v0.5.1 were licensed under the GPLv3. From release v0.5.2 onwards, these services are being licensed under the Apache License, Version 2.0.

You can use and/or modify the software for free (no license costs apply).

iotagent-json's Issues

IoT agent may be too sensible to payloads

On the current tip (74ac1b5) of branch cpqd_master is a little worse for wear than the original version regarding the processing of malformed messages coming from devices.

Instead of discarding messages it cannot parse as its definition of JSON, it will crash, causing all initialization procedure to run again, possibly making its operation harder to keep.

The log for a sample of the behavior described follows.

> node build/main.js "./config.json"

Connecting to broker...
... connected.
Registering callbacks...
... callbacks registered.
... MQTT connection configured.
Subscribing to all topics...
... subscribed.
kafka connection initialized
Got new MQTT message
Topic: /admin/dummy/attrs
Content:{a:"foo", b:"baz"}
undefined:1
{a:"foo", b:"baz"}
 ^

SyntaxError: Unexpected token a in JSON at position 1
    at JSON.parse (<anonymous>)
    at Agent.processMessage (/mmagr/build/iotagent-json.js:48:31)
    at MqttClient.mqttContext.mqttHandler.start (/mmagr/build/iotagent-json.js:52:93)
    at emitThree (events.js:136:13)
    at MqttClient.emit (events.js:217:7)
    at MqttClient._handlePublish (/mmagr/node_modules/mqtt/lib/client.js:905:12)
    at MqttClient._handlePacket (/mmagr/node_modules/mqtt/lib/client.js:301:12)
    at process (/mmagr/node_modules/mqtt/lib/client.js:257:12)
    at Writable.writable._write (/mmagr/node_modules/mqtt/lib/client.js:267:5)
    at doWrite (/mmagr/node_modules/readable-stream/lib/_stream_writable.js:406:64)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: `node build/main.js "./config.json"`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2017-11-22T11_32_05_240Z-debug.log

IoT Agent get stuck if mongodb is not ready

When starting the IoT Agent, if the Mongo DB was unreachable for whatever reason the IoT Agent reaches a state where it gets stuck.

When the mongo db is within range, nothing changes and the IoT Agent remains on that idle state from where it never recovers.

Improve Kafka support for device notifications (from device manager) and device updates (from the device itself)

The current Kafka support is nice, but it can be improved.

Currently, there are two "communication sides":

• Receiving device notifications from device manager: this is when iotagent gets to know all devices whose messages should be processed by it.
• Sending device updates to all components interested in them: when the physical device sends a message to iotagent, it processes it and send a nicer version of this message to other components. #16 allows this update to be sent through Kafka or through Orion.

There are a bunch of configurable things in these two scenarios, but I don't think they are in the same page. For instance, I can configure which topic will be used to receive the device notifications, its maximum read buffer, and so on, but from the device update side there is no configuration whatsoever.

iotagent Kafka support should depend on zookeeper

iotagent connects to Kakfa directly. Instead, it should contact zookeeper to select an instance and, only then, connect to the Kafka instance.

Error publishing to a device

Add support for device-defined timestamp

In the new version of iotagent-json, the device should be able to send all its readings data in batch, in just one (or a few) message. Therefore, it must also include the timestamp of each data, which should be considered by iotagent.

If too many errors occur during initialization, iotagent stalls and does nothing

How to reproduce:

• Remove kafka dependency on docker-compose file (from dojot/docker-compose repository)
• Decrease the amount of time that iotagent waits before trying to connect to Kafka again.

When starting up (with other components from docker-compose), a lot of errors will occur - they mainly indicate that iotagent can't connect to kafka (which is OK, as Kafka is not yet up). When it finally is able to connect to it, it won't receive any further message from it (such as device creation notifications)