balena-labs-projects / connector Goto Github PK
View Code? Open in Web Editor NEWAuto-configured data connector block based on Telegraf
Auto-configured data connector block based on Telegraf
When the block detects an input (sensor, HTTP, MQTT...etc) it should log the input. The user should be very clear about what the block as auto-wired.
Instead of enabling or disabling json_query = "payload_fields"
, allow the value set in the env var to pass through to this query to allow the user to specify their own value for json_query
.
It looks like python is not reading packages under /root/.local
. I get this error when running autowire
root@cceab6e:~# balena run --rm -ti --privileged balenablocks/connector sh
/app # python3 autowire.py
Traceback (most recent call last):
File "/app/autowire.py", line 6, in <module>
from pluginbase import PluginBase
ModuleNotFoundError: No module named 'pluginbase'
This seems to work on pi 3
I tried to run the internetspeedtest fleet on my NUC and got the error
Error: no matching manifest for linux/amd64 in the manifest list entries
The connector service references this block's pre-built images, so it seems they haven't been built for all architectures
https://github.com/balenalabs-incubator/internetspeedtest/blob/d8e94b670971f68919962de19cf24943eae4f942/docker-compose.yml#L17
The block is currently a black box after generating the config. If, for example, you push data to MQTT but it's not hitting your data sink - there's no way of knowing why. Implement debug logging of some sort.
The published Readme refers to Browser on both the BalenaHub published build as well as the source.
Use case: I want to pull two different types of data from two completely different endpoints with different authorization.
Should we assume that if a user sets the name for the external HTTP listener service that they want it enabled?
Currently you can provide a custom telegraf.conf
file by extending the block image but that disables autowire feature. Consider allowing both behaviours simultaneously.
Hello! This issue is about your example configuration for a mqtt client.
The code you have states that you should be able to connect via the localhost keyword:
client = mqtt.Client("1")
client.connect("localhost")
while(True):
value = GetReading() # code omitted for brevity
client.publish("sensors",json.dumps(value))
time.sleep(5)
However, when I try this, I get the following error:
14.06.21 19:15:52 (-0700) co2 Traceback (most recent call last):
14.06.21 19:15:52 (-0700) co2 File "/usr/src/co2.py", line 85, in
14.06.21 19:15:52 (-0700) co2 mqtt_client.connect("localhost", 1883, 60)
14.06.21 19:15:52 (-0700) co2 File "/usr/local/lib/python3.9/site-packages/paho/mqtt/client.py", line 941, in connect
14.06.21 19:15:52 (-0700) co2 return self.reconnect()
14.06.21 19:15:52 (-0700) co2 File "/usr/local/lib/python3.9/site-packages/paho/mqtt/client.py", line 1075, in reconnect
14.06.21 19:15:52 (-0700) co2 sock = self._create_socket_connection()
14.06.21 19:15:52 (-0700) co2 File "/usr/local/lib/python3.9/site-packages/paho/mqtt/client.py", line 3546, in _create_socket_connection
14.06.21 19:15:52 (-0700) co2 return socket.create_connection(addr, source_address=source, timeout=self._keepalive)
14.06.21 19:15:52 (-0700) co2 File "/usr/local/lib/python3.9/socket.py", line 843, in create_connection
14.06.21 19:15:52 (-0700) co2 raise err
14.06.21 19:15:52 (-0700) co2 File "/usr/local/lib/python3.9/socket.py", line 831, in create_connection
14.06.21 19:15:52 (-0700) co2 sock.connect(sa)
14.06.21 19:15:52 (-0700) co2 OSError: [Errno 99] Cannot assign requested address
I'm running a device with the sensor service from balenaSense, I've changed the port to 7575 in the code, tried EXPOSE 7575
in the Dockerfile, and tried setting the port in the compose file. The service is accessible on port 7575 externally (which I don't actually want in this case), but it isn't automatically detected by transport. Could this be because my device is in local mode?
10.02.21 10:52:17 (+0000) connector 2021-02-10T10:52:17Z W! [inputs.mqtt_consumer] Server "mqtt:1883" should be updated to use scheme://host:port
format
10.02.21 10:52:17 (+0000) connector 2021-02-10T10:52:17Z I! [inputs.mqtt_consumer] Connected [mqtt:1883]
For some HTTP API endpoints the client will be required to authenticate, we should add the ability to specify this header. https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Authorization
Currently when autowire finds a MQTT server with the name "mqtt" it connects as a data input source using both "sensors/#" and "balena/#" as topics. Consider allowing the user to add/change/delete topics w/o having to disable autowire and using a custom telegraf.conf file.
Currently, when using the External HTTP Listener plugin, data posted to this endpoint from multiple sources ends up in the same place within InfluxDB. This works if you have two different data sources, but if you had two sources the same (two identical sensors in different locations, for example), that post the same data structure, it won't be possible to differentiate between the two.
[Logs] [10/5/2020, 7:13:35 PM] Restarting service 'connector sha256:5c237eec5a5eed3dc95ac86b8abdd25370976d5da59c928fc16045e7e67f0594'
[Logs] [10/5/2020, 7:12:33 PM] [connector] Traceback (most recent call last):
[Logs] [10/5/2020, 7:12:33 PM] [connector] File "/root/.local/lib/python3.5/site-packages/balena/models/release.py", line 98, in get
[Logs] [10/5/2020, 7:12:33 PM] [connector] rt = self.__get_by_raw_query(raw_query)
[Logs] [10/5/2020, 7:12:33 PM] [connector] File "/root/.local/lib/python3.5/site-packages/balena/models/release.py", line 75, in __get_by_raw_query
[Logs] [10/5/2020, 7:12:33 PM] [connector] raise exceptions.ReleaseNotFound(raw_query)
[Logs] [10/5/2020, 7:12:33 PM] [connector] balena.exceptions.ReleaseNotFound: $filter=startswith(commit, 'None')
[Logs] [10/5/2020, 7:12:33 PM] [connector]
[Logs] [10/5/2020, 7:12:33 PM] [connector] During handling of the above exception, another exception occurred:
[Logs] [10/5/2020, 7:12:33 PM] [connector]
[Logs] [10/5/2020, 7:12:33 PM] [connector] Traceback (most recent call last):
[Logs] [10/5/2020, 7:12:33 PM] [connector] File "./autowire.py", line 82, in <module>
[Logs] [10/5/2020, 7:12:33 PM] [connector] config = autowire.GetConfig()
[Logs] [10/5/2020, 7:12:33 PM] [connector] File "./autowire.py", line 48, in GetConfig
[Logs] [10/5/2020, 7:12:33 PM] [connector] services = self.GetServices()
[Logs] [10/5/2020, 7:12:33 PM] [connector] File "./autowire.py", line 24, in GetServices
[Logs] [10/5/2020, 7:12:33 PM] [connector] release = self.balena.models.release.get(commit)
[Logs] [10/5/2020, 7:12:33 PM] [connector] File "/root/.local/lib/python3.5/site-packages/balena/models/release.py", line 105, in get
[Logs] [10/5/2020, 7:12:33 PM] [connector] raise exceptions.ReleaseNotFound(commit_or_id)
[Logs] [10/5/2020, 7:12:33 PM] [connector] balena.exceptions.ReleaseNotFound: None
[Logs] [10/5/2020, 7:13:35 PM] [connector]
[Logs] [10/5/2020, 7:13:35 PM] [connector] Changing hostname to aa5da0b
[Logs] [10/5/2020, 7:13:35 PM] [connector] Generating config
[Logs] [10/5/2020, 7:13:35 PM] [connector] /root/.local/lib/python3.5/site-packages/pluginbase.py:439: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release ofcryptography. Please upgrade your Python.
[Logs] [10/5/2020, 7:13:35 PM] [connector] fromlist, level)
[Logs] [10/5/2020, 7:13:37 PM] [connector] balenablocks/connector
[Logs] [10/5/2020, 7:13:37 PM] [connector] ----------------------
[Logs] [10/5/2020, 7:13:37 PM] [connector] Intelligently connecting data sources with data sinks
[Logs] [10/5/2020, 7:13:38 PM] [connector] Traceback (most recent call last):
[Logs] [10/5/2020, 7:13:38 PM] [connector] File "/root/.local/lib/python3.5/site-packages/balena/models/release.py", line 98, in get
[Logs] [10/5/2020, 7:13:38 PM] [connector] rt = self.__get_by_raw_query(raw_query)
[Logs] [10/5/2020, 7:13:38 PM] [connector] File "/root/.local/lib/python3.5/site-packages/balena/models/release.py", line 75, in __get_by_raw_query
[Logs] [10/5/2020, 7:13:38 PM] [connector] raise exceptions.ReleaseNotFound(raw_query)
[Logs] [10/5/2020, 7:13:38 PM] [connector] balena.exceptions.ReleaseNotFound: $filter=startswith(commit, 'None')
[Logs] [10/5/2020, 7:13:38 PM] [connector]
[Logs] [10/5/2020, 7:13:38 PM] [connector] During handling of the above exception, another exception occurred:
[Logs] [10/5/2020, 7:13:38 PM] [connector]
[Logs] [10/5/2020, 7:13:38 PM] [connector] Traceback (most recent call last):
[Logs] [10/5/2020, 7:13:38 PM] [connector] File "./autowire.py", line 82, in <module>
[Logs] [10/5/2020, 7:13:38 PM] [connector] config = autowire.GetConfig()
[Logs] [10/5/2020, 7:13:38 PM] [connector] File "./autowire.py", line 48, in GetConfig
[Logs] [10/5/2020, 7:13:38 PM] [connector] services = self.GetServices()
[Logs] [10/5/2020, 7:13:38 PM] [connector] File "./autowire.py", line 24, in GetServices
[Logs] [10/5/2020, 7:13:38 PM] [connector] release = self.balena.models.release.get(commit)
[Logs] [10/5/2020, 7:13:38 PM] [connector] File "/root/.local/lib/python3.5/site-packages/balena/models/release.py", line 105, in get
[Logs] [10/5/2020, 7:13:38 PM] [connector] raise exceptions.ReleaseNotFound(commit_or_id)
[Logs] [10/5/2020, 7:13:38 PM] [connector] balena.exceptions.ReleaseNotFound: None
[Logs] [10/5/2020, 7:13:39 PM] Service exited 'connector sha256:5c237eec5a5eed3dc95ac86b8abdd25370976d5da59c928fc16045e7e67f0594'
This applies at the least to HTTP pull and push, unsure about other inputs.
If we add something like json_string_fields = ['string_*', 'boolean_*']
to the telegraf configuration it means fields in the received JSON with these prefixes will be allowed to pass to the database, without this, strings and boolean values are dropped indiscriminately.
Under the "Extended image configuration" section of the readme.md it suggests that you can include your own Telegraf config file by using the dockerfile.template option. Currently this appears to not work due to an issue with how entry.sh generates the final exec command.
Per the readme.md file, using this in the template file:
FROM balenablocks/connector:%%BALENA_MACHINE_NAME%%
COPY customTelegraf.conf .
CMD ["--config customTelegraf.conf"]
generates the following error in the container:
<...snip...>
21.03.22 14:22:41 (-0400)
21.03.22 14:22:41 (-0400) balenaBlocks connector version: 1.1.2
21.03.22 14:22:41 (-0400) Changing hostname to c5ab484
21.03.22 14:22:41 (-0400) flag provided but not defined: -config ./customTelegraf.conf
21.03.22 14:22:41 (-0400) Telegraf, The plugin-driven server agent for collecting and reporting metrics.
21.03.22 14:22:41 (-0400)
21.03.22 14:22:41 (-0400) Usage:
<...snip...>
note the missing dash in front of the config option, should be "--config ./customTelegraf.conf"
Changing the CMD line to: CMD [" --config customTelegraf.conf"] (note the space between the double quote and the first dash) generates a different error:
<...snip...>
21.03.22 14:27:38 (-0400)
21.03.22 14:27:38 (-0400) balenaBlocks connector version: 1.1.2
21.03.22 14:27:38 (-0400) Changing hostname to c5ab484
21.03.22 14:27:38 (-0400) /app/entry.sh: line 27: /app/ --config ./customTelegraf.conf: No such file or directory
21.03.22 14:27:39 (-0400) Service exited 'connector sha256:12372dfc2677cb971debefb074f19408c5ed77b7d10d65cce930745220980e99'
<...snip...>
The option flag is now correct but the command to run appears to be "/app/" and not "./telegraf". I suspect there is something new with how this bit of code below from entry.sh is executed which I do not understand:
<...snip...>
if [[ "${1#-}" != "$1" ]]; then
set -- ./telegraf "$@"
fi
exec "$@"
<...snip...>
This was working correctly until yesterday when it suddenly started getting the following error. As far as I know, nothing had changed on the device. It might have lost power at some point, so potentially things got corrupt but this failure seems like maybe something changed on the balena API?
22.11.22 18:43:28 (+0100) connector Traceback (most recent call last):
22.11.22 18:43:28 (+0100) connector File "/app/./autowire.py", line 96, in <module>
22.11.22 18:43:28 (+0100) connector config = autowire.GetConfig()
22.11.22 18:43:28 (+0100) connector File "/app/./autowire.py", line 62, in GetConfig
22.11.22 18:43:28 (+0100) connector services = self.GetServices()
22.11.22 18:43:28 (+0100) connector File "/app/./autowire.py", line 33, in GetServices
22.11.22 18:43:28 (+0100) connector device = self.balena.models.device.get_with_service_details(device_id, True)
22.11.22 18:43:28 (+0100) connector File "/root/.local/lib/python3.9/site-packages/balena/models/device.py", line 309, in get_with_service_details
22.11.22 18:43:28 (+0100) connector raw_data = self.base_request.request(
22.11.22 18:43:28 (+0100) connector File "/root/.local/lib/python3.9/site-packages/balena/base_request.py", line 197, in request
22.11.22 18:43:28 (+0100) connector raise exceptions.RequestError(response._content)
22.11.22 18:43:28 (+0100) connector balena.exceptions.RequestError: b'Unauthorized'
The telegraf Azure IOT Hub output plugin is await further development, and may be some time before it's usable. Develop a way to get data to Azure via IOT Hub, with no effort from the Edge developer.
When using livepush or if the fleet has no releases yet the autowire will fail (https://github.com/balenablocks/connector/blob/9e40e50439db37053b79ac0f0187f18aadbfb61e/autowire.py#L16-L26)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.