os2iot / os2iot-backend Goto Github PK

Example without any sensitive information (emails names have been removed):

stdout,"�[95m[Nest] 18 - �[39m10/07/2021, 1:58:51 PM �[38 5 3m[PayloadDecoderExecutorService] �[39m�[95mReturned: '{""payload"":{""applicationID"":""1"",""applicationName"":""os2iot-2132e4b0-549d-4d39-9431-d1202889a1d6"",""deviceName"":""os2iot-70b3d563800001d5"",""devEUI"":""70b3d563800001d5"",""rxInfo"":[{""gatewayID"":""7276ff002e050870"",""uplinkID"":""c22d1d91-f063-4773-a187-f67aef492616"",""name"":""2074-LRW-01"",""time"":""2021-10-07T13:58:50.779646Z"",""rssi"":-115,""loRaSNR"":-7,""location"":{""latitude"":56.288482666015625,""longitude"":10.216209411621094,""altitude"":86}}],""txInfo"":{""frequency"":868500000,""dr"":3},""adr"":true,""fCnt"":10743,""fPort"":2,""data"":""00BA5FDC""},""metadata"":{""id"":11,""createdAt"":""2021-08-09T08:48:14.307Z"",""updatedAt"":""2021-08-09T08:48:14.307Z"",""name"":""70B3D563800001D5"",""location"":{""type"":""Point"",""coordinates"":[10.2282129,56.2363043]},""commentOnLocation"":null,""comment"":null,""metadata"":null,""type"":""LORAWAN"",""deviceEUI"":""70b3d563800001d5"",""chirpstackApplicationId"":1,""application"":{""id"":7,""createdAt"":""2021-08-06T11:10:55.750Z"",""updatedAt"":""2021-08-11T08:12:06.937Z"",""name"":""MTM - Cykelsti (udeklima)"",""description"":""OPI samarbejde med Sensade omkring måling af temperaturer i cykelstier. \n\nStatus: Projektforløb\n\nStart: primo 2020\nSlut: Forventet slutdato Q3 2021\n\nHardware: 8 x sensorer fra Sensade (+ 5 fra tidligere projekt der gerne må slettes når de løber tør)\nData: Temperatur\nForbindelsesteknologi: LoRaWan\n\nEjer: Drift @ MTM\n\nKontakt:\nITK, NAME OF PERSON, EMAIL OF SAID PERSON ""},""deviceModel"":null}}'�[39m","10/7/2021, 3:58:51.214 PM","aks-defaultpool-23477378-vmss000005","os2iot/os2iot-backend","7434902b-2a09-47cf-a6c2-3b1d327eb0ba/os2iot-backend",7ab8aced70f9d4ea71c76f438b9a9ed003f5add2df216c52190a3d998a947716

This is saved in the log. First of all, it's a lot to log. 8mb data from 4 hours. Purge that bad boy.

Second of all, the actual time-series data is saved in the log. Although we technically do not save the data as time-series, it is possible to recreate all data that have been sent through the platform through the log. This is problematic.

We've just tested a bulk import of exactly 50 sensors.

We had to import the CSV several times to import all sensors - and technically failed to do so with one file.

The first import managed to import 29 sensors.

At first, we waited. Then we said, let there be light... and slowly a few sensors were added to the application count at a time.
First 1, then like 8, then a few refreshes more... etc.
The next day we were at 29. Which we probably reached within a few minutes, but since we were unaware of the issue at the time, we kept waiting.

So the next day we added the entire CSV again. Got the following errors on most sensors. Even some that were successfully added, so not sure what they all mean.

SensorX | LoRaWAN | Failed | Id skal være unikt. Det pågældende Id er allerede brug
SensorX+1 | LoRaWAN | Failed | write tcp XX.x.0.154:46986->XXX.x.0.56:5432: write: connection reset by peer
SensorX+2 | LoRaWAN | Failed | EOF

Unique ID makes perfect sense. We are adding the same sensor several times, so this one should be there. But it wasn't consistent.

About 4 more sensors were added during the 2nd import.

We persisted with the same CSV file, and after 5 imports we had 49/50 sensors on.

The last refused to be added through the bulk import file. No matter how many times we tried.

The error would still read a mixture between the error codes shown above, so just because there were 49 added already, it didn't just read unique ID error for all but one.

We had to manually figure out which was missing, and add it on its own.

DATABASE_ENABLE_SSL parameter has been set to true on deployment (and propagated to pod) level:
$ kubectl -n my-project get pod os2iot-backend-779874f6b4-nm64j -o yaml | grep -i ssl -A 1
- name: DATABASE_ENABLE_SSL
value: "true"
$

Despite of this fact os2iot-backend still tries to establish non-ssl connection with Postrges (SSL off entry in logs):
$ kubectl -n my-project logs os2iot-backend-779874f6b4-nm64j

[email protected] prestart:prod /tmp/os2iot/backend
npm run run-migrations

[email protected] run-migrations /tmp/os2iot/backend
npm run typeorm migration:run

[email protected] typeorm /tmp/os2iot/backend
ts-node -r tsconfig-paths/register ./node_modules/typeorm/cli.js --config ./ormconfig.js "migration:run"

Error during migration run:
error: no pg_hba.conf entry for host "10.1.0.130", user "psqladmin", database "os2iot", SSL off
at Parser.parseErrorMessage (/tmp/os2iot/backend/node_modules/pg-protocol/src/parser.ts:357:11)
at Parser.handlePacket (/tmp/os2iot/backend/node_modules/pg-protocol/src/parser.ts:186:21)
at Parser.parse (/tmp/os2iot/backend/node_modules/pg-protocol/src/parser.ts:101:30)
at Socket. (/tmp/os2iot/backend/node_modules/pg-protocol/src/index.ts:7:48)
at Socket.emit (events.js:314:20)
at Socket.EventEmitter.emit (domain.js:483:12)
at addChunk (_stream_readable.js:297:12)
at readableAddChunk (_stream_readable.js:272:9)
at Socket.Readable.push (_stream_readable.js:213:10)
at TCP.onStreamRead (internal/stream_base_commons.js:188:23) {
length: 151,
severity: 'FATAL',
code: '28000',
detail: undefined,
hint: undefined,
position: undefined,
internalPosition: undefined,
internalQuery: undefined,
where: undefined,
schema: undefined,
table: undefined,
column: undefined,
dataType: undefined,
constraint: undefined,
file: 'auth.c',
line: '597',
routine: 'ClientAuthentication'
}
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] typeorm: ts-node -r tsconfig-paths/register ./node_modules/typeorm/cli.js --config ./ormconfig.js "migration:run"
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] typeorm script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/node/.npm/_logs/2022-08-03T07_28_53_504Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] run-migrations: npm run typeorm migration:run
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] run-migrations script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/node/.npm/_logs/2022-08-03T07_28_53_602Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] prestart:prod: npm run run-migrations
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] prestart:prod script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/node/.npm/_logs/2022-08-03T07_28_53_619Z-debug.log
$

Ps. I have disabled ssl mode on Postrges server, just for testing purposes. Then connection between os2iot-backend and Postgres was established.

Server returns 500 when creating an IoT-device without a type.
Response should be 400, with a validation error of missing/incorrect type.
As is done, if name is missing.

Submitting w/o a type returns 500

Submitting w/o a name returns 400

Check that server returns 201, when all three params is present

When using the above mentioned call, I can read the device-model name attached to each iot-device, but unfortunately not the device-model id, so I have to make a call to each iot-device with GET /api/v1/iot-device/{id}, to get the device-model id.
It would be nice if this information was added in the future.
Regards Kenneth

When you have an application where you can have multiple different data targets in this example called target A which is supposed to receive Data 1 - and target B with Data 2.

The original time-series data will be called Data 0.

1. If you use no decoder on all these data targets, no issue. Everything works as intended. Data 1 and Data 2 are unaffected. as they are exactly the same. Both target A and target B receive them normally.
2. If you use the same data decoder on the data targets. It seems to work normally. Data 1 and Data 2 seem to be unaffected (we currently do not have logs to ensure it, so we only have the data received by the target B which should still prove the point as it's the latter in the hierarchy)
3. If you use different data decoders on these data targets, the time-series Data 1 that is sent to target A will be correct, but the Data sent to target B, will now be a new data package that have been sent through both decoders. The result - in our case - is a wrong Data "3" where the data have been changed from base64 twice, resulting in a non-sensical hex string.

We believe the issue - in this case - is that Data 1 is the product of decoder A manipulating data 0. We however believe that the decoder to target B is now manipulating Data 1 and not Data 0 - thus subsequently resulting in data "3".

Log example:

note that data on line 18 was supposed to read 00BA5FDC not D34040E450C3 which is 00BA5FDC converted from base64 to HEX.

When running the staging version of the OS2IoT backend with the DB from Release v.1.0.1 we are getting the error:

...
0 migrations are already loaded in the database.
3 migrations were found in the source code.
3 migrations are new migrations that needs to be executed.
...
Error during migration run:
QueryFailedError: relation "open_data_dk_dataset" already exists

To reproduce this please run the Release v.1.0.1 version of the backend with a new database and then run the Stage on the same DB.

Data are not correctly processed by the OS2IoT. The log shows that the message is received by the OS2IoT, but then cannot be sent to kafka for further processing as the opening connection to kafka fails (os2iot opens a new connection every time a message need to be sent to kafka). The longer OS2IoT works, the issue is more common. After a few hours of operation, almost all packages are lost during processing. It works fine again after the OS2IoT backend restart.

{"level":"ERROR","timestamp":"2023-03-15T06:00:04.6630000Z","logger":"kafkajs","message":"[Connection] Connection error: write after end","broker":"kafka-cluster-kafka-bootstrap:9092","clientId":"os2iot-client","stack":"Error [ERR_STREAM_WRITE_AFTER_END]: write after end\n at writeAfterEnd (_stream_writable.js:266:14)\n at Socket.Writable.write (_stream_writable.js:315:5)\n at Object.sendRequest (/tmp/os2iot/backend/node_modules/kafkajs/src/network/connection.js:312:27)\n at SocketRequest.send [as sendRequest] (/tmp/os2iot/backend/node_modules/kafkajs/src/network/requestQueue/index.js:139:23)\n at SocketRequest.send (/tmp/os2iot/backend/node_modules/kafkajs/src/network/requestQueue/socketRequest.js:85:10)\n at RequestQueue.sendSocketRequest (/tmp/os2iot/backend/node_modules/kafkajs/src/network/requestQueue/index.js:168:19)\n at RequestQueue.push (/tmp/os2iot/backend/node_modules/kafkajs/src/network/requestQueue/index.js:148:12)\n at /tmp/os2iot/backend/node_modules/kafkajs/src/network/connection.js:307:29\n at new Promise (<anonymous>)\n at sendRequest (/tmp/os2iot/backend/node_modules/kafkajs/src/network/connection.js:302:14)\n at async Connection.send (/tmp/os2iot/backend/node_modules/kafkajs/src/network/connection.js:321:53)\n at async Broker.[private:Broker:sendRequest] (/tmp/os2iot/backend/node_modules/kafkajs/src/broker/index.js:890:14)\n at async Broker.metadata (/tmp/os2iot/backend/node_modules/kafkajs/src/broker/index.js:191:12)\n at async /tmp/os2iot/backend/node_modules/kafkajs/src/cluster/brokerPool.js:155:25\n at async Cluster.refreshMetadata (/tmp/os2iot/backend/node_modules/kafkajs/src/cluster/index.js:134:5)\n at async Cluster.addMultipleTargetTopics (/tmp/os2iot/backend/node_modules/kafkajs/src/cluster/index.js:192:11)"}

We have tested it with kafka versions: 3.2.0, 3.3.2, 2.7.0 with the very same results.