incosistent display of data in dashboard about thingsboard

[rem-electric]

Update: At the end of the daily interval, the data for one device is missing. Both devices are configured identically, and are feeding into the same aggregate stream node. I can see the all messages pass through in the debug correctly, but for one device the data does not persist.

[mykhailokornieiev]

Hi
Please check the "Scip latest persistence" in the Save Timeseries rule node that is responsive to the problem value. If it is enabled, then please disable it and check if everything works correctly after it.

[rem-electric]

It is currently disabled

[mykhailokornieiev]

Hi
I was unable to reproduce such behavior
Please provide full detailed steps to reproduce with screenshots of Rule Chain and its setup

[rem-electric]

The devices are 2 identical energy meters with the same configuration. They push data over http at 5 minute intervals. I'll attach the rule chain exported json file to this message. Let me know if you need any other details. Thanks for your help.
egauge_push.json

[mykhailokornieiev]

I was unable to reproduce such behavior even on thingsboard.cloud
It looks like you tried to modify the Root Rule Chain logic. I don't recommend doing that since it can lead to unpredicted behavior.

Default rule nodes must stay the first one. Only after them you can add your custom logic
Please check if everything works correctly with these test rule chain and dashboard. Add test devices to generator rule node and entity alias of dashboard. It will generate telemetry once a minute, so you can see if there are any errors
timeseries.json
aggregation_generator.json
Let me know if you get errors in this test rule chain

[rem-electric]

The test rule chain and dashboard that you uploaded work correctly. The rule chain that I uploaded is only used for the device profile I created for these 2 devices. To create it, I copied the root rule chain and added the other logic and gave it a unique name. It is not set to 'root'. Will this cause unpredictable behavior even though it's not set to root?

[rem-electric]

Some additional observations:
-This only seems to affect the timeseries charts. It happens on bar charts and time series tables. If I use a card widget or a gauge widget for example, the latest telemetry data is immediately available.
-When using the bar chart or timeseries table, the data will populate at the next telemetry posting, however, the data for previous days (interval is set to day) does not populate. So previous days' data is missing, or I'm unable to access it.

[mykhailokornieiev]

Answering your previous question: yes, it can, but I don't think that this is the cause of the problem with this ticket.
Please specify if this issue occurs only when you use 7-day timewindow + 1-day interval? Did you try to test it with 1 day /1 hour timewindow / interval respectively, for example ?
If it is possible, please attach your dashboard, so I can check it locally.

[rem-electric]

I think I may have resolved the issue. Today is the first day where data populates correctly, and the previous day's data is present. I'm not certain as to how this was fixed, but I'll explain what I did. On my rule chain that I posted here, there were some script nodes using TBEL. I tried to use Javascript instead, and when switched, there were errors in my syntax that didn't show up when I used TBEL. I'm not an expert at either language, but here is the original script:

var total = msg.ranges[0].rows[0][0];
var timeStamp = msg.ranges[0].ts;
timeStamp += "000";
msg = {"total": total};
metadata = {
"deviceType":
metadata.deviceType,
"deviceName": metadata.deviceName,
"ts": timeStamp
};
return {msg: msg, metadata: metadata, msgType: msgType};

TBEL didn't show any problem, but Javascript showed an error that msg and metadata are read only. So, I changed my script to use new variable assignments, and continued to use TBEL. I deleted all of the telemetry data for the devices, and so far everything is working. Again, I can't say for certain that this was the problem, but it seems to be resolved. Do you have any thought on this?
Thanks

[rem-electric]

Unfortunately, the issue has already returned. Yesterday's data is still displayed, but there is a delay in the current interval data populating in the bar chart. However, if I delete the telemetry data for the aggregate stream node, it starts working correctly. So it seems that the issue takes some time to occur, and deleting the telemetry data is a temporary fix. I've attached the current rule chain and dashboard.
4d_and_16b.json
egauge_push.json

[mykhailokornieiev]

Hi
Additionally, please provide an example of your payload, both message and metadata. You can take it from input (IN) debug messages of "extract total" rule node. It will be great, if you attach an example for both devices.

[rem-electric]

Here's the first device
{
"registers": [{
"name": "Power",
"type": "P",
"did": 0
}, {
"name": "Votls",
"type": "V",
"did": 1
}],
"ranges": [{
"ts": "1714165569",
"delta": 60.000,
"rows": [
["1737367903488", "393933414904"],
["-17228435", "-17241998"],
["-17225471", "-17281781"],
["-17224112", "-17295468"],
["-17222654", "-17271306"]
]
}]
}

{
"deviceName": "Ducor Egauge 4D",
"deviceType": "egauge push",
"ts": "1714165571051"
}

[rem-electric]

and Here's the other device
{
"registers": [{
"name": "Power",
"type": "P",
"did": 0
}, {
"name": "volts l1",
"type": "V",
"did": 1
}],
"ranges": [{
"ts": "1714165517",
"delta": 60.000,
"rows": [
["1507176991191", "741301020928"],
["-14474684", "-17603465"],
["-14468454", "-17589951"],
["-14473036", "-17554300"],
["-14473099", "-17563039"]
]
}]
}

{
"deviceName": "Ducor Egauge 16B",
"deviceType": "egauge push",
"ts": "1714165518257"
}

[rem-electric]

Any thoughts on what this issue may be? I've tried everything I can think of with no success. I have other rule chains using the aggregate stream node without issue. Is there something I'm missing?

[mykhailokornieiev]

Hi
Sorry for the delay in response. It looks like one of your devices has a negative aggregated value from time to time. I think it can have a negative value most of the day, and only at the end can it get a positive value. Even more, this negative value can be something like -1.5, when the value for another device will be something like 100-10000kwh, so it will be poorly visible on the dashboard until it get similar big value. I more then sure that you will see your "lost" value if you change this setting in widget

[rem-electric]

The issue is not the negative values. Please see the attached video, and let me know if that helps explain.

https://www.dropbox.com/scl/fi/ku7vej38dtcd00ewocmz8/agg_bug.mp4?rlkey=3z088a03smvghx0b607yhpqeq&st=6bd9txp8&dl=0

[AlexDoanTB]

@rem-electric I do not see any bug. But I do see the problem with your data (calculations, more likely).
The bar chart renders your actual values for daily and monthly power consumption from the TS database. In your Value cards, you render the latest values taken from another DB. I'm more than sure you have a short TTL configured for these time series because the chart shows zeros (as well as negative values). Make a GET TELEMETRY call to fetch the telemetry data curve for one month, for instance.

[rem-electric]

@AlexDoanTB If you look at the rule chain, there is a script node after the aggregate stream node that is used to add metadata.TTL = 126230400 I did this to override the default TTL on thingsboard cloud. It works fine on other devices. So I don't see how this could be a short TTL issue. Also, some days data persists, and other days it's missing. I didn't know that latest values are stored in a different DB than time series, so thank you for pointing that out. I still can't get my head around this issue. I've never had any problem aggregating values from other devices.

[AlexDoanTB]

@rem-electric, re-check your RCs. I have that gut feeling that you rewrite your agg values multiple times per day, either.
As mentioned, check your dailyKwh values for Ducor Egauge 4D device (Ducor Egauge 4D — Latest telemetry tab — dailyKwh key — Show on widget — Table (last 7 days).
There will be zeros, for sure. And the value will change over time (upon a new message, according to the video).