Question - is offline buffering working OK? about android

[growse]

Just tested this against the otrecorder, and the queue seemed to behave as expected. This is a gif of the app in move mode showing the queue as the endpoint goes from stopped to listening. The track on the otrecorder shows no missing points.

I tested this as well with putting the emulator into airplane mode, and the points were correctly queued then sent.

'm wondering if, when it gets home and reconnects, it sees the location is the same (to within a metre or so!) of the previously sent location so "ignores" the fact it's been out and about..

It shouldn't work like this, every time the device gives OT a location, it updates its "last seen" location regardless of the status of the network or how many pending locations to be sent there are.

Can you check the status screen to see that the "Endpoint Queue" value is a positive number next time you try this? Before you reconnect the device back to the network, this should be non-zero, and then when you connect it should (at some point) drain to zero.

[growse]

@JFingerle slightly confused as to which bit you think isn't working?

From your log file:

• There's 1 message in the queue at 18:36: 2023-04-17 18:36:53.759 E MessageProcessor: Message delivery failed. queueLength: 1, messageId: 1681749295046-a83404
• The a83404 message is eventually sent at 18:37: 2023-04-17 18:37:54.003 I MessageProcessorEndpointMqtt: MessageLocation id=1681749295046-a83404 sent in 122ms
• There's a new location that's queued/not sent at 18:44: 2023-04-17 18:44:43.841 E MessageProcessor: Message delivery failed. queueLength: 1, messageId: 1681749764601-404eac
• After some repeated attempts to deliver the message, by 19:09:54 there's another location in the queue: 2023-04-17 19:09:54.141 E MessageProcessor: Message delivery failed. queueLength: 2, messageId: 1681749764601-404eac
• By 19:14:31, the queue length is now 3: 2023-04-17 19:14:31.959 E MessageProcessor: Message delivery failed. queueLength: 3, messageId: 1681749764601-404eac
• 19:41, we're up to 4 2023-04-17 19:41:15.530 E MessageProcessor: Message delivery failed. queueLength: 4, messageId: 1681749764601-404eac
• Between 19:44:04 and 19:44:08, they all get delivered
• 19:51 there's a new message delivery failure and re-queue: 2023-04-17 19:51:06.526 E MessageProcessor: Message delivery failed. queueLength: 1, messageId: 1681753644471-b34e8

etc.

It seems like OT is buffering just fine.

Is your issue the fact that you're not getting locations often enough? If so, from the log, it looks like this is because the device isn't delivering locations to OT very often, rather than a buffering / connectivity issue. There's a whole raft of causes for this, depending on your device & configuration.

[JFingerle]

I think you are right, the location determination on my device (Redmi Note 8T, Android 11, "Pixel Experience" Rom) seems not to work reliably without a mobile connection.

owntracks-log (2).txt

• Today I walked the same track (about 3.6km)
  • "Significant Changes" mode (because at the time I went I had not seen your comment yet)
  • Airplane mode enabled, Wifi on. A connection to the MQTT broker was only possible at home via Wifi.
• I left home at 12:04. I've returned at 12:51.
• On the Owntracks map I can see that only two location points have been recorded near the end of my walk. Their accuracy is 4m, so somehow GPS was used.
  • 12:38:09, "acc: 4"
  • 12:39:40, "acc: 4"

-> I will repeat the test in "Move" mode tomorrow

Update 2022-04-20: In "Move" mode the offline buffering is working perfectly. IMHO this issue can be closed. Maybe another issue about the missing location updates in "Significant Changes" could be opened - but as this depends on the Google API I don't know if this can even be fixed by Owntrack.

[growse]

In "Move" mode the offline buffering is working perfectly. IMHO this issue can be closed. Maybe another issue about the missing location updates in "Significant Changes" could be opened - but as this depends on the Google API I don't know if this can even be fixed by Owntrack.

Glad to see you narrowed it down somewhat. If, like you say, it's a case of location updates not being delivered to OT frequently in Significant Changes mode, then this is something that's determined by the device. I'd suggest playing with both locatorInterval (lower this) and locatorDisplacement (take this to 0) in the settings to see if that changes things. Also, try setting the Android "Ignore Battery Optimizations" setting (from the Status screen).

There's also a pattern of people on devices of certain vendors (looking at you, Samsung) for whom they get basically no updates at all when the app's activity is not foregrounded and in Significant Changes mode. Not sure what we can do about that.

Are you using MQTT with at least QoS 1 for publishing, I think buffering only happens when QoS >=1

MQTT QoS settings determines what's buffered on the MQTT broker. Ie after OT has successfully sent the message to the broker. If you're offline, you're not going to be able to send anything, so I'm not sure the QoS setting changes much.

apprently looking at the code, nothing has been implemented in HTTP Mode.
It is working for MQTT out of the box with Paho.

The offline buffering is generic regardless of the transport, and implemented here: https://github.com/owntracks/android/blob/master/project/app/src/main/java/org/owntracks/android/services/MessageProcessor.kt#L169

[jpmens]

Offline buffering of position data typically works very well, and other than storage space, I think there’s no limit to the number of positions we store until we can connect to the backend server.

Are you running the app in “Significant Changes” mode, i.e. not in Move Mode?

[growse]

Definitely should work, I'll double check in HTTP mode next chance I get. I know it works in MQTT as that's what I use daily.

There is a limit on the buffer size (no-one likes an unbounded queue) of 10,000 points.

[jpmens]

There is a limit on the buffer size of 10,000 points.

Sorry for lying. :-)

[DerekTP]

The app was in Significant Move mode. I'm testing on a tablet that has only wifi connection, and by leaving home and returning and reconnecting to wifi. I'm wondering if, when it gets home and reconnects, it sees the location is the same (to within a metre or so!) of the previously sent location so "ignores" the fact it's been out and about...
If that's the case then in fact for my situation that's fine, I'm just trying to check that it's doing what I expect it to do.
I've put it in move mode now and will do a trip out and return; then later do a trip out, connect to wifi, then return...
Thanks for your input so far. I'll update here with what I find!

[DerekTP]

Thanks for the info, growse, really appreciate your detailed answer. After a trip today in move mode still no updates, but I hadn't checked the endpoint queue. The log doesn't show any additional sends after reconnecting to wifi, however.
I'm wondering whether, because the device is a tablet with no cellphone connectivity, whether OT is not getting a location when out and about? The tablet has GPS but if in "low accuracy" mode is OT just using cell data (supplemented by wifi SSIDs) even though GPS is enabled... Yet Move mode should be GPS-based, right? Also your screenshots show you're using MMQT; my setup is using HTTP, so could that be contributing to the problem?

[growse]

It's possible that without a GSM radio, that no locations arrive at the app in Singificant mode. That would be .... interesting, and something we'd have to work out what to do with.

Yet Move mode should be GPS-based, right?

Move mode is high-accuracy, high-power usage GPS, so it's really odd that you're not seeing any updates in move mode.

lso your screenshots show you're using MMQT; my setup is using HTTP, so could that be contributing to the problem?

The gif in #1362 (comment) is in HTTP mode.

If you're happy collecting logs (debug, in move mode, and going from connectivity to no connectivity for a bit, then back again) and sending them over to [email protected], we can hopefully take a closer look.

[JFingerle]

I can confirm that offline buffering does not work with app version 2.4.10 in "Significant Changes" mode.

• I left home exactly at 18:30. I returned around 19:50. The mobile data connection during my walk was really bad.
• MQTT Broker IP (via VPN): 192.168.10.36
• Phone Wifi IP: 192.168.2.200
• Phone mobile data IP: 10.137.213.99

owntracks-log.txt

[JFingerle]

@growse Thank you for checking my log. I think the offline buffering isn't working because with a working mobile data connection the location tracking is working perfectly (new point every 100m, as configured). In the areas without a good data connection the tracking does not work at all for me.

Today I will walk the track again and I will try to figure out what is happening

[growse]

That smells a little bit like the device is not able to get location data properly with a bad mobile connection.

Device location determination is pretty complicated, and the mysterious workings of Google's FusedLocationProviderClient don't help. I remember reading somewhere (don't remember where) that using the client with a priority of PRIORITY_BALANCED_POWER_ACCURACY (which is what Significant Changes mode does) tends to lean away from using the actual GPS in favour of network-based location services. Obviously, if you've got a bad network connection, then the frequency / accuracy of these locations is going to be not great.

If you can repeat the test in Move mode (which is Priority = PRIORITY_HIGH_ACCURACY), that should force it to use the GPS (at the cost of some battery usage), so at least the device should get better locations more frequently.

Every device does this a bit differently though, and without a full suite of devices to test on, it's pretty hard to debug what's going on sometimes.

[jules-ch]

Are you using MQTT with at least QoS 1 for publishing, I think buffering only happens when QoS >=1

[jules-ch]

See https://www.eclipse.org/paho/files/javadoc/org/eclipse/paho/client/mqttv3/MqttClientPersistence.html

apprently looking at the code, nothing has been implemented in HTTP Mode.
It is working for MQTT out of the box with Paho.

[github-actions]

Marking issue with request for more data as stale, due to no updates.

[DerekTP]

I raised the issue, but it seems there's still uncertainty as to status / cause. As it happens I was reviewing it for potential use by a client but the project is now on hold due to the unreliability of updates when no internet connection available.

[github-actions]

Marking issue with request for more data as stale, due to no updates.