The mower integration lost connection, and is not getting itself back online. about husqvarna_automower

[Din-BH]

Correction. Seems sending commands to the mower still has worked, despite errors turning up in log - my do-not-cut-during-rain timer has managed to update the the park time.

[Thomas55555]

The logs indicate, that there are no websocket updates anymore. Aren't there any REST updates in the full logs? They should come every five minutes.

[Din-BH]

I don't have the log from this available anymore unfortunately, but I checked right now and the last aioautomower.rest response was now over 2 hours ago. Until that they came in rather predictably. Then my "it's raining' automation sent a parking command

2023-07-31 14:17:44.866 DEBUG (MainThread) [aioautomower.rest] Mower Action URL: https://api.amc.husqvarna.dev/v1/mowers/291ffbd0-7aed-4f03-9567-433ba5924d7d/actions
2023-07-31 14:17:44.867 DEBUG (MainThread) [aioautomower.rest] Sent payload: {
 "data": {
   "type": "Park",
   "attributes": {"duration": 120}
}
}
2023-07-31 14:17:44.867 DEBUG (MainThread) [aioautomower.rest] Resp status mower command: 202

and after that no rest resposes... And the

2023-07-31 16:37:28.641 DEBUG (MainThread) [aioautomower.session] Age in sec: 8960
2023-07-31 16:37:28.641 DEBUG (MainThread) [aioautomower.session] No ws updates anymore and mower connected, ws probably down or mower shortly before disconnecting

started appearing instead.

[Din-BH]

There is also the following. I did some restarting on some inhouse network components, so connection might have been lost, which might have created some state that the integration could not recover from?

2023-07-31 14:13:38.970 ERROR (MainThread) [homeassistant.components.automation.park_husqvarna_on_rain] While executing automation automation.park_husqvarna_on_rain
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1155, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 874, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 867, in getaddrinfo
    return await self.run_in_executor(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/socket.py", line 962, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -3] Try again

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 659, in async_trigger
    await self.action_script.async_run(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1569, in async_run
    return await asyncio.shield(run.async_run())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 420, in async_run
    await self._async_step(log_exceptions=False)
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 470, in _async_step
    self._handle_exception(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 493, in _handle_exception
    raise exception
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 468, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 703, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 665, in _async_run_long_action
    return long_task.result()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 1965, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2005, in _execute_service
    return await cast(
           ^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 866, in handle_service
    await service.entity_service_call(
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 848, in entity_service_call
    response_data = task.result()  # pop exception if have
                    ^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1192, in async_request_call
    return await coro
           ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 892, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/config/custom_components/husqvarna_automower/vacuum.py", line 363, in async_custom_command
    await self.coordinator.session.action(
  File "/usr/local/lib/python3.11/site-packages/aioautomower/session.py", line 184, in action
    return await a.async_mower_command()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aioautomower/rest.py", line 213, in async_mower_command
    async with session.post(self.mower_action_url, data=self.payload) as resp:
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 1141, in __aenter__
    self._resp = await self._coro
                 ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client.py", line 536, in _request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 540, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 901, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/connector.py", line 1169, in _create_direct_connection
    raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host api.amc.husqvarna.dev:443 ssl:default [Try again]

[Thomas55555]

Yes, that might be. I've never tested what happens if there is no internet connection. It's on my 2do list.

[Din-BH]

Interesting new log, now on Version: 2023.8.0

2023-08-22 08:35:16.179 DEBUG (MainThread) [aioautomower.session] Age in sec: 2360
2023-08-22 08:35:16.180 DEBUG (MainThread) [aioautomower.session] No ws updates anymore and mower connected, ws probably down or mower shortly before disconnecting
2023-08-22 08:40:16.181 DEBUG (MainThread) [aioautomower.session] Age in sec: 2660
2023-08-22 08:40:16.181 DEBUG (MainThread) [aioautomower.session] No ws updates anymore and mower connected, ws probably down or mower shortly before disconnecting
2023-08-22 08:45:16.182 DEBUG (MainThread) [aioautomower.session] Age in sec: 2960
2023-08-22 08:45:16.183 DEBUG (MainThread) [aioautomower.session] No ws updates anymore and mower connected, ws probably down or mower shortly before disconnecting
2023-08-22 08:47:07.730 DEBUG (MainThread) [aioautomower.session] Websocket (re)connected
2023-08-22 08:47:07.775 DEBUG (MainThread) [aioautomower.session] Websocket ready=True (id='KDDKXeUFjoECEQQ=')
2023-08-22 08:50:16.185 DEBUG (MainThread) [aioautomower.session] Age in sec: 3260
2023-08-22 08:50:16.185 DEBUG (MainThread) [aioautomower.session] No ws updates anymore and mower connected, ws probably down or mower shortly before disconnecting
2023-08-22 08:55:16.186 DEBUG (MainThread) [aioautomower.session] Age in sec: 3560
2023-08-22 08:55:16.186 DEBUG (MainThread) [aioautomower.session] No ws updates anymore and mower connected, ws probably down or mower shortly before disconnecting
2023-08-22 08:55:58.500 DEBUG (MainThread) [aioautomower.session] Got positions-event, data: {'id': '291ffbd0-7aed-4f03-9567-433ba5924d7d', 'type': 'positions-event', 'attributes': {'positions': [{'latitude': 60.1995382, 'longitude': 24.8052128}]}}
2023-08-22 08:55:58.529 DEBUG (MainThread) [aioautomower.session] Got status-event, data: {'id': '291ffbd0-7aed-4f03-9567-433ba5924d7d', 'type': 'status-event', 'attributes': {'battery': {'batteryPercent': 100}, 'mower': {'mode': 'MAIN_AREA', 'activity': 'PARKED_IN_CS', 'state': 'IN_OPERATION', 'errorCode': 0, 'errorCodeTimestamp': 0}, 'planner': {'nextStartTimestamp': 0, 'override': {'action': 'NOT_ACTIVE'}, 'restrictedReason': 'NOT_APPLICABLE'}, 'metadata': {'connected': True, 'statusTimestamp': 1692683758402}}}
2023-08-22 08:55:58.558 ERROR (MainThread) [homeassistant.config_entries] Error unloading entry husqvarna_automower for husqvarna_automower
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 544, in async_unload
    result = await component.async_unload_entry(hass, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/husqvarna_automower/__init__.py", line 104, in async_unload_entry
    await coordinator.session.close()
  File "/usr/local/lib/python3.11/site-packages/aioautomower/session.py", line 176, in close
    await asyncio.gather(*tasks)
  File "/usr/local/lib/python3.11/site-packages/aioautomower/session.py", line 372, in _rest_task
    self.data = await self.get_status()
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aioautomower/session.py", line 191, in get_status
    return await d.async_mower_state()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aioautomower/rest.py", line 176, in async_mower_state
    raise MowerApiConnectionsError(
aioautomower.rest.MowerApiConnectionsError: Error 403, the mower state can't be fetched: {'errors': [{'id': 'f7a0c434-2310-437c-ae28-77c727525d90', 'status': '403', 'code': 'invalid.credentials', 'title': 'Invalid credentials', 'detail': 'The supplied credentials are invalid.'}]}

So, the connetion is open with the given credentials, but then the connection is attempted to be reopened, and all of the sudden, credentials are not valid. After that, it seems there is no further attempt to revive, and there is no info about the connection being broken. The sensors look like they are just stuck.

Restarting HA fixes teh issue, but there is really no other way to do it. Cannot restart extension, and it seems extension cannot even restart itself.

Also interesting; After this happens, then the mower still responds to commands sent - for awhile..

2023-08-22 14:53:10.291 DEBUG (MainThread) [aioautomower.rest] Mower Action URL: https://api.amc.husqvarna.dev/v1/mowers/291ffbd0-7aed-4f03-9567-433ba5924d7d/actions
2023-08-22 14:53:10.292 DEBUG (MainThread) [aioautomower.rest] Sent payload: {
 "data": {
   "type": "Park",
   "attributes": {"duration": 120}
}
}
2023-08-22 14:53:10.292 DEBUG (MainThread) [aioautomower.rest] Resp status mower command: 202

Until then not..

2023-08-22 15:33:52.283 DEBUG (MainThread) [aioautomower.rest] Mower Action URL: https://api.amc.husqvarna.dev/v1/mowers/291ffbd0-7aed-4f03-9567-433ba5924d7d/actions
2023-08-22 15:33:52.283 DEBUG (MainThread) [aioautomower.rest] Sent payload: {
 "data": {
   "type": "Park",
   "attributes": {"duration": 120}
}
}
2023-08-22 15:33:52.283 DEBUG (MainThread) [aioautomower.rest] Resp status mower command: 404

and then again.. All this while no new measurements are coming in on sensors..

2023-08-22 15:38:06.760 DEBUG (MainThread) [aioautomower.rest] Mower Action URL: https://api.amc.husqvarna.dev/v1/mowers/291ffbd0-7aed-4f03-9567-433ba5924d7d/actions
2023-08-22 15:38:06.760 DEBUG (MainThread) [aioautomower.rest] Sent payload: {
 "data": {
   "type": "Park",
   "attributes": {"duration": 120}
}
}
2023-08-22 15:38:06.760 DEBUG (MainThread) [aioautomower.rest] Resp status mower command: 202

Until finally starts to be forbidden...

2023-08-22 22:25:48.181 DEBUG (MainThread) [aioautomower.rest] Mower Action URL: https://api.amc.husqvarna.dev/v1/mowers/291ffbd0-7aed-4f03-9567-433ba5924d7d/actions
2023-08-22 22:25:48.181 DEBUG (MainThread) [aioautomower.rest] Sent payload: {
 "data": {
   "type": "Park",
   "attributes": {"duration": 120}
}
}
2023-08-22 22:25:48.181 DEBUG (MainThread) [aioautomower.rest] Resp status mower command: 403
2023-08-22 22:25:48.181 ERROR (MainThread) [custom_components.husqvarna_automower.vacuum] Command couldn't be sent to the command que: 403, message='Forbidden', url=URL('https://api.amc.husqvarna.dev/v1/mowers/291ffbd0-7aed-4f03-9567-433ba5924d7d/actions')
2023-08-22 22:25:48.188 ERROR (MainThread) [homeassistant.components.automation.park_husqvarna_on_rain] Park Husqvarna on Rain: Error executing script. Error for call_service at pos 1: Command not allowed.

Bit of a mess?

[Thomas55555]

There are still no rest updates every five minutes. That's strange, because this is hard-coded.
Also it's strange, that there were no websocket updates after 8:00
What happens at 8:55? Did you reload the integration or restarted HA? Because as you can see in the logs, HA tried to unload the integration.
Did you received any logs between 8:55 and 14:53?

[Din-BH]

I added an automation lately that restarts the integration if the PARKED_IN_CS is reported to be in the past. The intention was to resolve situations when the connection was stale. In this case it seems that the parking was set to until further notice (?) since next start timestamp is "0". Need to update my automation to not trigger for that. But is it so that restarting the integration is not supported at all?

[Din-BH]

And no; no [aioautomower.session] logs after the failed restart.

[Thomas55555]

I added an automation lately that restarts the integration if the PARKED_IN_CS is reported to be in the past. The intention was to resolve situations when the connection was stale. In this case it seems that the parking was set to until further notice (?) since next start timestamp is "0". Need to update my automation to not trigger for that. But is it so that restarting the integration is not supported at all?

It's supported. But right now it relies on a working connection. So reloading only works if the connection is closed properly. I can change that, that reloading works all the time. It's just three lines of code. I can do this next week.

[Din-BH]

Sounds like a good idea. Even if of course the main thread should be to keep the stuff working, it is always good to have possibilities for failsafe.

[benjR]

I have some kind of similar error, don't know if it's related:
It happens that I'm not in my house for a few days, and sometimes the mower gets stuck, if the mower gets to no more battery before I come back, the integration does not seem to work anymore (even is the mower is back up and running) and asks me to log again to Husqvarna servers, when I do so, the warning asking to login is gone but the integration still does not work.

The only way to make it come back is to delete everything (integration, repo in HACS) and add everything back... there is not a lot in the logs...

2023-08-29 09:22:10.959 ERROR (MainThread) [aioautomower.rest] Response mower data: {'Message': 'User is not authorized to access this resource with an explicit deny'}

When I reload

2023-08-29 09:20:35.859 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration husqvarna_automower which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2023-08-29 09:20:50.442 ERROR (MainThread) [aioautomower.rest] Response mower data: {'Message': 'User is not authorized to access this resource with an explicit deny'}
2023-08-29 09:22:10.959 ERROR (MainThread) [aioautomower.rest] Response mower data: {'Message': 'User is not authorized to access this resource with an explicit deny'}
2023-08-29 10:45:50.557 ERROR (MainThread) [aioautomower.rest] Response mower data: {'Message': 'User is not authorized to access this resource with an explicit deny'}
2023-08-29 10:46:01.714 ERROR (MainThread) [aioautomower.rest] Response mower data: {'Message': 'User is not authorized to access this resource with an explicit deny'}
2023-08-29 11:25:45.608 DEBUG (MainThread) [aioautomower.rest] Response mower data: <ClientResponse(https://api.amc.husqvarna.dev/v1/mowers/) [403 Forbidden]>
2023-08-29 11:25:45.611 ERROR (MainThread) [aioautomower.rest] Response mower data: {'Message': 'User is not authorized to access this resource with an explicit deny'}
2023-08-29 11:25:45.614 DEBUG (MainThread) [custom_components.husqvarna_automower] Exception in async_setup_entry: Error 403, the mower state can't be fetched: {'Message': 'User is not authorized to access this resource with an explicit deny'}
2023-08-29 11:25:45.616 DEBUG (MainThread) [custom_components.husqvarna_automower] Finished fetching husqvarna_automower data in 0.142 seconds (success: False)
2023-08-29 11:25:45.619 DEBUG (MainThread) [custom_components.husqvarna_automower.config_flow] USER INPUT NONE

When I click on the reauth needed alert on integrations page:

2023-08-29 11:30:10.338 DEBUG (MainThread) [custom_components.husqvarna_automower.config_flow] USER INPUT NONE
2023-08-29 11:30:13.429 DEBUG (MainThread) [custom_components.husqvarna_automower.config_flow] user_input: {}
2023-08-29 11:30:17.533 INFO (MainThread) [custom_components.husqvarna_automower.config_flow] Successfully authenticated
2023-08-29 11:30:17.586 DEBUG (MainThread) [aioautomower.rest] Response mower data: <ClientResponse(https://api.amc.husqvarna.dev/v1/mowers/) [403 Forbidden]>
<CIMultiDictProxy('Content-Type': 'application/json', 'Content-Length': '82', 'Connection': 'keep-alive', 'Date': 'Tue, 29 Aug 2023 09:30:17 GMT', 'x-amzn-RequestId': 'bdb64497-c523-4096-9539-345dda24952f', 'x-amzn-ErrorType': 'AccessDeniedException', 'x-amz-apigw-id': 'KaoaiGY6DoEFkYQ=', 'X-Cache': 'Error from cloudfront', 'Via': '1.1 59d6f4cc9bf66b7389662cc7f6d33f70.cloudfront.net (CloudFront)', 'X-Amz-Cf-Pop': 'CDG50-C2', 'X-Amz-Cf-Id': 'EZuj2nqDM_yMlk9X4-Lwjp1D1LV1mbsftLk2L6cGLgmrW2dbOUzpdQ==')>

2023-08-29 11:30:17.587 ERROR (MainThread) [aioautomower.rest] Response mower data: {'Message': 'User is not authorized to access this resource with an explicit deny'}
2023-08-29 11:30:17.590 DEBUG (MainThread) [custom_components.husqvarna_automower] Exception in async_setup_entry: Error 403, the mower state can't be fetched: {'Message': 'User is not authorized to access this resource with an explicit deny'}
2023-08-29 11:30:17.590 DEBUG (MainThread) [custom_components.husqvarna_automower] Finished fetching husqvarna_automower data in 0.056 seconds (success: False)

Will see if this current bugfix will do anything for me :)

[OlwinFroon]

I'm having similar issues and I didn't notice for a while.
Might be helpful to have a 'last API lifesign' sensor or something like that.

[OlwinFroon]

Never mind. That sensor "Next start: 10 hours ago" is a dead giveaway :P

[benjR]

I still have this issue, only solution is to delete integration and add it again, and add IDs via the config flow as the reconfigure button seems to do the same (sending me through husqvarna dev portal) but doesn't manage to restart the connection (popping 403 errors as shared in my previous post).
Husqvarna app doesn't have any issue

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[benjR]

still an issue

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Din-BH]

While not season for mowing right now, this did seem to improve after fixes (plus the added option to repair via automation - thanks!) so closing this one now.

[benjR]

While not season for mowing right now, this did seem to improve after fixes (plus the added option to repair via automation - thanks!) so closing this one now.

Hi there, could you explain the repair via automation ? Still have a 500 this morning, will go through delete the integration and add it back again

[Din-BH]

I made an automation to check for template {{ (as_timestamp(now()) - states('sensor.xxxx_next_start')|as_timestamp(0))>600 }} to find out if the next start time was in the past. If it was, I check that the device is Docked and not in parking 'until further notice' and if all these match i call Homeassistant reload config entry on my mower entity.

[benjR]

thanks @Din-BH will try it and report but that's pretty much a hack, it shouldn't be necessary to have this or if it is (because of a server issue or whatever) we should have this behavior in the integration itself ? cheers