tronikos / google_assistant_sdk_custom

Fork of Google Assistant SDK integration to allow getting responses from commands to Google Assistant

Google Assistant SDK Custom integration for Home Assistant

About

This integration is a copy of Google Assistant SDK integration with the following additional features:

• HTML responses as text from commands to Google Assistant

Why aren't these changes in the core Google Assistant SDK integration?

Due to a bug in the Google Assistant API, not all responses contain text, especially for home control commands, like turn on the lights. The workaround, per the linked bug, is to enable HTML output and then parse the HTML for the text. Because the core integrations are not allowed to parse HTML, this custom integration is needed.

Installation

HACS

1. Add custom integrations repository: https://github.com/tronikos/google_assistant_sdk_custom
2. Select "Google Assistant SDK Custom" in the Integration tab and click download
3. Restart Home Assistant
4. Enable the integration

Manual

1. Copy directory custom_components/google_assistant_sdk_custom to your <config dir>/custom_components directory
2. Restart Home-Assistant
3. Enable the integration

Enable the integration

1. Go to Settings / Devices & Services / Integrations. Click + ADD INTEGRATION
2. Search for "Google Assistant SDK Custom" and click on it

Examples

Nest Guard

Following example allows controlling Nest Guard and getting its status in a text helper. A similar approach, especially the automation that polls the status, can be used for other devices, for which a template, e.g. template light, might make more sense. Here a Template Alarm Control Panel wouldn't work because Google Assistant doesn't allow disarming Nest Guard.

Create a text helper with:

Name: Nest Guard Status
Icon: mdi:shield-home
Entity ID: input_text.nest_guard_status

Create following scripts:

nest_guard_refresh:
  alias: "Nest Guard: Refresh"
  sequence:
    - service: google_assistant_sdk_custom.send_text_command
      data:
        command: what is the status of nest guard
      response_variable: response
    - service: input_text.set_value
      data:
        value: "{{ response.responses[0].text }}"
      target:
        entity_id: input_text.nest_guard_status
  mode: single
  icon: mdi:shield-refresh
nest_guard_away:
  alias: 'Nest Guard: Away'
  sequence:
  - service: google_assistant_sdk_custom.send_text_command
    data:
      command: Set Nest Guard to away and guarding
  - service: script.nest_guard_refresh
    data: {}
  mode: single
  icon: mdi:shield-lock
nest_guard_home:
  alias: 'Nest Guard: Home'
  sequence:
  - service: google_assistant_sdk_custom.send_text_command
    data:
      command: Set Nest Guard to home and guarding
  - service: script.nest_guard_refresh
    data: {}
  mode: single
  icon: mdi:shield-home

Create automation:

alias: "Nest Guard: status"
description: ""
trigger:
  - platform: time_pattern
    minutes: "10"
condition: []
action:
  - service: script.nest_guard_refresh
    data: {}
mode: queued
max: 10

Add the following Entities card in the lovelace dashboard:

type: entities
entities:
  - input_text.nest_guard_status
footer:
  type: buttons
  entities:
    - entity: script.nest_guard_home
      show_icon: true
      show_name: true
    - entity: script.nest_guard_away
      show_icon: true
      show_name: true
    - entity: script.nest_guard_refresh
      show_icon: true
      show_name: true

Other examples

(You will have to replace google_assistant_sdk.send_text_command with google_assistant_sdk_custom.send_text_command).

• 360 vacuum
• robot mowers
• Hisense air conditioning

google_assistant_sdk_custom's Issues

Parsed tekst is not propery handeld for speech assistant

A question about the weather results in a multi line responce.

Can this be fixed. I can give more info in required

See my video https://youtu.be/NHb0Zp8_1eU

And trace log

Possible issue with multiple listeners

I'm trying to use this to monitor the status of my two robot mowers while they are mowing. I have one working, however the second one doesn't seem to work. And it may be my misunderstanding of how the events/listeners work.

I have this for my text command:

And then for the trigger/event data - I have this setup:

I assume the "data" is how we select which automation to trigger (there are two, the other is identical except the name is not Mary) since both are using the same event type (google_assistant_sdk_custom_event).

No longer loading after most recent HA update

This error originated from a custom integration.

Logger: custom_components.google_assistant_sdk_custom
Source: custom_components/google_assistant_sdk_custom/init.py:33
Integration: Google Assistant SDK Custom (documentation, issues)
First occurred: 21:12:35 (2 occurrences)
Last logged: 21:34:34

Failed to apply patch :(

---

This error originated from a custom integration.

Logger: custom_components.google_assistant_sdk_custom
Source: custom_components/google_assistant_sdk_custom/init.py:27
Integration: Google Assistant SDK Custom (documentation, issues)
First occurred: 21:12:35 (2 occurrences)
Last logged: 21:34:34

Error running command: git apply -p4 -- /config/custom_components/google_assistant_sdk_custom/google_assistant_sdk.patch - stderr:b'error: patch failed: init.py:2\nerror: init.py: patch does not apply\nerror: patch failed: helpers.py:2\nerror: helpers.py: patch does not apply\nerror: helpers2.py: already exists in working directory\n', stdout:b''

---

any ideas?

Web Scraping?

Not really an issue but I don't see why this can't be included in the main integration. Your rationale is that parsing HTML is not allowed, but that is explicitly listed as an exception to the web scraping rule, on the page you linked:

So it seems there is no reason for this not to be included in the Google Assistant SDK integration. Did you submit a pull request and have it denied?

[Request] 'request:' to be able to parse variables

I am trying to make a google home compatible device work with HA using the google assistant sdk and a generic thermostat.

I tried to debug how it would work but the event never gets fired:
the entities are 2 text inputs (one for the request and one for the response)

the script

alias: google text send
sequence:
  - service: google_assistant_sdk.send_text_command
    data:
      command: "{{states('input_text.google_input_debug')}}"
mode: single

the automation that should catch the response:

alias: google test
description: ""
trigger:
  - platform: event
    event_type: google_assistant_sdk_custom_event
    id: response
    event_data:
      request: "{{states('input_text.google_input_debug')}}"
condition: []
action:
  - choose:
      - conditions:
          - condition: trigger
            id: response
        sequence:
          - service: input_text.set_value
            data:
              value: "{{ trigger.event.data.response }}"
            target:
              entity_id: input_text.google_response
mode: queued
max: 10

I can see my responses in the https://myactivity.google.com/myactivity yet the automation never gets triggered and starting it manually does not result any response

[edit] I also tried handling the request and response commands in a single automation using a wait_for event flow, but that does nots seem to fix the problem either.

[edit] fater further testing it seems that the 'request:' object does not parse any variables. Is there a way to change that? It would make it much more flexible when it comes to other devices.

Unable to set device name

Tried following https://github.com/tronikos/google_assistant_sdk_custom#enable-personal-results. At the step where I tap the notification, the screen quickly flashes as if opening another app, but ends up just displaying my home screen so I am unable to set a device name.

Tried restarting the phone, but that didn't help. Tried opening the assistant app and going into settings for personal results but don't see any new device there.

Error setting up entry Google Assistant SDK Custom for google_assistant_sdk_custom

Error setting up entry Google Assistant SDK Custom for google_assistant_sdk_custom
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 402, in async_setup
    result = await component.async_setup_entry(hass, self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/google_assistant_sdk_custom/__init__.py", line 66, in async_setup_entry
    implementation = await async_get_config_entry_implementation(hass, entry)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/config_entry_oauth2_flow.py", line 402, in async_get_config_entry_implementation
    implementation = implementations.get(config_entry.data["auth_implementation"])
                                         ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'auth_implementation'

I deleted the integration before updating as instructed. Now, I am getting the above error in the logs and this
The automation xyz has an action that calls an unknown service: google_assistant_sdk_custom.send_text_command

Test Script Not Working

I'm using your (fantastic) custom integration to ask my Volvo EV how much fuel (battery) it has through Google Assistant. The install went ok I think and I've duplicated your automations and a script to emulate what I want to achieve. When I run my script (below), I get a prompt on my mobile phone to "Set up voice match". I've done all the setup I can on Google Assistant on the phone by following the prompt. I'm fresh out of ideas. Any suggestions please?

volvo_battery_refresh:
alias: 'Volvo Battery : Refresh'
sequence:

• service: google_assistant_sdk.send_text_command
  data:
  command: how much fuel does my car have
  mode: single
  icon: mdi:car

Unable to run google-oauthlib-tool

Hi, tried following windows instructions for getting personal results at https://github.com/tronikos/google_assistant_sdk_custom#create-credentials-file, but unable to resolve the google-oauthlib-tool. I get an error:

'google-oauthlib-tool' is not recognized as an internal or external command,
operable program or batch file.

Event not triggering and response text is null

Hi! I'm trying to figure out how to use this patch to do something similar to #3.

I updated Home Assistant to the latest version (2023.10.3) and installed this patch (v0.0.7). I did not setup personal results (that's not needed for general commands, right?).

To test the integration I'm sending commands through Developer Tools invoking the Google Assistant service like this:

service: google_assistant_sdk.send_text_command
data:
  command: is the AC turned on

I suppose I should be able to see an event called google_assistant_sdk_custom_event. When I listen for such event in the Developer Tools I don't see any trigger (neither in my automation). However I can see the commands in https://myactivity.google.com/myactivity?product=31 with the expected text response.

Another stange thing I also see is that the service response contains null text. I don't fully understand yet how service responses work, but I guess I should be seeing the text response in here, right?

responses:
  - text: null

Am I missing something? I'd like to know how to debug the issue, but I'm not sure how. I tried looking the debug logs from Google Assistant SDK integration and all I see related to this is the following:

2023-10-22 13:13:36.155 DEBUG (MainThread) [homeassistant.components.google_assistant_sdk.helpers] command: is the AC turned on
response: None

I would appreciate some help figuring out how to proceed. Thanks for your time!

Failed to apply patch :( since ha 2023.7

It seems like there is an incompatible issue in homeassistant 2023.7.
Installation method is HAOS

event_type changed

Had to do some debugging since after i've reinstalled the integration my automations stopped working. I don't know if it's a bug or a rewrite, but the event_type changed from google_assistant_sdk_custom_event to google_assistant_sdk_custom_custom_event

event_type: call_service
data:
  domain: google_assistant_sdk_custom
  service: send_text_command
  service_data:
    command: is the ac on
origin: LOCAL
time_fired: "2023-11-24T11:52:25.231174+00:00"
context:
  id: 01HG0JS2CDE17RZKAA8VZA2FGW
  parent_id: null
  user_id: 4dc189dd28ca45c48432cb945f568c77

event_type: google_assistant_sdk_custom_custom_event
data:
  request: is the ac on
  response: AC is turned off.
origin: LOCAL
time_fired: "2023-11-24T11:52:26.878462+00:00"
context:
  id: 01HG0JS3ZY50872XVNWZQJ1QEQ
  parent_id: null
  user_id: null

Action which requires Voice Match

Hi want to create an automation from which I get a response from google assistant telling me to authenticate using voice match. Is there a way to prerecord my request and play it back to the google assistant sdk custom to trigger my command?

There are still cases in which the response is not valid.

I think the main issue of Google Assistant SDK is that the integration is invoking the Google APIs for a Smart Display device and not for a device like a Smart Speaker with no display...

I tried to get the HTML as output from the assist response, and it has the standard low bar darker area, with the reply and the buttons you can press on a Smart Display... with the Google Assistant Logo... and sometimes, the answer is "Here it is what I found on the web..." This is actually not an answer...

I think you should try to change the API to invoke it as a Smart Speaker instead of Smart Display... this would make the trick...

Thank you

Allow personal answers notification disappears after tapping in Android

When I tap the "Allow personal answers" notification that appears on Android, the screen sort of flashes as if an app is being opened and closed, and then I'm back at whatever app was opened before. There is no place to enter a device name. I'm not seeing anything new in my Google security activity on my phone.

Delete

Does not work since the update from 2023.10.5 to 2023.11.0b0

Does not work since the update from 2023.10.5 to 2023.11.0b0

ErrorLog:

Logger: homeassistant.setup
Source: setup.py:215
First occurred: 09:32:13 (1 occurrences)
Last logged: 09:32:13

Setup failed for custom integration google_assistant_sdk_custom: Unable to import component: cannot import name 'SERVICE_HOMEASSISTANT_RESTART' from 'homeassistant.const' (/usr/src/homeassistant/homeassistant/const.py)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 215, in _async_setup_component
    component = integration.get_component()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 816, in get_component
    ComponentProtocol, importlib.import_module(self.pkg_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/config/custom_components/google_assistant_sdk_custom/__init__.py", line 11, in <module>
    from homeassistant.const import SERVICE_HOMEASSISTANT_RESTART
ImportError: cannot import name 'SERVICE_HOMEASSISTANT_RESTART' from 'homeassistant.const' (/usr/src/homeassistant/homeassistant/const.py)

It looks like the service "SERVICE_HOMEASSISTANT_RESTART" has been moved from the /homeassistant/const.py to the /homeassistant/components/homeassistant/const.py. This most likely triggers the error.

Source:
https://github.com/home-assistant/core/compare/2023.10.5...2023.11.0b0?diff=split#diff-902818bfbf29bed3a8f93d3514499e76921cc8c8798431e0a64b7f5077ec0e80
https://github.com/home-assistant/core/compare/2023.10.5...2023.11.0b0?diff=split#diff-07e20eb7c6560330f9774eb23343f5dd3002106bb8b6523ccb18d95857990d77

Failed to apply patch :( since 2023.9

It seems like there is an incompatible issue in homeassistant 2023.9.
Installation method is HAOS

Specific device actions

Would it somehow be possible to also have device specific commands, like setting an alarm on the bedroom hub, or initiating a duo call to grandma on the living room nest mini?

Access blocked: This app’s request is invalid

When I go to enable the intergration in Devices & Services > Add Intergration > Google Assistant SDK Custom It takes we to a website where it says:

Access blocked: This app’s request is invalid

You can’t sign in because this app sent an invalid request. You can try again later, or contact the developer about this issue. [Learn more about this error](https://developers.google.com/identity/protocols/oauth2/web-server#authorization-errors-redirect-uri-mismatch) If you are a developer of this app, see [error details](https://accounts.google.com/). Error 400: redirect_uri_mismatch

Any help is good