createcandle / voco

Privacy friendly voice control for the Candle Controller / WebThings Gateway, with fully local AI assistant

Voco

Privacy friendly, 100% local voice control in your home. For the Candle Controller / WebThings Gateway.

For more details, visit:

https://www.candlesmarthome.com/voco-privacy-friendly-voice-control

New 100% local AI assistant features using "Large Language Models" (LLM's)

Installation

Visit the add-ons page in your WebThings Gateway, and add the 'Voco' add-on.

Abilities

These are all examples. You don't have to phrase things exactly like this, the AI will understand different variations. Any command should always be preceded by saying "Hey Snips" or "Hey Candle".

Turn things on and off

Toggle devices

• "Turn on the table lamp"
• "Lock the back door"
• "Set the color of the living room light to green"
• "Set the brightness of the kitchen light to 70 percent"
• "Set the kitchen light to 70"
• "Turn off (all) the lights"

Toggle devices at a certain time

• "Turn on the blender in 3 minutes"
• "Close the garage door at 23 minutes past 4"

Toggle devices for a time period

• "Turn on the bedroom heater for 20 minutes"
• "Turn on the bubble blowing machine between 3 and 4 o'clock"
• "Unlock the front door until 3 pm"
• "Turn on the air filter tomorrow afternoon"
• "Lock the front door at lunchtime" (11-13 o'clock)
• "At 5 o clock turn on the radio for 3 minutes"

Ask questions

Query a specific device

• "Is the front door locked?"
• "Is the bedroom heater on or off?"
• "What is the humidity level of the weather station?"

More general queries

• "What are the levels of the plant sensor?" -> Will give you all sensor values of the plant sensor
• "Increase the volume of the radio"
• "Lower the volume of the radio"

Timers

Set timers

• "Set a timer for 5 minutes"
• "Set an alarm for a quarter past 9"
• "Wake me up at 7 am"
• "Set a countdown to midnight"

Set reminders

• "At 5 o'clock remind me to wash the car"
• "Remind me to go to the dentist in 40 minutes"

Query timers

• "How many alarms have been set?"
• "How many timers are there?"

Query timer details

• "Tell me about all my reminders"
• "Can you give me a list of all the timers"

Remove timers

• "Remove the last timer"
• "Delete the last 3 alarms"
• "Disable the countdown"
• "Cancel all timers"

Other

• "What time is it?"
• "Set the volume of snips to 95"

Features

• Strong focus on privacy. Once installed, there is no internet connection required or used. Listening can be toggled on and off.
• Supports satellite functionality, so you can expand voice control to all rooms in your home. The devices then communicate over the local network only.

Recommended hardware

The Playstation Eye is a 4-microphone array that can be plugged into your Raspberry Pi via USB. It can be bought new for about $12. https://en.wikipedia.org/wiki/PlayStation_Eye

Technology

Voco is built on Snips https://snips.ai/

Thanks to

Voco is inspired by the original Voice Addon made by Andre Natal. https://github.com/andrenatal/voice-addon/

Work on Voco was supported by the SHERPA project, a H2020 research project into AI issues by the European Union. https://project-sherpa.eu

You can learn mote about the Candle privacy friendly smart home project at: https://www.candlesmarthome.com and http://www.sherpapieces.eu/overview/candle

Work on Voco was also supported by the Creative Industries Fund NL https://stimuleringsfonds.nl/en

This project has received funding from the European Union’s Horizon 2020 research and innovation programme, under grant agreement No 786641.

voco's Issues

Install failed

Followed instructions in readme file. Install failed. Using Gateway ver. 0.12. Raspberry Pi 3B.

Voco - does not recognise any device

Hello,
I am trying to install and configure Voco add-on on WebThings Gateway.

Voco performs functions like, alarms, time, timer etc. But does not recognise any "things".

Playstation eye microphone array

Could improve the array functionality later:
https://blog.michaelamerz.com/wordpress/trying-respeaker-mic-array-v2-0/

Respeaker 4 hat

I have a respeaker 4 hat and a usb-speaker for a raspberry pi 3+ with snips. That works fine. I would like to use this hardware for mozilla-iot with voco. Is there a way to do this?

nanotts segfault

2019-08-25 19:08:26.187 ERROR : voco: 3300 Segmentation fault | ${DIR}/nanotts -l ${DIR}/lang --speed 0.9 --pitch 1.2 --volume 1 -p -m -o "$2" nanotts does segfault a lot. this leave the snips-audio-server not being able to play the audio and eventually the dialogue timeout

MQTT error

Unplugged router and gateway. Now getting MQTT error. MQTT client is connecting to: 10.0.0.33
2020-10-12 18:27:25.289 INFO : voco: Error creating MQTT client connection: [Errno 111] Connection refused
2020-10-12 18:27:25.291 INFO : voco: Setting status on thing to: MQTT error
2020-10-12 18:27:25.293 INFO : voco: property -> update. Value = MQTT error
2020-10-12 18:27:25.307 INFO : voco: property updated to new value
2020-10-12 18:27:25.313 INFO : voco: GET PATH = /things
2020-10-12 18:27:25.314 INFO : voco: Error doing http request/loading returned json: HTTPConnectionPool(host='127.0.0.1', port=8080): Read timed out. (read timeout=3)
2020-10-12 18:27:25.327 INFO : voco:
2020-10-12 18:27:25.328 INFO : voco: Speaking locally: 'I could not connect. ' at: default

Voco not working

I updated to new Gateway, now Voco adapter doesn't respond. Uninstalled then reinstalled.

Voco stopped working

I didn't use voco for a while. But yesterday I tried to call "snips" and got no answer. I tried an USB microphone also. The same behaviour.
When I change the configuration of voco, voco says "I am listening", but no reaction on the wake word.

I did a text input in the voco-menu point (set alarm to 20:25pm).

Some logs:
2021-02-09 20:26:12.038 INFO : voco: Teaching Snips the updated thing titles: 2021-02-09 20:26:12.040 INFO : voco: ['lamp', 'Reading Lamp', 'Voice control'] 2021-02-09 20:26:12.048 INFO : voco: /\ /\ /\ Injecting names into Snips! update_request json: {"operations": [["addFromVanilla", {"Thing": ["lamp", "Reading Lamp", "Voice control"]}]]} 2021-02-09 20:26:12.050 INFO : voco: Saving to persistence data store at path: /home/pi/.webthings/data/voco/persistence.json 2021-02-09 20:26:12.052 INFO : voco: Persistence file existed. Will try to save to it. 2021-02-09 20:26:12.053 INFO : voco: Data stored 2021-02-09 20:26:12.055 INFO : voco: Injection: self.mqtt_client exists, will try to inject 2021-02-09 20:26:12.057 INFO : voco: [["addFromVanilla", {"Thing": ["lamp", "Reading Lamp", "Voice control"]}]] 2021-02-09 20:26:12.059 INFO : voco: Handling request to /init 2021-02-09 20:26:12.061 INFO : voco: Warning, the current persistent_data['mqtt_server'] IP was not actually spotted in the network by the ARP scan! 2021-02-09 20:26:12.063 INFO : voco: - satellite_targets: {} 2021-02-09 20:26:12.065 INFO : voco: - has_token: True 2021-02-09 20:26:12.067 INFO : voco: - is_satellite: False 2021-02-09 20:26:12.069 INFO : voco: - hostname: gateway 2021-02-09 20:26:12.073 INFO : voco: - mqtt_server: localhost 2021-02-09 20:26:12.076 INFO : voco: 15 seconds have passed. Time: 36 2021-02-09 20:26:12.078 INFO : voco: Periodic check: self.mqtt_connected = false - will try to run_mqtt 2021-02-09 20:26:12.080 INFO : voco: in run_mqtt 2021-02-09 20:26:12.082 INFO : voco: disconnecting mqtt first 2021-02-09 20:26:12.085 INFO : voco: self.persistent_data['mqtt_server'] = localhost 2021-02-09 20:26:12.087 INFO : voco: This device is NOT a satellite, so MQTT client is connecting to 127.0.0.1:1883 2021-02-09 20:26:12.089 INFO : voco: Error creating MQTT client connection: [Errno 111] Connection refused 2021-02-09 20:26:12.091 INFO : voco: /\ /\ /\ inject_updated_things_into_snips: starting an attempt 2021-02-09 20:26:12.093 INFO : voco: GET PATH = /things 2021-02-09 20:26:12.095 INFO : voco: intent in api_get: default 2021-02-09 20:26:12.097 INFO : voco: API GET: 200, OK 2021-02-09 20:26:12.098 INFO : voco: \/ \/ \/ No need for injection 2021-02-09 20:26:12.100 INFO : voco: 15 seconds have passed. Time: 51 2021-02-09 20:26:12.102 INFO : voco: Periodic check: self.mqtt_connected = false - will try to run_mqtt 2021-02-09 20:26:12.104 INFO : voco: in run_mqtt 2021-02-09 20:26:12.106 INFO : voco: disconnecting mqtt first 2021-02-09 20:26:12.108 INFO : voco: self.persistent_data['mqtt_server'] = localhost 2021-02-09 20:26:12.110 INFO : voco: This device is NOT a satellite, so MQTT client is connecting to 127.0.0.1:1883 2021-02-09 20:26:12.112 INFO : voco: Error creating MQTT client connection: [Errno 111] Connection refused 2021-02-09 20:26:12.114 INFO : voco: /\ /\ /\ inject_updated_things_into_snips: starting an attempt 2021-02-09 20:26:12.116 INFO : voco: GET PATH = /things 2021-02-09 20:26:12.118 INFO : voco: intent in api_get: default 2021-02-09 20:26:12.119 INFO : voco: API GET: 200, OK 2021-02-09 20:26:12.121 INFO : voco: \/ \/ \/ No need for injection 2021-02-09 20:26:12.123 INFO : voco: 15 seconds have passed. Time: 6 2021-02-09 20:26:12.125 INFO : voco: Periodic check: self.mqtt_connected = false - will try to run_mqtt 2021-02-09 20:26:12.127 INFO : voco: in run_mqtt 2021-02-09 20:26:12.128 INFO : voco: disconnecting mqtt first 2021-02-09 20:26:12.129 INFO : voco: self.persistent_data['mqtt_server'] = localhost 2021-02-09 20:26:12.130 INFO : voco: This device is NOT a satellite, so MQTT client is connecting to 127.0.0.1:1883 2021-02-09 20:26:12.131 INFO : voco: Error creating MQTT client connection: [Errno 111] Connection refused 2021-02-09 20:26:12.132 INFO : voco: /\ /\ /\ inject_updated_things_into_snips: starting an attempt 2021-02-09 20:26:12.133 INFO : voco: GET PATH = /things 2021-02-09 20:26:12.134 INFO : voco: intent in api_get: default 2021-02-09 20:26:12.135 INFO : voco: API GET: 200, OK 2021-02-09 20:26:12.136 INFO : voco: \/ \/ \/ No need for injection 2021-02-09 20:26:12.137 INFO : voco: handling /parse. Incoming text: set alarm to 20:25pm 2021-02-09 20:26:12.138 INFO : voco: Handling request to /init 2021-02-09 20:26:12.139 INFO : voco: Warning, the current persistent_data['mqtt_server'] IP was not actually spotted in the network by the ARP scan! 2021-02-09 20:26:12.140 INFO : voco: - satellite_targets: {} 2021-02-09 20:26:12.141 INFO : voco: - has_token: True 2021-02-09 20:26:12.142 INFO : voco: - is_satellite: False 2021-02-09 20:26:12.143 INFO : voco: - hostname: gateway 2021-02-09 20:26:12.145 INFO : voco: - mqtt_server: localhost 2021-02-09 20:26:12.146 INFO : voco: 15 seconds have passed. Time: 22 2021-02-09 20:26:12.147 INFO : voco: Periodic check: self.mqtt_connected = false - will try to run_mqtt 2021-02-09 20:26:12.148 INFO : voco: in run_mqtt 2021-02-09 20:26:12.149 INFO : voco: disconnecting mqtt first 2021-02-09 20:26:12.150 INFO : voco: self.persistent_data['mqtt_server'] = localhost 2021-02-09 20:26:12.151 INFO : voco: This device is NOT a satellite, so MQTT client is connecting to 127.0.0.1:1883 2021-02-09 20:26:12.152 INFO : voco: Error creating MQTT client connection: [Errno 111] Connection refused 2021-02-09 20:26:12.153 INFO : voco: /\ /\ /\ inject_updated_things_into_snips: starting an attempt 2021-02-09 20:26:12.154 INFO : voco: GET PATH = /things 2021-02-09 20:26:12.155 INFO : voco: intent in api_get: default 2021-02-09 20:26:12.156 INFO : voco: API GET: 200, OK

Voco feedback played too fast

Hello,
I am trying to install and configure Voco add-on on WebThings Gateway. For this I am using Jabra USB speaker / microphone combo. Speaker and microphone seems to be detected and working fine, I tested it via "aplay " and “record”.

I also here feedback when I say “hey snip” however feedback sound is played very fast. For example if I say “what is current time?” I do hear output but it playbacks very fast and it is hard to understand. If I connect speaker via 3.5mm jack and change Things --> Voice Control --> Audio output --> From " USB Audio" to “Built-in Headphone jack” it works fine.

I just wanted to offer a thanks for doing this

I am totally creeped out by all the microphones everywhere. I'd been doing good speech recognition (various tools) and output (cepstral) for decades prior to the explosion of AI in the cloud and being told that it was needed "out there" I simply could not believe or buy into.

thx for this tool.

Can't find addon

Hi,

It seems like I can't find the addon through the gateway

Is it still there?

custom tts and addons install path

the tts cmd part is hardcoded in snips/snips.toml , in version 9.2 addons are installed in /home/pi/.mozilla-iot/addons

2019-08-25 17:38:24.847 ERROR : voco: ERROR:snips_tts_hermes: Error while trying to process event 'Say(SayMessage { text: "Do you want it to be on or off?", lang: Some("en"), id: Some("70b29f0f-cea1-476d-b8ee-c1a48a08e316"), site_id: "default", session_id: Some("036e5df9-ed9a-4b80-a554-4f7191366290") })': Can't execute `/home/pi/.mozilla/addons/voco/snips/speak.sh` 2019-08-25 17:38:24.849 ERROR : voco: -> caused by: No such file or directory (os error 2)

Voco doesn't hear me at all, but my mic worked with Voice Control

Hello,

I am trying to use Voco voice control with my webthings gateway, but I encountered an issue I just can't overcome.

I tried before with the Voice Control add on, and it could speak and hear me, so I assume that both my mic and loudspeaker work fine with my gateway. I made it to make Voice Control work with my mic by configuring the addon with "hw:CARD=Device,DEV=0". Since Voice Control had some issues, and didn't understand me, I decided to try with Voco.

The loudspeaker with Voco works alright too, and it says "I am listening". But it doesn't seem to work with the mic: it doesn't seem to hear me at all, as its Snips thing never marks the "Sound detected" ticker.

In the addon I've tried to switch to a different microphone setting from the default one "Attached device(1,0)", but the reconfiguration option doesn't seem to work. Whenever I choose a different option of the 4 available (1, 0) (1,1) (2,0) (2,1), either if the addon is enabled or disabled, when I try to do the change, the mic config value returns to its default configuration "Attached device(1,0)".

So, I know my mic works on my webthings gateway, but it doesn't work with Voco, and I don't know if this is due to any other issue, or if this happens because I should be able to change the mic config of the addon to another option, but I can't because after trying to change it, the mic config change doesn't work.

Any clue? Thx a lot!

Languages support

Hi,

Since snips support a couple other languages besides english, is there a way to make use of them?
What needs to be translated in order to accomplish this goal?

If you can point me out which files needs to be edited, I can try in making a patch for another language

is there supposed to be an audible response?

Hi I believe I have successfully installed web things gateway onto a raspberry pi 3 vB

I have installed this add-on with a Playstation eye and a mini spreadsheet plugged in to the audio jack of the Pi, and in general it works. I do not understand what is supposed to happen, though, when I say "hey snips, what time is it?" Whenever I say "hey snips ....," the mini speaks emits a rising tone but nothing else. Is the mini-speaker supposed to tell me the time audibly?

Maybe I have something configured wrong?

Sean

working but issues

First of all, nice work.
Your snips adapter is working. The female voice responds. However, i cant control any devices.

Hey snips..
DOESNT WORK:
... turn on office light, response: i couldnt find a match
....turn office light on, response: do you want it to be on or off (then quick ding) any response i give to answer is just ignored.
....is the office light on or off? response: i couldnt find a match for your request.
note: tried all 10 devices, same result. I can however turn them on/off and change lamp color from the web interface.

WORKS:
....what time is it? response: gives me the proper time
....set a timer for one minute, response: the timer has been set for one minute, and then timer works.

I am running on a Raspi 3B+ with Eye Camera you suggested. Audio in and out work fine
I am running Current version: 0.10.0 of webthings
I have 8 phillips lamps and 1 Wemo wall switch.
Any ideas?
I really like what you have done. Im guessing im pretty close, maybe you can point me in the right direction.