treethought / flask-assistant

Framework for Building Virtual Assistants with Dialogflow and python

Create Virtual Assistants with Python

A flask extension serving as a framework to easily create virtual assistants using Dialogflow which may be integrated with platforms such as Actions on Google (Google Assistant).

Flask-Assistant allows you to focus on building the core business logic of conversational user interfaces while utilizing Dialogflow's Natural Language Processing to interact with users.

Now supports Dialogflow V2!

This project is heavily inspired and based on John Wheeler's Flask-ask for the Alexa Skills Kit.

Features

• Mapping of user-triggered Intents to action functions
• Context support for crafting dialogue dependent on the user's requests
• Define prompts for missing parameters when they are not present in the users request or past active contexts
• A convenient syntax resembling Flask's decoratored routing
• Rich Responses for Google Assistant

Hello World

from flask import Flask
from flask_assistant import Assistant, ask

app = Flask(__name__)
assist = Assistant(app, project_id="GOOGLE_CLOUD_PROJECT_ID")

@assist.action("Demo")
def hello_world():
    speech = "Microphone check 1, 2 what is this?"
    return ask(speech)

if __name__ == "__main__":
    app.run(debug=True)

How-To

1. Create an Assistant object with a Flask app.
2. Use action decorators to map intents to the proper action function.
3. Use action view functions to return ask or tell responses.

Documentation

• Check out the Quick Start to jump right in
• View the full documentation

flask-assistant's Issues

How to retrieve the option in actions_intent_option

from the doc https://developers.google.com/actions/assistant/helpers it looks like we should be able to access actions_intent_option value by app.getContextArgument('actions_intent_option', 'OPTION').value

How do we do the same thing with flask-assistant? Thanks

install failing in virtualenvironmet

Hi. I'm getting this error when trying to install in a virtual environment: Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/j4/h1bl7sdn54qgg21hfwr974l40000gn/T/pip-build-h0o9zgpp/ruamel.yaml/

Any suggestions on how to troubleshoot this? Preferably without running as sudo? I have the latest pip and updated my setuptools. The virtual environment is running Python 3.5.0 on OSX 10.11.6

Thank you!

Contexts

Hi! First of all thanks for providing this. It's a great tool for quickly getting up and running.

I seem to have an issue with the contexts or maybe I don't understand how you intended them to be used.
The intents.json always has "contexts" and "affectedContexts" void for me. Is there a way to define the input/output contexts using flask-assistant?

The problem is that even if I:

• define the input and output contexts in the API.AI console
• and then later add another entry in the user_says.yaml
• run the schema command
• the previously defined contexts get deleted, as the intents.json contains no contexts.

Would love your input on this if you have time, thanks

reprompt does not seem to work

I was testing out sample reprompt code i found here

flask-assistant/samples/actions_demo/webhook.py

Line 45 in a724c10

return ask(speech).reprompt('Do you want to see some examples?').suggest('Show card', 'Show List')

it does not seem like the reprompt works. i tested the code using my phone. note other features like ask, card, and build_list all seem to work fine

no module named schema_handlers

I attempt to run the "schema" command, I get an error saying "no modules names schema_handlers)

Programmatically Add entities

There is no way to add entities dynamically to API.AI using a function call. It would be ideal to have one which could download/upload the JSON payload to add entities to API.AI. This will allow us to manipulate entities in a system where the attributes are constantly changing.

For Eg: A list of connected devices can be made an entity using this feature, so a service can poll for the list of connected devices and keep updating the entities at regular intervals of time.

I will see if I can implement this and submit a PR, meanwhile food for thought for you :)

Converstation End

Great project and great work!
I'm new to the topic, how can I end a conversation? I created a function but what should I use a "return" to exit the app? using the "Set this intent as end of conversation" on API AI makes the app hangs.

Any help is much appreciated.

Accessing the sessionId

I've dug around through the docs and haven't found a way to access things in the request JSON from api.ai such as the sessionId. How would I go about doing that? My app requires user auth, so I'll need to set the sessionId as well as read it.

Triggering functions from a scheduler?

How does one go about using Flask's app_context w/ flask-assistant? I'm trying to trigger a function on a scheduler. So w/ flask I can wrap the scheduler job in with app.app_context(): and trigger functions (that might be horrible).

But from an extension like flask-assistant? Not sure. Posted over on stackoverflow, figured I'd ask here too. Though this might be the complete wrong way to go about it.

https://stackoverflow.com/questions/45132878/trigger-a-function-inside-flask-assistant

Check permissions and ask for it

How can we check for permissions like location and ask for it from the user?

StopIteration when running schema

I tried running schema webhook.py after following the instructions in the Quick Start:
http://flask-assistant.readthedocs.io/en/latest/quick_start.html#schema

I got:
Traceback (most recent call last): File "/usr/local/bin/schema", line 11, in <module> sys.exit(schema()) File "/Users/advak/apiainew/lib/python3.6/site-packages/api_ai/cli.py", line 74, in schema entities.generate() File "/Users/advak/apiainew/lib/python3.6/site-packages/api_ai/schema_handlers.py", line 289, in generate entity = self.push_entity(entity) File "/Users/advak/apiainew/lib/python3.6/site-packages/api_ai/schema_handlers.py", line 281, in push_entity entity = self.register(entity) File "/Users/advak/apiainew/lib/python3.6/site-packages/api_ai/schema_handlers.py", line 263, in register entity.id = next(i.id for i in self.api.agent_entities if i.name == entity.name)

API.AI setup

Hey, I am new to Google Home Actions and API.AI. Could you please tell me how to set up API.AI for the Helloworld sample to work. When I run the code in the sample, I get the following warning.

"No API.AI Developer Access Token set.\n You will not be able to register or retrieve resources from API.AI"
"No API.AI Client Access Token set. You will be able to query the agent"
WARNING:
No route was provided for the Assistant object, but a flask app object given
The Assistant will be mapped to the app's '/' endpoint.
If this is a problem please initialize the Assitant with the 'route' parameter

Suggestion chips cause SSML not to render

Hi, and thanks for the great work! I use Flask-Ask for the Alexa version of my skill :)

The issue I found is this: when using Suggestion Chips, Google Assistant is rendering the ask as the full SSML and not processed. I don't know if this is a limitation, but I couldn't find anything specifically about it.

response.py looks like it sets the display text to the same text as the speech passed in, which also breaks <audio> directives I use.

Is there a way to bypass this? I can write a way to strip out the SSML to make display text, but not sure how to override the mechanism at runtime.

Thanks!

Adding Bot Router Logic

Hey, this is more an idea rather than an issue with flask-assistant, is there a possibility where flask-assistant can have a bot router which can route requests with context and user session information to different agents in API.AI?
It would be very helpful to query different agents using a single webhook

Flask-assistant on Heroku

I'm building an app for google assistant which uses multiple calendars. I've set up flask-assistant, api.ai, and ngrok and tested it and everything works great.

The next phase was deploying it to Heroku and updating the webhook inside api.ai's settings to use Heroku's webhook instead of ngrok (https://myappname.herokuapp.com/).

Heroku creates a random port rather than using 5000, so I added:

port = int(os.environ.get("PORT", 5000))
print("Starting app on port %d" % port)
app.run(debug=True, port=port)

Looking on the logs I see that there is no connection between api.ai and Heroku - the webhook doesn't work.
Is there anything I need to change?
Thanks a lot for this great framework.

I found an issue.

When I followed quick start guide, I got a small issue.
terminal :

Traceback (most recent call last):
File "my_assistant.py", line 2, in
from flask_assistant import Assistant, tell
File "/home/opqrstu/anaconda3/lib/python3.5/site-packages/flask_assistant/init.py", line 9, in
from .core import (
File "/home/opqrstu/anaconda3/lib/python3.5/site-packages/flask_assistant/core.py", line 12, in
from .api_ai import ApiAi
ImportError: No module named 'flask_assistant.api_ai'

So, I checked the directory(~/anaconda3/lib/python3.5/site-packages/flask_assistant/) and then I couldn't find 'api_ai' directory.
I copied 'api_ai' folder from this repository, pasted it in my directory. and then it worked.

possible issue with link_out

I have the following code but it looks like only the last one will be shown.

    resp.link_out('Wiki', data['wiki_link'])
    resp.link_out('TraiConditions', data['trailconditions_link'])
    resp.link_out('Weather', data['weather_link'])

Unknown error when running schema

Hi there, just trying to follow the quick start, have followed all the steps but the schema webhook.py command is failing with errors below:

schema webhook.py
Generating intent schema...
Registering greeting intent
{'id': 'e2f91470-1651-46ba-849d-ae26c9038a0f', 'timestamp': '2018-03-01T07:03:57.609Z', 'lang': 'en', 'status': {'code': 400, 'errorType': 'bad_request', 'errorDetails': 'Unknown error errorid=766a2412-f5ba-42d1-aca3-86b5c6708e36', 'webhookTimedOut': False}}

Registering give-gender intent
{'id': '4266d750-3098-4b11-b368-60d91530e509', 'timestamp': '2018-03-01T07:03:57.873Z', 'lang': 'en', 'status': {'code': 400, 'errorType': 'bad_request', 'errorDetails': 'Unknown error errorid=fe0589e1-d201-4712-a853-a01f8809e0df', 'webhookTimedOut': False}}

Registering give-color intent
{'id': 'ca4c1e35-5b41-4510-9b6b-6a5d54750dd9', 'timestamp': '2018-03-01T07:03:58.139Z', 'lang': 'en', 'status': {'code': 400, 'errorType': 'bad_request', 'errorDetails': 'Unknown error errorid=fa8cda8d-ca83-4a5b-b8d2-5aa888e21d18', 'webhookTimedOut': False}}

Writing schema json to file
Generating entity schema...
Registering gender entity
{'id': '7349396e-b421-418f-88ac-94c7c599e31d', 'timestamp': '2018-03-01T07:03:58.409Z', 'lang': 'en', 'status': {'code': 400, 'errorType': 'bad_request', 'errorDetails': 'Unknown error errorid=5fa4784f-913d-4577-8f65-8136020dee4d', 'webhookTimedOut': False}}

Writing schema json to file

The credentials are correct as if I intentionally set them wrong I get a different "Not Authorized" message.

Unsure what else to try.

Thanks!

“User Says” examples not being published to console

Went through the QuickStart. After a successful run of “schema app.py” (with the changes noted in #59), the DialogueFlow console is populated with everything except the user utterances. Once they are typed in manually from the schema into the console, the app works as expected.

Django compatibility

Is there a way to make this project Django-compatible without forking it outright ?

I'm currently building custom code for my Django project, and have that lingering feeling of rebuilding the wheel.

How to generate following json using yaml

I have the following setup in Dialogflow, i am trying to see how i can reproduce it in the user_says.yaml

Exporting the agent shows the following related json content
usersays.json

    "data": [
      {
        "text": "gender",
        "alias": "gender_id",
        "meta": "@gender",
        "userDefined": true
      }
    ],

intent.json

"parameters": [
        {
          "id": "8a88f79d-7001-457d-9616-e3092f83928a",
          "required": false,
          "dataType": "@gender",
          "name": "gender_id",
          "value": "$gender_id",
          "prompts": [],
          "defaultValue": "",
          "isList": false
        },
        {
          "id": "81c29436-1f4a-424f-a5d2-a2d51128adca",
          "dataType": "@gender",
          "name": "gender",
          "value": "$gender_id.original"
        }
      ],

Looking through the source code, i am not sure if I can create this setup in the yaml
https://github.com/treethought/flask-assistant/blob/master/api_ai/models.py#L179-L181

        annotation['text'] = word
        annotation['meta'] = '@' + self.entity_map[word]
        annotation['alias'] = self.entity_map[word].replace('sys.', '')

from the code, it looks like meta and alias basically have to be the same name, but in my generated json, i have "alias": "gender_id", "meta": "@gender", Let me know if i am mistaken.

More notes:
The way it is setup manually via Dialogflow works the exact the way I want to, but I want to reproduce it in yaml so i can version it better. (I could version the export zip file but not as good).

I will give more info on what i want to do and why and maybe someone can tell me there is better way to do it. I basically have 2 variables for gender. gender_id which can be male or female, this is key that i will use to query database. and gender this is the $gender_id.original which can be man, boy, girl, gal which i will use in response to the user in a more friendly way.

unmatched event raises KeyError

I am getting an error like this self.intent = self.request['result']['metadata']['intentName'] Disconnected from log stream. There may be events happening that you do not see here! Attempting to reconnect... 2017-05-25T06:32:32.802182+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1614, in full_dispatch_request 2017-05-25T06:32:32.802182+00:00 app[web.1]: rv = self.handle_user_exception(e) 2017-05-25T06:32:32.802183+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1517, in handle_user_exception 2017-05-25T06:32:32.802184+00:00 app[web.1]: reraise(exc_type, exc_value, tb) 2017-05-25T06:32:32.802184+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1612, in full_dispatch_request 2017-05-25T06:32:32.802185+00:00 app[web.1]: rv = self.dispatch_request() 2017-05-25T06:32:32.802186+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask/app.py", line 1598, in dispatch_request 2017-05-25T06:32:32.802186+00:00 app[web.1]: return self.view_functions[rule.endpoint](**req.view_args) 2017-05-25T06:32:32.802187+00:00 app[web.1]: File "/app/.heroku/python/lib/python2.7/site-packages/flask_assistant/core.py", line 187, in _flask_assitant_view_func whenever I try to access the an unmatched intent. I also have an default fallback intent defined in api.ai

match action by intent action instead of intent name.

The matching by intent name is not ideal as intent names can be quite arbitrary in api.ai.
Matching by intent action would also mean we could use regex in the future to route particular requests according to intent action and group intents accordingly.

feature request easy way to ask user for their location

Not sure if it is on the road map, but i was wondering what would it take to implement a helper method like ask to ask for permission.

Reference
https://developers.google.com/actions/reference/rest/Shared.Types/Permission
https://developers.google.com/actions/assistant/helpers
https://github.com/actions-on-google/actions-on-google-nodejs/blob/ef01c100c4c9e2c95744842c4238127e1417262d/assistant-app.js#L910

NoneType error when running schema command

I get the following error when trying to run the schema command for the pizza topping sample in Python 2.7.13:

"No API.AI Client Access Token set. You will be able to query the agent"
"No API.AI Client Access Token set. You will be able to query the agent"
"No API.AI Client Access Token set. You will be able to query the agent"
"No API.AI Client Access Token set. You will be able to query the agent"
"No API.AI Client Access Token set. You will be able to query the agent"
Generating intent schema...
Traceback (most recent call last):
  File "c:\python27\scripts\schema-script.py", line 9, in <module>
    load_entry_point('Flask-Assistant==0.2.3', 'console_scripts', 'schema')()
  File "c:\python27\lib\site-packages\api_ai\cli.py", line 72, in schema
    intents.generate()
  File "c:\python27\lib\site-packages\api_ai\schema_handlers.py", line 218, in generate
    for intent in self.app_intents:
  File "c:\python27\lib\site-packages\api_ai\schema_handlers.py", line 114, in app_intents
    intent = self.build_intent(intent_name)
  File "c:\python27\lib\site-packages\api_ai\schema_handlers.py", line 123, in build_intent
    self.build_user_says(new_intent)  # TODO
  File "c:\python27\lib\site-packages\api_ai\schema_handlers.py", line 173, in build_user_says
    intent_data = raw.get(intent.name)
AttributeError: 'NoneType' object has no attribute 'get'

function to get the current api.ai entities

There is no way I can get the entities already defined in api.ai currently, it would be useful to get the list of entity objects, especially when the backend logic needs to have the same state variables

"No API.AI Client Access Token set. You will be able to query the agent"

Hey, so that's the error I'm getting when I'm trying to run the starting script with schema {filename}.py. I've defined my API key with export DEV_ACCESS_TOKEN='{api key}' (I'm on Ubuntu).

templates in samples

Hi,

I know that it usually to be generated, but if you can add the templates you used for other examples (e.g. Hotel Reservations, or Pizza order) it may help understanding the logic?

I'm having a bit of hard time putting things together to make it work (from a logical point of few)

just if you have it somewhere :-)

again thanks for the excellent Job!

BR

Making an entity 'anything'?

Say I have this:

from flask import Flask
from flask_assistant import Assistant, ask, tell

app = Flask(__name__)
assist = Assistant(app, '/')

@assist.action("say-hello")
def say_hello(name):
    return ask('Hello, {}'.format(name))

if __name__ == '__main__':
    app.run(debug=True)

How could I edit my entities.yaml file to make name be 'anything', as I won't know the users name.

Quickstart problems?

Newbie.. so take these with a grain of salt..on this page
http://flask-assistant.readthedocs.io/en/latest/quick_start.html#testing-the-assistant

i) Had to export CLIENT_ACCESS_TOKEN to make it stop complaining?
ii) In the code you have
@assist.action("user-gender")
def ask_for_color(gender):

But in the yaml you have: user-gives-gender: ?? I had to change that to make it work?
iii) if gender is 'male': should be if gender == 'male': ??

Nice work! Thanks for sharing.

Implement a helper to access a parameter's original value

Relates to #64

deploy and test on a Raspberry Pi

HI i have followed the guide and i am able to write a hello world action in python and i am able to test it using the Try it now option from the Api.AI Console UI. But i wanted to know how to run the same python script in a raspberry pi 3 and then wake the google home assistant and perform the same actions as i was doing here. ?

Is there any expectation for API V2?

Hi, Is there an expectation of updating the library to support the 2beta1 version of DialogFlow?

Slack?

I have been able to get this working with Slack, but everything is one-way right now. I am guessing that a new action would need to be added to send to the slack webhook instead of google assistant, but I am not sure.

How hard would it be to do this?

yaml formatting

This isn't issue per se with flask-assistant but friendly warning to fellow chatbot builders. flask-assistant and underlying libraries will quietly skip entire parts due to some odd space and wrong indent in yaml files.

In my case entire parts of user says had been empty and suddenly your chatbot will get dumb and unable to recognize phraes that had been working just minutes before. Very frustrating and you could waste hours.

So when in doubt check intents and entities in Dialogflow console or inspect json files in schema folder. And get proper yaml formatter for your editor.

On a positive note perhaps a little more verbose out from schema command will help.

Can't get matched intent as a variable

There is no way to get the the matched intent name in the code workflow other than having a decorator for each intent.If there is a way to extract the name of the matched intent then we could do a dynamic matching of intents and possibly map functions to match intent names.Cleaner and efficient code this way!

is there a way to check the source of the user input

i would like to know if the intent is being trigger by "google-assistant" app or text via twillio integration.

The reason I would like to know the source is so i can customize the output. If it is google-assistant i will use less text in "ask" and use suggest, but if intent is trigger via text, i would like to add more text in the "ask" method since there is no suggest or card

"ImportError: cannot import name core" when generating schema/intents

Getting the following error when trying to use the intent generation via schema:

Traceback (most recent call last):
File "/usr/local/bin/schema", line 11, in
load_entry_point('Flask-Assistant==0.1.1', 'console_scripts', 'schema')()
File "/usr/local/lib/python2.7/site-packages/pkg_resources/init.py", line 560, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/local/lib/python2.7/site-packages/pkg_resources/init.py", line 2642, in load_entry_point
return ep.load()
File "/usr/local/lib/python2.7/site-packages/pkg_resources/init.py", line 2296, in load
return self.resolve()
File "/usr/local/lib/python2.7/site-packages/pkg_resources/init.py", line 2302, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/usr/local/lib/python2.7/site-packages/api_ai/cli.py", line 5, in
from flask_assistant.core import Assistant
File "/usr/local/lib/python2.7/site-packages/flask_assistant/init.py", line 9, in
from .core import (
File "/usr/local/lib/python2.7/site-packages/flask_assistant/core.py", line 10, in
from .response import _Response
File "/usr/local/lib/python2.7/site-packages/flask_assistant/response.py", line 2, in
from . import core
ImportError: cannot import name core

Ability to do some stuff on every incoming request to the webhook

Hey @treethought , I am not able to figure out how I can do some stuff such as logging on a global level on each incoming POST request to the webhook. It would be great if you can add that feature.

Getting user's input language

I'm beginning in learning how works your module, and i didn't find a way to get the user's input language. is there any way to get it?

ngrok question

Hi,

I've used the quick start guide and so far worked great! using ngrok.

Do you have best practice if I want to put this in a container? let's say in AWS or Azure? with Public IP.

I won't need ngrok I assume, right? but also http://CONTAINER_IP:5000 will probably not work well

Any guidance is much appreciated.

Thank you

Context Issues

It looks like flask-assistant uses Contexts using its own library (managed in core.py) rather than that of the one used by api.ai.. Although this seems okay initially, I have come across cases where this causes issues and makes it quite hard to use.

For e.g. If I have a "yes" or "no" response for a question that will only need to be received upon a particular intent being hit, I cant do that because api.ai seems to be sending it anyway to the webhook since there are no contexts defined. Inside core.py, it seems to be checking if there are any view_funcs that match the output and if there arent any, it just returns the first function configured (Line 291: core.py) . Im expecting the "Default fallback intent" to get fired in this case because the required primary intent has not been hit yet, but a subsequent followup is being picked up.

Is there a better way of handling this?

My usecase is a workflow asking the end-users a series of yes/no questions. Without context, this is just picking up the first method that response to a "yes" or "no". (Based on my above description)

Pls let know if you need any more information about my usecase.

Asking for User Location and Name

I have been working with the package and so far it has been very helpful for building quick prototypes. Given my limited programming background it works out perfect. I am now thinking of using some more advanced features such as asking user permission for location and name. I can solve it trivially by asking for user address and name in an intent but is there a way of getting device location and account name?

Need help

Hi,

I have copied the templates folder and webhook.py under samples/hass, when I tried to perform $schema webhook.py, I get the below pasted error.

Generating intent schema...
Traceback (most recent call last):

re_value = r".\b{}\b".format(value[1:]) if value.startswith(('$', '¥', '￥', '€', '£')) else r"\b{}\b".format(value)

AttributeError: 'int' object has no attribute 'startswith'

But if I comment these 2 lines from user_says.yaml
turn-on-light:
UserSays:

• turn on the fan
• turn on the lamp to 100 percent
• turn on fan to 50 brightness
  Annotations:
• fan: light
• lamp: light
• lights: light
  #- 100: brightness
  #- 50: brightness

Then it works fine.

Thanks for the help.

any plans to support using this SDK without API.ai

default value does not seem to work for me

I tried to use the default but it does not seem to work for me. Not sure if this has to do with how i setup intent

# @assist.action('find-event', default={'committee_id': '60', 'committee': 'hiking and backpacking'}) # did not seem to work, so i did my own
@assist.action('find-event')
def get_event_for_committee(committee, committee_id, activity, activity_id, state, state_id):

    if committee_id == '':
        committee_id = '60'
        committee = 'Hiking and Backpacking'

Please merge advanced responses for assistant.

Hey, I came across your other branch with the additional rich responses, could you please merge it in the near future, It would be very helpful for my project. Thanks

Bug in Add synonyms method

There is a bug in the following code where self.entries is a list and the each entry in the list is a dict, however you are trying to fetch the entry in the list by the value of the entry and not by index.
You need to fetch the dict which has the value of the entry and then extend the synonyms

def add_entry(self, value, synonyms=[]):
        if self.isEnum:
            entry = {'value': value, 'synonyms': value}
        entry = {'value': value, 'synonyms': synonyms}
        self.entries.append(entry)


    def add_synonyms(self, entry, synonyms):
        self.entries[entry].extend(synonyms)

can you provide an example for card response with facebook integration

Not sure if facebook integration is in the near future. In the meantime can you give some guidance on how to provide a facebook card response. even if it is hacky and temporary.

basically i have a voice assistant for google assistant with facebook integration, it seems like the text response is showing fine in both google and facebook but when it is card response, it only shows in google assistant.

my guess is that i could manually copy the card response json/dict value and add one for facebook as part of the overall response.

Thanks again

More detailed instructions.

Hello!
This project looks awesome, but I feel a little lost. I've never used API.AI, and don't understand where the Google Developer Console project comes into the equation for getting this to work with Google Home. I've got a good knowledge of Python, etc, but some more detailed instructions on how to set up the API.AI and Google Developer Console portions would be greatly appreciated.
Thanks!