cltl / pepper Goto Github PK

View Code? Open in Web Editor NEW

29.0 18.0 10.0 109.66 MB

VU-CLTL Pepper/Nao Application Repository (Python 2)

License: MIT License

Python 98.33% Java 0.49% Batchfile 0.06% Shell 0.05% HTML 1.07%

robot framework nlp rdf pepper naoqi theory-of-mind cognitive-robotics python2

pepper's People

Contributors

Stargazers

Watchers

Forkers

jasperrr91 jenbishop spijkervet uva-robotics intuitionmachine roferretti leolani jayrob101

pepper's Issues

Re-visit nlp module

"One utterance to many triples"

Currently, the triples generated contain entities extracted as "lenka-sister" (eg. lenka-sister_is-from_serbia), which might need to be broken down into triples themselves (lenka_has-sister_x and x_is-from_serbia)

This can be noted through the litst of types extracted. Usually when the list of types for a single entity contains certain combinations (eg. adjective and noun)

pepper/pepper/brain/infrastructure/rdf_builder.py

Line 147 in e40435d

 # TODO here we know if two types are different category (aka noun and verb) we might need to split the triple 

Factual responder not working

The Wolfram Responder is throwing an error regarding number of arguments during init.

Language Test: Scenarios with BrainResponder

Hey,

I manually tested the language test scenarios in a live application: worked through them all and looked if they were working. All entries with a tick box are working! For every entry that is not working 100%, the log is provided. Hope this is useful for debugging :)

These tests were done on the latest Develop branch!

Here we go:

you are a robot
are you a robot
who is a robot
who is the robot (Google picked this up, resulting in no answer, which might be okay, actually)

10/16/19 13:25:16 INFO     pepper.Chat (Bram)              006       Bram: "who is the robot"
10/16/19 13:25:16 DEBUG    pepper.pepper.language.analyzer     initial RDF: {'predicate': u'is', 'object': u'the-robot', 'subject': ''}
10/16/19 13:25:16 DEBUG    pepper.pepper.language.analyzer     after VP: {'predicate': u'be', 'object': u'the-robot', 'subject': ''}
10/16/19 13:25:16 DEBUG    pepper.pepper.language.analyzer     after NP: {'predicate': u'be', 'object': u'the-robot', 'subject': ''}
10/16/19 13:25:16 DEBUG    pepper.pepper.language.analyzer     final RDF: {'predicate': {'text': u'be', 'type': u'verb.stative'}, 'object': {'text': u'the-robot', 'type': [u'noun.artifact']}, 'subject': {'text': '', 'type': []}} {'neg': False, 'wh_word': {u'response': u'person'}}
10/16/19 13:25:16 INFO     pepper.pepper.language.analyzer     RDF    subject: {"text": "", "type": []}
10/16/19 13:25:16 INFO     pepper.pepper.language.analyzer     RDF  predicate: {"text": "be", "type": "verb.stative"}
10/16/19 13:25:16 INFO     pepper.pepper.language.analyzer     RDF     object: {"text": "the-robot", "type": ["noun.artifact"]}
10/16/19 13:25:16 DEBUG    pepper.RdfBuilder                   Booted
10/16/19 13:25:17 DEBUG    pepper.BrainResponder               TRIPLE: ?_be_the-robot [_->_artifact or object])
10/16/19 13:25:17 INFO     pepper.LongTermMemory               Triple: ?_be_the-robot [_->_artifact or object])
10/16/19 13:25:17 DEBUG    pepper.LongTermMemory               Posting query
10/16/19 13:25:17 DEBUG    pepper.BrainResponder               REPLY: I know things usually be artifact, but I do not know this case
10/16/19 13:25:17 INFO     pepper.ResponsePicker               BrainResponder (0.18s)
10/16/19 13:25:17 INFO     pepper.Chat (Bram)              007    Leolani: "I know things usually be artifact, but I do not know this case"

you live in Amsterdam

works, but does not form a reply, resulting in another responder to be called.

10/16/19 13:26:06 INFO     pepper.Chat (Bram)              014       Bram: "you live in Amsterdam"
10/16/19 13:26:06 DEBUG    pepper.pepper.language.analyzer     initial RDF: {'predicate': u'live', 'object': u'in-Amsterdam', 'subject': u'you'}
10/16/19 13:26:06 DEBUG    pepper.pepper.language.analyzer     after VP: {'predicate': u'live', 'object': u'in-Amsterdam', 'subject': u'you'}
10/16/19 13:26:06 DEBUG    pepper.pepper.language.analyzer     after NP: {'predicate': u'live', 'object': u'in-Amsterdam', 'subject': u'leolani'}
10/16/19 13:26:06 DEBUG    pepper.pepper.language.analyzer     final RDF: {'predicate': {'text': u'live-in', 'type': [u'verb.stative', 'preposition']}, 'object': {'text': u'Amsterdam', 'type': u'noun.location'}, 'subject': {'text': u'leolani', 'type': ['robot']}} {'neg': False}
10/16/19 13:26:06 INFO     pepper.pepper.language.analyzer     RDF    subject: {"text": "leolani", "type": ["robot"]}
10/16/19 13:26:06 INFO     pepper.pepper.language.analyzer     RDF  predicate: {"text": "live-in", "type": ["verb.stative", "preposition"]}
10/16/19 13:26:06 INFO     pepper.pepper.language.analyzer     RDF     object: {"text": "Amsterdam", "type": "noun.location"}
10/16/19 13:26:06 DEBUG    pepper.RdfBuilder                   Booted
10/16/19 13:26:06 DEBUG    pepper.BrainResponder               TRIPLE: leolani_live-in_amsterdam [robot_->_location])
10/16/19 13:26:06 DEBUG    pepper.LocationReasoner             Posting query
10/16/19 13:26:06 INFO     pepper.LongTermMemory               Triple: leolani_live-in_amsterdam [robot_->_location])
10/16/19 13:26:06 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:26:06 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:26:06 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:26:06 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:26:06 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:26:07 INFO     pepper.LongTermMemory               Posting triples
10/16/19 13:26:07 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:26:07 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:26:07 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:26:07 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:26:07 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:26:07 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:26:07 DEBUG    pepper.LongTermMemory               Posting query
10/16/19 13:26:07 DEBUG    pepper.BrainResponder               REPLY: 
10/16/19 13:26:07 INFO     pepper.ResponsePicker               GreetingResponder (0.00s)
10/16/19 13:26:07 INFO     pepper.Chat (Bram)              015    Leolani: "What's new?, Bram!"

who lives in Amsterdam
do you live in Amsterdam
I am a man

again: works, but does not form a reply, resulting in another responder to be called.

10/16/19 13:27:01 INFO     pepper.Chat (Bram)              024       Bram: "I am a man"
10/16/19 13:27:01 DEBUG    pepper.pepper.language.analyzer     initial RDF: {'predicate': u'am', 'object': u'a-man', 'subject': u'I'}
10/16/19 13:27:01 DEBUG    pepper.pepper.language.analyzer     after VP: {'predicate': u'be', 'object': u'a-man', 'subject': u'I'}
10/16/19 13:27:01 DEBUG    pepper.pepper.language.analyzer     after NP: {'predicate': u'be', 'object': u'a-man', 'subject': 'Bram'}
10/16/19 13:27:01 DEBUG    pepper.pepper.language.analyzer     final RDF: {'predicate': {'text': u'be', 'type': u'verb.stative'}, 'object': {'text': u'a-man', 'type': [u'noun.person']}, 'subject': {'text': 'Bram', 'type': ['person']}} {'neg': False}
10/16/19 13:27:01 INFO     pepper.pepper.language.analyzer     RDF    subject: {"text": "Bram", "type": ["person"]}
10/16/19 13:27:01 INFO     pepper.pepper.language.analyzer     RDF  predicate: {"text": "be", "type": "verb.stative"}
10/16/19 13:27:01 INFO     pepper.pepper.language.analyzer     RDF     object: {"text": "a-man", "type": ["noun.person"]}
10/16/19 13:27:01 DEBUG    pepper.RdfBuilder                   Booted
10/16/19 13:27:01 DEBUG    pepper.BrainResponder               TRIPLE: bram_be_a-man [person_->_person])
10/16/19 13:27:01 DEBUG    pepper.LocationReasoner             Posting query
10/16/19 13:27:01 INFO     pepper.LongTermMemory               Triple: bram_be_a-man [person_->_person])
10/16/19 13:27:01 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:27:01 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:27:01 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:27:01 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:27:01 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:27:02 INFO     pepper.LongTermMemory               Posting triples
10/16/19 13:27:02 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:27:02 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:27:02 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:27:02 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:27:02 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:27:02 DEBUG    pepper.LongTermMemory               Posting query
10/16/19 13:27:02 DEBUG    pepper.BrainResponder               REPLY: 
10/16/19 13:27:02 INFO     pepper.ResponsePicker               UnknownResponder (0.00s)
10/16/19 13:27:02 INFO     pepper.Chat (Bram)              025    Leolani: "Why?"

The <object> part of the triple is empty? Resulting in strange reply!

10/16/19 13:28:09 INFO     pepper.Chat (Bram)              036       Bram: "birds can sing"
10/16/19 13:28:09 DEBUG    pepper.pepper.language.analyzer     initial RDF: {'predicate': u'can', 'object': u'sing', 'subject': u'birds'}
10/16/19 13:28:09 DEBUG    pepper.pepper.language.analyzer     after VP: {'predicate': u'can', 'object': u'sing', 'subject': u'birds'}
10/16/19 13:28:09 DEBUG    pepper.pepper.language.analyzer     after NP: {'predicate': u'can', 'object': u'sing', 'subject': u'birds'}
10/16/19 13:28:09 DEBUG    pepper.pepper.language.analyzer     final RDF: {'predicate': {'text': u'can-sing', 'type': [u'verb.creation']}, 'object': {'text': '', 'type': []}, 'subject': {'text': u'birds', 'type': u'noun.animal'}} {'neg': False}
10/16/19 13:28:09 INFO     pepper.pepper.language.analyzer     RDF    subject: {"text": "birds", "type": "noun.animal"}
10/16/19 13:28:09 INFO     pepper.pepper.language.analyzer     RDF  predicate: {"text": "can-sing", "type": ["verb.creation"]}
10/16/19 13:28:09 INFO     pepper.pepper.language.analyzer     RDF     object: {"text": "", "type": []}
10/16/19 13:28:09 DEBUG    pepper.RdfBuilder                   Booted
10/16/19 13:28:09 DEBUG    pepper.BrainResponder               TRIPLE: birds_can-sing_? [animal_->_])
10/16/19 13:28:09 DEBUG    pepper.TypeReasoner                 Posting query
10/16/19 13:28:09 DEBUG    pepper.TypeReasoner                 Posting query
10/16/19 13:28:12 DEBUG    pepper.TypeReasoner                 Posting query
10/16/19 13:28:12 DEBUG    pepper.TypeReasoner                 Posting query
10/16/19 13:28:12 DEBUG    pepper.TypeReasoner                 Posting query
10/16/19 13:28:12 DEBUG    pepper.TypeReasoner                 Posting query
10/16/19 13:28:12 DEBUG    pepper.LocationReasoner             Posting query
10/16/19 13:28:12 INFO     pepper.LongTermMemory               Triple: birds_can-sing_? [animal_->_])
10/16/19 13:28:12 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:28:12 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:28:12 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:28:12 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:28:12 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:28:13 INFO     pepper.LongTermMemory               Posting triples
10/16/19 13:28:13 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:28:13 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:28:13 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:28:13 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:28:13 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:28:13 DEBUG    pepper.LongTermMemory               Posting query
10/16/19 13:28:13 DEBUG    pepper.BrainResponder               REPLY: Exciting news! I had never heard about  before!
10/16/19 13:28:13 INFO     pepper.ResponsePicker               BrainResponder (4.45s)
10/16/19 13:28:13 INFO     pepper.Chat (Bram)              037    Leolani: "Exciting news! I had never heard about  before!"

can birds sing
what can birds do

no answer from brain

10/16/19 13:28:23 INFO     pepper.Chat (Bram)              038       Bram: "what can birds do"
10/16/19 13:28:23 DEBUG    pepper.pepper.language.analyzer     initial RDF: {'predicate': u'can-do', 'object': '', 'subject': u'birds'}
10/16/19 13:28:23 DEBUG    pepper.pepper.language.analyzer     after VP: {'predicate': u'can', 'object': '', 'subject': u'birds'}
10/16/19 13:28:23 DEBUG    pepper.pepper.language.analyzer     after NP: {'predicate': u'can', 'object': '', 'subject': u'birds'}
10/16/19 13:28:23 DEBUG    pepper.pepper.language.analyzer     final RDF: {'predicate': {'text': u'can', 'type': None}, 'object': {'text': '', 'type': []}, 'subject': {'text': u'birds', 'type': u'noun.animal'}} {'neg': False, 'aux': {u'person': [u'first', u'second', u'third'], u'tense': u'present', u'number': [u'singular', u'plural']}, 'wh_word': {u'response': u'object'}}
10/16/19 13:28:23 INFO     pepper.pepper.language.analyzer     RDF    subject: {"text": "birds", "type": "noun.animal"}
10/16/19 13:28:23 INFO     pepper.pepper.language.analyzer     RDF  predicate: {"text": "can", "type": null}
10/16/19 13:28:23 INFO     pepper.pepper.language.analyzer     RDF     object: {"text": "", "type": []}
10/16/19 13:28:23 DEBUG    pepper.RdfBuilder                   Booted
10/16/19 13:28:23 DEBUG    pepper.BrainResponder               TRIPLE: birds_can_? [animal_->_])
10/16/19 13:28:23 INFO     pepper.LongTermMemory               Triple: birds_can_? [animal_->_])
10/16/19 13:28:23 DEBUG    pepper.LongTermMemory               Posting query
10/16/19 13:28:23 DEBUG    pepper.BrainResponder               REPLY: I know animal usually can things, but I do not know this case
10/16/19 13:28:23 INFO     pepper.ResponsePicker               BrainResponder (0.13s)
10/16/19 13:28:23 INFO     pepper.Chat (Bram)              039    Leolani: "I know animal usually can things, but I do not know this case"

who can sing
Robots can't eat food

Think all goes well here, because {'neg': True}
But again, no reply is formed, resulting in another responder to respond

10/16/19 13:29:01 INFO     pepper.Chat (Bram)              044       Bram: "robots can't eat food"
10/16/19 13:29:01 DEBUG    pepper.pepper.language.analyzer     initial RDF: {'predicate': u'can-not-eat', 'object': u'food', 'subject': u'robots'}
10/16/19 13:29:01 DEBUG    pepper.pepper.language.analyzer     after VP: {'predicate': u'can-eat', 'object': u'food', 'subject': u'robots'}
10/16/19 13:29:01 DEBUG    pepper.pepper.language.analyzer     after NP: {'predicate': u'can-eat', 'object': u'food', 'subject': u'robots'}
10/16/19 13:29:01 DEBUG    pepper.pepper.language.analyzer     final RDF: {'predicate': {'text': u'can-eat', 'type': [u'verb.consumption']}, 'object': {'text': u'food', 'type': u'noun.Tops'}, 'subject': {'text': u'robots', 'type': u'noun.artifact'}} {'neg': True}
10/16/19 13:29:01 INFO     pepper.pepper.language.analyzer     RDF    subject: {"text": "robots", "type": "noun.artifact"}
10/16/19 13:29:01 INFO     pepper.pepper.language.analyzer     RDF  predicate: {"text": "can-eat", "type": ["verb.consumption"]}
10/16/19 13:29:01 INFO     pepper.pepper.language.analyzer     RDF     object: {"text": "food", "type": "noun.Tops"}
10/16/19 13:29:01 DEBUG    pepper.RdfBuilder                   Booted
10/16/19 13:29:01 DEBUG    pepper.BrainResponder               TRIPLE: robots_can-eat_food [artifact or object_->_Tops])
10/16/19 13:29:01 DEBUG    pepper.LocationReasoner             Posting query
10/16/19 13:29:01 INFO     pepper.LongTermMemory               Triple: robots_can-eat_food [artifact or object_->_tops])
10/16/19 13:29:01 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:29:01 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:29:01 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:29:01 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:29:01 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:29:01 INFO     pepper.LongTermMemory               Posting triples
10/16/19 13:29:02 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:29:02 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:29:02 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:29:02 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:29:02 DEBUG    pepper.ThoughtGenerator             Posting query
10/16/19 13:29:02 DEBUG    pepper.LongTermMemory               Posting query
10/16/19 13:29:02 DEBUG    pepper.BrainResponder               REPLY: 
10/16/19 13:29:02 INFO     pepper.ResponsePicker               UnknownResponder (0.00s)
10/16/19 13:29:02 INFO     pepper.Chat (Bram)              045    Leolani: "I'm guessing you told me: robots can't eat food, but I have no clue!"

can robots eat food

Pepper thinks robot CAN eat food, while I just told her robots cannot.
Observe that {'neg': False}

10/16/19 13:29:21 INFO     pepper.Chat (Bram)              046       Bram: "can robots eat food"
10/16/19 13:29:21 DEBUG    pepper.pepper.language.analyzer     initial RDF: {'predicate': u'can-eat', 'object': u'food', 'subject': u'robots'}
10/16/19 13:29:21 DEBUG    pepper.pepper.language.analyzer     after VP: {'predicate': u'can-eat', 'object': u'food', 'subject': u'robots'}
10/16/19 13:29:21 DEBUG    pepper.pepper.language.analyzer     after NP: {'predicate': u'can-eat', 'object': u'food', 'subject': u'robots'}
10/16/19 13:29:21 DEBUG    pepper.pepper.language.analyzer     final RDF: {'predicate': {'text': u'can-eat', 'type': [u'verb.consumption']}, 'object': {'text': u'food', 'type': u'noun.Tops'}, 'subject': {'text': u'robots', 'type': u'noun.artifact'}} {'neg': False}
10/16/19 13:29:21 INFO     pepper.pepper.language.analyzer     RDF    subject: {"text": "robots", "type": "noun.artifact"}
10/16/19 13:29:21 INFO     pepper.pepper.language.analyzer     RDF  predicate: {"text": "can-eat", "type": ["verb.consumption"]}
10/16/19 13:29:21 INFO     pepper.pepper.language.analyzer     RDF     object: {"text": "food", "type": "noun.Tops"}
10/16/19 13:29:21 DEBUG    pepper.RdfBuilder                   Booted
10/16/19 13:29:21 DEBUG    pepper.BrainResponder               TRIPLE: robots_can-eat_food [artifact or object_->_Tops])
10/16/19 13:29:21 INFO     pepper.LongTermMemory               Triple: robots_can-eat_food [artifact or object_->_Tops])
10/16/19 13:29:21 DEBUG    pepper.LongTermMemory               Posting query
10/16/19 13:29:21 DEBUG    pepper.BrainResponder               REPLY: you told me Robots can eat food
10/16/19 13:29:21 INFO     pepper.ResponsePicker               BrainResponder (0.15s)
10/16/19 13:29:21 INFO     pepper.Chat (Bram)              047    Leolani: "you told me Robots can eat food"

who can eat food

Same problem as above, the negation doesn't come through

10/16/19 13:29:29 INFO     pepper.Chat (Bram)              048       Bram: "who can eat food"
10/16/19 13:29:29 DEBUG    pepper.pepper.language.analyzer     initial RDF: {'predicate': u'can-eat', 'object': u'food', 'subject': ''}
10/16/19 13:29:29 DEBUG    pepper.pepper.language.analyzer     after VP: {'predicate': u'can-eat', 'object': u'food', 'subject': ''}
10/16/19 13:29:29 DEBUG    pepper.pepper.language.analyzer     after NP: {'predicate': u'can-eat', 'object': u'food', 'subject': ''}
10/16/19 13:29:29 DEBUG    pepper.pepper.language.analyzer     final RDF: {'predicate': {'text': u'can-eat', 'type': [u'verb.consumption']}, 'object': {'text': u'food', 'type': u'noun.Tops'}, 'subject': {'text': '', 'type': []}} {'neg': False, 'wh_word': {u'response': u'person'}}
10/16/19 13:29:29 INFO     pepper.pepper.language.analyzer     RDF    subject: {"text": "", "type": []}
10/16/19 13:29:29 INFO     pepper.pepper.language.analyzer     RDF  predicate: {"text": "can-eat", "type": ["verb.consumption"]}
10/16/19 13:29:29 INFO     pepper.pepper.language.analyzer     RDF     object: {"text": "food", "type": "noun.Tops"}
10/16/19 13:29:29 DEBUG    pepper.RdfBuilder                   Booted
10/16/19 13:29:29 DEBUG    pepper.BrainResponder               TRIPLE: ?_can-eat_food [_->_Tops])
10/16/19 13:29:29 INFO     pepper.LongTermMemory               Triple: ?_can-eat_food [_->_Tops])
10/16/19 13:29:29 DEBUG    pepper.LongTermMemory               Posting query
10/16/19 13:29:29 DEBUG    pepper.BrainResponder               REPLY: you told me Robots can eat food
10/16/19 13:29:29 INFO     pepper.ResponsePicker               BrainResponder (0.14s)
10/16/19 13:29:29 INFO     pepper.Chat (Bram)              049    Leolani: "you told me Robots can eat food"

Elena can see a laptop
what can Selene see
who can see a laptop

Location recognition

Use scene classification from: https://github.com/IBM/MAX-Scene-Classifier

Available as Docker

Clean incoming types from NLP

The outcome of analysing an utterance generates a triple representation, which includes types for any detected entities. These types could come from the internal parsing mechanisms and are not always useful. An extensive analysis on the input for triple representation is needed to filter out unnecessary noise.

pepper/pepper/brain/infrastructure/rdf_builder.py

Line 146 in e40435d

def _fix_nlp_types(self, types):

Not able to install docker image for face detetction and run the speech-recognition

How to install openface docker on windows 7 ?

Please also mention the detailed instruction to run this repo.

Improve face recognition module

Migrate to Facenet, as it is the current and stable state-of-the-art

Pose recognition

Use pose recognition from:

Available as pre-trained model: https://github.com/tensorflow/tfjs-models/tree/master/posenet#loading-a-pre-trained-posenet-model

Available as Docker: https://github.com/deephdc/posenet-tf#docker-installation

Fix when trust is calculated

Currently, a trust value for the speaker is computed on every parsed utterance. Since trust computation is expensive (6 queries), producing a response is time-consuming and the robot behaviour is severely affected.

The trust computation should be moved to be computed across the network, on initialization/call.

Add ability to guess location

The Location responder has the ability to set the name of a location given a cue, but it does not make use of the brain ability to reason about previous experiences to guess the location.

In this class

pepper/pepper/responder/sensory.py

Line 181 in 233d39d

class LocationResponder(Responder):

add the cues for "Guess where you are" and 2) add a clause for it in the respond event

pepper/pepper/responder/sensory.py

Line 203 in 233d39d

def respond(self, utterance, app):

then call

pepper/pepper/brain/reasoners/location_reasoner.py

Line 115 in 233d39d

def reason_location(self, cntxt):

and set the label, checking for None. For an example see:

pepper/test/brain/location disambiguation.py

Line 23 in 233d39d

y = brain.reason_location(capsule.context)

Set identity for "people who I heard about and I personally met"

We need to formally express that two entities are actually the same thing. In this case
A) a person who Leolani interacted with (LeolaniFriends:X)
B) a person who Leolani heard about through someone else (LeolaniWorld:X)

Currently done through owl:sameAs, but it is not working/in use

pepper/pepper/brain/LTM_statement_processing.py

Line 201 in 233d39d

self.claim_graph.add((actor.id, OWL.sameAs, person.id))

Create folders on installation

The following folders are needed to run the main responder, but are not created on installation:

backups/brain
people/friends

Thoughts: Hyponomy

Formal modelling of hyponomy as Python objects. These should work at the ontology level, including any learning the robot has acquired through conversation or linked data.

For expanding the ontology, we could include wordnet or the querying of it.

Create Baratheon backend

Create extra backend to work with a NAO robot. Robust against functionality available in Pepper but not for Nao (e.g tablet)

Improve text recognition

Improve text recognition by using several hypotheses

pepper/pepper/language/language.py

Line 418 in 233d39d

def _choose_hypothesis(self, hypotheses):

Voice person identity

Identify speakers by their voice

Fix brain info by correction

This is a dog, not a cat: change object type and label to cat (give utterance, since we have context info there)
- Add functionality from brain side
- Call from vision responder
  
  pepper/pepper/responder/sensory.py
  
  Line 101 in 233d39d
  
  for cue in self.SEE_SPECIFIC:
This is piek’s office not armando’s office: change label of location in the current context. Like so:

pepper/pepper/responder/sensory.py

Line 211 in 233d39d

utterance.context.location.label = location

Client-Server infrastructure

Since this project is becoming very big and complex, we would like to break it up into smaller services that can be managed in an easier way.

The idea is to break them down in a similar manner as

object recognition component calling services from a different repository:
pepper/pepper/framework/sensor/obj.py

Line 304 in 233d39d

class ObjectDetectionClient(object):
pepper/pepper/framework/component/object_detection.py

Line 33 in 233d39d

clients = [ObjectDetectionClient(target) for target in ObjectDetectionComponent.TARGETS]
pepper/pepper/framework/component/object_detection.py

Line 58 in 233d39d

objects = [obj for obj in client.classify(image) if obj.confidence > config.OBJECT_RECOGNITION_THRESHOLD]
face recognition component calling services from a docker:
pepper/pepper/framework/sensor/face.py

Line 58 in 233d39d

class OpenFace(object):
pepper/pepper/framework/component/face_detection.py

Line 30 in 233d39d

open_face = OpenFace()
pepper/pepper/framework/component/face_detection.py

Line 61 in 233d39d

on_face = [self.face_classifier.classify(r, b, image) for r, b in open_face.represent(image.image)]

This method may be applied to other components like the brain, context, scene, etc

Update Data Model to follow SEM

Copy of #10
Our current Data Model is entity-centric, but this must be changed to be event-centric.

For this we should use SEM

Expand thoughts

Query for - [ ]
- hunger for knowledge / exploration
  - broad: entity that I do not know much about
  - deep: entity that gets the most attention (people talk about it, I see it a lot)
- knowing if I have heard about this person but have not met them (until now?)
- many other people do X so perhaps you too?
- somebody recently told me something about you
- differences in the objects and people observed in a location across different contexts
- hyponomy (see this issue)

Hello Are you using Windows platform to implement this

I am working with pepper for a simple task on windows platform and I see your work is very interesting. Would love to hear if this work is done on Windows platform. Would you be interested to have short email exchange ? Here is my email id [email protected]
I am currently working on some educational research exploration. So, I thought of having any guidance from you would be great head-start.

Thank you!!

Looking forward to hear from you soon.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.

Jobs

Jooble