moe-serifu-circle / moe-serifu-agent

Anime-themed personal assistant and goal-oriented intelligent agent

Moe Serifu Agent

Overview

Project Website Documentation

Moe Serifu Agent (MSA) is an event-driven personal assistant system that presents itself as existing in a particular location (like a house or a smartphone) and performs various tasks as directed by the user.

At a high-level, this system provides an anime-themed character that exists in cyberspace. It runs around the location it's installed in and appears at the end-users' beck and call in order to perform whatever services are needed, including timed reminders, checking and reporting on the state of its location, conversation, and performing in an entertainment role.

As an example, a user might tell the MSA to greet them when they return from work, or to wake them up in a customized way in the mornings. With its plugin API, new sensors and interfaces can be added to allow the MSA to interact with the world in just about any way the user desires.

---

Anime AI

The MSA project is inspired by various fictional artificial entities, such as the Virtual Intelligences from the Mass Effect Series, the Persocoms from the Chobits series, the Tachikoma from the Ghost in the Shell series, and the AnthroPCs from the Questionable Content webcomic. The primary goal of the project is to create a system that carries out commands for the user and that gives the appearance of being an independent intelligent entity.

The anime theme was chosen because the author believes that the demograph that consumes anime tends to have a lower barrier to their willing suspension of disbelief in ascribing emotions to fictional characters than that of the general population.

Exchangable Personality and Appearance

The MSA system at its core represents itself as an anime-themed character. An intelligent agent system is used to determine how to accomplish goals set by the user, as well as to control the character's state, including the appearance of emotions and how to react to events. The AI is driven partially by a personality module, which can be exchanged in order to make the character act differently. Different personality modules are created with different behaviors in mind; each would fall under a different anime character archetype, such as tsundere, kuudere, yandere, deredere, etc.

An avatar of the character is presented to the end-user for interfacing with the system. This initial project narrows the goal of the avatar system to exist purely in cyberspace; there is no physical device (such as a robotic assembly) that the MSA can manipulate, although this functionality could certainly be added using the plugin system.

This MSA avatar can be interacted with using a variety of methods including voice recognition and via command-line interface, and it is shown to the user as a 3D model or 2D character on whichever devices are included in an instance of the system.

The specific details regarding what the avatar looks like visually, how it sounds, and how it demonstrates emotions are controlled by an avatar module within the MSA. This module can be exchanged with other such modules in order to change the appearance of the avatar.

A personality module and avatar module are intended to be combined into a set and distributed as a complete 'character pack', though there is nothing in the system design that would prevent the personality module of one pack from being used with the avatar module of another.

Physical Representation

In a complete MSA installation, a device (such as a screen/monitor) is set up in each of the rooms that it is to be interacted with. The MSA maintains a 'room' that the character resides in, and the character 'travels' between rooms by its avatar exiting a device and entering another one in an adjacent physical room. In general, the avatar will only travel between adjacent devices, e.g. if the system is set up such that device A is next to device B which is next to device C, then in order to travel from device A to device C, the avatar will move from A to B, then B to C.

Additionally, the user may download an app that allows their mobile device to be used as an output device. In this case, the avatar could travel directly to the user in order to interact with them. The MSA system would use a variety of sensors in order to detect the physical location of the mobile device and track which other output devices it should be considered adjacent to.

moe-serifu-agent's Issues

CLI command arg quoting/escaping

The shell currently tokenizes input by space; it is not currently possible to include space characters in an argument.

The shell should use double-quotes to mark arguments, and allow double-quotes, spaces, and option-dashes to be escaped with the backslash character. The backslash may itself be escaped, to send a literal backslash.

Single quotes instead of double quotes to indicate a non-parsed string, in which substitution and escaping does not occur.

Convert output module to python

Output module is a series of functions for delivering output information to the installed devices in a device-agnostic encapsulated way. Output devices will take the information and use the metadata they can while ignoring metadata that is not relevant.

A/C: Output module accepted into python codebase, and has feature parity with original C++ module. Output module is able to write to a TTY via stdout.

Hire artist (3D modeling, personality module)

Hire artist for contract to do 3D character art and animation, based on pre-existing designs created by artists (see #21).

Command return values

Allow command handler functions to return a value. Allow them to return their status, and optionally a string with whatever they want as well. If no string specified, it defaults to empty string.

If return status is 0 (success), message is used as the retval. If nonzero, message is used as error message.

Mascot design ideas

Need to get some ideas for unique features for Masa-chan in order to differentiate her from other characters. This space can be used for brainstorming.

Convert Event type to Python class

The class msa::event::Event needs to be converted to an appropriate python class.

A/C: New python class and module is provided which gives same functionality as msa::event::Event class.

Command option parsing

Right now, the command parser always tokenizes input by whitespace, and then considers the first symbol to be the command and all following symbols to be arguments, which is then passed to the command handlers. If a command wants to accept an option, its handler must parse it out of the options manually.

The command parser must be modified to parse out options, and the command handler signature must be modified to accept the options separately from the arguments.

Convert msa::cmd types to own module

Currently, the command module and its types is coupled in a way that cannot be combined in python, due to circular dep formed with the timer module and the dispatch module. In order to fix this, we will pull the necessary types found in cmd.hpp into their own module in python.

Determine which types can be pulled out into own module (it will be the classes whose implementations do not depend on the mentioned modules), and then create that module.

A/C: New module with all non-dependent classes converted to python classes is created and accepted into the codebase. New module must have no dependencies on dispatch module or timer module.

Race condition during log file shutdown

There is a race condition which has occured at least once where the log module can be shutdown while another thread is in the process of writing to it, which causes a logic_error to be thrown during an otherwise clean shutdown. The log module should not be shutdown until all other threads have terminated, and additional synchronization should be added to ensure that even if this is not the case, the log module does not cause the entire calling application to crash.

Python config module documentation

Now that the system is up and running, we need some documentation before the system grows too large.

A/C: Add a doc comment for each class and method. Include type annotations suitable for use with pycharm.

Pull Timer classes/types out of dispatch.cpp

Dispatch is a bit bloated. We should pull the timer code out and put it into its own file in the event module.

Fail when config reading fails

• Create a new exception type specifically for use with reading the configuration.
• Ensure each module's init() function fails properly when config reading fails.

Create Architecture Diagrams

Current architecture is vague. Need diagrams / description of system as is.

differentiate CLI command events from auto-generated ones

CLI command events should have special status and should be able to do everything. Auto-generated input events ('synthetic' commands) should be discriminated on based on source, and disallowed from privileged function if the origin is a plugin.

Output maintenance stats

After system startup, schedule a timer to run every 500 ticks or so that logs system statistics. Should be togglable on/off via config file and the duration should also be configurable.

Switch CI/CD Pipeline to drone

Currently, pipeline is set to build through Travis CI. For unity with website builder and full control over build pipeline, this needs to be changed to Drone (pointing at the build server at https://moedrone.dekarrin.com).

A/C: .travis.yaml is converted to .drone.yaml; all pipeline steps build successfully using drone.

Timer command allows negative period

At CLI, the TIMER command will not complain if the period given is a negative number. This is unwanted behavior, and could theoretically cause the timer to fire on every tick since last_fired + period would always be less than now().

Build failing on MacOS

Environment:
macOS Sierra 10.12.6

Xcode version: 9.2 (9C40b)
make version: 3.81

Git commit: 3a99a04

Output:

$ make
c++ -c -o obj/platform/thread.o compat/platform/thread/thread.cpp -std=c++11 -Wall -Wextra -Wpedantic -pthread -Isrc -Icompat -include compat/compat.hpp
In file included from compat/platform/thread/thread.cpp:8:
compat/platform/thread/unix.cpp:120:16: error: no matching function for call to 'pthread_setname_np'
                int status = pthread_setname_np(thread, name);
                             ^~~~~~~~~~~~~~~~~~
/usr/include/pthread.h:500:5: note: candidate function not viable: requires 1 argument, but 2 were provided
int     pthread_setname_np(const char*);
        ^
1 error generated.
make: *** [obj/platform/thread.o] Error 1

Remove typedef

In C++, typedef is not necessary. There are now enough C++ features such that compilation with C-compilers is an unfeasible goal, so the typedefs should be removed entirely.

Convert msa::agent module to python

The agent module provides agent behavior and is the planned location for swappable personalities. For now, only a single behavior type is hard-coded into the system.

A/C: Create python module that contains feature parity to original module.

BLOCKERS:

• var module conversion must be completed
• output module conversion must be completed

Learn OpenGL for graphics output

Take two weeks to learn to develop OpenGL and prepare self to apply knowledge to MSA's output system.

Move var substitution into command

Variable substitution doesn't make sense for sitting in the agent module. It should be moved to the cmd module, with the agent submitting its substitutions at setup().

Hire Artist (2D characters; branding)

Need to find artist(s) who can do character design of different mascots. Contract must grant full usage and rights, including ability to modify, to MSA organization (or be more permissive).

Design to be used for branding; only need 2D art for that; no need for 3D modeling.

• Note that branding artist does not need to be willing to do NSFW, but should be made aware that character designs will almost certainly be used in such aspects when 3D modeling is added.

Location information on I/O devices

Add location information to I/O devices along with map manager to handle links between I/O nodes when character moves between them.

Convert src/event/topics.hpp into Python enum

Priorities and names given in src/event/topics.hpp file need to be converted to python enum.

A/C: Python enum exists that contains all topic definitions

Input module API not suitable for external extension

As is, the input module cannot be extended by adding additional types/handlers. See output module for a better model of how to be extendable.

Plugins should be able to provide new sources of input and handlers for the input, but the current system does not allow for this due to input module not exposing enough of its interface.

Decoupling "client" and "server"

Have you considered decoupling the system into a series of "clients" and a "server". e.g. One websocket server that is free to run asynchronously which is then communicating with clients asynchronously via websockets? This would allow for client/frontend development to be relatively free of the operational constraints of the server.

It would also allow clients to be implemented using other languages, and if networked plugins were allowed, it would allow for better interoperability with other IOT services e.g. frontend is a WS client, plugin is a WS client, server handles some commands internally but WS clients can register new commands with the server, that when the command is issued by the frontend client it is relayed to the server which then broadcasts that command to the WS plugin which manages its own state and handles the operation and then can make core api calls to the server. if a response to the frontend is required, it is sent to the server which routes the response to the client.

This would also make the stability of the system independent of the stability of a plugin. If the plugin doesn't respond the server can tell the frontend there was an error and continue on without additional error handling.

Frontend clients e.g. the terminal client, a voice client, etc. could then be self contained and the implementation for each would not be wholly dependent on the server.

Just food for thought. If you were interested in going this route, i would be wiling to look into websocket server implementations. I already have experience in python with Autobahn and Asyncio which make the process such a breeze. However i am not personally sure the state of C++ async abilities or websocket libraries.

May help provide a way to make #29 , #30 , and #31 server independent.

Remove system timers from CLI

CLI should have the power to kill a system timer, but only if it is a true CLI event (as opposed to a synthetic event).

This task is blocked by #16

Move config validation and parsing into config

Config sections should have methods for accessing data as different types, and for validating data as a particular type.

Convert msa::cmd module-level functions into own module

Convert all remaining functionality of the msa::cmd module (those that are dependent on other modules) to a new python module.

A/C: New python module for msa::cmd is created and accepted into the codebase.

BLOCKERS:

• Need the non-dependent parts of the cmd module to be pulled out first.

Flooding timer system causes hang

The system becomes unresponsive (I/O does not seem to work) when the timer system has too many tasks assigned to it.

To repro, execute timer -r 0 timer -r 0 echo Hi at the CLI.

Convert the msa::cfg::config_error exception class to Python

This class should be converted to an Exception class in python that is capable of locating and precisely reporting the location of an error in a structured configuration file-format (ini-style, with custom arrays)

A/C:
New class committed to the repository and accepted by code reviewers. Class must accept and return the following information:

• A message that explains the problem
• The key that there was a problem in
• The value that there was an issue with
• The section of config that there was a problem with
• The index of the key that there was an issue with (optional, defaults to 0)

Convert msa::var module to Python class/module

Convert the variable expander class to python equivalent. May be able to replace outright with shlex module; research into whether or not we can get variable substitution with shlex.

A/C: Need module that supports:

• creating new variable values
• unsetting existing variables
• checking if variables are set
• string shell tokenization with support for variable substitution and escape sequences

Create load() and save() methods in config module.

In python configuration module, create two new methods that can load and save the configuration file.

save() should accept a filename and a config.Config object, and iterate over all sections and write them to disk.

load() should accept a filename and read the config file at that file location into a config.Config object, which is then returned.

A/C: save() and load() are written and tested to work properly. Module should be able to both load and save the configuration. Prove that this is the case by creating a test program that uses the config module to do so, and commit it to the repository.

Convert all enums to scoped enums

Since project is compiled using C++11 and all enum literals are intended to be fully qualified anyways, all enum types should be updated to use enum class instead of just typedef enum to stop polluting the enclosing namespace.

TIMER -r command dereferences invalid pointer

When executing a recurring timer command, the second occurance will always result in dereferencing bad memory (because the first one would have caused a delete of the event arg (std::string *) in the TEXT_INPUT handler).

Convert msa::event's timer module to python

Timer class provides tracking of system timers.

A/C: new python module is accepted into the codebase.

Allow new types of I/O

Allow plugins to create new sources of input and output instead of just relying on TCP/UDP/TTY.

This may involve using a local socket to connect to self, which might be okay... research to find what would be best.

Add Drone job for testing python branches

Add a new set of pipeline jobs to test out the python rewrite code.

A/C: New automated tests are drafted for drone, and are able to execute in the pipeline.

Convert the msa::cfg::Section data class to Python

The msa::cfg::Section class is a multi-data format container class that holds a set of keys and values from a section of configuration.

In order to simplify the conversion process, we will split its conversion into multiple stories. In this first one, we just want to get the basic operations supported.

A/C: A new class written in python is accepted into the codebase. The class must accept a name as a parameter, for the name of the section. It must accept the following operations:

• Accessing a key by the index operator [], which returns the value of that key as a string
• a set() operation, which sets a key to a single value.
• a push() operation, which adds a value to an existing key (or sets the first value if the key does not exist)
• a create_key() operation, which creates a key without setting a value
• a has() operation, which returns whether a key exists in the section.

Add 'clear' command to clear terminal.

I was playing with it and the terminal gets very cluttered very quickly, and my eyes are bad. A clear command might be helpful for increasing readability of long terminal sessions.

For reference here is an article (to save you time) on different ways to clear the screen in cpp.

Secure plugin API

Plugins can modify system state to override settings of other plugins. Plugin calls should be segmented to avoid this.

A/C: A malicious plugin that attempts to modify the state of other plugins has no way to do so. Check this by first creating a malicious plugin which does such modification, then show that such a plugin doesn't work after changes are applied.

Convert the msa::cfg::Config typedef to Python class

The Config class is merely a type alias (a typedef) to a mapping of strings to msa::cfg::Section instances.

Python doesn't have a concept of a 'typedef', so we will use a normal python class.

Create a class that has as instance variables the name of the file it was loaded from, and a map of section names to Section objects.

A/C: Code is accepted into codebase.

Convert event handler module to python module

The event handler module, located in src/event/handler.{hpp,cpp}, is responsible for setting up synchronization primitives and allowing handlers to indicate when they are ready to be interrupted.

We may want to attempt to research a suitable replacement for this system in the future (such as asyncio or another such framework), but for now we will keep the older system for compatibility as we get through the python re-write.

A/C: Event handler and synchronization object have been converted to new python module.

Find 3D rendering engine

Need to find a suitable output system for agent. Must be free. Ensure license does not conflict with primary codebase.

Segregate system-level and user-level timers

Add a system to the event module whereby the core can add timers for system behavior. These timers have special status and cannot be stopped normally.

Add a terminal prompt character

From my experience it is common that when user input is expected to print a prompt character to make it easily visible to determine that the program is waiting for input and not hanging.

e.g.

give me input:

vs

give me input:
>

Add terminal history.

It might be helpful to add the ability to cycle through past commands with the up/down keys. May save a few keystrokes.

Replace platform-dependant threading solutions with std::thread

Platform-dependent thread start/detach/join calls should all be replaced by using the std::thread API of C++11.

Standardize module functions and return codes

Make sure each module is returning the same error codes for the same error conditions in each of the primary module functions:

• init()
• quit()
• setup() (if exists)
• teardown() (if exists)