fuelrats / pipsqueak

ED Fuel rats IRC bot

pipsqueak 2

ED Fuel rats sopel module package.

Looking to contribute to the next version of mecha?! Go to FuelRats/pipsqueak3

The below information is outdated, but will be updated soon™

Requirements

Python 3.4

In addition to Python 3.4 itself, pipsqueak uses the following modules which will be installed for you during part of the setup process:

• Sopel for the bot framework itself
• requests for API calls and fetching EDSM system data.
• iso8601 for date parsing.
• sqlalchemy for database access.
• alembic for database schema creation/migration across updates.
• psycopg2 for PostgreSQL database support in SQLAlchemy and Alembic.

You may need to install the development versions of the PostgreSQL client libraries to build psycopg2 on your platform.

PostgreSQL

All testing has been performed with version 9.5.1. Versions as low as 9.1 should theoretically work. Versions earlier then that may also work with some updates to the query used by bot's !search command.

PostgreSQL will need:

• A user account for the bot.
• A database instance owned by that user.
• The fuzzystrmatch extension loaded into the bot's database. This is included with PostgreSQL, though it may require installation of your platform's postgresql-contrib package or similar. It can be added by any superuser with CREATE EXTENSION fuzzystrmatch;

Alternate database options

Theoretically, another database can be used instead of PostgreSQL provided that SQLAlchemy and Alembic support it. Using another database will require adjusting the query used in !search to use an appropriate fuzzy-matching function, and has some possible issues when refreshing the systemlist.

There are potential locking issues with SQLite and possibly MySQL, notably during the couple of minutes it takes to refresh the starsystem list.

Installation instructions

Requires Sopel to be installed, for more information on this see Sopel's website. Following the virtualenv setup procedure should install sopel.

Acquiring source

git clone https://github.com/FuelRats/pipsqueak.git

Create a virtual environment

1. Most Python distributions include a built-in module for creating virtualenvs. If yours does not:

• # pip install virtualenv

2. # cd pipsqueak
3. # python -m venv *PATH* or virtualenv *PATH*

• PATH can be . to create the virtualenv in the current directory. Using 'venv' as a path is also fine, and will ensure virtual environment files are ignored by git.

Configure the bot

1. Copy sopel.cfg-dist to sopel.cfg

• # cp sopel.cfg-dist sopel.cfg

2. Edit sopel.cfg

• # vim sopel.cfg

Activate the virtual environment and install dependencies

1. # source *PATH*/bin/activate
2. # pip install -r requirements.txt

Start the bot

1. # source *PATH*/bin/activate
2. # python start.py -c sopel.cfg

• Using the built-in sopel command is not recommended, as it won't set PYTHONPATH correctly for imports.

rat-search.py

Commands

Command	Parameters	Explanation
search	System	Searches for the given system in the system database.
sysstats		Returns some statistics on the current system database. (Temporary command for debugging)
sysrefresh		Rebuilds the system database by pulling new data from EDSM, provided the existing data is old enough (12 hours by default).

         | -f         | Does the above, regardless of the age of the existing data.

sysstats and sysrefresh exist mainly for debugging and will probably go away or be redesigned in a later build.

Detailed module information

The system search compares the input with a large list of systems, downloaded from EDSM, if no list present this will fail.

rat-board.py

Commands

ref in the below table refers to a reference to a case. This can be: the client's nickname, the client's CMDR name (if known), a case number, or a case API ID (beginning with an @-sign)

Commands that add quotes to a case will create a new case when ref looks like a nickname or CMDR name and no active case can be found.

Command	Parameters	Explanation
quote	ref	Recites all information on Nick's case.
clear, close	ref	Mark the referenced case as closed.
list		List the currently active cases.
-i	Also list open, inactive cases.
-@	Show API IDs in the list in addition to case numbers.
grab	Nick	Grabs the last message Nick said and add it to their case, creating one if it didn't already exist.
inject	ref, message	Injects a custom message into the referenced case's quotes. Creates the case if it doesn't already exist.
sub	ref, index, [message]	Substitute or delete line index to the referenced case.
active, activate, inactive, deactivate	ref	Toggle the referenced case between inactive and active. Despite the command names, all of these perform the same action (e.g. deactivate will happily re-activate an inactive case)
assign, add, go	ref, rats...	Assigns rats to the referenced case. Separate rats with spaces.
unassign, deassign, rm, remove, standdown	ref, rats...	Removes rats from the referenced case if they were assigned to it.
cr, codered, casered	ref	Toggle the code red status of the referenced case.
pc	ref	Sets the referenced case to be in the PC universe.
xbox, xb, xb1, xbone, xbox1	ref	Set the referenced case to be in the Xbox One universe.

Detailed module information

pipsqueak includes a tool to keep track of the current board of rescues, called 'cases'.

Every message that starts with the word 'ratsignal' (case insensitive) is automatically used to create a new case.

Bonus Features

Ratsignals and lines added with inject perform some behind-the-scenes magic when they add lines to a case:

• If the system coordinates trigger System Name Autocorrection, the system name is automatically corrected and an additional line is added to the case indicating the correction. This fixes simple cases of accidental letter/number substitution in procedurally-generated system names, but does not otherwise guarantee the system name is correct.
• If the platform is unknown and a new line contains 'PC' as a whole word somewhere, the platform is automatically set to PC. If a new line contains XB, XBox, XB1, Xbone, XboxOne, Xbox1, XB-1, or any of several other variations, the platform is automatically set to XBox. If a line matches both the PC and XBox patterns, the platform is unchanged.

In all situations where this magic occurs, the bot' confirmation message will tell you about it. For instance, a new case where the system name was corrected and platform autodetected will end with something like (Case 4, autocorrected, XB)

sub does not perform any of this magic, and may be used to correct the bot in the unlikely case of false positives.

rat-facts.py

Commands

Command	Parameters	Explanation
fact / facts		Shows a list of all known facts.
fact	Reports translation statistics on the listed fact.
fact full	As above, but also PMs you with all translations.
lang	Reports translation statistics on the listed language.
lang full	As above, but also PMs you with all facts in that language.

Privileged Commands

Commands listed here are only usable if you have halfop or op on any channel the bot is joined to. You do not need to send the command from that channel, but must be currently joined to it.

Command	Parameters	Explanation
fact / facts	(add	set) fact-lang message
fact / facts	(del[ete]	remove) set fact-lang
fact / facts	import	Tells the bot to (re)import legacy JSON files into the database. This will not overwrite existing facts.

Config

Name	Purpose	Example
filename	the name (and absolute path) to the JSON file containing the facts, or a directory containing .json files. Any files found will be imported to the database on startup	/home/pipsqueak/facts.json
lang	Comma-separated list of languages to search for facts when no language specifier is present.	en,es,de,ru

Detailed module information

Scans incoming message that start with ! for keywords specified in the database and replies with the appropriate response. Also allows online editing of facts.

If the language search order is "en,es":

• !xwing: Searches for the 'xwing' fact using the default search order (English, Spanish). The fact command will display matching facts as xwing-en and xwing-es.
• !xwing-es: Searches for the 'xwing' fact in Spanish first. If this fails, falls back to the default search order.
• !xwing-ru: Searches for the 'xwing' fact in Russian first. If this fails, falls back to the default search order.

When adding or deleting facts the full fact+language specifier must be used (xwing-en rather than xwing). fact will tell you this if you forget.

rat-drill.py

Commands

Command	Parameters	Explanation
drill		Print out both drill lists.
-b	See above
-r	Print only the [R]atting drills list.
-d / -p	Print only the [D]is[P]atch drills list.
drilladd	-r name	Add name to the [R]atting drills list.
-d name / -p name	Add name to the [D]is[P]atch drills list.
-b name	Add name to [B]oth the ratting and dispatch drills list.
drillrem	name	Remove name from both drill lists (if applicable).
	To remove name from only 1 list, use drilladd

Config

Name	Purpose	Example
drilllist	The name of the JSON file containing the drill lists	drills.json

rat-socket.py

Commands

Command	Parameters	Explanation
connect connectsocket	none	Connects to the configured Websocket Server and starts dumping information to chat

Config

Name	Purpose	Example
websocketurl	The URL of the WebSocket to connect to	ws://dev.api.fuelrats.com
websocketport	The Port of the WebSocket to connect to	80

Detailed module information

Used to Connect to the WebSocket Part of the API to listen for Updates from RatTracker. Currently only Dumping gotten messages into Chat only though. Will Attempt to reconnect to the WebSocket if the initial connection fails or the connection gets lost at any point. If a reconnect fails, it will retry indefinitely with an always increasing delay which grows exponentially.

pipsqueak's Issues

Add @command as quiet alias of !command

Old Mecha has a !silence comment that quiets the bot in channel.

Rather than a global silence, we can add @ to run a command with no (non-error) feedback on commands where it makes sense.

Alternatively, this can be a user preference if we ever add those.

Remove !search cooldown, replace with PM-only response.

[2245.50Z] <DerryBear[AWAY]> Allow systems to be searched multiple times but have 2.0 pm the rat instead of post in the main channel 
[2245.56Z] <~Tyrope> Yeah, no, I typo'd.
[2246.00Z] <~Tyrope> And PM is no problem.
[2246.04Z] <~Tyrope> No cooldown at all then?
[2246.37Z] <DerryBear[AWAY]> No cooldown timer , but prevent channel spam by having the bot pm the rat who searched for thr system 

'2.0' mentioned here is sopelsqueak

Mecha auto correction

Currently gives out a shortened version of the system name (dropping of the start of the system name) which makes it a pain to copy and paste the corrected name. Please give out full corrected system name.

!filtergalmap

[client name], to learn about Filtering the Galaxy Map click here: http://bit.ly/1GKXCWv

Add an new rat command

Hello i think it would be good when we have an new rat command what give the new rat the most important links.

Like:

1. http://goo.gl/UXt8Ic | 2. http://goo.gl/kZ21WC | 3. http://goo.gl/3S8p8I | 4. http://goo.gl/1J1F6b | 5. http://goo.gl/4Y7QNB | 6. http://goo.gl/mDF03t

Make getRat/ClientName/Id smart

Cache information about id/name relations to drastically reduce the amount of api calls and improve the speed of which messages will be relayed from the api to irc

Make r@signal trigger message configurable

Since we're using a separate instance of Mecha as a dedicated #DrillRats bot, it should be possible to customize the r@signal message so that it can respond to 'drillsignal' and such like the normal bot responds to r@signal

Add details for contributing

Github will automatically link to a CONTRIBUTING.md file if one is present in your repo. @duk3luk3, I'd love to see one detailing the requirements you have for working on pipsqueak. It should include the forking requirement, any code style standards, PR requirements, and anything else necessary for a PR to be accepted.

Check out puppet's CONTIRIBUTING.md for a great example.

Accept empty commands if there's only one case

Should commands like !quote require an identifier if there's only one case? I feel like that's unnecessary.

Practice bot

A second copy of the bot running in only a specific channel (E.G. #RatBot) independent from the main bot.

To allow Rats and Dispatchers to practice with the bot, learn commands, hands on without affecting live ops and impeding dispatch.

Purge case with !invalidate or similar

It should be possible to close a case and immediately mark it as Going/deletable, for those instances where someone misfires a signal or our friendly neighborhood trolls.

Current status of rat-board overhaul branch (work/dragons)

I'll be out of town for several days and won't be able to get the current code in my work branch up to even-remotely-stable by then (returning Sunday evening PST, probably Monday in actuality), so here's an update in case someone takes it on while I'm gone. This should probably be split into multiple issues, but I have too much to do IRL.

Deep magic at the end, basic stuff at the beginning

sopel-modules/rat-board.py

RescueBoard class

This replaces the various dicts and tracking mechanisms we've used so far with a more integrated management. Key features:

• Can be used as a context manager (proxies to its lock controller). All access to a board sould be within its context manager, since it doesn't use SopelMemory().
• Case number assignment now uses a hybrid available-id pool + an autoincrement in case of overflow.
• Adding or removing a case with add/remove autopopulates various dicts that maintain an index of cases.
• change() is a context manager for ensuring indexes update if they change on an existing case. !cmdr ... David Braben should at some point do with board.change(): rescue.client['CMDRname'] = 'David Braben'
• find() is a unified method for finding (and possibly creating) rescues. It supports, in order from what is searched first.
  • Lookup by number. ("4" or "#4")
  • Lookup by API id by prefixing it with an @. ("@0248dead...beef0672")
  • Lookup by CMDR name and nickname, in that order.
    If create is True and find() doesn't find anything, it creates a case and sats the client CMDRname and nickname to whatever the search term was.

It's possible to create multiple RescueBoards. Among other uses, we can use an extra one as a recently closed cases queue, and one for a "special behavior in #drillrats" board that is isolated and disconnected from the API so drillspatch can use normal case management commands.

Rescue class

This represents an individual case. Has some convenience methods, and currently largely relies on ratlib/api/props.py (see Deep Magic at the end of this)

Overhaul

Starting from the top, I've been refactoring code to use the new models -- but I haven't reached the end of the file yet. Notable changes beyond the general refactor are:

• append_quotes searches for (and possibly creates) a case, including filtering the line(s) through autocorrect (and any additional filters we might add). It returns a tuple of (rescue, newlines). newlines is useful to report what was actually added (i.e. the autocorrected system name, rather than the original), and rescue can be examined. (A rescue where id is None was newly created, vs added on to.)
• Channel logging uses a collections.OrderedDict. Among other things, this sets an upper bound for how many distinct nicks the bot will maintain history for.
• Interface to API calls was overhauled and moved to ratlib/api/http

Plan

In addition to finishing the overhaul, fixing bugs and finishing TODO:

• When gathering changes to a Rescue to submit to the API, throw the resulting gathered changes into a queue. Objects in the queue are sent to the API in order (retrying as needed); if the API is down the changes will just queue indefinitely. Also, actually implement the gathering mechanism. (rescue.commit() is the planned mechanism for this, things that modify rescue should be able to mostly pretend the API doesn't exist.)
• Save the state of the API submission queue (hopefully empty) and board on exit, so data can be recovered on startup if the API happens to be unavailable.
• All the websocket stuff every, which should simplify a lot of the synchronization. (Query all relevant cases on startup, and then just listen for changes we care about).

Oh, huge chunks of this are at least partially tested in isolation, but not at a large scale. Like I said, I really ran out of time.

Known bugs

• RescueBoard.change() doesn't lock.
• RescueBoard._lock probably should be reentrant.
• RescueBoard.find() should return immediately upon failure to find a case by ID or Number.

TODO

• Implement functionality so Rescue can actually gather its changes and send them back to the API.

ratlib/api/http.py

Just a refactor and move of the code involved with API calls. Named 'http' in hopes of an eventual api/ws for websocket.

Oh, throw more detailed exceptions too.

ratlib/init.py

Adds a 'format_timedelta' function that returns exact results. Used for reporting how long a case has existed when it is closed, for giggles.

ratlib/api/props.py (HERE THERE BE DRAGONS)

This is a quick and dirty implementation of something that kind of mimics the change tracking of SQLAlchemy's ORM. Consider this a separate project from Mecha on par with something you might install with pip, so some portions of what it implements aren't relevant to us.

Properties and Metaclass

• TrackedProperty is the basic property implementation, using Python's descriptor interface. Assignments to TrackedProperty are stored internally in a private _data dict on their instance, keyed to the property name. They also add themselves to the instance's _changed set, which allows one to examine an instance and see what properties have change. TrackedProperty has some framework for how the property should be converted to and from JSON as well. (Well, to/from a type that is JSON-serializable; they don't produce the JSON themselves.)
• There are a few subclasses of TrackedProperty, used for some specific cases (like ensuring the Instrumented* properties remain the correct type even if they're blanket assigned.)
• TrackedMeta is a metaclass. When a class uses this metaclass, any not-yet-named properties will be named based of their key in the class's namespace dict (so foo = TrackedProperty() in the class body yields a property named 'foo'), and the resulting TrackedProperty is added to the set cls._props
• TrackedBase implements basic support for all of the above, and is intended to be the base class of anything doing change tracking. (Currently just Rescue, but this may see use elsewhere/in other projects).

Instrumented{List,Set,Dict}

These are subclasses of their corresponding Python base type. They do limited change tracking: when there's a clear way on how an operation should perform even if the instance's current state was different, they produce information on how to merge their current changes with a refreshed data source. When there's no clear way to resolve the conflict (basically, any modification that they can't handle), set their replaced property to True and just overwrite the new incoming data upon merge instead instead.

• Lists remember their appended elements, whether through append or extend.
• Dicts remember keys that were added, changed or deleted.
• Sets remember what items were added and removed.

Changes to Instrumented objects that are not deterministic result will set the replace attribute to True, which means that it's interpreted as replacing any new data that comes from a refresh rather than reconciling, as do operations like obj.clear() or any other modification not supported.

The merge(other) method determines what to do if the underlying data is updated. The commit() method discards the change history and resets the replace -- intended to be called when we've successfully reconciled our changes with the other source (e.g. the API), like right after a merge.

Example: I want to assign Trezy to a case. I don't care who else is assigned. The case is currently stored in rescue, and `rescue.rats = {'Tyrope', 'Absolver'}

1. I do rescue.rats.add('Trezy'). Behind the scenes, this adds a dict entry {'Trezy': True}, where the True indicates that it was an addition rather than a deletion.
2. Before writing the change, Mecha refreshes the case. Looks like Burwellian was added too.
3. rescue.rats.merge(refreshed_rats) checks to see if it needs to do a full replace. Since it doesn't, it clears itself, adds everything in rescue_rats, and then adds everything in its own list of additions. rescue.rats now equals {'Tyrope', 'Absolver', 'Burwellian', 'Trezy'}

Example 2: I royally botched up and assigned a bunch of rats to the wrong case. So I want to clear who's assigned.

1. I do rescue.rats.clear() Internally, the replace property is set to True.
2. A refresh happens.
3. replace is true, so the refresh is ignored, and I send whatever my current rescue.rats value is.

Known Bugs

• Instrumented classes don't have a way of notifying their parent property that they've changed. This is one of the major reasons for having them, but should be an easy fix.
• Assigning to something that should have a instrumented class won't trigger replacement, though it will be coerced to the correct type. This means that rescue.rats = set() and rescue.rats.clear() do not yield the same result if data is merged in.

Run work/sadb in offline mode in #drillrats

With my current work/sadb branch being largely stable, we should consider running it in #drillrats in lieu of Mecha's current presence there.

In addition to allowing early testing, it'll allow drillspatch to use real bot case management commands without affecting live rescues. And the new !search becomes available for general use.

!assign command doubts

Hey, I have my doubts about the assign command now it automatically suggests the client to send a friend request. Some rats have a underscore instead of a space, and clients might get confused about the part in brackets rats usually have behind their names.

Imho, it also negates the human part of a dispatcher. They're humans, too! 👨

instancing issues.

Change commands to not address their sender as much.

Most Mecha commands respond using bot.reply, which prefixes the response with the sender's nickname. This causes excessive pings in most cases.

We should remove this, except in cases where it makes sense to keep it.

drillsqueak crashed

@Marenthyu

Drillsqueak crashed with an input error. Can't for the life of me get it to restart.

When able please restart the bot, and write here the details of starting it.

Command to set Client:CMDRname value in the API

Because things.

Replace all uses of % string formatting by str.format

% is deprecated, please stop using it. str.format may be a bit clunkier and more verbose in some cases but it's a much better interface.

Code Red Instructions Fact

We could use code red instructions (The, "Log into OPEN play (NOT YET), enable your wing beacon, add your rats to wing" thing) to the facts. Mainly so we can get translations in place for it as it's not uncommon to get a non-English speaking CR from time to time.

Multilingual !assign

The add person to friends-message when doing !assign would be nice if it could be multilingual like all the facts are.

Private group instructions command

!pg = Log out to main menu, click Start, then Private Group, then the rat's group.

Add different Languages for Facts/Instructions

We got a "lot" of Russian Clients lately, so adding the Facts/Instructions in Russian to pull from them would be great.

Off the top of my head the following at least should be included:

• !ruo2 - Russian Emergency o2 Question and instruction to log out asap if true
• !ruprep - Russian Prep
• !rufr - Russian Friendslist add instruction
• !ruwing - Russian Wing Instruction
• !rubeacon - Russian Beacon Instruction

Helpful would be to also have a general "!askru" to ask if the client speaks russian to simply confirm that fact.

Other maybe helpful languages to add:

• German (I could do that since im native german)
• Had one in my mind but forgot it

Implement block for creating cases with client name "ratsignal".

Do not allow pipsqueak to create cases with a client name of "Ratsignal". Yell (loudly) at dispatchers when they attempt to inject cases with that name.

Station search implementation

I'd like to suggest a few additions to pipsqueak to enable rats to search for a few vital points of information.

!scoopable [Jump range]

Returns the closest class KGBFOAM star to specified system, alternatively with a specified jump range.

Example output:

!scoopable COL 123 AB-C 1-33
Mechasqueak: Nearest fuel source is COL 123 AB-C 2-4 (11.6 LY, class M)
!scoopable COL 123 AB-C 1-33 5.35
Mechasqueak: Nearest fuel source is COL 123 AB-C 2-4 (11.6 LY, Class M, 3 jumps)

!refuel [pad size] [jump range]

Returns the closest station with refuel capabilities and optionally specified minimum pad size.

Example output:

!refuel COL 123 AB-C 1-33
Mechasqueak: Nearest station is Happy Terminal, COL 123 AB-C 2-4 (11.6 LY, Size M pad)
!refuel COL 123 AB-C 1-33 Large
Mechasqueak: Nearest station is Bigpad Orbiter, COL 123 AB-C 2-4 (11.6 LY, Size L pad)

!rearm [pad size] [jump range]

Returns the closest station with rearm capabilities and optionally specified minimum pad size.

Example output:

!refuel COL 123 AB-C 1-33
Mechasqueak: Nearest station is Happy Terminal, COL 123 AB-C 2-4 (11.6 LY, Size M pad)
!refuel COL 123 AB-C 1-33 Large
Mechasqueak: Nearest station is Bigpad Orbiter, COL 123 AB-C 2-4 (11.6 LY, Size L pad)

Alot has already most of the groundwork that would be needed for this in his pathfinding Python, so this should be fairly easy to implement.

New Facts: !modules !invites

Add new facts for convenience:
!modules: you can disable modules on the right panel by pressing then q then space select active press space
!invites: how to accept invites: marenthyu.de/invites.png

French translation of facts.json

French Facts.txt

System name detection

While I've generally considered it to be "not feasible" in the past, I have some ideas on what could make parsing a known system name from a line of text an actual possibility. This is a collection of ideas that might aid that:

This would likely require the system database for !search to be stored in the sqlite database rather than in a JSON file. After the system names are stored, an analysis pass runs that compiles the following statistics:

For each group of systems, grouped by first_word and normalized to lowercase:
min_words: The number of words contained by the system in the group with the fewest total words.
max_words: As above, but for greatest total words.
const_words: Words that EVERY system matching first_word have in common, starting with the second word. (e.g. for "Core" this may be "Sys Sector"). This may be empty.
bloom_filter: The result of applying a bloom filter against all matching systems, using a hash function that only covers the 'unique' part of the system name as the input data.

This prefix data may be stored in-memory if it ends up being a small dataset, otherwise in a secondary database table.

Scanning user input for a system name would then consist of:

1. Split input into words.
2. Query the statistics table for entries that are in words
3. For each matching entry, check the following words against const_words (if any)
4. If they match, for each amount of words from min_words to max_words, check the system name against the bloom filter. If it matches, check it against the actual system list.

This parsing would only happen when input is being added via ratsignal or !grab, since that's the only time we'd be doing anything with that data anyways.

Define the running environment

I'd like to see either:

1. A list of requirements for the ideal pipsqueak environment, or
2. A Vagrantfile/Ansible playbook that sets up such an environment.

I'm gonna be pushing for this on all of our repos so that anybody can pull down the code, run vagrant up, and have a functioning project.

I'll be doing this with the API repo shortly so the work for this ticket can be based on that. I can also do the legwork for this ticket, I'd just need number 1. :-)

Lightweight system name autocorrection

There's a common issue with clients misreading system names and entering 0/1/2/5/8 when they meant o/i/z/s/b or vice versa.

There's a particular pattern that all procedurally-generated systems appear to follow with their ending coordinates that means we can automatically detect and correct these in many cases without needing to resort to !search:

sectorname LL-L LD-D
or
sectorname LL-L LD

Where "L" is a letter and D is any number of digits. (The actual pseudo-regex I'm using is \w+\s+(?P<l>LL-L\s+L)(?P<d>D?-?D+)\b, which expands to \w+\s+(?P<l>[a-z85120][a-z85120]-[a-z85120]\s+[a-z85120])(?P<d>[0-9siobz]?-?[0-9siobz]+)\b after figuring substitutions)

I've been running a Hexchat script that detects and autocorrects cases of this for the last week or two and it seems to be reliable -- so after some discussions I think it ought to be integrated into Mecha as follows:

• Check for 'wrong' system names in all incoming chat
• If a wrong system name is found, correct it in channel and add it to a temporary blacklist to avoid channel spam
• Regardless of the blacklist, check for 'wrong' system names in !grab, !inject and ratsignal and correct them before they reach the case notes
• Add a note explaining the autocorrection and the original location (in the event someone does find an exception to the pattern)

Space at end of message

Spaces after autocompletion at the end of commands get taken as arguments so parsing fails.

Allow ! commands in PM

Since a lot of CMDRs are so used to typing ! before their commands, would it be possible for the bot to parse the message and remove a leading exclamation mark (if present) before processing the text that was passed to it?

End result: " list " and " !list " would both result in the bot replying with the list of cases.

Cheers!

Add ability to import facts from URL

Add the ability to import facts from a URL to simplify contributions of translations.

Debriefing SOP command

I created a !kgbfoam-like image, that can help (new) rats debrief a client. My idea was to add a command to it, something like !db , but that could be changed. Almost all of the rats I showed it to liked it, so Cpt_Shinobi told me to post it here.

Link: http://goo.gl/VT6qyc

Tell client to send friend request to rats

When rats are assigned to a rescue the client is immediately told to send requests to the rats. This could all be part of the !go or !assign commands.

Mecha resiliency and the API

Poking all of you since this is somewhat multidisciplinary
@trezy @tyrope @xlexi @kenneaal

(Wall of text crits for 9001 damage).

The current API model for "skunkworks" will be stripping out all of the HTTP functionality and speaking only Websockets. This drastically simplifies some of the design bits, considering it needs WS anyways for notifications and not having to implement both is... convenient.

It's been theoretically possible for Mecha to operate in a hybrid "online, with offline fallback" mode for awhile now, though I hadn't given it much thought of how it'd look. I also don't have a convenient way to test various failure modes since I haven't been able to get the API to run locally to be able to do things like... 'crash' it in the middle of Mecha talking to it.

One part of this is figuring out some of the specifics of when Mecha should consider itself offline vs online, how exactly it should go about checking for restored connectivity, and how it should reconcile vs the API after being disconnected for a set amount of time.

To that end, here's some tidbits on the current setup:

Property Change Tracking

Mecha tracks "pending" property changes. That is, I can do this:

rescue.platform = 'pc'
rescue.quotes.append("quote")

and Mecha will know that both platform and quotes have changed. Furthermore, for some simple cases of modification (like appending to a list, add/remove from a set, or add/remove/set in a dict) it maintains a limited amount of state that can be used to re-apply the same change against an updated collection. (In essence, it's kind of like git rebase). More complicated modifications that can't be reliably repeated (like removing or altering an item in a list) flag the collection in such a way where Mecha knows it can't reliably merge and instead will overwrite whatever the API provides in its entirety.

This is currently used by the mechanism for saving cases to determine which attributes to send -- once a case is successfully saved, the particular properties are 'committed' which essentially removes them from the set of changed properties and tells them to discard any pending state they might have.

This is all handled under the hood, individual commands just change properties and call rescue.save() without having to deal with the minutiae.

Async saving and applying updates from the API

Mecha immediately applies any changes locally, reports on their affects, and then queues the relevant API call. It does have the ability to report any subsequent failures, but does not have the ability to roll back state.

Mecha also applies any updated rescue messages it receives from the API against a rescue, with one exception: Any properties with pending changes keep their existing (Mecha-supplied) values rather than what the API says. This is probably the correct behavior, because if they're flagged as changed in Mecha that means it's probably trying to tell the API about the change and hasn't yet. The exception to this exception is collections: as mentioned above, they'll replay their changes against the API version of the data if they believe it is feasible.

There is one potential problem here: If there are multiple pending updates to a case and they end up executing out of order. In theory the change protection should prevent any mayhem from happening. In practice, Mecha has a per-rescue "lock" -- while a case is locked no other command may modify it until the lock is released. If everything is healthy with the API, this should be unnoticeable -- but if there's issues saving cases, it'll take longer for the lock to be released (if the API is slow or calls are otherwise timing out) and may somewhat slow down multiple actions on the same case.

The rescue lock only applies to writes -- !quote, !list, etc are unaffected.

Timeouts and Retries.

Currently, Mecha allows for a 30 second timeout before giving up on a change. There's not yet any reply mechanism -- the case will just stay out of sync until something triggers it to save again and that save succeeds. This is bad and needs to be fixed.

There's also issues retrying some requests -- appending quotes as a convenience method is great, until this scenario occurs:

1. Mecha tries to append a quote.
2. The request times out or the connection is lost.
3. Unknown to Mecha, the quote is successfully appended.
4. Mecha retries the action
5. The quote gets duplicated.

(Replace "append a quote" with "Create a new case" for a bigger issue.)

One option here might be to treat a timeout as "Okay, we're totally offline" until some other connectivity test proves otherwise and then reconcile state (but we need to figure out how to reconcile state.)

Currently, only appending quotes and opening new cases are not idempotent and thus cannot safely be retried.

Reconciliation after downtime.

The big question here is... what happens when whatever issue kept Mecha away from the API is resolved?

There's a few options here:

• Mecha can assume it's authoritative and completely overwrite the case with its own state. (This can merge in anything that doesn't have pending changes as normal). This is currently what I'm leaning towards, but is not without drawbacks -- anyone using !sub on a case in Mecha will overwrite all of the changes to that case's quotes made via some other API user if the issue is due to Mecha losing its network connection rather than the server.
• Mecha can discard all of its changes and use the API's versions instead. This is probably the wrong option.
• Mecha can pick some hybrid approach (but what?) based on case attributes -- like what Mecha has for dateModified vs what the server has. This would be super easy if individual attributes were versioned, but that's also overkill.
• Something else you guys can come up with.

Something else Mecha will need to do is remember all of its closed cases until API connectivity is restored -- but I want a way to be able to access recently closed cases from Mecha anyways. (My thought is they'd use negative index numbers and rotate between, e.g. -1..-10)

What exactly is downtime, and what is an error.

While connection loss is easy to identify, there's a fine between "the API is being slow/timing out" and "the API is allowing connections, but is completely unresponsive to commands." Figuring out where to draw this line in Mecha for it to switch from online to offline mode is going to be key. Figuring out when functionality is restored is also important.

Also, Mecha needs to be able to distinguish from an error message that is -- say -- complaining about MongoDB or Elasticsearch being down (which corresponds to "API is borked, go offline!"), error messages telling it to retry something/etc (e.g versioning conflicts, if/when the API gets them), and error messages saying "Nope, you screwed up, don't ever try that again." Right now most API errors are pretty vague and nonspecific.

Add alternative fact import/export format

Since people contributing translations may not understand the semantics of JSON and it's quotation marks/etc, add an alternative format that's easier to write

Commands for troubleshooting

For example

!xnat

: please can you tap on xbox button, tap left on d pad and scroll down to settings, choose all setting and then choose network. Confirm NAT type is open.

!xrelog
: please restart the game client by press xbox button, pressing the start button and choosing quit.

!pcrelog

: please completely quit the game by choosing save and exit to desktop

!xreset

Please turn your xbox off, unplug power, then plug back in and restart.

etc etc :)

System search causes SQL query to be output to chat instead of result

<Harkon_fry[Dispatch]> !search Col 258 Sector IJ-C a 15-3.
<DrillSqueak[BOT]> [SQL: 'SELECT starsystem.id AS starsystem_id, starsystem.name_lower AS starsystem_name_lower, starsystem.name AS starsystem_name, starsystem.word_ct AS starsystem_word_ct, starsystem.x AS starsystem_x, starsystem.y AS starsystem_y, starsystem.z AS starsystem_z, starsystem.prefix_id AS starsystem_prefix_id, levenshtein_less_equal(starsystem.name_lower, %(levenshtein_less_equal_1)s, %(levenshtein_less_equal_2)s) AS distance \nFROM sta

German translation of !commands

{
"xbeacon": "Um das Wingsignal zu aktivieren halte X und drücke dann rechts auf dem D-Pad. Dann drücke einmal Q, wähle Signal, drücke Leertaste und ändere es von "Aus" zu "Geschwader",
"pcwing": "Um eine Geschwadereinladung zu senden, gehe zum Kommunikationsmenü (\u0002Taste 2\u000f), \u0002drücke ESC\u000f um das Chat Eingabefeld zu schließen, und gehe dann zum zweiten Panel (\u0002Default key E\u000f). Dann wähle den CMDR aus dem du die Winganfrage senden willst, drücke die Leertaste und wähle \u0002Geschwadereinladung\u000f.",
"pcbeacon": "Um das Wingsignal zu aktivieren, gehe zum rechten Menü (\u0002Taste 4\u000f), gehe zum Funktionsmenü (\u0002Taste Q\u000f), wähle \u0002Signal\u000f und setze es auf \u0002Geschwader\u000f",
"xwing": "Um die Rats zu deinem Geschwader hinzuzufügen, halte X und drücke dann hoch auf dem D-Pad, drücke einmal RB, dann wähle den Namen der Rat, drücke Leertaste und wähle [Geschwadereinladung]",
"pcfr": "Um eine Freundesanfrage zu senden, gehe zum Menü (\u0002Drücke ESC\u000f), klicke auf \u0002Freunde und Private Gruppen\u000f, und drücke dann \u0002Freund hinzufügen\u000f",
"prep": "Bitte droppe vom Supercruise, komm zum stehen und deaktiviere alle Module AUSSER der LEBENSERHALTUNG",
"xfr": "Um die Rats zu deiner Freundesliste hinzuzufügen drücke doppelt auf den XBox Button, dann drücke Links auf dem D-Pad. Jetzt drücke auf A und suche nach dem Namen der Rat.",
"xquit": "Bitte logge dich SOFORT aus, indem du \u0002START\u000f drückst und \u0002Save and Exit to Main Menu\u000f!",
"pcquit": "Please log out of the game immediately by pressing \u0002ESC\u000f and selecting \u0002Speichern und zum Hauptmenü\u000f auswählst!",
"filtergalmap": "Um zu lernen wie man die Galaxiekarte filtert, klicke auf diesen Link: http://bit.ly/1GKXCWv",
"kgbfoam": "Um zu lernen wie man die Galaxiekarte filtert, klicke auf diesen Link: http://bit.ly/1GKXCWv",
"galnet": "Hier ist ein Link zum GalNet: https://community.elitedangerous.com/galnet",
"sc": "Bitte reaktiviere deine Schubdüsen und deinen Frameshiftantrieb, wähle alle Ziele AB und gehe in den Supercruisemodus. Flieg für circa fünf Sekunden weg vom Stern des Systems, dann droppe zurück in den normalen Raum.",
"ratchat": "HEY! #fuelrats ist nur für Rettungsoperationen, wenn du/ihr reden möchtest, tritt bitte #RatChat bei indem du '/join #RatChat' in den Chat schreibst.",
"rc": "HEY! #fuelrats ist nur für Rettungsoperationen, wenn du/ihr reden möchtest, tritt bitte #RatChat bei indem du '/join #RatChat' in den Chat schreibst.",
"pw": "Zeit für den Papierkram! http://bit.ly/1lBp53V"

}

// German translation of the current mechasqueak commands I did a month ago for Phidora_Eclase.
//Questions, Feedback and everything else please to Becci in #RatChat.

Make sopelsqueak ready for production use

I would like to retire my old jaraco.irc based version of mecha asap.

What is needed to bring the new version to feature parity?

!close / !clear safety switch

[0009.14Z] <&Ravenov> hmm, i just thought of a feature for mechasqueak[BOT] next time i see DukeOfEarl[Off] 
[0009.40Z] <Tyrope[PC]> Ravenov: Tell me, I'll post it on the bug tracker.
[0009.56Z] <&Ravenov> a !protect option, so that a !clear requires a 2 step so a case doesn't get deleted by mistake

This sounds like it should be default. The new bot already requires the username to be entered (since the IDs from the web API are just stupid) Perhaps it could go something like this:

<Tyrope> !clear testCase
<pipsqueak> To clear "testCase" confirm with code 72478
<Tyrope> !clear testCase 72477
<pipsqueak> Confirmation code incorrect
<Tyrope> !clear testCase 72478
<pipsqueak> Confirmation code correct, case "testCase" closed.

edit: this 5 digit code would of course be randomized per case.

Add Websocket Capabilities and Handle API events

Add Websocket capabilites. Make API Events recognised. (see https://app.asana.com/0/88474121069780/92104607825103 )

Translated commands

Add translated versions of commands such as !prep (as E.G. !prep-de to translate to German) for clients who struggle to speak English. Russian prob most needed.

Rats/Rescue Languages

ensure compatability with https://github.com/FuelRats/api.fuelrats.com/issues/126

Supercruise hop command

Something to the tune of !sc clientname : Clientname Reactivate your thrusters and Frameshift drive. Un-target everything and jump to supercruise. Fly away from the star for about 5ls then drop back down.

Everything in !list red after (CR)

As you can see, everything after (CR) is in red in !list response.

Not the biggest or most urgent issue, but thought I would report it.

Operations support in Mecha 2.0

Add support for the new API support for Operations. This can help separate out long-term cases from the main rescue board, cleaning up IRC board a bit.

!list should show for instance:
Myclient [0] Myclient2 [1](Plus 1 inactive and 1 operation)
!list -i would pull the inactive, but leave the operations.
!list -o would pull the operation as well.
!operations or !ops lists currently active operations.
!opname <casenumber/name> - Converts a normal case to an operation and sets the name for that operation. Moves it to the operations queue.

!list should probably exclude rescues marked as operations in its default list, and only enumerate them.

Extra commands for Russian without English language knowledge

We need more commands exclusiverly for Russian, that will exist only in Russian language, such as:

!hello-ru - greets and asks about fuel. Can be merged with !o2-ru

!o2-ru - asking about timer on russian

!prepfr-ru - the command will explain what should the client do and why (These CMDRS are heading your way - please add them to friends)

!prepfrcr-ru special command which is going to copy the previous one with emphasize on DO NOT LOG IN

!prepwing-ru the command will explain what should the client do after he added rats to friends and will state why (Now, please, add CMDRs to your wing and drop beacon so that our rats can drop yo your position and fuel you)

!prepwingcr-ru - special command which is going to copy the previous one with emphasize on DO NOT LOG IN

!preplog1-ru: first step of log in in case of CR - adding to wing (after that it is useful to use !pcwing-ru)

!preplog2-ru second step of log in in case of CR - dropping beacon (after that it is useful to use !pcbeacon-ru)

!preplog3-ru final step of log in in case of CR - making sure that client understood anything and has questions. Asking if client is ready

Comment and suggest anything else

Wrong german !pcquit Translation!

In Pipsqueak it is:
"pcquit": "Please log out of the game immediately by pressing \u0002ESC\u000f and selecting \u0002Speichern und zum Hauptmenü\u000f auswählst!"
and this is mostly English.
Here my German Translation:
"pcquit": "Bitte gehe sofort zum Hauptmenü, drücke dafür \u0002ESC\u000f und wähle \u0002Speichern und zum Hauptmenü\u000f aus!"
I think it's more German Language.