Minecraft and RocketChat are both new features that will have high resource consumption if used as intended (i.e., by many people at the same time). I restarted the minecraft container tonight in response to complaints of lag + logs say can't keep up. We should probably rearrange so they're not both on the same server.

If we consider rocketchat as equally critical as the mail server, it might make sense to move it to server2, or we can see if server1 can handle anything else, or we could reduce the maximum players/RAM allocation to minecraft, or we could look into getting an additional server (or VPS).

@clonm @closoft @clohard

cloynemongodb

This was on Christopher's computer.

Forked version of mongodb container (version 3.6) with the recommended settings from the rocket-chat container (step one):

oplog size: 128
replset: rs0

There is a third option "smallfiles" but we haven't done anything with that yet.

First, localhost and localhost:3000 give "conection was reset" at the moment, which is different from "connection refused" previously with clonmmongodb.

Following step two, we got some kind of syntax error claiming a missing curly bracket, but there's doesn't seem to be a missing one. New error after adding escape characters around the quotation marks. Something might have been misnamed.

Added to mongo 3.6 dockerfile echo -e "replication:\n replSetName: \ "rs01\"" | sudo tee -a /etc/mongod.conf

In /etc/service/rocketchat/run added to the last line (so the same line): --replSet rs01 before 2>&1.

Ran rs.initiate() in the mongo shell. Oddly enough replicaset first showed up as secondary, but then showed up as primary upon runningrs.isMaster().

Ran export MONGO_UPLOG_URL: mongodb://mongo:27017/local?replSet=rs01 MONGO_URL: mongodb://mongo:27017/rocketchat node main.js in the mongo shell.

After this, rocketchat worked on localhost:3000.

server2 was responsive to ping but unresponsive to SSH or HTTP. We went to Fremont to manually power cycle it. This fixed ssh, sympa, phpmyadmin, and phppgadmin, but not postfix or blog.

using salt-ssh 'server2' state.highstate was not showing output until forced to quit with control-c, which, if allowed to run long enough, did show the normal output. After fixing the routes, this ran normally and showed output on its own.

Mailserver

Nullmailer was showing temporary error in name resolution for connecting to mail.cloyne.org. This suggested either a DNS issue or an issue with postfix. First, I tried to test mail.cloyne.org using telnet. This worked from my laptop but failed from server2, so that meant it wasn't an issue with mail.cloyne.org. Next, I checked /etc/resolv.conf to find the IP of our DNS server and then tried pinging it. Ping failed.

I re-added the routes we added when we first moved the servers:

ip route add 64.62.133.41 dev [eth0 for server2, p1p1 for server3]
ip route replace default via 64.62.133.41 dev [eth0 for server2, p1p1 for server3] src 64.62.133.[44 for server2, 45 for server3]

This fixed the mailserver but not the blog.

Blog

After fixing the route, I ran salt-ssh server2 state.highstate again with just mysql and its dependents and this fixed it. Probably because it upgraded the tozd images which had been upgraded? Not sure what was wrong.

all the past versions of pgsql were running, which prevented sympa from loading correctly. Stopping the extra containers fixed it. Removing them for good measure.

Ubuntu 14.04 Trusty is no longer supported, so we need to upgrade.

server1:

• powerdns-secondary: pushed updated image to run on Xenial, but not sure if it's actually working. See notes on powerdns for server2.

server2:

• powerdns-primary: works but requires manual intervention, see readme. the secondary image's issues are similar.
• sympa: running on Bionic (18.04.1)
• backup: on Xenial (16.04.6)
• blog: on Xenial (16.04.5)
• pgsql: Running on Bionic (18.04.2).
• phppgadmin: Running on Xenial (16.04.5) from clonm, waiting for pull request acceptance on tozd.
• mysql: Upgraded to bionic, but still need to update github. See pull request
• phpmyadmin: Still on Trusty Updated to Xenial
• mail: still on trusty. tozd has been cloned but not changed upgraded to bionic at clonm/docker-postfix
• web: Still on Trusty. Updated to Xenial
• iperf: Still on Trusty. Testing new image with Xenial

server3:

• mongo: Xenial (16.04.5)
• iperf: Running Xenial (16.04.6).
• rocketchat: Xenial (16.04.6), but runit doesn't fully work yet Now on Bionic, runit works
• nginx-proxy: Still on Trusty Upgraded to Xenial via clonm fork
• owncloud: Upgraded but not yet deployed
• postgresql: Image updated but not deployed Deployed and documented

I was getting

          ID: states                                                                                                                                                                              
    Function: no.None                                                                                                                                                                             
      Result: False                                                                                                                                                                               
     Comment: No Top file or external nodes data matches found.                                                                                                                                   
     Changes:  

and I added top_file_merging_strategy: merge_all to /config/master to fix this.

I was also getting Failed to create path "/var/log/salt/ssh" - [Errno 13] Permission denied: '/var/log/salt', which stopped happening when I added priv: /path/to/private/key to /config/roster.