restmail.net is down about restmail.net

[rfk]

@jbuck mentioned this in a meeting earlier so reflecting it here - SMTP is still down on the box, and we may need to do a full re-deploy in order for him to get the access to fix it. Re-opening.

[stephendonner]

Thanks, @jbuck - the site is up, and account creation works, but the verification-email functionality is either missing or broken[0]:

RuntimeError: Verification email was not received

Stacktrace

tests/test_fxa_test_account.py:19: in test_default_environment_should_be_dev
    account = FxATestAccount()
fxapom/fxapom.py:59: in __init__
    raise RuntimeError("Verification email was not received")
E   RuntimeError: Verification email was not received

Standard Output

fxapom created an account for email: [email protected] at https://stable.dev.lcip.org/auth/ on 2018-04-03 20:09:10.971632

[0] https://github.com/mozilla/fxapom/blob/6d346e342d93efa68a5b91b1e1dc7072b31ecd38/tests/test_fxa_test_account.py#L18

[jbuck]

Alright, it's working now. I don't understand why it stopped working, and I couldn't access the original box so here's what I did to get things working again:

1. Create AMI of restmail-20170515 in us-east-1
2. Launch said AMI as restmail-20180404
3. Notice that port 25 is only listening on localhost
4. Change instance security group restmail-20170515-InstanceSecurityGroup-1NJ7YRNO90I0A to allow 9025 inbound
5. Change ELB listener on TCP port 25 to forward to port 9025

[karlht]

@jbuck Have we verified SMTP dlivery on the new stack? I'm hearing rumblings that mail delivery is still not working.

[jbuck]

We fixed it!

[jrbenny35]

Getting that same error as well.

[stephendonner]

Thanks, everyone!

[viorelaioia-zz]

Hi! Do you have any updates on this? Most of mozillians tests are failing because we use restmail for passwordless login.

[jbuck]

Yeah, I confirmed that delivery is working but we're seeing delays in delivery for reasons I'm not sure of yet

[m8ttyB]

Thanks @jbuck, we're seeing an error similar to @stephendonner's comment that is consistent across our test runs.

[m8ttyB]

\o/ our test suites are happily consuming restmail goodness. ty for the quick remedy.

[stephendonner]

Already mentioned this in IRC, but for posterity, all of our affected testsuites are happy, now (Mozillians, Kinto, FxAPOM, etc.)

[jbuck]

https://github.com/mozilla/restmail.net/pull/25/files has the fix

[vladikoff]

I think we crashed again

➜  notes git:(master) date
Fri  6 Apr 2018 10:02:09 EDT
➜  notes git:(master) wget restmail.net
--2018-04-06 10:02:17--  http://restmail.net/
Resolving restmail.net... 52.73.149.191, 52.44.243.160
Connecting to restmail.net|52.73.149.191|:80... connected.
HTTP request sent, awaiting response... 503 Service Unavailable: Back-end server is at capacity
2018-04-06 10:02:17 ERROR 503: Service Unavailable: Back-end server is at capacity.

[jbuck]

Indeed we did! Redis was unable to save data to disk and eventually was killed by the Linux OOM killer. This caused the restmail web and email processes to shutdown

[vladikoff]

Went down again:

➜  Desktop wget restmail.net        
--2018-04-08 10:37:10--  http://restmail.net/
Resolving restmail.net... 52.73.149.191, 52.44.243.160
Connecting to restmail.net|52.73.149.191|:80... connected.
HTTP request sent, awaiting response... 503 Service Unavailable: Back-end server is at capacity
2018-04-08 10:37:10 ERROR 503: Service Unavailable: Back-end server is at capacity.

➜  Desktop 

date
Sun  8 Apr 2018 10:37:34 EDT

[jbuck]

Bleh, working again now

[vladikoff]

Thank you @jbuck !