Seeing Restmail timeouts after 60 seconds, in mozillians-tests builds about restmail.net

[jrgm]

I assume https://qa-master.fxtest.jenkins.stage.mozaws.net/ is behind a firewall, since I can't reach it.

What were the FxA origins involved in these tests. latest.dev.lcip.org in late afternoon PDT be any chance? (That ec2 failed and I had to reboot it). Something else?

restmail.net used in other tests is working normally at the moment, and also earlier today.

[stephendonner]

1. Yeah, behind firewall: https://mana.mozilla.org/wiki/display/TestEngineering/qa-master.fxtest.jenkins.stage.mozaws.net

2. Not sure the FxA origin, but I know it's https://restmail.net, due to https://github.com/mozilla/mozillians-tests/blob/25ccf77c75cfb30f0120fcfff77b73293926a092/tests/conftest.py#L65 and https://github.com/mozilla/mozillians-tests/blob/25ccf77c75cfb30f0120fcfff77b73293926a092/tests/restmail.py

3. We also saw another one at around 4:55am PDT today:

E   Exception: Timeout after 60 seconds getting restmail for c8977c1d-0751-4988-934b-f2dac4422947. Expected 1 messages but there were 0.

[shane-tomlinson]

@stephendonner - we see this quite often ourselves. We suspect it's not a problem with restmail itself, rather messages sometimes take a while to get from SES (our sender) to restmail.

[shane-tomlinson]

We suspect it's not a problem with restmail itself, rather messages sometimes take a while to get from SES (our sender) to restmail.

You know, on my walk to the office I was thinking about this claim, which was based on now ancient info. I also see that you and @jrgm are working through some restmail limiting over in this bug.

[jrgm]

Anyways, there was some unwanted traffic that was probably triggering early evictions from redis. I've made some changes to address this, and there have been zero evictions in the past ~24 hours. I'm calling this one fixed, but thanks for pointing this out.

[stephendonner]

We're seeing this again: https://qa-master.fxtest.jenkins.stage.mozaws.net/job/mozillians.prod/487/testReport/junit/tests.test_profile/TestProfile/test_that_links_in_the_services_page_return_200_code/

01:45:27 _______ TestProfile.test_that_links_in_the_services_page_return_200_code _______
01:45:27 [gw6] linux2 -- Python 2.7.12 /usr/bin/python2.7
01:45:27 tests/test_profile.py:215: in test_that_links_in_the_services_page_return_200_code
01:45:27     home_page.login(vouched_user['email'])
01:45:27 pages/base.py:58: in login
01:45:27     login_link = conftest.login_link(email)
01:45:27 tests/conftest.py:66: in login_link
01:45:27     mail = restmail.get_mail(username)
01:45:27 tests/restmail.py:33: in get_mail
01:45:27     'ACTUAL_MESSAGE_COUNT': len(restmail)})
01:45:27 E   Exception: Timeout after 60 seconds getting restmail for 0f8018ac-cce4-4d0c-a1d2-4f2dc10863ea. Expected 1 messages but there were 0.
01:45:27 ------------------------------- pytest-selenium --------------------------------
01:45:27 URL: https://auth.mozilla.auth0.com/login?state=z5cnVtu0juWNWxwoM7wxUgRyTGYlKU7F&client=HdfEiM1SZibaQnOYTxLoMdxSh4a6ZKD3&protocol=oauth2&nonce=XuWMIJOqzFASoCPq0jQoWD71SNgYh2Cw&redirect_uri=https%3A%2F%2Fmozillians.org%2Foidc%2Fcallback%2F&response_type=code&scope=openid%20email%20profile
01:45:27 WARNING: Failed to gather log types: Message: GET /session/XXXXXXXX-XXXX-XXXX-XXXX-XXXX663bbef9/log/types did not match a known command
01:45:27 Build info: version: '3.4.0', revision: 'unknown', time: 'unknown'
01:45:27 System info: host: 'WIN-SB3ER6JQ6ME', ip: '172.20.57.163', os.name: 'Windows 10', os.arch: 'x86', os.version: '10.0', java.version: '1.8.0_91'
01:45:27 Driver info: org.openqa.selenium.firefox.FirefoxDriver
01:45:27 Capabilities [{moz:profile=C:\Users\Administrator\AppData\Local\Temp\rust_mozprofile.d00BF2tyYjOX, rotatable=false, timeouts={implicit=0.0, pageLoad=300000.0, script=30000.0}, pageLoadStrategy=normal, moz:headless=false, platform=ANY, moz:accessibilityChecks=false, moz:useNonSpecCompliantPointerOrigin=false, acceptInsecureCerts=false, browserVersion=61.0, platformVersion=10.0, moz:processID=3872.0, browserName=firefox, javascriptEnabled=true, platformName=windows_nt, moz:webdriverClick=true}]
01:45:27 Session ID: XXXXXXXX-XXXX-XXXX-XXXX-XXXX663bbef9
01:45:27 Stacktrace:
01:45:27     at sun.reflect.NativeConstructorAccessorImpl.newInstance0 (None:-2)
01:45:27     at sun.reflect.NativeConstructorAccessorImpl.newInstance (None:-1)
01:45:27     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance (None:-1)
01:45:27     at java.lang.reflect.Constructor.newInstance (None:-1)
01:45:27     at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException (W3CHttpResponseCodec.java:150)
01:45:27     at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode (W3CHttpResponseCodec.java:115)
01:45:27     at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode (W3CHttpResponseCodec.java:45)
01:45:27     at org.openqa.selenium.remote.HttpCommandExecutor.execute (HttpCommandExecutor.java:164)
01:45:27     at org.openqa.selenium.remote.service.DriverCommandExecutor.execute (DriverCommandExecutor.java:82)
01:45:27     at org.openqa.selenium.remote.RemoteWebDriver.execute (RemoteWebDriver.java:637)
01:45:27     at org.openqa.selenium.remote.RemoteWebDriver.execute (RemoteWebDriver.java:694)
01:45:27     at org.openqa.selenium.remote.RemoteExecuteMethod.execute (RemoteExecuteMethod.java:33)
01:45:27     at org.openqa.selenium.remote.RemoteLogs.getAvailableLogTypes (RemoteLogs.java:103)
01:45:27     at org.openqa.selenium.remote.server.handler.GetAvailableLogTypesHandler.call (GetAvailableLogTypesHandler.java:37)
01:45:27     at org.openqa.selenium.remote.server.handler.GetAvailableLogTypesHandler.call (GetAvailableLogTypesHandler.java:28)
01:45:27     at java.util.concurrent.FutureTask.run (None:-1)
01:45:27     at org.openqa.selenium.remote.server.DefaultSession$1.run (DefaultSession.java:176)
01:45:27     at java.util.concurrent.ThreadPoolExecutor.runWorker (None:-1)
01:45:27     at java.util.concurrent.ThreadPoolExecutor$Worker.run (None:-1)
01:45:27     at java.lang.Thread.run (None:-1)
01:45:27 Sauce Labs Job: https://saucelabs.com/jobs/ddebd3259c64442ba4b77a0278f29f2d

@jrgm mind reopening? It's happening often enough, after seemingly not happening at all, for a bit. Thanks!

[jrgm]

You may be racing yourself. I see a delete and a delivery within the same second, and then >100 attempts to fetch that address. If the delete was after the delivery, you deleted your own email.