AMO prod updates are applied with a delay about addons

[vcarciu]

This is not reproducible on Stage

[kumar303]

For context, this is how the code is expected to work:

• user performs an action that causes a DB write (such as submitting a review)
• they get a cookie that pins them to the master MySQL DB for 15 seconds
• each subsequent request should get data from the master DB instead of a slave DB
• this prevents them from seeing any noticeable lag

Some theories:

• maybe 15 seconds is too short on AWS and we need to increase it. We're still trying to find evidence of laggy slaves or not.
• maybe django-cache-machine is involved? That deals with selecting data from memcache instead of the db. Not sure if it's related though.

[muffinresearch]

This is causing problems with login related to new users see #1618 - ops are investigating.

[kumar303]

Update: @jasonthomas increased the size of redis and the cache lag seems to have gone away. That is, I can no longer reproduce the add/delete/refresh test on the review page. This suggests that the key flush lists which were stored in redis were getting evicted before the associated memcache keys were evicted, resulting in memcache keys that never got flushed. Redis is at version 2.8.23 and is currently configured for volatile-lru. Jason is monitoring it and we'll have to see if the problems come back once the larger redis instance hits its max memory.

[kumar303]

Thanks for verifying. I am skeptical that this is actually fixed :) But we'll see what happens when the larger redis instance hits its memory max and has to start evicting the cache. We can always re-open though.