Issues running unit tests on rhel6 and Mac about tokenserver

[jbonacci]

(from @rfk )

I see a similar hang when trying to run the tests from tokenserver
master, and it appears to be caused by gevent monkey-patching. I don't
know exactly how or why. But doing this makes all the tests pass
reliably without hanging for me:

 ./bin/pip uninstall gevent

Reinstalling gevent brings back the hang.

Both tokenserver and circus do an unconditional monkeypatch if gevent is
available, along the lines of:

 try:
     from gevent import monkey
     from gevent_zeromq import monkey_patch
     monkey.patch_all()
     monkey_patch()
 except ImportError:
     pass

I think we should consider doing it only if something else (e.g. the
gunicorn worker code) has already initiated a monkeypath, something like:

 if "gevent.monkey" in sys.modules:
     try:
         from gevent import monkey
         from gevent_zeromq import monkey_patch
         monkey.patch_all()
         monkey_patch()
     except ImportError:
         pass

Cheers,

 Ryan

[jbonacci]

(from @tarekziade )

On 7/17/12 9:22 AM, Ryan Kelly wrote:

I see a similar hang when trying to run the tests from tokenserver
master, and it appears to be caused by gevent monkey-patching. I
don't know exactly how or why. But doing this makes all the tests
pass reliably without hanging for me:

./bin/pip uninstall gevent

Reinstalling gevent brings back the hang.

the hang is generally happening because pyzmq uses sockets in a way
that's not compatible with the gevent sockets.

gevent_zmq solves this by making pyzmq green. The usual issue is that
the zmq patching occurs after a pyzmq context was created.

I suspect a bad nose import order, will try to reproduce.

Both tokenserver and circus do an unconditional monkeypatch if gevent
is available, along the lines of:

try:
    from gevent import monkey
    from gevent_zeromq import monkey_patch
    monkey.patch_all()
    monkey_patch()
except ImportError:
    pass

I think we should consider doing it only if something else (e.g. the
gunicorn worker code) has already initiated a monkeypath, something like:

I don't understand why you want to patch again only if gevent was
already patched.

The thing is, if gevent is installed, we don't know how the code will
use it and potentially lock everything, so we do need to monkey patch

Will have a look, cheers

[jbonacci]

(from @rfk )

Hmmm, right, I guess when you put it like that it doesn't make much
sense :-)

What I'm trying to suggest is that we somehow detect that gevent
monkeypatching is already in place, and if it is, then do the necessary
extra gevent_zeromq monkeypatching as well:

 if GEVENT_IS_ALREADY_MONKEYPATCHED:
      from gevent_zeromq import monkey_patch
      monkey_patch()

Ideally this code would live inside pyzmq itself, so that pyzmq Just
Works when gevent monkey-patching is in place.

The thing is, if gevent is installed, we don't know how the code will
use it and potentially lock everything, so we do need to monkey patch

We don't know whether the code will use it.

If the calling code uses gevent and we don't monkey-patch then we'll get
hangs. But conversely, if the calling code doesn't use gevent and we do
monkey-patch then we can get hangs like seen here.

So I feel like there's something a little more nuanced that we can do to
work well in both situations. The "if 'gevent.monkey' in sys.modules"
thing is just one suggestion.

[jbonacci]

(from @tarekziade )

We don't know whether the code will use it.

If the calling code uses gevent and we don't monkey-patch then we'll
get hangs. But conversely, if the calling code doesn't use gevent and
we do monkey-patch then we can get hangs like seen here.

That's the part I don't agree with => if the calling code doesn't use
gevent and we do monkey-patch then we can get hangs like seen here.

I think it's rather => If gevent and pyzmq are both present, make sure
both are patched before any one of them is ever used.

So I feel like there's something a little more nuanced that we can do
to work well in both situations. The "if 'gevent.monkey' in
sys.modules" thing is just one suggestion.

Unfortunately that does not really prove gevent monkey patch was or was
not called -- just imported . Even if it's called, we don't know if the
socket module was patched or not.

Cheers

[jbonacci]

Errors and traceback reproduced on all of the clientX boxes this morning.
So at least the errors I see are consistent.

[jbonacci]

(per tarek)
I moved lib/python2.6/site-packages/gevent_z* in the tokenserver directory (after doing make build)

$ git clone git://github.com/mozilla-services/tokenserver
$ cd tokenserver
$ make build
$ mv lib/python2.6/site-packages/gevent_zeromq-0.2.2-py2.6.egg lib/python2.6/site-packages/gevent_zeromq-0.2.2-py2.6.egg-SAVE
(or similar)
$ make test

All functional tests passed.

So, we have a workaround.

[jbonacci]

OK, so on Mac install:

Workaround:
$ mv lib/python2.7/site-packages/gevent_zeromq-0.2.2-py2.7-macosx-10.7-intel.egg lib/python2.7/site-packages/gevent_zeromq-0.2.2-py2.7-macosx-10.7-intel.egg-SAVE

$ ls -al lib/python2.7/site-packages/gevent*

lib/python2.7/site-packages/gevent-0.13.7-py2.7-macosx-10.7-intel.egg:
lib/python2.7/site-packages/gevent_zeromq-0.2.2-py2.7-macosx-10.7-intel.egg-SAVE:

Still fails, but does not hang: http://jbonacci.pastebin.mozilla.org/1713799

[tarekziade]

@jbonacci The "AttributeError: Socket has no such option: HWM" might mean that you have the wrong version of zmq installed on your system, thus the failures

[tarekziade]

ok please try again on a fresh master

$ make build test

[jbonacci]

This is the latest from qa2 (rhel6) and client4 (rhel6):
FAIL: test_ttled_dict (tokenserver.tests.test_crypto_pyworker.TestTTledDict)

Traceback (most recent call last):
File "/opt/tokenserver/tokenserver/tokenserver/tests/test_crypto_pyworker.py", line 128, in test_ttled_dict
self.assertFalse('foo' in cache)
AssertionError: True is not False

---

Ran 25 tests in 17.194s

FAILED (failures=1)
make: *** [test] Error 1

[jbonacci]

On Mac, still getting this issue: http://jbonacci.pastebin.mozilla.org/1715031

(not really sure full functionality on Mac is that important)

[tarekziade]

@jbonacci you did not fix your zmq / pyzmq mismatch : AttributeError: Socket has no such option: HWM

[tarekziade]

it's working for me on OS X

[jbonacci]

OK, once I "downgraded" libzmq from 3.3 to 2.2, the functional tests started working - no errors on Mac.
Ran 25 tests in 14.245s
OK

So, on Linux, I still see the one error above:
#12 (comment)

[jbonacci]

@tarekziade if you are ok with the one error on Linux, we can close this issue.
Otherwise, let me know what other tests, steps, information you need from me.

[rfk]

@jbonacci it would be great if you can pick this up and see what the current status is on various platforms. Updated versions of our dependencies may have made the situation better, or worse :-)

[jbonacci]

@rfk
Wow, a blast from the past!
Sure, if you can tell me the correct values for this:
$ make build CHANNEL=prod TOKENSERVER=rpm-0.8.2

[rfk]

Let's forget about channels for now, and just see where we're at with a vanilla make build.

[jbonacci]

Ran into some external dependency issues on Mac - will assume same for Linux.
Checking other OS just to see what errors come up...

[jbonacci]

Not sure if it will help you, but here are some "make build" failure snapshots:
Mac: https://jbonacci.pastebin.mozilla.org/3954804
Ubuntu 13: https://jbonacci.pastebin.mozilla.org/3955504
CentOS: https://jbonacci.pastebin.mozilla.org/3955803
RHEL AWS: https://jbonacci.pastebin.mozilla.org/3955165
Ubuntu AWS: https://jbonacci.pastebin.mozilla.org/3955009

[rfk]

@jbonacci the deps have been updated, you should have more luck with the builds now.

[jbonacci]

OK. As you know, I got Mac to build.
Still tinkering with AWS (rhel and ubuntu) - issues have come up around the following
memcached
libmemcached
libevent-dev

[jbonacci]

OK. I got this to "make build" twice (two different Macs).
I had to pre-install libmemcached (but not memcached directly) and easy_install.
...etc...
Building the app

Checking the environ                                                             [ok]
Updating the repo                                                                  [ok]
Building Services dependencies                                            [ok]
Building External dependencies                                             [ok]
Now building the app itself                                                     [ok]
                                                                                               [done]

[jbonacci]

OK, so I can run unit test on the following:
Mac
AWS ubuntu
AWS rhel

There are errors of various kinds, but I can open separate issues for each platform if you want...

PS still working on SciLinux...