destructors don't appear to be called: leaking memory & file descriptors about ccs-pykerberos

[jmehnle]

@dreness, any chance you could make that PYPI release soon? I'm really hoping requests-kerberos can switch from the old pykerberos to ccs-pykerberos once this release is out.

[iynehz]

@dreness Could you release this to PyPI? Latest on PyPI is still 1.3.0

[behackett]

I took a look at the necessary changes. authenticate_gss_client_init and authenticate_gss_server_init both set the gss_*_state members to reasonable default values before doing anything else. The cleanest way to fix this is to change O to N, and call Py_DECREF on pystate if either of the init functions fails. The destructor will do the cleanup. It probably also makes sense to check if PyCObject_FromVoidPtr fails (pystate will be NULL) and free state if it does, before returning NULL.

[behackett]

@dreness - ping! It would be great to get this fix into master and released.

[rgbyrnes]

I think the cause is that authGSSServerInit and authGSSClientInit call Py_BuildValue with "O", and that increments the refcount of pystate (from 1 to 2), so it will never be reclaimed, even when it goes out of scope in the caller, because the refcount will drop back to 1 (not 0). This could be fixed by using "N" instead of "O".

There seems to be another leak in each function if (result == AUTH_GSS_ERROR) ... the functions should call Py_DECREF(pystate) before returning early. Or (perhaps better) construct pystate later, when Py_BuildValue is called.

[behackett]

Nice catch. It looks like that leak has been in ccs-pykerberos for years. It was caught and fixed in one of the forks a while ago: 02strich/pykerberos@088d453

[dreness]

Good detective work, and thanks for the repro cases @mkomitee! I'll get these going in the existing test harness, so I should be able to check in both the fix for the FD leak and the proof that the fix works...

[behackett]

@mkomitee can you try the patch in #71?

[mkomitee]

Going to test it shortly, but if you want to test the returns from PyCObject_FromVoidPtr like that, you may want to do it in channelBindings as well.

Not sure whether you'd want to do so in the MOD_INIT when you're adding GSS_MECH_OID_{KRB5,SPEGNO} to the module dictionary.

[rgbyrnes]

Also, at the end of channelBindings, I think ...

return Py_BuildValue("N", pychan_bindings);

... could be replaced by ...

return pychan_bindings;

[behackett]

There is a lot of error checking that could be added. I'm happy to add more as time allows, but I'm not a maintainer of this project, just a user.

[dreness]

@behackett thanks for your efforts :) I'm currently messing around with running the scripts @mkomitee provided as part of the automated tests, which should lend additional confidence to the fix.

[dreness]

Ok check it out, I think the fixes look good for 3.x, although for 2.x the client-side test is still a little leaky? There's one travis CI run that includes just the additional tests (which are not fully hooked up in the sense that they don't assert anything, they just print output when they run), and then another one that includes the tests plus the fixes.

Travis CI runs of just the additional tests:
https://travis-ci.org/dreness/ccs-pykerberos/builds/364874847

Travis CI runs of the additional tests + fixes:
https://travis-ci.org/dreness/ccs-pykerberos/builds/364893362

Sample output from a 3.3.6 build with the fixes:

tests/test_kerberos.py ....fd 0 => pipe:[19143]
fd 1 => pipe:[19144]
fd 2 => pipe:[19145]
.Leaked 0 total in 1000 calls: ~0.0 bytes per call
Leaked 0 total in 10000 calls: ~0.0 bytes per call
Leaked 0 total in 100000 calls: ~0.0 bytes per call
Leaked 0 total in 1000000 calls: ~0.0 bytes per call
.
=========================== 6 passed in 1.12 seconds ===========================

The output from 3.3.6 without the fix is much longer, so I'll just link to it.

2.7.13 with the fix looks like this:

tests/test_kerberos.py ....fd 0 => pipe:[19231]
fd 1 => pipe:[19232]
fd 2 => pipe:[19233]
fd 10 => /dev/urandom
.Leaked 0 total in 1000 calls: ~0 bytes per call
Leaked 0 total in 10000 calls: ~0 bytes per call
Leaked 1335296 total in 100000 calls: ~13 bytes per call
Leaked 21430272 total in 1000000 calls: ~21 bytes per call
.

I'd say the results look pretty good for python3. For python2, although the number of discrete FDs we generate are reduced to the expected amount, and the leaked bytes have been reduced by a lot, there are still some leaked bytes. The leaked bytes output for 2.7.13 before the fix is:

.Leaked 0 total in 1000 calls: ~0 bytes per call
Leaked 1892352 total in 10000 calls: ~189 bytes per call
Leaked 33533952 total in 100000 calls: ~335 bytes per call
Leaked 334688256 total in 1000000 calls: ~334 bytes per call

Once I have a solid understanding of the expected output for the passing case, I'll add the appropriate asserts to the tests.

[behackett]

Change range(count) to xrange(count) in the test for Python 2. That solves the remaining problem for me.

[mkomitee]

The patch looks good to me.

I suspect the leak in 2.7 with the fix has more to do with python 2.7's range() than it has to do with kerberos:

import psutil
count = 10000
for mechanism in (range, xrange):
    before = psutil.Process().memory_info().rss
    for _ in mechanism(count):
        pass
    after = psutil.Process().memory_info().rss
    print("{} leaked {} bytes per iteration".format(
        mechanism, (after - before) /  count))

whern run prints out:

<built-in function range> leaked 27 bytes per iteration
<type 'xrange'> leaked 0 bytes per iteration

Also, @behackett I didn't realize you weren't a maintainer. Thanks for your help.

[dreness]

Cool, thanks for the tip about range / xrange. I'll make that change, add some asserts to the tests, merge and do a pypi release... soon - hopefully tomorrow.

On the topic of maintenance, I do have the authority of a maintainer, although I don't really do much maintenance on account of not really having a lot of time to spend on this project these days. However, I do generally believe in the GitHub dream and am willing to push buttons here and there to leverage the contributions of helpful and interested users.

[omegagx]

when will this be fixed in an official release?

[dreness]

I've updated my fork of @behackett 's PR to finish up the test cases contributed by @mkomitee by adding asserts to check for memory or FD leaks. With the new asserts in place, if I revert the fix, the tests fail as expected:

tests/test_kerberos.py::test_service_principal PASSED                    [ 16%]
tests/test_kerberos.py::test_basic_check_password PASSED                 [ 33%]
tests/test_kerberos.py::test_gssapi PASSED                               [ 50%]
tests/test_kerberos.py::test_http_endpoint PASSED                        [ 66%]
tests/test_kerberos.py::test_leaks_server_linux FAILED                   [ 83%]
tests/test_kerberos.py::test_leaks_client FAILED                         [100%]
... (elided output) ...
====================== 2 failed, 4 passed in 0.21 seconds ======================

... and with the fix in place, all the tests pass:

tests/test_kerberos.py::test_service_principal PASSED                    [ 16%]
tests/test_kerberos.py::test_basic_check_password PASSED                 [ 33%]
tests/test_kerberos.py::test_gssapi PASSED                               [ 50%]
tests/test_kerberos.py::test_http_endpoint PASSED                        [ 66%]
tests/test_kerberos.py::test_leaks_server_linux PASSED                   [ 83%]
tests/test_kerberos.py::test_leaks_client PASSED                         [100%]
=========================== 6 passed in 1.08 seconds ===========================

I think this is ready to go, although I wouldn't complain if anyone wanted to glance at my fork as a final sanity check before release.

[belyaev-pa]

guys this fix breaks authGSSServerStoreDelegate() this always return NULL after any forks or patch injected