low-frequency error on module object with `use_dill` about ipyparallel

[minrk]

I can't see where this would be coming from. There isn't any .__main__ attribute-access that I know of. The only .__main__ I see in the code is in process startup with ipcluster, where the code:

python -c 'from ipyparallel.engine.__main__ import main; main()'

is how engines are started (only on Windows with ipcluster). Is this seen only on Windows? If not, I'm perplexed, especially at the absence of a traceback. If so, I'm not sure what IPython would be doing wrong, since the exec command shouldn't be relevant at any point once the process is running.

[jakirkham]

@minrk, for some context, I have seen this primarily with code that looks like this ( uqfoundation/dill#135 ) on Python 2 where I am using the workaround. Though I haven't yet seen this when using cloudpickle, which doesn't suffer from that bug. So, am curious if maybe one causes the other. This was on a CentOS 6.x (mixture of 6.3 and 6.6) cluster.

[mmckerns]

Serializers reach out and touch __main__ primarily through the global dict which is required for serializing any function especially in a closure (as referenced above). dill has several variants that handle this slightly differently, and cloudpickle handles it with yet another variant (very similarly to dill.settings['recurse'] = True).

I don't have enough info from the reporter to reproduce the issue, and I'm hoping that maybe some of that could be uncovered here. Then we'd see if it's something that needs patching at the dill level or if it's something that needs patching within ipyparallel.

[minrk]

Touching __main__ via sys.modules['__main__'] makes sense. What's weird to me is that it's an AttributeError, and I don't see a way for IPython to trigger access to main via attr access.

[mmckerns]

I've had some feedback from @MaximilianR, that celery also (infrequently?) sees this issue when using celery (as opposed to ipyparallel) in a notebook. So, it still could be something with how dill in interacting with "the artist formerly known as ipython".

Apparently, the workaround is to use dill.settings['recurse']=True, which is very similar to the cloudpickle way of interacting with the objects in globals. I'll see if I can get some updates on the open dill ticket, and report back here.

[mmckerns]

I don't have a minimal example yet, for the "low-frequency" case… but do have something interesting to report. If dill.dumps is used withpickle.loads (not dill.loads), then this error is seen. Could it be the wrong Unpickler is being used (possibly infrequently)?

def n(a):
    return a*2

import dill
p = dill.dumps(n)
dill.loads(p)

import pickle
pickle.loads(p)

Is such a mismatch possible in ipyparallel (or in the larger scope, in a notebook)?

[jakirkham]

Interesting on Python 3 one will get a different error message.

ImportError: No module named '__builtin__'

[minrk]

@mmckerns it shouldn't be if client.use_dill() is working correctly, but that's a good data point, because that is what would happen if it is not doing what it's supposed to. I will look for some direct calls to pickle that may be missed.

[mmckerns]

(I randomly saw this...) related? https://stackoverflow.com/questions/35913975/pickle-load-works-in-ipython-and-breaks-in-jupyter-notebook (at least to the issue with celery) -- I expect not, but anyway just another datapoint