Comments (13)
Also happends when sending large buffers
In [9]: def write(data):
...: print(data)
...:
In [10]: r = d.apply_async(write,'a'*1024000)
In [11]: r
Out[11]: <AsyncResult: finished>
In [12]: r.get()
In [13]: r.display_outputs()
<memory at 0x0303E260
from ipyparallel.
@frmdstryr what version of IPython and Python are you using, and what platform? I'm getting the expected bytes back.
from ipyparallel.
Python 2.7.10, ipython 4.0 (just installed with pip today)
from ipyparallel.
Windows 8.1
from ipyparallel.
I have a similar problem on Windows 7 although it is happening on strings which are not all that big (1421 bytes in fact)
In [8]: print sys.version
2.7.3 (default, Apr 10 2012, 23:31:26) [MSC v.1500 32 bit (Intel)]
In [9]: print ipyparallel.version
4.0.2
In [12]: print zmq.version
14.7.0
from ipyparallel.
Thanks, appears to be a Python 2 issue. I was testing on Python 3. Fix incoming.
from ipyparallel.
I tried upping the session limit directly in the client code:
c.session.buffer_threshold = 16 * 1024 * 1042
However, my string (1421 characters long) is getting pickled into something fancy
(Pdb) p arg_bufs[0]
"\x80\x02cipykernel.pickleutil\nCannedBytes\nq\x01)\x81q\x02}q\x03U\x07buffersq\x04]q\x05T\x8d\x05\x00\x00# [-]
I tried to customize the pickling of strings
class CannedString(CannedObject):
wrap = str
ipykernel.pickleutil.can_map[str] = CannedString
and the pickler did indeed get used:
["\x80\x02c__main__\nCannedString\nq\x01)\x81q\x02}q\x03(U\x04keysq\x04]q\x05U\x04hookq\x06NU\x03objq\x07T\x8d\x05\x00\x00#
but the remote end was unable to deal with it.
Do you have any recommendations on how to disable the fancy stuff?
from ipyparallel.
When I trimmed down my file-size to 875 bytes, the s/w works just fine.
I notice that jupyter_client.session.MAX_BYTES is set to 1024 (and that this seems to be used within the pack_apply_message function) and although I tried to increase it (monkey patching), it does not seem to work.
from ipyparallel.
I suspect it's a buffer/memoryview confusion on Python 2.
from ipyparallel.
Should be fixed by ipython/ipykernel#44
from ipyparallel.
If you want a mitigation while you wait for that patch to get a release, you can apply this patch at runtime:
def patch_issue_30():
""""CannedBytes.wrap should turn buffers into bytes"""
from ipykernel.pickleutil import CannedBytes
from ipython_genutils.py3compat import buffer_to_bytes
CannedBytes.wrap = staticmethod(buffer_to_bytes)
patch_issue_30() # patch locally
rc[:].apply_sync(patch_issue_30) # patch on all engines
from ipyparallel.
Confirmed working. awesome! Thanks very much!
from ipyparallel.
Works now, thanks!
from ipyparallel.
Related Issues (20)
- No module named 'jupyter_server' HOT 2
- How to make it work with torch DDP HOT 4
- Transition from `CompositeError` to builtin `ExceptionGroup` HOT 1
- ipcluster nbextension enable not working after notebook upgrade HOT 2
- Print in multiprocessing.Process crashing the engine HOT 7
- Windows ssh support by ipcluster HOT 39
- map_sync with pandas operation function does not finish. HOT 1
- Py3.10 code serialization does not work on PyPy3.10
- sync_imports not working as intended HOT 9
- ipyparallel and pymoo doesn't work HOT 2
- AsyncResult.join doesn't work
- AsyncResult.abort() call hangs if not all jobs can be stopped HOT 1
- Question: engines and databases HOT 1
- BroadcastView map Not Implemented HOT 3
- Cannot run ipythonparallel with openmpi HOT 7
- 60s timeout on get_connection_info() is not configurable HOT 1
- please release/tag/pypi the current version as it supports JupyterLab 4.x HOT 2
- SSHEngineLauncher does not work as expected HOT 2
- Outstanding task on client but hub says completed when using broadcast view
- Entrypoints should be phased out HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ipyparallel.