Performance degrade in unpickling fairly large objects about pfio

[belldandyxtq]

Thank you for reporting and finding the root cause.

If we remove the wrapper, what is your plan on supporting pickle on HDFS?

There are a few cases we need to replace the original file object to support some functionalities. For example, reading as text from HDFS and zip

And the internal zip for Python < 3.7

I am thinking about making the functionality of __init__ of file object, which is currently used to determine the proper file object type for opening, to a separate function to be called by open decorator.

In this way, we can for now get rid of the following I/O calls going to file object wrapper while still having the ability to solve the issues like text read on HDFS and internal zip

[belldandyxtq]

To make it more clear, I am thinking about creating a file_object_maker as a replacement of __init__ in the current file_object.py, which is used to cover those cases where some functionalities are not supported by the original file objects like the text read cases in HDFS and zip.

The currently implementation of open_wrapper returns the FileObject defined in fileobject.pyor its derived classes. And the file object replacement takes places in such __init__ of the FileObject, hence the replacement is bound to the file object.

If we extract the __init__ from the FileObject and make it a function (e.g. file_object_maker), we can still have the ability to control which kind of file object to return to user while not returning a always-wrappered FileObject to workaround the issues we have now.

[kuenishi]

Wrapping where we need it is fine, but FileObject in fileobject.py is unnecessary for now especially for "posix" filesystem and zip container. This is maybe because aligning other filesystems and file objects' behaviour to "posix" is the best way to prevent potential performance issues like this, just until profiler.

[belldandyxtq]

I think we need to wrap zip container for text reading and internal zip.
The only two don't need wrapper are POSIX and HTTP.

And how about pickle on HDFS? an simple plan will be extracting the content and putting into io.bufferedreader in the file_object_maker

[kuenishi]

This issue for POSIX filesystems is addressed by #38 .