Bootstrapped exe error: "ValueError: bad marshal data" about esky

[rfk]

It looks like python3.4 made some changes to how .pyc files are formatted:

https://docs.python.org/3/whatsnew/3.4.html#marshal

So there's probably some new incompatibility with the way esky is building them.

The code in question is e.g. this: https://github.com/cloudmatrix/esky/blob/master/esky/bdist_esky/f_cxfreeze.py#L133

It's trying to generate the contents of a .pyc from some python source code, so that it can be stuffed into a zipfile.

If you're feeling adventurous, you could try re-implementing this logic using https://docs.python.org/3/library/py_compile.html and see if that makes a difference. It could probably do with a refactoring into a utility function anyway.

I'm a bit confused as to what eskycode and eskybscode are supposed to have in them. I see on line 145 that maincode will have the compiled code_source in it, but with the other two the compile function is getting passed an empty string.

[rfk]

This is intentional, they're supposed to be empty modules (like putting an empty __init__.py to mark a directory as a package) so that the import environment for the bootstrapping code looks like it has esky installed.

I believe I've found a solution (at least for python 3.4.1). This is my first time contributing to an online project, so what are the next steps I should take? Pull request?

[rfk]

Pull request?

Yes, thanks, that would be great.

After looking at doing a pull request, I realized I don't know what the standard is when importing something that is only available in certain python versions.

Here is what I have so far, it works for the versions of everything I have installed at the moment, but I haven't tested it otherwise.

        #  Since Python 3.3 the .pyc file format contains the source size.
        #  It's not used for anything at all except to check if the file is up to date.
        #  We can set this value to zero to make Esky also work for Python 3.3
        if sys.version_info[:2] < (3, 1):
            maincode = imp.get_magic() + struct.pack("<i",0)
            eskycode = imp.get_magic() + struct.pack("<i",0)
            eskybscode = imp.get_magic() + struct.pack("<i",0)

            maincode += marshal.dumps(compile(code_source,INITNAME+".py","exec"))    
            eskycode += marshal.dumps(compile("","esky/__init__.py","exec"))
            eskybscode += marshal.dumps(compile("","esky/bootstrap.py","exec"))
        else:
            import importlib._bootstrap
            loader = importlib._bootstrap.SourceLoader()

            code = loader.source_to_code(code_source, '<string>')
            empty_code = loader.source_to_code(b"", '<string>')
            empty_bytecode = importlib._bootstrap._code_to_bytecode(empty_code)

            maincode = importlib._bootstrap._code_to_bytecode(code)
            eskycode = empty_bytecode
            eskybscode = empty_bytecode

[rfk]

Great! Using stuff from importlib seems much nicer than trying to keep up-to-date with the evolving bytecode format.

It's OK to start a pull-request even if the code is incomplete, since you can always add more commits to the branch to update it after further discussion.

I'd go for three-way branching logic here, which is pretty ugly but safer than anything else I can think of:

if sys.version_info[:2] < (3, 1):
    ...existing code...
elif sys.version_info[:2] < (3, 4):
    ...existing code...
else:
    ...the new code for 3.4 and later...

The docs say that loader.source_to_code is new in 3.4 so we may have to keep both existing branches of this code, as well as the new one.

Ideally we can refactor this into a helper function in esky.util, e.g. something like compile_to_bytecode where you just give it the source and it returns the bytes for the .pyc file.

[mgrandi]

I'm using the latest version of esky available at the moment, and it looks like this was included (through pull request or whatever), however the version on pypi is outdated and this bug is still present in that build, and now i'm having to specify the git commit here on github in my requirements.txt rather then just 'esky', is there any chance you can update the build on pypi?

[flutefreak7]

Doen't having it on pypi also help Python 3 users since 2to3 has to be run on the new code before it will work? (This is based on a fuzzy memory of having to do that a few months ago - I apologize if this is incorrect.)

[mgrandi]

yes, esky is on pypi and it uses 2to3 to produce a python 3 version, but that just runs the same setup.py that is present in this github repo, but i like pypi more cause its just one less thing to keep track of (what git commit do i need to make this work, etc, is the current commit broken, etc)

but currently the version on pypi doesn't have this fix so anyone on python3.4+ who downloads it from pip will get a broken version and be sad =(

[flutefreak7]

Yeah I remember that sadness... took me a while to get all my stuff working
on 3.4, and esky wasn't a very smooth transition. Also had to switch to
cxfreeze from py2exe, but I don't think that was esky's fault.

[mgrandi]

Hilariously, i also had problems with cx_freeze, because apparently python internal stuff changed with python3.4 and it builds fine, but running the generated exe gives some import error. Turns out if you just 'recompile' cx_freeze then it works again, but they don't seem to think that updating the version on pypi is worth it: https://bitbucket.org/anthony_tuininga/cx_freeze/issue/81/python-34-venv-importlib-attributeerror

why do i have to maintain so many custom copies of all these libraries =(

[mvbentes]

Just to note thas as of 07/30/2015 the issue remais on the pypi version of esky, but it seems corrected on the master branch.
Installing from the repo fixed it for me. Thank you all for the great work and valuable contributions!

Environment Python 3.4.1 x64, cx_freeze 4.3.4, win7

[timeyyy]

Pypi has been updated