amyreese / fissix

backport of lib2to3, with enhancements

fissix

Backport of latest lib2to3, with enhancements.

Installation

pip3 install fissix or pip install fissix, depending on the operating system's Python policy. On Debian ≥11 and derivatives such as Ubuntu 21.04, sudo apt install python3-fissix.

License

fissix is licensed under the Python Software Foundation License Version 2. I am providing code in this repository to you under an open source license. Because this is my personal repository, the license you receive to my code is from me and not from my employer. See the LICENSE file for details.

Used by

• https://pypi.org/project/bowler/ Safe code refactoring for modern Python projects
• https://pypi.org/project/modernize/ A hack on top of fissix for modernizing code for hybrid codebases.

fissix's Issues

catch StopIteration around 1-arg next in generator functions

Description

catch StopIteration around 1-arg next in generator functions

example from https://www.python.org/dev/peps/pep-0479/#converting-the-exception-inside-next

input:

def my_generator():
    ...
    yield next(it)
    ...

expected output:

def my_generator():
    try:
        ...
        yield next(it)
        ...
    except StopIteration
        return

Details

• OS: N/A
• Python version: 2.7 -> 3.8
• fissix version: master
• Can you repro on master? yes (feature request)
• Can you repro in a clean virtualenv? yes (feature request)

Plans on 3.9+ support?

Hi! Do you have any plans on supporting new Python versions?
Or since this is a backport - this is out of scope?

Feature request: helper(s) to avoid syntax errors

(Not sure if this belongs in bowler or fissix)

I wrote a thing to convert self.assertFalse(x) to assert not x, etc. Trouble is, it falls over easily when the expression in the parentheses isn't valid without the parentheses:

self.assertFalse(
    'a'
    'b'
)

My code naively converts this to

assert not 'a'
    'b'

• I think a parenthesize_if_necessary helper would be really useful for this kind of situation. I imagine this bug is widespread. I tried to write one, but I'm a bit out of my depth.
• It'd be useful to have access to a function which asserts a list of nodes has no syntax errors. I realise that would vary depending on targeted python version and __future__ imports, but perhaps something can be done?

Thanks

fissix.tests (unittest.loader._FailedTest) ... FAIL

Description

fissix.tests (unittest.loader._FailedTest) fail. This one stumped me and I had to consult a colleague. P.S. sorry it's taking so long to get Bowler into Debian!

python3.8 -m unittest discover -v 
fissix.tests (unittest.loader._FailedTest) ... FAIL

======================================================================
FAIL: fissix.tests (unittest.loader._FailedTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_fissix/build/fissix/tests/__init__.py", line 9, in load_tests
    return load_package_tests(os.path.dirname(__file__), *args)
  File "/usr/lib/python3.8/test/support/__init__.py", line 243, in load_package_tests
    package_tests = loader.discover(start_dir=pkg_dir,
AssertionError: Path must be within the project

----------------------------------------------------------------------
Ran 1 test in 0.000s

My colleague writes that this test fails due to a cause related to https://github.com/python/cpython/blob/master/Lib/test/support/__init__.py#L138-L140 and assumptions about a specific cpython layout. This sounds like an actionable bug with the following two resolution cases:

1. the system cpython loader should be executed.
2. fissix should included a forked version of the loader or somehow advise/hint the loader.

Details

• OS: Debian unstable
• Python version: 3.8
• fissix version: 20.5.1
• Can you repro on master? yes
• Can you repro in a clean virtualenv? NA

Fix PR #40 (except fixer: don't do anything if the "as" is already there) by checking against value of node

Description

My fix PR #40 didn't actually do anything. I must have dropped the .value during some refactoring.

Luckily this doesn't cause any regression.

Clarify changes and local patches from lib2to3 in readme.

Parse error on argument called "print"

Related to this lib2to3 bug: https://bugs.python.org/issue35383

When having a kwarg called print, parsing fails. Minimal example:

from fissix.pgen2.driver import Driver
from fissix import pytree
from fissix.pygram import python_grammar
import sys
d = Driver(grammar=python_grammar, convert=pytree.convert)
d.parse_string('print("a", file=sys.stderr)')

Results in:

ParseError                                Traceback (most recent call last)
<ipython-input-1-617e4f2e4fd8> in <module>
      4 import sys
      5 d = Driver(grammar=python_grammar, convert=pytree.convert)
----> 6 d.parse_string('print("a", file=sys.stderr)')

/nix/store/qh5vvjgvs792gb67k5aywndqrkgp3j2w-python3-3.7.2-env/lib/python3.7/site-packages/fissix/pgen2/driver.py in parse_string(self, text, debug)
    101         """Parse a string and return the syntax tree."""
    102         tokens = tokenize.generate_tokens(io.StringIO(text).readline)
--> 103         return self.parse_tokens(tokens, debug)
    104 
    105 

/nix/store/qh5vvjgvs792gb67k5aywndqrkgp3j2w-python3-3.7.2-env/lib/python3.7/site-packages/fissix/pgen2/driver.py in parse_tokens(self, tokens, debug)
     70                     "%s %r (prefix=%r)", token.tok_name[type], value, prefix
     71                 )
---> 72             if p.addtoken(type, value, (prefix, start)):
     73                 if debug:
     74                     self.logger.debug("Stop.")

/nix/store/qh5vvjgvs792gb67k5aywndqrkgp3j2w-python3-3.7.2-env/lib/python3.7/site-packages/fissix/pgen2/parse.py in addtoken(self, type, value, context)
    159                 else:
    160                     # No success finding a transition
--> 161                     raise ParseError("bad input", type, value, context)
    162 
    163     def classify(self, type, value, context):

ParseError: bad input: type=22, value='=', context=('', (1, 15))

upload docs to rtfd etc

Description

I'd like to link from https://modernize.readthedocs.io/en/latest/fixers.html#to3-fixers to fissix but there's no docs page on https://fissix.readthedocs.io/

StdoutRefactoringTool sets files attribute even when no files need to be modified

Description

Hi there! While trying to enforce 2/3 compatibility on a large codebase, I ran across an issue with modernize's --enforce flag:

$ python-modernize --future-unicode --enforce enqueue_function.py
 Loading the following fixers:
    fissix.fixes.fix_apply  (apply)
    fissix.fixes.fix_except  (except)
    fissix.fixes.fix_exec  (exec)
    fissix.fixes.fix_execfile  (execfile)
    fissix.fixes.fix_exitfunc  (exitfunc)
    fissix.fixes.fix_funcattrs  (funcattrs)
    fissix.fixes.fix_has_key  (has_key)
    fissix.fixes.fix_idioms  (idioms)
    fissix.fixes.fix_long  (long)
    fissix.fixes.fix_methodattrs  (methodattrs)
    fissix.fixes.fix_ne  (ne)
    fissix.fixes.fix_numliterals  (numliterals)
    fissix.fixes.fix_operator  (operator)
    fissix.fixes.fix_paren  (paren)
    fissix.fixes.fix_reduce  (reduce)
    fissix.fixes.fix_renames  (renames)
    fissix.fixes.fix_repr  (repr)
    fissix.fixes.fix_set_literal  (set_literal)
    fissix.fixes.fix_standarderror  (standarderror)
    fissix.fixes.fix_sys_exc  (sys_exc)
    fissix.fixes.fix_throw  (throw)
    fissix.fixes.fix_tuple_params  (tuple_params)
    fissix.fixes.fix_types  (types)
    fissix.fixes.fix_ws_comma  (ws_comma)
    fissix.fixes.fix_xreadlines  (xreadlines)
    libmodernize.fixes.fix_basestring  (basestring)
    libmodernize.fixes.fix_dict_six  (dict_six)
    libmodernize.fixes.fix_file  (file)
    libmodernize.fixes.fix_filter  (filter)
    libmodernize.fixes.fix_import  (import)
    libmodernize.fixes.fix_imports_six  (imports_six)
    libmodernize.fixes.fix_input_six  (input_six)
    libmodernize.fixes.fix_int_long_tuple  (int_long_tuple)
    libmodernize.fixes.fix_itertools_imports_six  (itertools_imports_six)
    libmodernize.fixes.fix_itertools_six  (itertools_six)
    libmodernize.fixes.fix_map  (map)
    libmodernize.fixes.fix_metaclass  (metaclass)
    libmodernize.fixes.fix_next  (next)
    libmodernize.fixes.fix_print  (print)
    libmodernize.fixes.fix_raise  (raise)
    libmodernize.fixes.fix_raise_six  (raise_six)
    libmodernize.fixes.fix_unichr  (unichr)
    libmodernize.fixes.fix_unicode_future  (unicode_future)
    libmodernize.fixes.fix_unicode_type  (unicode_type)
    libmodernize.fixes.fix_urllib_six  (urllib_six)
    libmodernize.fixes.fix_xrange_six  (xrange_six)
    libmodernize.fixes.fix_zip  (zip)
 Applying the following explicit transformations:
    (None)

RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: No changes to ebill/ebill/tasks/enqueue_function.py
RefactoringTool: Files that need to be modified:
RefactoringTool: ebill/ebill/tasks/enqueue_function.py

As you can see in the last 3 lines, "no changes" but also "need to be modified". Seems like this is a longstanding issue with 2to3.

I think the "need to be modified" is translated into the rt.files attribute that powers the --enforce flag.

I haven't dug into fissix yet, but maybe it represents a suitable layer at which to undo the damage? If a path says "no changes", remove it from the files list? Just an idea!

Details

• OS: Ubuntu 18.04
• Python version: 3.6.9
• fissix version: 20.8.0
• Can you repro on master? yes
• Can you repro in a clean virtualenv? yes

fix iterables to list when indexed in iterated context

Description

PyCQA/modernize#204

input:

print(set(zip(*[(1, 2)])[0]))

output:

print(set(zip(*[(1, 2)])[0]))

expected:

print(set(list(zip(*[(1, 2)]))[0]))

Details

• OS:
• Python version:
• fissix version:
• Can you repro on master?
• Can you repro in a clean virtualenv?

Rename function arg async to async_

A lot of pre py37 codebases use async which needs to be renamed on Python 3.7+. Detecting when it was used before it was claimed by the language may be a bit tricky to perfect, but the most common case of function args seems quite doable.

[upstream issue] missing license and permission to distribute fissix/pgen2/tokenize.py

Hi,

I've begun packaging Fissix for Debian, as a dependency of Bowler, and as you may know Debian is very strict about licenses. Unfortunately the following makes Fissix undistribuable:

# Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006 Python Software Foundation.                        
# All rights reserved.

because a license grant permitting distribution is missing, and because redistribution is explicitly prohibited by "All rights reserved". Eg: the right to redistribute has been reserved by the Python Software Foundation.

Would you please work with upstream lib2to3 to resolve this issue? An additional line with "License: PSF" will solve it.

Thanks!
Nicholas

fix_dict does not wrap enumerate(d.items()) in a list

Description

I was reading the source code of fixer_util.py and I noticed that one item enumerate was not like the others in consuming_calls because it does not actually consume the iterable in python 2 or python 3.

For example, fissix leaves this unchanged (without wrapping in list()) because it thinks that enumerate will consume the python 3 iterable: enumerate(d.items()). But it should wrap it in list because you might mutate d inside a loop: for i, x in enumerate(d.items()): del d[x] which will raise “RuntimeError: dictionary changed size during iteration” in python 3.

Instead, enumerate should be handled the same as iter.

Details

• OS: any
• Python version: any
• fissix version: 21.6.6
• Can you repro on master? yes
• Can you repro in a clean virtualenv? yes

Import lib2to3 from v3.8.2+

@jreese specifically I'm after the fix for https://bugs.python.org/issue38641

I tried to run "make update" but it failed with

git submodule update --init
git -C cpython checkout -f master
Previous HEAD position was 7b3ab5921f Python 3.8.2
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
git -C cpython clean -xfd
export VERSION=$(awk -F '"' '/#define PY_VERSION /{print $2}' cpython/Include/patchlevel.h) && \
sed -i "" -e "s/__base_version__ = \".*\"/__base_version__ = \"$VERSION\"/" fissix/__init__.py
sed: can't read : No such file or directory
make: *** [makefile:15: version] Error 2

What is the purpose of this module?

fissix claims to be a backport of lib2to3. Maybe it would be better named as backports.lib2to3?

I'm unclear what "with enhancements" is supposed to mean. As far as I can tell, fissix.__init__ monkeypatches some things purely based around loading/saving the grammar, but the only other differences compared to stock lib2to3 are trivial formatting changes, because apparently running black on someone else's code is more important than being able to diff changes.

...

I'm considering packaging bowler for Arch Linux, but it seems like it requires me to create an additional fissix package too, and I don't understand why.

Please make fissix/tests relocatable and able to load from system libs rather than source tree

Description

Please make fissix/tests relocatable and able to load from system libs rather than source tree. This is required for Debian's CI, because we must test packages as-installed rather than from their source trees. Additionally, we're supposed to load libs and modules from their build directory rather than from the source repo during the initial build. Setting and exporting PYTHONPATH has no effect on this issue at present (a67281e). Ideally it would be nice to be able to skip the loading of a helper script that sets up the existing strict load-from-source-tree expectation.

Details

• OS: Debian sid (currently 11)
• Python version: 3.9.0
• fissix version: v20.8.0
• Can you repro on master? v20.8.0-15-ga67281e (a67281e) does not appear to contain commits that would allow successful execution of self-tests on our infrastructure.
• Can you repro in a clean virtualenv? NA

Thank you for your consideration,
Nicholas

FP in fix_except

Description

fix_except returns FP even when the code contains try..except according to Python 3 specification

Details

• OS: Win10
• Python version: 3.8
• fissix version: 20.8.0
• Can you repro on master? Yes
• Can you repro in a clean virtualenv? yes

After looking into the issue, i think the problem is with the transform function in fix_except, where it only looks for the length of the except clause.
https://github.com/jreese/fissix/blob/main/fissix/fixes/fix_except.py

if len(except_clause.children) == 4:
                (E, comma, N) = except_clause.children[1:4]
                comma.replace(Name("as", prefix=" "))
                changed = True

A fix for this can be checking if there is a command being used.

Reproduction code:

Taken from https://docs.python.org/3/tutorial/errors.html

def test():    
    try:
        raise NameError('HiThere')
    except Exception as e:
        print('An exception flew by!')

New release?

Description

Is it possible to get a new minor release?

UserDict

A fixer for Python 2 UserDict would be helpful.