amyreese / fissix Goto Github PK
View Code? Open in Web Editor NEWbackport of lib2to3, with enhancements
License: Other
backport of lib2to3, with enhancements
License: Other
A lot of pre py37 codebases use async
which needs to be renamed on Python 3.7+. Detecting when it was used before it was claimed by the language may be a bit tricky to perfect, but the most common case of function args seems quite doable.
Related to this lib2to3 bug: https://bugs.python.org/issue35383
When having a kwarg called print
, parsing fails. Minimal example:
from fissix.pgen2.driver import Driver
from fissix import pytree
from fissix.pygram import python_grammar
import sys
d = Driver(grammar=python_grammar, convert=pytree.convert)
d.parse_string('print("a", file=sys.stderr)')
Results in:
ParseError Traceback (most recent call last)
<ipython-input-1-617e4f2e4fd8> in <module>
4 import sys
5 d = Driver(grammar=python_grammar, convert=pytree.convert)
----> 6 d.parse_string('print("a", file=sys.stderr)')
/nix/store/qh5vvjgvs792gb67k5aywndqrkgp3j2w-python3-3.7.2-env/lib/python3.7/site-packages/fissix/pgen2/driver.py in parse_string(self, text, debug)
101 """Parse a string and return the syntax tree."""
102 tokens = tokenize.generate_tokens(io.StringIO(text).readline)
--> 103 return self.parse_tokens(tokens, debug)
104
105
/nix/store/qh5vvjgvs792gb67k5aywndqrkgp3j2w-python3-3.7.2-env/lib/python3.7/site-packages/fissix/pgen2/driver.py in parse_tokens(self, tokens, debug)
70 "%s %r (prefix=%r)", token.tok_name[type], value, prefix
71 )
---> 72 if p.addtoken(type, value, (prefix, start)):
73 if debug:
74 self.logger.debug("Stop.")
/nix/store/qh5vvjgvs792gb67k5aywndqrkgp3j2w-python3-3.7.2-env/lib/python3.7/site-packages/fissix/pgen2/parse.py in addtoken(self, type, value, context)
159 else:
160 # No success finding a transition
--> 161 raise ParseError("bad input", type, value, context)
162
163 def classify(self, type, value, context):
ParseError: bad input: type=22, value='=', context=('', (1, 15))
@jreese specifically I'm after the fix for https://bugs.python.org/issue38641
I tried to run "make update" but it failed with
git submodule update --init
git -C cpython checkout -f master
Previous HEAD position was 7b3ab5921f Python 3.8.2
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
git -C cpython clean -xfd
export VERSION=$(awk -F '"' '/#define PY_VERSION /{print $2}' cpython/Include/patchlevel.h) && \
sed -i "" -e "s/__base_version__ = \".*\"/__base_version__ = \"$VERSION\"/" fissix/__init__.py
sed: can't read : No such file or directory
make: *** [makefile:15: version] Error 2
Hi,
I've begun packaging Fissix for Debian, as a dependency of Bowler, and as you may know Debian is very strict about licenses. Unfortunately the following makes Fissix undistribuable:
# Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006 Python Software Foundation.
# All rights reserved.
because a license grant permitting distribution is missing, and because redistribution is explicitly prohibited by "All rights reserved". Eg: the right to redistribute has been reserved by the Python Software Foundation.
Would you please work with upstream lib2to3 to resolve this issue? An additional line with "License: PSF" will solve it.
Thanks!
Nicholas
fissix claims to be a backport of lib2to3. Maybe it would be better named as backports.lib2to3?
I'm unclear what "with enhancements" is supposed to mean. As far as I can tell, fissix.__init__
monkeypatches some things purely based around loading/saving the grammar, but the only other differences compared to stock lib2to3 are trivial formatting changes, because apparently running black on someone else's code is more important than being able to diff changes.
...
I'm considering packaging bowler for Arch Linux, but it seems like it requires me to create an additional fissix package too, and I don't understand why.
I'd like to link from https://modernize.readthedocs.io/en/latest/fixers.html#to3-fixers to fissix but there's no docs page on https://fissix.readthedocs.io/
Please make fissix/tests relocatable and able to load from system libs rather than source tree. This is required for Debian's CI, because we must test packages as-installed rather than from their source trees. Additionally, we're supposed to load libs and modules from their build directory rather than from the source repo during the initial build. Setting and exporting PYTHONPATH
has no effect on this issue at present (a67281e). Ideally it would be nice to be able to skip the loading of a helper script that sets up the existing strict load-from-source-tree expectation.
Thank you for your consideration,
Nicholas
A fixer for Python 2 UserDict
would be helpful.
fissix.tests (unittest.loader._FailedTest) fail. This one stumped me and I had to consult a colleague. P.S. sorry it's taking so long to get Bowler into Debian!
python3.8 -m unittest discover -v
fissix.tests (unittest.loader._FailedTest) ... FAIL
======================================================================
FAIL: fissix.tests (unittest.loader._FailedTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_fissix/build/fissix/tests/__init__.py", line 9, in load_tests
return load_package_tests(os.path.dirname(__file__), *args)
File "/usr/lib/python3.8/test/support/__init__.py", line 243, in load_package_tests
package_tests = loader.discover(start_dir=pkg_dir,
AssertionError: Path must be within the project
----------------------------------------------------------------------
Ran 1 test in 0.000s
My colleague writes that this test fails due to a cause related to https://github.com/python/cpython/blob/master/Lib/test/support/__init__.py#L138-L140 and assumptions about a specific cpython layout. This sounds like an actionable bug with the following two resolution cases:
input:
print(set(zip(*[(1, 2)])[0]))
output:
print(set(zip(*[(1, 2)])[0]))
expected:
print(set(list(zip(*[(1, 2)]))[0]))
My fix PR #40 didn't actually do anything. I must have dropped the .value
during some refactoring.
Luckily this doesn't cause any regression.
Hi! Do you have any plans on supporting new Python versions?
Or since this is a backport - this is out of scope?
(Not sure if this belongs in bowler or fissix)
I wrote a thing to convert self.assertFalse(x)
to assert not x
, etc. Trouble is, it falls over easily when the expression in the parentheses isn't valid without the parentheses:
self.assertFalse(
'a'
'b'
)
My code naively converts this to
assert not 'a'
'b'
parenthesize_if_necessary
helper would be really useful for this kind of situation. I imagine this bug is widespread. I tried to write one, but I'm a bit out of my depth.__future__
imports, but perhaps something can be done?Thanks
Is it possible to get a new minor release?
Hi there! While trying to enforce 2/3 compatibility on a large codebase, I ran across an issue with modernize's --enforce
flag:
$ python-modernize --future-unicode --enforce enqueue_function.py
Loading the following fixers:
fissix.fixes.fix_apply (apply)
fissix.fixes.fix_except (except)
fissix.fixes.fix_exec (exec)
fissix.fixes.fix_execfile (execfile)
fissix.fixes.fix_exitfunc (exitfunc)
fissix.fixes.fix_funcattrs (funcattrs)
fissix.fixes.fix_has_key (has_key)
fissix.fixes.fix_idioms (idioms)
fissix.fixes.fix_long (long)
fissix.fixes.fix_methodattrs (methodattrs)
fissix.fixes.fix_ne (ne)
fissix.fixes.fix_numliterals (numliterals)
fissix.fixes.fix_operator (operator)
fissix.fixes.fix_paren (paren)
fissix.fixes.fix_reduce (reduce)
fissix.fixes.fix_renames (renames)
fissix.fixes.fix_repr (repr)
fissix.fixes.fix_set_literal (set_literal)
fissix.fixes.fix_standarderror (standarderror)
fissix.fixes.fix_sys_exc (sys_exc)
fissix.fixes.fix_throw (throw)
fissix.fixes.fix_tuple_params (tuple_params)
fissix.fixes.fix_types (types)
fissix.fixes.fix_ws_comma (ws_comma)
fissix.fixes.fix_xreadlines (xreadlines)
libmodernize.fixes.fix_basestring (basestring)
libmodernize.fixes.fix_dict_six (dict_six)
libmodernize.fixes.fix_file (file)
libmodernize.fixes.fix_filter (filter)
libmodernize.fixes.fix_import (import)
libmodernize.fixes.fix_imports_six (imports_six)
libmodernize.fixes.fix_input_six (input_six)
libmodernize.fixes.fix_int_long_tuple (int_long_tuple)
libmodernize.fixes.fix_itertools_imports_six (itertools_imports_six)
libmodernize.fixes.fix_itertools_six (itertools_six)
libmodernize.fixes.fix_map (map)
libmodernize.fixes.fix_metaclass (metaclass)
libmodernize.fixes.fix_next (next)
libmodernize.fixes.fix_print (print)
libmodernize.fixes.fix_raise (raise)
libmodernize.fixes.fix_raise_six (raise_six)
libmodernize.fixes.fix_unichr (unichr)
libmodernize.fixes.fix_unicode_future (unicode_future)
libmodernize.fixes.fix_unicode_type (unicode_type)
libmodernize.fixes.fix_urllib_six (urllib_six)
libmodernize.fixes.fix_xrange_six (xrange_six)
libmodernize.fixes.fix_zip (zip)
Applying the following explicit transformations:
(None)
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: No changes to ebill/ebill/tasks/enqueue_function.py
RefactoringTool: Files that need to be modified:
RefactoringTool: ebill/ebill/tasks/enqueue_function.py
As you can see in the last 3 lines, "no changes" but also "need to be modified". Seems like this is a longstanding issue with 2to3.
I think the "need to be modified" is translated into the rt.files
attribute that powers the --enforce
flag.
I haven't dug into fissix yet, but maybe it represents a suitable layer at which to undo the damage? If a path says "no changes", remove it from the files list? Just an idea!
fix_except returns FP even when the code contains try..except according to Python 3 specification
After looking into the issue, i think the problem is with the transform function in fix_except, where it only looks for the length of the except clause.
https://github.com/jreese/fissix/blob/main/fissix/fixes/fix_except.py
if len(except_clause.children) == 4:
(E, comma, N) = except_clause.children[1:4]
comma.replace(Name("as", prefix=" "))
changed = True
A fix for this can be checking if there is a command being used.
Reproduction code:
Taken from https://docs.python.org/3/tutorial/errors.html
def test():
try:
raise NameError('HiThere')
except Exception as e:
print('An exception flew by!')
catch StopIteration around 1-arg next in generator functions
example from https://www.python.org/dev/peps/pep-0479/#converting-the-exception-inside-next
input:
def my_generator():
...
yield next(it)
...
expected output:
def my_generator():
try:
...
yield next(it)
...
except StopIteration
return
I was reading the source code of fixer_util.py and I noticed that one item enumerate
was not like the others in consuming_calls
because it does not actually consume the iterable in python 2 or python 3.
For example, fissix leaves this unchanged (without wrapping in list()
) because it thinks that enumerate
will consume the python 3 iterable: enumerate(d.items())
. But it should wrap it in list
because you might mutate d
inside a loop: for i, x in enumerate(d.items()): del d[x]
which will raise “RuntimeError: dictionary changed size during iteration” in python 3.
Instead, enumerate
should be handled the same as iter
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.