google / yapf Goto Github PK
View Code? Open in Web Editor NEWA formatter for Python files
License: Apache License 2.0
A formatter for Python files
License: Apache License 2.0
Before installing yapf, I'd usually want to see some examples of what it does. A weirdly formatted file should come out quite nicely, and an already well-formatted file should come out at least just as well-formatted. I suggest adding some sample input/output combinations either to the Readme or a sepeate Examples file.
The examples could just be copied from some of the tests, but it's much more nice to see some stuff without browsing though the code.
This means that code like this will cause a SyntaxError:
from __future__ import print_function
def call_my_function(the_function):
the_function("hi")
if __name__ == "__main__":
call_my_function(print)
with output/traceback like so:
INTERNAL ERROR:
call_my_function(print)
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/flyte/dev/yapf/yapf/__main__.py", line 18, in <module>
sys.exit(yapf.main(sys.argv))
File "yapf/__init__.py", line 102, in main
print_diff=args.diff)
File "yapf/__init__.py", line 124, in FormatFiles
filename, style_config=style_config, lines=lines, print_diff=print_diff)
File "yapf/yapflib/yapf_api.py", line 68, in FormatFile
print_diff=print_diff)
File "yapf/yapflib/yapf_api.py", line 113, in FormatCode
reformatted_source = reformatter.Reformat(uwlines)
File "yapf/yapflib/reformatter.py", line 73, in Reformat
verifier.VerifyCode(formatted_code[-1])
File "yapf/yapflib/verifier.py", line 45, in VerifyCode
compile(normalized_code.encode('UTF-8'), '<string>', 'exec')
File "<string>", line 1
call_my_function(print)
^
SyntaxError: invalid syntax
The problem is that each line is taken on its own merit without taking into account imports from future.
Given the following where the indents are tabs:
def strip_fragment(url):
"""
"""
yapf converts the first tab into spaces but not the second.
Tabs are merely discouraged, but mixing tabs with spaces is definitely illegal in pep8.
This is somewhat of an aesthetic preference, so obviously feel free to close this as notabugwontfix, but the following code:
def aaaaaaaaaaa(*args):
pass
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb = aaaaaaaaaaa(
("...", "."), "..",
".............................................."
)
is reformatted as
def aaaaaaaaaaa(*args):
pass
bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb = aaaaaaaaaaa(("...", "."
), "..", "..............................................")
Having the tuple argument split over the line like that with the closing bracket on the next line looks really bad. Note that putting the closing bracket on the same line as the opening bracket would not even come close to the 80 character column limit.
The following code:
class NAryTreeStrategy():
def produce_template(self, context, pv):
if True:
children = tuple((self.branch_key_strategy.draw_template(
context, pv.branch_key_parameter),
self.produce_template(context, pv))
for _ in range(10))
return children
Is reformatted as:
class NAryTreeStrategy():
def produce_template(self, context, pv):
if True:
children = tuple((self.branch_key_strategy.draw_template(
context, pv.branch_key_parameter),
self.produce_template(context, pv))
for _ in range(10))
return children
Which causes the pep8 checker to report "E126 continuation line over-indented for hanging indent"
If I have a python file with just the following line:
#"
Then running yapf on the file will eat the comment and produce an empty file.
yapf formats:
class eld(d):
if str(geom.geom_type).upper(
) != self.geom_type and not self.geom_type == 'GEOMETRY':
ror(code='om_type')
as:
class eld(d):
if str(geom.geom_type).upper(
) != self.geom_type and not self.geom_type == 'GEOMETRY':
ror(code='om_type')
This is both pep8 wrong and really confusing to read.
The following:
def feed(request, url, feed_dict=None):
instance.title_template = f.title_template or (
'feeds/%s_title.html' %
slug)
gets reformatted as:
def feed(request, url, feed_dict=None):
instance.title_template = f.title_template or ('feeds/%s_title.html' % slug)
To address requests like #42
The following:
def foo(x):
return x
@foo
# A comment
@foo
def inner(self):
pass
Produces the following error:
INTERNAL ERROR:
# A comment
@foo
Traceback (most recent call last):
File "/home/david/.pyenv/versions/2.7.8/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/home/david/.pyenv/versions/2.7.8/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/david/yapf/yapf/__main__.py", line 18, in <module>
sys.exit(yapf.main(sys.argv))
File "/home/david/yapf/yapf/__init__.py", line 102, in main
print_diff=args.diff)
File "/home/david/yapf/yapf/__init__.py", line 124, in FormatFiles
filename, style_config=style_config, lines=lines, print_diff=print_diff)
File "/home/david/yapf/yapf/yapflib/yapf_api.py", line 67, in FormatFile
print_diff=print_diff)
File "/home/david/yapf/yapf/yapflib/yapf_api.py", line 110, in FormatCode
reformatted_source = reformatter.Reformat(uwlines)
File "/home/david/yapf/yapf/yapflib/reformatter.py", line 73, in Reformat
verifier.VerifyCode(formatted_code[-1])
File "/home/david/yapf/yapf/yapflib/verifier.py", line 45, in VerifyCode
compile(normalized_code.encode('UTF-8'), '<string>', 'exec')
File "<string>", line 2
@foo
^
SyntaxError: invalid syntax
The following:
def w(context): {
'e_link': (
not is_popup and context['has_delete_permission'] and
ange and t('e', True)
),
}
Gets reformatted as:
def w(context):
{
'e_link': (not is_popup and context['has_delete_permission'] and ange
and t('e', True)),
}
Which is wrong. The "and" should not be starting a line according to pep8.
The following code:
def noop(x):
return x
timeout_settings = None
# This is a comment not a blank line
# This is a comment not a blank line
@noop
@noop
def test_slow_failing_test_1(x):
pass
is reformatted to:
def noop(x):
return x
timeout_settings = None
# This is a comment not a blank line
# This is a comment not a blank line
@noop
@noop
def test_slow_failing_test_1(x):
pass
With a single blank line between timeout_settings and the start of the comment. This is incorrect. ( E302 expected 2 blank lines, found 1)
Both comments and both decorators appear to be necessary to trigger this behaviour.
yapf will error if you run it on a file that cannot be handled by the encoding set by your LANG environment variable. e.g. consider the following file:
# encoding: utf-8
x = "☃"
If you set LANG=C and run yapf on it this will crash with
Traceback (most recent call last):
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/david/yapf/yapf/__main__.py", line 18, in <module>
sys.exit(yapf.main(sys.argv))
File "/home/david/yapf/yapf/__init__.py", line 102, in main
print_diff=args.diff)
File "/home/david/yapf/yapf/__init__.py", line 124, in FormatFiles
filename, style_config=style_config, lines=lines, print_diff=print_diff)
File "/home/david/yapf/yapf/yapflib/yapf_api.py", line 59, in FormatFile
original_source = ReadFile(filename, logging.warning)
File "/home/david/yapf/yapf/yapflib/yapf_api.py", line 142, in ReadFile
source = fd.read()
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 24: ordinal not in range(128)
(Took me forever to figure out this was what was going on here. I was experiencing a lot of crashes when running yapf in a subprocess but couldn't reproduce them. Finally figured out it was that the environment was getting overridden)
In the interests of not having github eat the important details, here is an example file encoded as a python string literal:
"\ufeff# -*- coding: utf-8 -*-\n\nresult = 'passed'\n"
(Yes, I know that a bom in a utf-8 file doesn't make sense. Don't ask me, I just find 'em).
Running on python 3.4.2 with the latest master this will produce the following lib2to3 error:
Traceback (most recent call last):
File "/home/david/yapf/yapf/yapflib/pytree_utils.py", line 85, in ParseCodeToTree
tree = parser_driver.parse_string(code, debug=False)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/lib2to3/pgen2/driver.py", line 106, in parse_string
return self.parse_tokens(tokens, debug)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/lib2to3/pgen2/driver.py", line 71, in parse_tokens
if p.addtoken(type, value, (prefix, start)):
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/lib2to3/pgen2/parse.py", line 116, in addtoken
ilabel = self.classify(type, value, context)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/lib2to3/pgen2/parse.py", line 172, in classify
raise ParseError("bad token", type, value, context)
lib2to3.pgen2.parse.ParseError: bad token: type=56, value='\ufeff', context=('', (1, 0))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/david/yapf/yapf/__main__.py", line 18, in <module>
sys.exit(yapf.main(sys.argv))
File "/home/david/yapf/yapf/__init__.py", line 102, in main
print_diff=args.diff)
File "/home/david/yapf/yapf/__init__.py", line 124, in FormatFiles
filename, style_config=style_config, lines=lines, print_diff=print_diff)
File "/home/david/yapf/yapf/yapflib/yapf_api.py", line 67, in FormatFile
print_diff=print_diff)
File "/home/david/yapf/yapf/yapflib/yapf_api.py", line 88, in FormatCode
tree = pytree_utils.ParseCodeToTree(unformatted_source.rstrip() + '\n')
File "/home/david/yapf/yapf/yapflib/pytree_utils.py", line 91, in ParseCodeToTree
tree = parser_driver.parse_string(code, debug=False)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/lib2to3/pgen2/driver.py", line 106, in parse_string
return self.parse_tokens(tokens, debug)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/lib2to3/pgen2/driver.py", line 71, in parse_tokens
if p.addtoken(type, value, (prefix, start)):
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/lib2to3/pgen2/parse.py", line 116, in addtoken
ilabel = self.classify(type, value, context)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/lib2to3/pgen2/parse.py", line 172, in classify
raise ParseError("bad token", type, value, context)
lib2to3.pgen2.parse.ParseError: bad token: type=56, value='\ufeff', context=('', (1, 0))
Running 2to3 itself on the file works correctly.
This one is not caused by whitespace as far as I can tell, or at least not at the end of the file:
def walk(top, topdown=True, onerror=None, followlinks=False):
yield from walk(new_path, topdown, onerror, followlinks)
Produces the error:
Traceback (most recent call last):
File "/home/david/yapf/yapf/yapflib/pytree_utils.py", line 85, in ParseCodeToTree
tree = parser_driver.parse_string(code, debug=False)
File "/usr/lib/python3.2/lib2to3/pgen2/driver.py", line 106, in parse_string
return self.parse_tokens(tokens, debug)
File "/usr/lib/python3.2/lib2to3/pgen2/driver.py", line 71, in parse_tokens
if p.addtoken(type, value, (prefix, start)):
File "/usr/lib/python3.2/lib2to3/pgen2/parse.py", line 159, in addtoken
raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=5, value=' ', context=('', (1, 0))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.2/runpy.py", line 160, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python3.2/runpy.py", line 73, in _run_code
exec(code, run_globals)
File "/home/david/yapf/yapf/__main__.py", line 18, in <module>
sys.exit(yapf.main(sys.argv))
File "/home/david/yapf/yapf/__init__.py", line 98, in main
lines=lines))
File "/home/david/yapf/yapf/yapflib/yapf_api.py", line 88, in FormatCode
tree = pytree_utils.ParseCodeToTree(unformatted_source.rstrip() + '\n')
File "/home/david/yapf/yapf/yapflib/pytree_utils.py", line 91, in ParseCodeToTree
tree = parser_driver.parse_string(code, debug=False)
File "/usr/lib/python3.2/lib2to3/pgen2/driver.py", line 106, in parse_string
return self.parse_tokens(tokens, debug)
File "/usr/lib/python3.2/lib2to3/pgen2/driver.py", line 71, in parse_tokens
if p.addtoken(type, value, (prefix, start)):
File "/usr/lib/python3.2/lib2to3/pgen2/parse.py", line 159, in addtoken
raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=5, value=' ', context=('', (1, 0))
The following code:
def response( using=None):
if (context_instance is _context_instance_undefined and
dirs is _dirs_undefined and dictionary is _dictionary_undefined):
return to
Gives me the following exception when run under python 3.4:
Traceback (most recent call last):
File "/home/david/yapf/yapf/yapflib/pytree_utils.py", line 85, in ParseCodeToTree
tree = parser_driver.parse_string(code, debug=False)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/lib2to3/pgen2/driver.py", line 106, in parse_string
return self.parse_tokens(tokens, debug)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/lib2to3/pgen2/driver.py", line 71, in parse_tokens
if p.addtoken(type, value, (prefix, start)):
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/lib2to3/pgen2/parse.py", line 159, in addtoken
raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=6, value='', context=('\n', (5, 0))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/david/yapf/yapf/__main__.py", line 18, in <module>
sys.exit(yapf.main(sys.argv))
File "/home/david/yapf/yapf/__init__.py", line 102, in main
print_diff=args.diff)
File "/home/david/yapf/yapf/__init__.py", line 124, in FormatFiles
filename, style_config=style_config, lines=lines, print_diff=print_diff)
File "/home/david/yapf/yapf/yapflib/yapf_api.py", line 67, in FormatFile
print_diff=print_diff)
File "/home/david/yapf/yapf/yapflib/yapf_api.py", line 88, in FormatCode
tree = pytree_utils.ParseCodeToTree(unformatted_source)
File "/home/david/yapf/yapf/yapflib/pytree_utils.py", line 91, in ParseCodeToTree
tree = parser_driver.parse_string(code, debug=False)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/lib2to3/pgen2/driver.py", line 106, in parse_string
return self.parse_tokens(tokens, debug)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/lib2to3/pgen2/driver.py", line 71, in parse_tokens
if p.addtoken(type, value, (prefix, start)):
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/lib2to3/pgen2/parse.py", line 159, in addtoken
raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=6, value='', context=('\n', (5, 0))
Running the normal 2to3 command line utility on this file works fine, so I think there's something about what you're passing in to 2to3.
Rather than duplicating it:
https://github.com/python/cpython/blob/master/Lib/configparser.py#L597
This isn't really such a bug report as more of an inquisition about how to approach this.
As requested here: pre-commit/pre-commit-hooks#50 we've had some interest in integrating yapf with our pre-commit framework.
There's two basic approaches to doing this so I'd like to get your opinions on this (I'd like to contribute to making either of them happen):
Option 1 is ideal for us as we don't have to deal with dependencies, breaking apis, etc. especially when the project is in an early stage and potentially volatile (and I don't think yapf is on pypi yet?)
Either way, the general interface for a pre-commit hook to work with our framework is as follows:
I'm not sure how much of that already happens but I'd be willing to pitch in to make it work (and test it to prevent regressions, etc.).
Given those assumptions it'd be as simple as checking in a file named hooks.yaml
which has contents probably similar to this:
- id: yapf
name: yapf
description: "Runs yapf 'Yet Another Python Formatter' over python source"
entry: yapf
language: python
files: \.py$
args: [-i]
Thanks in advance,
Anthony
yapf chokes with SyntaxError: unexpected EOF while parsing
on input like the following:
from asyncio import coroutine, Lock
lock = Lock()
@coroutine
def foo():
with (yield from lock):
return 42
See https://gist.github.com/jab/8df5bfae6ba13c0d51d5.
This syntax is given as an example in https://docs.python.org/dev/library/asyncio-sync.html#lock (search for "Context manager usage:").
The following code:
class YapfTest(unittest.TestCase):
def testSimple(self):
unformatted_code = textwrap.dedent(u"""\
print('foo')
""")
Gives the following error:
INTERNAL ERROR:
unformatted_code = textwrap.dedent(u"""\
print('foo')
""")
Traceback (most recent call last):
File "/home/david/yapf/yapf/yapflib/verifier.py", line 38, in VerifyCode
compile(textwrap.dedent(code).encode('UTF-8'), '<string>', 'exec')
File "<string>", line 4
""")
^
SyntaxError: invalid syntax
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/david/yapf/yapf/yapflib/verifier.py", line 41, in VerifyCode
ast.parse(textwrap.dedent(code.lstrip('\n')).lstrip(), '<string>', 'exec')
File "/usr/lib/python3.2/ast.py", line 36, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
File "<string>", line 3
""")
^
SyntaxError: invalid syntax
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.2/runpy.py", line 160, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python3.2/runpy.py", line 73, in _run_code
exec(code, run_globals)
File "/home/david/yapf/yapf/__main__.py", line 18, in <module>
sys.exit(yapf.main(sys.argv))
File "/home/david/yapf/yapf/__init__.py", line 102, in main
print_diff=args.diff)
File "/home/david/yapf/yapf/__init__.py", line 124, in FormatFiles
filename, style_config=style_config, lines=lines, print_diff=print_diff)
File "/home/david/yapf/yapf/yapflib/yapf_api.py", line 67, in FormatFile
print_diff=print_diff)
File "/home/david/yapf/yapf/yapflib/yapf_api.py", line 110, in FormatCode
reformatted_source = reformatter.Reformat(uwlines)
File "/home/david/yapf/yapf/yapflib/reformatter.py", line 73, in Reformat
verifier.VerifyCode(formatted_code[-1])
File "/home/david/yapf/yapf/yapflib/verifier.py", line 45, in VerifyCode
compile(normalized_code.encode('UTF-8'), '<string>', 'exec')
File "<string>", line 3
""")
^
SyntaxError: invalid syntax
...as you can probably tell, by this point I have a fuzzer doing most of the work for me (about the first half + a few of the subsequent ones have been hand produced) and this is from turning it on the yapf codebase itself. If you want it I'd be happy to give you a copy of the fuzzer.
The following code:
class Foo(object):
def bar(self):
if self.solo_generator is None and len(
self.generators + self.next_batch) == 1:
pass
reformats as
class Foo(object):
def bar(self):
if self.solo_generator is None and len(
self.generators + self.next_batch) == 1:
pass
Which the pep8 error checker says "E125 continuation line with same indent as next logical line" to
yapf
seems to break the indentation of Python 3.4's bisect.py
. Note that the lines after the else:
should be indented one more level than what yapf
is currently doing.
$ python -m yapf -d bisect.py
--- bisect.py (original)
+++ bisect.py (reformatted)
@@ -1,4 +1,5 @@
"""Bisection algorithms."""
+
def insort_right(a, x, lo=0, hi=None):
"""Insert item x in list a, and keep it sorted assuming a is sorted.
@@ -14,12 +15,15 @@
if hi is None:
hi = len(a)
while lo < hi:
- mid = (lo+hi)//2
+ mid = (lo + hi) // 2
if x < a[mid]: hi = mid
- else: lo = mid+1
+ else:
+ lo = mid + 1
a.insert(lo, x)
-insort = insort_right # backward compatibility
+
+insort = insort_right # backward compatibility
+
def bisect_right(a, x, lo=0, hi=None):
"""Return the index where to insert item x in list a, assuming a is sorted.
@@ -37,12 +41,15 @@
if hi is None:
hi = len(a)
while lo < hi:
- mid = (lo+hi)//2
+ mid = (lo + hi) // 2
if x < a[mid]: hi = mid
- else: lo = mid+1
+ else:
+ lo = mid + 1
return lo
-bisect = bisect_right # backward compatibility
+
+bisect = bisect_right # backward compatibility
+
def insort_left(a, x, lo=0, hi=None):
"""Insert item x in list a, and keep it sorted assuming a is sorted.
@@ -58,9 +65,10 @@
if hi is None:
hi = len(a)
while lo < hi:
- mid = (lo+hi)//2
- if a[mid] < x: lo = mid+1
- else: hi = mid
+ mid = (lo + hi) // 2
+ if a[mid] < x: lo = mid + 1
+ else:
+ hi = mid
a.insert(lo, x)
@@ -80,9 +88,10 @@
if hi is None:
hi = len(a)
while lo < hi:
- mid = (lo+hi)//2
- if a[mid] < x: lo = mid+1
- else: hi = mid
+ mid = (lo + hi) // 2
+ if a[mid] < x: lo = mid + 1
+ else:
+ hi = mid
return lo
# Overwrite above definitions with a fast C implementation
Here is a minimal example:
if True: pass
else: pass
--- foo.py (original)
+++ foo.py (reformatted)
@@ -1,2 +1,3 @@
if True: pass
-else: pass
+else:
+pass
Similarly:
class Foo(object): pass
--- foo.py (original)
+++ foo.py (reformatted)
@@ -1 +1,2 @@
-class Foo(object): pass
+class Foo(object):
+pass
Thanks
When running yapf over this code:
class CookieJar(RAMCookieJar):
def purge_old_cookies(self):
cookies = [c for c in self.allCookies()
if c.isSessionCookie() or c.expirationDate() >= now]
I get:
Traceback (most recent call last):
File "/usr/lib/python3.4/runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.4/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/florian/proj/qutebrowser/.venv/lib/python3.4/site-packages/yapf/__main__.py", line 18, in <module>
sys.exit(yapf.main(sys.argv))
File "/home/florian/proj/qutebrowser/.venv/lib/python3.4/site-packages/yapf/__init__.py", line 102, in main
print_diff=args.diff)
File "/home/florian/proj/qutebrowser/.venv/lib/python3.4/site-packages/yapf/__init__.py", line 124, in FormatFiles
filename, style_config=style_config, lines=lines, print_diff=print_diff)
File "/home/florian/proj/qutebrowser/.venv/lib/python3.4/site-packages/yapf/yapflib/yapf_api.py", line 68, in FormatFile
print_diff=print_diff)
File "/home/florian/proj/qutebrowser/.venv/lib/python3.4/site-packages/yapf/yapflib/yapf_api.py", line 101, in FormatCode
uwl.CalculateFormattingInformation()
File "/home/florian/proj/qutebrowser/.venv/lib/python3.4/site-packages/yapf/yapflib/unwrapped_line.py", line 77, in CalculateFormattingInformation
token.split_penalty += _SplitPenalty(prev_token, token)
File "/home/florian/proj/qutebrowser/.venv/lib/python3.4/site-packages/yapf/yapflib/unwrapped_line.py", line 342, in _SplitPenalty
if cur_token.node_split_penalty > 0:
TypeError: unorderable types: NoneType() > int()
The following:
import sys
# THING
# THING
Reformats as:
import sys # THING
# THING
Which is both almost certainly not what was intended and two pep8 violations for the price of one
(pep8 checker says E111 and E113)
You're calling parser.parse_args in https://github.com/google/yapf/blob/master/yapf/__init__.py#L71, which uses sys.argv instead of the argv argument. Given the usage pattern elsewhere I expect this is not intended behaviour.
$ echo 'from . import foo' | python -myapf
from .import foo
Oops.
This will create a clean command line tool in any OS : http://stackoverflow.com/questions/774824/explain-python-entry-points
yapf currently chokes on raw bytestring literals:
bash-4.3$ cat /tmp/rb_str_literal.py
rb'literal'
bash-4.3$ python3 -m yapf -i /tmp/rb_str_literal.py
Traceback (most recent call last):
File "/usr/local/lib/python3.4/site-packages/yapf/yapflib/pytree_utils.py", line 96, in ParseCodeToTree
tree = parser_driver.parse_string(code, debug=False)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib2to3/pgen2/driver.py", line 106, in parse_string
return self.parse_tokens(tokens, debug)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib2to3/pgen2/driver.py", line 71, in parse_tokens
if p.addtoken(type, value, (prefix, start)):
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib2to3/pgen2/parse.py", line 159, in addtoken
raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=3, value="'literal'", context=('', (1, 2))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/usr/local/lib/python3.4/site-packages/yapf/__main__.py", line 18, in <module>
sys.exit(yapf.main(sys.argv))
File "/usr/local/lib/python3.4/site-packages/yapf/__init__.py", line 109, in main
print_diff=args.diff, verify=args.verify)
File "/usr/local/lib/python3.4/site-packages/yapf/__init__.py", line 133, in FormatFiles
print_diff=print_diff, verify=verify)
File "/usr/local/lib/python3.4/site-packages/yapf/yapflib/yapf_api.py", line 72, in FormatFile
verify=verify)
File "/usr/local/lib/python3.4/site-packages/yapf/yapflib/yapf_api.py", line 95, in FormatCode
tree = pytree_utils.ParseCodeToTree(unformatted_source.rstrip() + '\n')
File "/usr/local/lib/python3.4/site-packages/yapf/yapflib/pytree_utils.py", line 101, in ParseCodeToTree
tree = parser_driver.parse_string(code, debug=False)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib2to3/pgen2/driver.py", line 106, in parse_string
return self.parse_tokens(tokens, debug)
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib2to3/pgen2/driver.py", line 71, in parse_tokens
if p.addtoken(type, value, (prefix, start)):
File "/usr/local/Cellar/python3/3.4.3/Frameworks/Python.framework/Versions/3.4/lib/python3.4/lib2to3/pgen2/parse.py", line 159, in addtoken
raise ParseError("bad input", type, value, context)
lib2to3.pgen2.parse.ParseError: bad input: type=3, value="'literal'", context=('', (1, 2))
This is using revision 7a01d00.
Reformatting the following (pep8 correct) code:
trailing_comma = [
1,
]
Results in the following:
trailing_comma = [1,]
PEP8 wants whitespace after the trailing comma. I can't immediately find a reference in PEP8 itself, but the pep8 checker program calls this E231.
Many Django projects have code like this:
urlpatterns = patterns('',
url(r'^$', 'homepage_view'),
url(r'^/login/$', 'login_view'),
url(r'^/login/$', 'logout_view'),
url(r'^/user/(?P<username>\w+)/$', 'profile_view')
)
With the PEP-8 preset, this shows the alignment algorithm's attempts to fit as much as possible on each line which tends to require a little more care scanning to distinguish the arguments:
urlpatterns = patterns('', url(r'^$', 'homepage_view'), url(r'^/login/$',
'login_view'),
url(r'^/login/$',
'logout_view'), url(r'^/user/(?P<username>\w+)/$',
'profile_view')
)
A quick look through style.py doesn't seem to offer a way to have it attempt to fit everything on a line but, when impossible, use a common starting position all of the elements at the same level to avoid what happens to the last element in the previous example:
urlpatterns = patterns(
'',
url(r'^$', 'homepage_view'),
url(r'^/login/$', 'login_view'),
url(r'^/login/$', 'logout_view'),
url(r'^/user/(?P<username>\w+)/$', 'profile_view'),
)
or a more compact version which avoids wrapping within a term:
urlpatterns = patterns('', url(r'^$', 'homepage_view'),
url(r'^/login/$', 'login_view'),
url(r'^/login/$', 'logout_view'),
url(r'^/user/(?P<username>\w+)/$', 'profile_view'))
The following takes 25 seconds to reformat on my machine. In comparison, many significantly larger files run in under a second:
specialElements = frozenset((
(namespaces['html'], 'address'),
(namespaces['html'], 'applet'),
(namespaces['html'], 'area'),
(namespaces['html'], 'article'),
(namespaces['html'], 'aside'),
(namespaces['html'], 'base'),
(namespaces['html'], 'basefont'),
(namespaces['html'], 'bgsound'),
(namespaces['html'], 'blockquote'),
(namespaces['html'], 'body'),
(namespaces['html'], 'br'),
(namespaces['html'], 'button'),
(namespaces['html'], 'caption'),
(namespaces['html'], 'center'),
(namespaces['html'], 'col'),
(namespaces['html'], 'colgroup'),
(namespaces['html'], 'command'),
(namespaces['html'], 'dd'),
(namespaces['html'], 'details'),
(namespaces['html'], 'dir'),
(namespaces['html'], 'div'),
(namespaces['html'], 'dl'),
(namespaces['html'], 'dt'),
(namespaces['html'], 'embed'),
(namespaces['html'], 'fieldset'),
(namespaces['html'], 'figure'),
(namespaces['html'], 'footer'),
(namespaces['html'], 'form'),
(namespaces['html'], 'frame'),
(namespaces['html'], 'frameset'),
(namespaces['html'], 'h1'),
(namespaces['html'], 'h2'),
(namespaces['html'], 'h3'),
(namespaces['html'], 'h4'),
(namespaces['html'], 'h5'),
(namespaces['html'], 'h6'),
(namespaces['html'], 'head'),
(namespaces['html'], 'header'),
(namespaces['html'], 'hr'),
(namespaces['html'], 'html'),
(namespaces['html'], 'iframe'),
# Note that image is commented out in the spec as "this isn't an
# element that can end up on the stack, so it doesn't matter,"
(namespaces['html'], 'image'),
(namespaces['html'], 'img'),
(namespaces['html'], 'input'),
(namespaces['html'], 'isindex'),
(namespaces['html'], 'li'),
(namespaces['html'], 'link'),
(namespaces['html'], 'listing'),
(namespaces['html'], 'marquee'),
(namespaces['html'], 'menu'),
(namespaces['html'], 'meta'),
(namespaces['html'], 'nav'),
(namespaces['html'], 'noembed'),
(namespaces['html'], 'noframes'),
(namespaces['html'], 'noscript'),
(namespaces['html'], 'object'),
(namespaces['html'], 'ol'),
(namespaces['html'], 'p'),
(namespaces['html'], 'param'),
(namespaces['html'], 'plaintext'),
(namespaces['html'], 'pre'),
(namespaces['html'], 'script'),
(namespaces['html'], 'section'),
(namespaces['html'], 'select'),
(namespaces['html'], 'style'),
(namespaces['html'], 'table'),
(namespaces['html'], 'tbody'),
(namespaces['html'], 'td'),
(namespaces['html'], 'textarea'),
(namespaces['html'], 'tfoot'),
(namespaces['html'], 'th'),
(namespaces['html'], 'thead'),
(namespaces['html'], 'title'),
(namespaces['html'], 'tr'),
(namespaces['html'], 'ul'),
(namespaces['html'], 'wbr'),
(namespaces['html'], 'xmp'),
(namespaces['svg'], 'foreignObject')
))
yapf formats:
def response():
if (defined and
dirs is _dirs_undefined and dictionary is _dictionary_undefined):
return to
as
def response():
if (defined and dirs is _dirs_undefined and
dictionary is _dictionary_undefined):
return to
pep8 checker gives E129. I also agree with it that this looks misleading.
pyformat thinks the original is correct and leaves it untouched. I would personally have formatted as
def response():
if (
defined and
dirs is _dirs_undefined and dictionary is _dictionary_undefined
):
return to
but I'm not wedded to that and suspect you like whitespace less than I do. :-)
In the continuing strategy of "here's a python string literal for a file exhibiting this problem to avoid github being clever", the following string passes ast.parse (in python 3.4.2) but causes yapf to crash when a file with precisely these contents is passed to it and run under the same python version. I think this is because yapf has an assumption baked in that all source is valid utf-8.
String:
"# а\x91а\x96б\x9fб\x80б\x81б\x82б\x83б\x84б\x85б\x86б\x87б\x88б\x89б\x8aб\x8bб\x8cб\x8dб\x8eб\x8f <- Cyrillic characters\n'а\x8eб\x82т\x84\x96аЄ'\n"
Error:
INTERNAL ERROR: # а�а�б�б�б�б�б�б�б
б�б�б�б�б�б�б�б�б�б� <- Cyrillic characters
Traceback (most recent call last):
File "/home/david/yapf/yapf/yapflib/verifier.py", line 38, in VerifyCode
compile(textwrap.dedent(code).encode('UTF-8'), '<string>', 'exec')
File "<string>", line 2
б�б�б�б�б�б�б�б�б�б� <- Cyrillic characters
^
SyntaxError: invalid character in identifier
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/david/yapf/yapf/yapflib/verifier.py", line 41, in VerifyCode
ast.parse(textwrap.dedent(code.lstrip('\n')).lstrip(), '<string>', 'exec')
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/ast.py", line 35, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
File "<string>", line 2
б�б�б�б�б�б�б�б�б�б� <- Cyrillic characters
^
SyntaxError: invalid character in identifier
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/david/yapf/yapf/__main__.py", line 18, in <module>
sys.exit(yapf.main(sys.argv))
File "/home/david/yapf/yapf/__init__.py", line 102, in main
print_diff=args.diff)
File "/home/david/yapf/yapf/__init__.py", line 124, in FormatFiles
filename, style_config=style_config, lines=lines, print_diff=print_diff)
File "/home/david/yapf/yapf/yapflib/yapf_api.py", line 67, in FormatFile
print_diff=print_diff)
File "/home/david/yapf/yapf/yapflib/yapf_api.py", line 110, in FormatCode
reformatted_source = reformatter.Reformat(uwlines)
File "/home/david/yapf/yapf/yapflib/reformatter.py", line 73, in Reformat
verifier.VerifyCode(formatted_code[-1])
File "/home/david/yapf/yapf/yapflib/verifier.py", line 45, in VerifyCode
compile(normalized_code.encode('UTF-8'), '<string>', 'exec')
File "<string>", line 1
б�б�б�б�б�б�б�б�б�б� <- Cyrillic characters
^
SyntaxError: invalid character in identifier
Reformatting the following valid (but gross) code:
def foo(x):
return x
def test_collapses_whitespace_nicely():
foo(
lambda x, y: 1 # pragma: no cover
)
Results in the following (better but invalid) code:
def foo(x):
return x
def test_collapses_whitespace_nicely():
foo(lambda x, y: 1 # pragma: no cover
)
The following pep8 correct code:
class A(object):
def b(self):
if self.aaaaaaaaaaaaaaaaaaaa not in self.bbbbbbbbbb(
cccccccccccccccccccc=True
):
pass
Will be reformatted to the following non pep8 compliant (and ugly) code:
class A(object):
def b(self):
if self.aaaaaaaaaaaaaaaaaaaa not in self.bbbbbbbbbb(cccccccccccccccccccc
=True):
pass
The line break around the equals is not kosher. pep8 requires keeping kwargs together without spaces.
Formatting the following crashes (under both 2.7.8 and 3.4.2):
methods.exec(sys.modules[name])
The error is:
lib2to3.pgen2.parse.ParseError: bad input: type=1, value='exec', context=('', (1, 8))
Running 2to3 on the same code produces the same parse error, so this is a limitation of the library.
Downloading it and running setup.py make deployment harder than it should be.
The following syntactically valid python:
class SplitPenaltyTest(unittest.TestCase):
def testUnbreakable(self):
self._CheckPenalties(tree, [
]) # yapf: disable
Is reformatted into the following invalid python:
class SplitPenaltyTest(unittest.TestCase):
def testUnbreakable(self):
self._CheckPenalties(tree, [
]) # yapf: disable
The following code will produce an internal error when yapf is run under 2.7:
class ABC(metaclass=type):
pass
The error is:
INTERNAL ERROR: class ABC(metaclass=type):
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/david/yapf/yapf/__main__.py", line 18, in <module>
sys.exit(yapf.main(sys.argv))
File "/home/david/yapf/yapf/__init__.py", line 102, in main
print_diff=args.diff)
File "/home/david/yapf/yapf/__init__.py", line 124, in FormatFiles
filename, style_config=style_config, lines=lines, print_diff=print_diff)
File "/home/david/yapf/yapf/yapflib/yapf_api.py", line 67, in FormatFile
print_diff=print_diff)
File "/home/david/yapf/yapf/yapflib/yapf_api.py", line 110, in FormatCode
reformatted_source = reformatter.Reformat(uwlines)
File "/home/david/yapf/yapf/yapflib/reformatter.py", line 73, in Reformat
verifier.VerifyCode(formatted_code[-1])
File "/home/david/yapf/yapf/yapflib/verifier.py", line 45, in VerifyCode
compile(normalized_code.encode('UTF-8'), '<string>', 'exec')
File "<string>", line 1
class ABC(metaclass=type):
^
SyntaxError: invalid syntax
The following code causes an error when yapf is run under python 3 (I've tried on both 3.2 and 3.4):
def given(y):
[k for k in () if k in y]
It gives the following error:
Traceback (most recent call last):
File "/home/david/.pyenv/versions/3.4.2/bin/yapf", line 11, in <module>
sys.exit(run_main())
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/site-packages/yapf/__init__.py", line 154, in run_main
sys.exit(main(sys.argv))
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/site-packages/yapf/__init__.py", line 102, in main
print_diff=args.diff)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/site-packages/yapf/__init__.py", line 124, in FormatFiles
filename, style_config=style_config, lines=lines, print_diff=print_diff)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/site-packages/yapf/yapflib/yapf_api.py", line 68, in FormatFile
print_diff=print_diff)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/site-packages/yapf/yapflib/yapf_api.py", line 101, in FormatCode
uwl.CalculateFormattingInformation()
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/site-packages/yapf/yapflib/unwrapped_line.py", line 77, in CalculateFormattingInformation
token.split_penalty += _SplitPenalty(prev_token, token)
File "/home/david/.pyenv/versions/3.4.2/lib/python3.4/site-packages/yapf/yapflib/unwrapped_line.py", line 342, in _SplitPenalty
if cur_token.node_split_penalty > 0:
TypeError: unorderable types: NoneType() > int()
I've tried this both against the latest version on pip and the current version of master.
The following code
d(ew);ing_new
is reformatted as:
d(ew) ; ing_new
The whitespace before ; is definitely forbidden by pep8. The multiple statements per line is "discouraged".
This code is pep8 correct, because timeout is a local function definition and thus has only one blank line in front of it:
import signal
try:
signal.SIGALRM
# ..................................................................
# ...............................................................
def timeout(seconds=1):
pass
except:
pass
However yapf reformats it as:
import signal
try:
signal.SIGALRM
# ..................................................................
# ...............................................................
def timeout(seconds=1):
pass
except:
pass
With two blank lines before timeout.
pep8 checker says "E303 too many blank lines (2)"
I suppose this one is more of a feature request. pyformat will strip blank lines from the end of a file (pep8 checker reports the blank line at the end of the file as W391), but yapf will not, meaning a file that could easily be formatted into PEP8 compliance will not be.
Consider e.g.
from __future__ import division, print_function, absolute_import, unicode_literals
(there's nothing special about this being a from future import, it's just an example of a long import from my project)
This import line is more than 82 characters but yapf puts it all on one line. It does this even if the code was originally broken across multiple lines.
It is not clear (to me at least) what is the default style if the --style
option is not set.
i.e. just exit immediately with a big message saying "yapf doesn't support running on this version of python", mostly so you don't get twits like me running it on 3.2 and getting confused when it produces strange error messages later. :-)
This produces the same pep error as #26 but I think is not the same bug. Obviously feel free to close it as a duplicate if I'm wrong.
The following code:
class A(object):
def f(self, aaaaaaaaa, bbbbbbbbbbbbb, row):
if True:
if row[4] is None or row[5] is None:
bbbbbbbbbbbbb['..............'] = row[
5] if row[5] is not None else 5
is reformatted by yapf --style=pep8 as
class A(object):
def f(self, aaaaaaaaa, bbbbbbbbbbbbb, row):
if True:
if row[4] is None or row[5] is None:
bbbbbbbbbbbbb['..............'] = row[5] if row[5
] is not None else 5
Note that the closing bracket on the following line is being lined up with the opening bracket from an entirely different indexing expression. This is both a pep8 violation and also rather misleading.
From an email from Terry Reedy:
[...] style.py has
The number of spaces required before a trailing comment.
SPACES_BEFORE_COMMENT=1,
PEP8 actually says "Inline comments should be separated by at least two spaces from the statement.", which is what you have in the google style.
https://www.python.org/dev/peps/pep-0008/#inline-comments states "Inline comments should be separated by at least two spaces from the statement"
Yapf in pep8 style mode appears to put inline comments one space after the end. e.g. the following is both a result and fixed point of yapf --style=pep8:
def bar(x): # inline comment
pass
Installing packages system-wide as root is problematic for several reasons. (Clobbering or shadowing system versions of the module, system packages clobbering or shadowing user-installed modules, version management issues.)
Instead of recommending 'sudo', recommend users use 'python setup.py install --user' -- or better yet, a virtualenv + pip.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.