GithubHelp home page GithubHelp logo

parso's People

Contributors

asmeurer avatar bgw avatar blueyed avatar carljm avatar davidhalter avatar gandhis1 avatar gousaiyang avatar hroncok avatar hugovk avatar isidentical avatar jarryshaw avatar jspricke avatar matanagasauce avatar mgorny avatar michael-k avatar mmorearty avatar naglis avatar peterjclaw avatar pjvandehaar avatar robinfrcd avatar robinro avatar robodair avatar sobolevn avatar sturmianseq avatar tacaswell avatar terseus avatar thatch avatar timgates42 avatar winchua avatar yuan-xy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

parso's Issues

Debug info in django shell

Hi.
When I use Django shell and try to use autocomplete I see this strings.

diff parser start
diff parser calculated                                                                        
diff: line_lengths old: 1, new: 1                                                             
diff replace old[1:1] new[1:1]                                                                
parse_part from 1 to 1 (to 0 in part parser)                                                  
diff parser end

Does your parser use the same logger as Django default?
I use Python 3.6.3, Django 1.10 and Ipython 6.2.1 which installed your package.

Can you change logger to prevent make output noise?

Possible misuse of `hashlib.sha256`

Parso includes calls to hashlib.sha256 with an argument.

The documentation doesn't mention arguments. In my Windows Python 3.8 it does take arguments, but here is a log from a package build in Fedora, which failed with:

parser = <class 'parso.python.parser.Parser'>
diff_parser = <class 'parso.python.diff.DiffParser'>
    def __init__(self, text, tokenizer, parser=BaseParser, diff_parser=None):
        self._pgen_grammar = generate_grammar(
            text,
            token_namespace=self._get_token_namespace()
        )
        self._parser = parser
        self._tokenizer = tokenizer
        self._diff_parser = diff_parser
>       self._hashed = hashlib.sha256(text.encode("utf-8")).hexdigest()
E       TypeError: sha256() takes no arguments
/usr/lib/python3.8/site-packages/parso/grammar.py:39: TypeError

Abnormal behavior when using get_defined_names

Hi,

I found that when I use get_defined_names for a atom_expr object, it just don't wok.

Then atom_expr code that I used to test is below:

detections['detection_masks'] = tf.to_float(
          tf.greater_equal(tf.sigmoid(mask_predictions), mask_threshold))

I thought it will return me detections['detection_masks] after get_defined_names, but it returns nothing.

I checked your codes in get_defined_names and _defined_names. I found that you handle it like this:

elif current.type in ('power', 'atom_expr'):
        if current.children[-2] != '**':  # Just if there's no operation
            trailer = current.children[-1]
            if trailer.children[0] == '.':
                names.append(trailer.children[1])

So, you just add to names only if it looks like xx.xxx. So I just want to ask why don't you consider situation that I met and how you define defined_name.

Thanks!

Travis Itself Fails

Invocation failed (exit code 1), logfile: /home/travis/build/davidhalter/parso/.tox/py33/log/py33-1.log

Check out the RecursiveChildPrinter pull request for more information.

Generally speaking, the testing method is failing itself, causing an overall build failure.

Escaped quotes produce incorrect SyntaxError in f-strings

I think I've found a small issue with quote escaping in f-strings. Consider the following code:

import parso
s = "f\"\\\"\""
eval(s) # this works
grammar = parso.load_grammar()
module = grammar.parse(s)
(err,) = grammar.iter_errors(module)
print(err.message)

The eval call works fine (and produces " as expected), while the err.message is:

SyntaxError: EOL while scanning string literal.

This is with:

  • python 3.7
  • parso 0.3.1

An option to disable cache completely

This cache takes a lot of space. I guess I'm ready to trade some speed for not having this eating space on my HDD. I guess there should be a global config file for parso and jedi where I should be able to disable caching completely.

Incorrect parsing behaviour with assignment expressions

Recently, we encountered several issues with parso when implementing a backport compiler for assignment expressions.

  1. Parsing f-strings with := formatter produces incorrect SyntaxError
  2. Parsing invalid use cases of assignment expressions do not raise SyntaxError

NB: all codes are running under Python 3.8.0; actual behaviours are running through parso.parse(code, error_recovery=False, version='3.8').

Case 1

Source code:

f'{x:=5}'

Expected behaviour: valid, passes =5 to formatter

Actual behaviour:

Traceback (most recent call last):
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/parser.py", line 181, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: ReservedString(:=)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/__init__.py", line 58, in parse
    return grammar.parse(code, **kwargs)
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/grammar.py", line 78, in parse
    return self._parse(code=code, **kwargs)
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/grammar.py", line 147, in _parse
    root_node = p.parse(tokens=tokens)
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/python/parser.py", line 82, in parse
    return super(Parser, self).parse(tokens)
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/parser.py", line 128, in parse
    self._add_token(token)
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/parser.py", line 187, in _add_token
    self.error_recovery(token)
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/python/parser.py", line 151, in error_recovery
    return super(Parser, self).error_recovery(token)
  File "/fakepath/.venv/lib/python3.8/site-packages/parso/parser.py", line 151, in error_recovery
    raise ParserSyntaxError('SyntaxError: invalid syntax', error_leaf)
parso.parser.ParserSyntaxError: ('SyntaxError: invalid syntax', <ErrorLeaf: TokenType(OP):':=', (1, 4)>)

Case 2

Source code:

(lambda: x := 1)

Expected behaviour:

SyntaxError: cannot use named assignment with lambda

Actual behaviour: parsed as valid code

Case 3

Source code:

(a[i] := x)

Expected behaviour:

SyntaxError: cannot use named assignment with subscript

Actual behaviour: parsed as valid code

Case 4

Source code:

(a.b := c)

Expected behaviour:

SyntaxError: cannot use named assignment with attribute

Actual behaviour: parsed as valid code

Case 5

Source code:

[i := 0 for i, j in range(5)]
[[(i := i) for j in range(5)] for i in range(5)]
[i for i, j in range(5) if True or (i := 1)]
[False and (i := 0) for i, j in range(5)]

Expected behaviour:

SyntaxError: assignment expression cannot rebind comprehension iteration variable 'i'

Actual behaviour: parsed as valid code

Case 6

Source code:

[i+1 for i in (i := range(5))]
[i+1 for i in (j := range(5))]
[i+1 for i in (lambda: (j := range(5)))()]

Expected behaviour:

SyntaxError: assignment expression cannot be used in a comprehension iterable expression

Actual behaviour: parsed as valid code

Case 7

Source code:

class Example:
    [(j := i) for i in range(5)]

Expected behaviour:

SyntaxError: assignment expression within a comprehension cannot be used in a class body

Actual behaviour: parsed as valid code

Index error when parsing parameters with wrong bare asterisk usage

The following code (the version parameter can be any Python 3):

import parso
parso.parse('def test(arg, *):\n    pass', version='3.6')

raises an IndexError exception:

Traceback (most recent call last):
  File "parso\parser.py", line 178, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: TokenType(ENDMARKER)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "script.py", line 9, in <module>
    parso.parse('def test(arg, *):\n    pass', version='3.6')
  File "parso\__init__.py", line 58, in parse
    return grammar.parse(code, **kwargs)
  File "parso\grammar.py", line 76, in parse
    return self._parse(code=code, **kwargs)
  File "parso\grammar.py", line 141, in _parse
    root_node = p.parse(tokens=tokens)
  File "parso\python\parser.py", line 82, in parse
    return super(Parser, self).parse(tokens)
  File "parso\parser.py", line 128, in parse
    self._add_token(token)
  File "parso\parser.py", line 182, in _add_token
    self._pop()
  File "parso\parser.py", line 206, in _pop
    new_node = self.convert_node(tos.dfa.from_rule, tos.nodes)
  File "parso\python\parser.py", line 93, in convert_node
    return self.node_map[nonterminal](children)
  File "parso\python\tree.py", line 570, in __init__
    parameters.children[1:-1] = _create_params(parameters, parameters.children[1:-1])
  File "parso\python\tree.py", line 540, in _create_params
    if param_children[0] == '*' and param_children[1] == ',' \
IndexError: list index out of range

This occurs on Parso e05d7fd.

dead code in `get_definition()`

this if condition is always false, so the branch never executes:

parso/parso/python/tree.py

Lines 217 to 220 in dcdd3bb

if type_ in ():
if self in node.get_defined_names():
return node
return None

(this may be intentional or temporary, I'm not sure there's an actual bug here, but I found this while investigating a case where variables that occur within a function are not finding their definition within the same function body.)

Short PythonErrorLeaf strings

I'm starting out using parso, and enjoying what I've used so far. Here's my question:

Given an invalid character, the PythonErrorLeaf eats up the rest of the expression. Is it possible to make the error more localized, like tokenize does? This is useful when parsing errors in f-strings because tokenize doesn't handle them as granularly. Specifically, instead of getting an error token '? + b' I'd love to get '?'. Would it be possible to get these smaller PythonErrorLeaf strings, or is that infeasible or a bad idea?

Thank you!

from pprint import pprint
import io
import ast
from tokenize import tokenize, untokenize
import inspect
import token
import re


import parso


def string_to_tokens(string):
    bytestring = string.encode('utf-8')
    bytesio = io.BytesIO(bytestring)
    tokens = tokenize(bytesio.readline)
    return list(tokens)


def iter_repeat(obj, func, sentinel=None):
    while obj is not None:
        yield obj
        obj = func(obj)


def iter_leaves(node):
    yield from iter_repeat(node, lambda x: x.get_next_leaf())


string = 'a + ? + b'


tokens = string_to_tokens(string)
pprint(tokens)
tree = parso.parse(string)
list(iter_leaves(tree.get_first_leaf()))
[TokenInfo(type=59 (BACKQUOTE), string='utf-8', start=(0, 0), end=(0, 0), line=''),
 TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a + ? + b'),
 TokenInfo(type=53 (OP), string='+', start=(1, 2), end=(1, 3), line='a + ? + b'),
 TokenInfo(type=56 (ERRORTOKEN), string=' ', start=(1, 3), end=(1, 4), line='a + ? + b'),
 TokenInfo(type=56 (ERRORTOKEN), string='?', start=(1, 4), end=(1, 5), line='a + ? + b'),
 TokenInfo(type=53 (OP), string='+', start=(1, 6), end=(1, 7), line='a + ? + b'),
 TokenInfo(type=1 (NAME), string='b', start=(1, 8), end=(1, 9), line='a + ? + b'),
 TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]
[<Name: a@1,0>,
 <Operator: +>,
 <PythonErrorLeaf: errortoken:'? + b', (1, 4)>,
 <EndMarker: prefix=''>]
string = 'f"{a + ? + b}"'
[TokenInfo(type=59 (BACKQUOTE), string='utf-8', start=(0, 0), end=(0, 0), line=''),
 TokenInfo(type=3 (STRING), string='f"{a + ? + b}"', start=(1, 0), end=(1, 14), line='f"{a + ? + b}"'),
 TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')]

Issue with the diff parser in update

Just have seen this:

[deoplete] Traceback (most recent call last):                          
  File "/home/user/.vim/plugged/deoplete.nvim/rplugin/python3/deoplete/child.py", line 229, in _gather_results
    ctx['candidates'] = source.gather_candidates(ctx)                                                           
  File "/home/user/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/sources/deoplete_jedi/profiler.py", line 39, in wrapper
    ret = func(self, *args, **kwargs)                                                                                           
  File "/home/user/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/sources/deoplete_jedi.py", line 124, in gather_candidates
  File "/home/user/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/sources/deoplete_jedi/profiler.py", line 39, in wrapper  
    ret = func(self, *args, **kwargs)                                                                                           
  File "/home/user/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/sources/deoplete_jedi.py", line 92, in get_completions
    # TODO: skip creating Script instances if not necessary.                                                                   
  File "/home/user/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/jedi/jedi/api/__init__.py", line 120, in __init__
    cache_path=settings.cache_directory                                                                                            
  File "/home/user/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/jedi/jedi/evaluate/__init__.py", line 383, in parse_and_get_code
    return self.grammar.parse(code=code, path=path, **kwargs), code                                                                               
  File "/home/user/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/parso/parso/grammar.py", line 76, in parse
    return self._parse(code=code, **kwargs)                                                                                 
  File "/home/user/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/parso/parso/grammar.py", line 126, in _parse
    new_lines=lines                                                                                                           
  File "/home/user/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/parso/parso/python/diff.py", line 162, in update
    % (last_pos, line_length, ''.join(diff))                                                                                      
Exception: There's an issue (294 != 293) with the diff parser. Please report:
---                                                                          
+++ 
@@ -102,7 +102,6 @@
                                filename,
                                environment=self._env).completions()
                                                                    
-    @profiler.profile
     @profiler.profile
     def massage_completions(self, completions):
         out = []                               
@@ -291,5 +290,4 @@
                     i -= 1
             return self.completion_dict(name, type_, comp)
                                                           
-        return self.completion_dict(name, type_, comp)
-    +        return self.completion_dict(name, type_, comp)

Maybe it is related to providing source together with an outdated (because currently being edited) filename?

Support for adding and removing nodes in the parsed tree

I'm working on support for this for isort now in their repo. If/when I finish, I'll try to make a PR to parso itself.

I would like to have a functionality to:

  1. Add/remove node (adjusting start_pos of nodes for subsequent nodes).
  2. Add/remove lines (same thing).

question: what's your timeline for supporting new python versions?

In this PR, we're trying to write a test that uses jedi. The overall project tests against 3.8-dev, but of course trying to use jedi under python 3.8 currently errors out with:

E                   FileNotFoundError: [Errno 2] No such file or directory: '/home/travis/virtualenv/python3.8-dev/lib/python3.8/site-packages/parso/python/grammar38.txt'

Totally reasonable, and we can skip the test on 3.8 for now.

But... I don't want to just mark the test as skipped on Python 3.8, because then it'll remain skipped and forgotten forever, past when 3.8 comes out :-).

Do you have a usual timeline for adding grammars for new python releases? For example, we could mark the test as skipped until 3.8-beta is released, or 3.8-final...

Interest in shipping type annotations with the project?

Hi! Thanks for parso.

We are using parso in a project that uses Python PEP 484 type annotations and a typechecker, so we have written fairly comprehensive type stubs for Parso's API. We'd like to share this work with the community, if you are amenable. There are many options for how to do this:

  1. Integrate type annotations directly into the codebase (need to use the comment-based syntax in order to preserve Python2 support). This is perhaps the best in terms of ensuring they stay up to date, but it depends if you want type hint comments in the code.

  2. Add .pyi stub files next to the .py files in the repo. This is next-best for ensuring they stay up to date, but keeps them out of the code for those who don't care.

  3. Keep them out of the parso repo entirely and include them in github.com/python/typeshed instead. This is the best option if you don't want to adopt any responsibility for maintaining them, and leave it up to the users of parso and typecheckers. But we won't do this either without your permission.

Let me know if any of those options sound acceptable to you, and which you prefer! Thanks again for your work on parso.

Possibly incorrect line at parso/python/pep8.py:148

I was looking at some of the new parso library just for fun and I found:

def _is_magic_name(name):
    return name.value.startswith('__') and name.value.startswith('__')

I guess the intention is

def _is_magic_name(name):
    return name.value.startswith('__') and name.value.endswith('__')

i.e.: "endswith" instead of repeating "startswith". :)

Good library by the way, keep it up! I use jedi-vim every day!!

Don't mutate the token.tok_name dictionary

The parso token library takes the standard library token.tok_name dictionary and mutates it here.

This results in some very odd behavior when using tokenize in my Python shell that uses Jedi. Before I do any tab completions, things work just fine. But as soon as I do a tab completion, parso is imported, and ENCODING is replaced with BACKQUOTE.

Parso should instead use its own copies of mutable data structures and leave the standard library alone.

Parsing f-strings with invalid quotes in expression part does not raise SyntaxError

Hello! I think I've found a small issue with parsing f-strings with invalid quotes in the expression part.

>>> import parso
>>> parso.parse("f'{ ''}'", error_recovery=False)
<Module: @1-1>

The code being parsed above is syntactically invalid, but parso is not generating an error.
However, I found that when I remove the leading the space in the expression part (before the invalid quote), parso works correctly.

>>> import parso
>>> parso.parse("f'{''}'", error_recovery=False)
Traceback (most recent call last):
  File "C:\Python38\lib\site-packages\parso\parser.py", line 181, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: TokenType(FSTRING_END)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python38\lib\site-packages\parso\__init__.py", line 58, in parse
    return grammar.parse(code, **kwargs)
  File "C:\Python38\lib\site-packages\parso\grammar.py", line 78, in parse
    return self._parse(code=code, **kwargs)
  File "C:\Python38\lib\site-packages\parso\grammar.py", line 147, in _parse
    root_node = p.parse(tokens=tokens)
  File "C:\Python38\lib\site-packages\parso\python\parser.py", line 82, in parse
    return super(Parser, self).parse(tokens)
  File "C:\Python38\lib\site-packages\parso\parser.py", line 128, in parse
    self._add_token(token)
  File "C:\Python38\lib\site-packages\parso\parser.py", line 187, in _add_token
    self.error_recovery(token)
  File "C:\Python38\lib\site-packages\parso\python\parser.py", line 151, in error_recovery
    return super(Parser, self).error_recovery(token)
  File "C:\Python38\lib\site-packages\parso\parser.py", line 151, in error_recovery
    raise ParserSyntaxError('SyntaxError: invalid syntax', error_leaf)
parso.parser.ParserSyntaxError: ('SyntaxError: invalid syntax', <ErrorLeaf: TokenType(FSTRING_END):"'", (1, 3)>)

Maybe there is a subtle bug when parsing the expression part? As for my test, this bug happens on both Python 3.8 and Python 3.6.
(I'm using parso 0.5.1)

cython for performance

I'm currently in a try-to-optimize-static-analysers mood, and I'm wondering whether we could Cythonise this? Here's an example for astroid where I basically change no code, and just compile it with cython for a 20% performance improvement. There's more relevant discussion in that PR about other options too.

Thoughts? I imagine parso is pretty performance critical (jedi-vim etc.), but if not, then there's not much point. Especially if you don't want to depend on users installing Cython.

If you're interested and can point to a standard performance test case, I can submit a PR like the above for you to look at, including quantified performance improvements.

Error when parsing f-strings with multiple arguments and formatting options

Thanks for the very useful library! I believe that there is an issue with parsing multiple arguments of an f-string when one has a formatting argument:

import parso

simple = "print(f'Some {x} and some {y}')"
grammar = parso.load_grammar()
module = grammar.parse(simple)
err = grammar.iter_errors(module)
print(f'`simple` f-string: {len(err)}')

with_formatting = "print(f'Some {x:.2f} and some {y}')"
grammar = parso.load_grammar()
module = grammar.parse(with_formatting)
(err,) = grammar.iter_errors(module)
print(f'`with_formatting` f-string: {err.message}')

with_formatting2 = "print(f'Some {x:,} and some {y}')"
grammar = parso.load_grammar()
module = grammar.parse(with_formatting2)
(err,) = grammar.iter_errors(module)
print(f'`with_formatting2` f-string: {err.message}')

single = "print(f'Some {x:,}')"
grammar = parso.load_grammar()
module = grammar.parse(single)
err = grammar.iter_errors(module)
print(f'`single` f-string: {len(err)}')

prints

`simple` f-string: 0
`with_formatting` f-string: SyntaxError: invalid syntax
`with_formatting2` f-string: SyntaxError: invalid syntax
`single` f-string: 0

No grammar for Python 3.7

Using IPython 6.2 with Python 3.7.0a1 (early adopter so to say), I get an error when trying to tab-complete which refers to a missing grammar text file. Copying the grammar for 3.6 to 3.7 resolves the problem.

Could you deliver a preliminary grammar for Python 3.7 by using the grammar from Python 3.6?

incorrect path string syntax

file = 'python/grammar%s%s.txt' % (version_info.major, version_info.minor)

'python/grammar%s%s.txt' does not work on all OS (e.g. Windows)
you should use
os.path.join('python', 'grammar%s%s.txt' % (version_info.major, version_info.minor)) instead

Hope this is helpful.

Parsing f-strings with nested format specifiers produces incorrect SyntaxError

Hello! I think I've found an issue with parsing f-strings with nested format specifiers (this should be a very rare use case though).

import parso
x = 1
y = 2
test_fmtspec_in_fmtspec = """f'{x:{y:1}}'"""
print(f'Result: {eval(test_fmtspec_in_fmtspec)!r}')
module = parso.parse(test_fmtspec_in_fmtspec, error_recovery=False)

The code test_fmtspec_in_fmtspec is syntactically valid according to Python document of f-strings:

Top-level format specifiers may include nested replacement fields. These nested fields may include their own conversion fields and format specifiers, but may not include more deeply-nested replacement fields. The format specifier mini-language is the same as that used by the string .format() method.

But the code above will produce an exception:

Result: ' 1'
Traceback (most recent call last):
  File "C:\Python37\lib\site-packages\parso\parser.py", line 181, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: TokenType(NUMBER)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test_fmtspec_in_fmtspec.py", line 6, in <module>
    module = parso.parse(test_fmtspec_in_fmtspec, error_recovery=False)
  File "C:\Python37\lib\site-packages\parso\__init__.py", line 58, in parse
    return grammar.parse(code, **kwargs)
  File "C:\Python37\lib\site-packages\parso\grammar.py", line 77, in parse
    return self._parse(code=code, **kwargs)
  File "C:\Python37\lib\site-packages\parso\grammar.py", line 146, in _parse
    root_node = p.parse(tokens=tokens)
  File "C:\Python37\lib\site-packages\parso\python\parser.py", line 82, in parse
    return super(Parser, self).parse(tokens)
  File "C:\Python37\lib\site-packages\parso\parser.py", line 128, in parse
    self._add_token(token)
  File "C:\Python37\lib\site-packages\parso\parser.py", line 187, in _add_token
    self.error_recovery(token)
  File "C:\Python37\lib\site-packages\parso\python\parser.py", line 151, in error_recovery
    return super(Parser, self).error_recovery(token)
  File "C:\Python37\lib\site-packages\parso\parser.py", line 151, in error_recovery
    raise ParserSyntaxError('SyntaxError: invalid syntax', error_leaf)
parso.parser.ParserSyntaxError: ('SyntaxError: invalid syntax', <ErrorLeaf: TokenType(NUMBER):'1', (1, 8)>)

I tested the code on both Python 3.7 and Python 3.6, both will produce this exception.
(I'm using parso 0.4.0)

AttributeError: 'PythonNode' object has no attribute 'string_prefix'

Using Python 3.6.7 and Parso 0.3.2

from parso import load_grammar

grammar = load_grammar()
module = grammar.parse('%p25%=" & echoc -y "F" & echoc -w "')
errors = grammar.iter_errors(module)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/grammar.py", line 165, in iter_errors
    return self._get_normalizer_issues(node, self._error_normalizer_config)
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/grammar.py", line 185, in _get_normalizer_issues
    normalizer.walk(node)
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/normalizer.py", line 34, in walk
    value = self.visit(node)
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/python/errors.py", line 280, in visit
    return super(ErrorFinder, self).visit(node)
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/normalizer.py", line 45, in visit
    return ''.join(self.visit(child) for child in children)
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/normalizer.py", line 45, in <genexpr>
    return ''.join(self.visit(child) for child in children)
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/python/errors.py", line 280, in visit
    return super(ErrorFinder, self).visit(node)
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/normalizer.py", line 45, in visit
    return ''.join(self.visit(child) for child in children)
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/normalizer.py", line 45, in <genexpr>
    return ''.join(self.visit(child) for child in children)
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/python/errors.py", line 280, in visit
    return super(ErrorFinder, self).visit(node)
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/normalizer.py", line 44, in visit
    with self.visit_node(node):
  File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/python/errors.py", line 285, in visit_node
    self._check_type_rules(node)
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/normalizer.py", line 54, in _check_type_rules
    rule.feed_node(node)
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/normalizer.py", line 182, in feed_node
    if self.is_issue(node):
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/python/errors.py", line 580, in is_issue
    if first_is_bytes != self._is_bytes_literal(string):
  File "/home/alexander/sandbox/src/git.udia.ca/alex/msr-2019-analysis/venv/lib/python3.7/site-packages/parso/python/errors.py", line 573, in _is_bytes_literal
    return 'b' in string.string_prefix.lower()
AttributeError: 'PythonNode' object has no attribute 'string_prefix'

Is this expected? I'm doing some data mining and my input source code is not guaranteed to be python source code.

Duplicate statements in `KeywordStatement` docstring

Specifically this part of the docstring repeats a pair at the end:

For the following statements: `assert`, `del`, `global`, `nonlocal`, `raise`, `return`, `yield`, `return`, `yield`.

return and yield are repeated, but I'm not sure what is meant to be in their place (if any). Someone more knowledgeable than I am about the Python grammar would be able to create a fix.

module.iter_imports() doesn't give all the imports in a file

First of all thank you for the wonderful tool.
I am trying to parse couple of python projects from the tool.
I wrote following test code to print all the import statements in a file.

path = "xxx"
    try:
        source = open(path, "r", encoding="ISO-8859-1").read()
    except FileNotFoundError as f:
        print(f)

        #
    module = parso.parse(source)
    for x in module.iter_imports():
        if isinstance(x,parso.python.tree.ImportFrom):
            print ([y.value for y in x.get_from_names()])
            print([y.value for y in x.get_defined_names()])
        else:
            importname = []
            imname = [y[0] for y in x._dotted_as_names()]
            for l in imname:
                for r in l:
                    importname.append(r.value)

It gives me correct results for all the imports which are at the start of the file.
However it doesn't provide me the import statements which are at the middle of the file

      def _build(self, model):
                """ Create the backend-specific placeholder.
                """
                backend = model.get_backend()
                if backend.get_name() == 'keras':

                        if self.highway:
                                raise ValueError('Backend does not support highway dense '
                                        'layers.')

                        import keras.layers as L                        # pylint: disable=import-error

                        if self.auto_flatten:
                                yield L.Flatten()

                        if backend.keras_version() == 1:
                                func = lambda x, **kwargs: L.Dense(output_dim=x, **kwargs)
                        else:
                                func = lambda x, **kwargs: L.Dense(units=x, **kwargs)

                        for v in self.size[:-1]:
                                yield func(v, trainable=not self.frozen)

                        yield func(
                                self.size[-1],
                                name=self.name,
                                trainable=not self.frozen
                        )

                elif backend.get_name() == 'pytorch':

The result of the iter_imports doesn't give me the stmt import keras.layers as L
Would this be a bug or error in my parsing code?

Question: Possible to serialize AST, or get the INDENT token in the parse tree?

Using Python3.6 with Parso v0.3.4.

I am looking for a way to serialize an AST into a list. Is it possible to get the INDENT and DEDENT tokens from the parse tree?

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
def main():
    print("Hello, world!")

if __name__ == "__main__":
    main()

I parse this code block by stepping through the siblings and children of each node, starting from the root module.

from parso import load_grammar

TYPE_TERMINAL = "T"
TYPE_STRUCTURE = "S" # todo handle structure start & end?

dummy_code = """#!/usr/bin/env python3
# -*- coding: utf-8 -*-
def main():
    print("Hello, world!")

if __name__ == "__main__":
    main()

"""

def parse_source_code(source_code):
    """take source code, convert into serialized AST
    """
    grammar = load_grammar()
    root = grammar.parse(source_code)
    ast_serialized = []

    to_parse_stack = [root, ]

    while to_parse_stack:
        struct_or_terminal_node = to_parse_stack.pop()
        if hasattr(struct_or_terminal_node, 'children') and struct_or_terminal_node.children:
            # this is a structure node
            struct_node = struct_or_terminal_node
            ast_serialized.append((TYPE_STRUCTURE, struct_node.type))
            for child in reversed(struct_node.children):
                to_parse_stack.append(child)
        else:
            # there are no children, this is a leaf
            terminal_node = struct_or_terminal_node
            child_code = terminal_node.get_code(include_prefix=False)
            child_type = terminal_node.type
            ast_serialized.append((TYPE_TERMINAL, child_type, child_code))
    return ast_serialized

print(parse_source_code(dummy_code))

The outputted list show the structure nodes and terminal nodes as 2,3-tuples.

[('S', 'file_input'), ('S', 'funcdef'), ('T', 'keyword', 'def'), ('T', 'name', 'main'), ('S', 'parameters'), ('T', 'operator', '('), ('T', 'operator', ')'), ('T', 'operator', ':'), ('S', 'suite'), ('T', 'newline', '\n'), ('S', 'simple_stmt'), ('S', 'atom_expr'), ('T', 'name', 'print'), ('S', 'trailer'), ('T', 'operator', '('), ('T', 'string', '"Hello, world!"'), ('T', 'operator', ')'), ('T', 'newline', '\n'), ('S', 'if_stmt'), ('T', 'keyword', 'if'), ('S', 'comparison'), ('T', 'name', '__name__'), ('T', 'operator', '=='), ('T', 'string', '"__main__"'), ('T', 'operator', ':'), ('S', 'suite'), ('T', 'newline', '\n'), ('S', 'simple_stmt'), ('S', 'atom_expr'), ('T', 'name', 'main'), ('S', 'trailer'), ('T', 'operator', '('), ('T', 'operator', ')'), ('T', 'newline', '\n'), ('T', 'endmarker', '')]

Is it possible to also output the indent and dedent tokens similar to how the newline tokens are provided?

The high level goal is to get a serialized AST that I can reconstruct the original source code from. Right now, I join all of the leaf node values together with a single white space separator. Thanks!

Enable to disable LOG.Debug

I embed IPython in blender, when I type a variable name and ' 'tab', parso's debug log shows up and this so annoying

In [1]: armDEBUG:THREE:pickle loaded: /home/gl/tools/blender-2.79a-linux-glibc219-x86_64/2.79/scripts/modules/bpy_types.py
atureDEBUG:THREE:diff parser start
DEBUG:THREE:diff parser calculated
DEBUG:THREE:diff: line_lengths old: 1, new: 1
DEBUG:THREE:diff replace old[1:1] new[1:1]
DEBUG:THREE:parse_part from 1 to 1 (to 0 in part parser)
DEBUG:THREE:diff parser end

My current solution is remove log from source code file. I think there is must be a better way to disable Logs

Async and await can't be used as identifiers in 3.6

It looks like parso's tokenizer unconditionally treats async and await as keywords under 3.6. However, while that's valid on 3.7+, on 3.6 it's conditionally treated as a keyword or identifier depending on if you're inside an async function.

Black has some logic to support this behavior when parsing as <3.7: https://github.com/psf/black/blob/40e8b3a231bade22d838858435a33d60a8325306/blib2to3/pgen2/tokenize.py#L359

This was reported by @zhammer on Instagram/LibCST#32

It might not be worth supporting because it's an older version of Python, and the hacks required to support it are pretty ugly, but I figured I'd upstream this issue.

Parsing print function with "file" argument results in error node in Python 2 grammers

Python 2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:22:17) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> code = """
... from __future__ import print_function
... try:
...     print(warn, file=sys.stderr)
... except IOError:
...     pass
... """
>>> import parso; parso.load_grammar(version='2.7').parse(code).children[1].children[2]
PythonNode(suite, [<Newline: u'\n'>, <PythonErrorNode: print(warn, file@4,4>, <PythonErrorLeaf: OP:u'=', (4, 20)>, <PythonErrorNode: sys.stderr@4,21>, <PythonErrorLeaf: OP:u')', (4, 31)>, <PythonErrorLeaf: NEWLINE:u'\n', (4, 32)>])

No requirements-dev.txt

I know it's not necessary but it would be a great help when installing sphinx, py.test, pylint, etc when setting up a new clone.

I'd make one but I'm not sure what other dev dependencies should be included.

Python 3.7 deprecation warning on collections.Mapping import

Since 3fa8630, parso uses from collections import Mapping, but that's deprecated since Python 3.7:

DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working

The plan is now to remove this starting with Python 3.9: https://docs.python.org/3.8/library/collections.html, but that still issues a warning on Python 3.7, and we fail our test suite on warnings. Since parso stills supports Python 2, I guess the fix looks like this (untested):

try:
    from collections.abc import Mapping
except ImportErrror:
    from collections import Mapping

Thanks!

cajarename

Yesterday I've installed ubuntu-mate 18.04.
Previosly jedi was installing without problems. I'm using it with geany (https://github.com/notetau/geany-jedi-complete)
sudo pip2 install parso
gives
...
Collecting parso
Downloading https://files.pythonhosted.org/packages/cd/3e/5908f9577dbd1e5df53e64349bfd11e46b726c1e4d8cd676bbe8aa4de316/parso-0.2.1-py2.py3-none-any.whl (91kB)
100% |โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 92kB 654kB/s
cajarename 17.3.28 requires caja, which is not installed.
cajarename 17.3.28 requires gir1.2-gtk-3.0, which is not installed.
cajarename 17.3.28 requires python-caja, which is not installed.
cajarename 17.3.28 requires python-gi, which is not installed.
Installing collected packages: parso
Successfully installed parso-0.2.1

all those packages are installed via apt, and cajarename is in apt caja-rename
alias cajarename='caja-rename' is not helping
Any suggestion?

False Positive: SyntaxError: trailing comma not allowed without surrounding parentheses

Hi,
I encountered a False positive about trailing comma not allowed without surrounding parentheses. I've made imports with trailing comma with surrounding parenthesis but the parser raises this error, even if it's valid.

import parso

string = """
from hashlib import (
    md5,
    sha256,
)
"""

grammar = parso.load_grammar(version='3.8')
module = grammar.parse(string, error_recovery=True)

print(grammar.iter_errors(module)[0].message)
#SyntaxError: trailing comma not allowed without surrounding parentheses

Improve Unicode name matching regex

Great that you're already matching Unicode names! However, looks like you're currently simply accepting one-or-more characters of class w:

Name = r'\w+'

AFAICT, a better regex for this would be:

Name = r'[^\d\W]\w*'

Uses root logger

Hi!

I noticed while using iPython that this library uses the root logger. That's typically poor practice for published packages. I recommend changing to a named logger (logging.getLogger('parso'), logging.getLogger(__name__), etc) to make it easier for others to filter out your messages.

Parsing f-strings with seeming assignment expressions produces incorrect SyntaxError

There is a subtle issue when parsing f-strings whose expression parts look like an assignment expression in Python 3.8. As illustrated in PEP 572, both f'{(x:=10)}' and f'{x:=10}' are valid expressions.

>>> f'{(x:=10)}'  # Valid, uses assignment expression
'10'
>>> x = 10
>>> f'{x:=10}'    # Valid, passes '=10' to formatter
'        10'

The former is an f-string with an assignment expression, while the latter is an f-string with =10 as the format specifiers of x. Both are valid, although this may be confusing and is not a good programming practice.

However, parso fails to parse f'{x:=10}':

>>> import parso
>>> parso.parse("f'{x:=10}'", error_recovery=False)
Traceback (most recent call last):
  File "C:\Python38\lib\site-packages\parso\parser.py", line 181, in _add_token
    plan = stack[-1].dfa.transitions[transition]
KeyError: ReservedString(:=)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python38\lib\site-packages\parso\__init__.py", line 58, in parse
    return grammar.parse(code, **kwargs)
  File "C:\Python38\lib\site-packages\parso\grammar.py", line 78, in parse
    return self._parse(code=code, **kwargs)
  File "C:\Python38\lib\site-packages\parso\grammar.py", line 147, in _parse
    root_node = p.parse(tokens=tokens)
  File "C:\Python38\lib\site-packages\parso\python\parser.py", line 82, in parse
    return super(Parser, self).parse(tokens)
  File "C:\Python38\lib\site-packages\parso\parser.py", line 128, in parse
    self._add_token(token)
  File "C:\Python38\lib\site-packages\parso\parser.py", line 187, in _add_token
    self.error_recovery(token)
  File "C:\Python38\lib\site-packages\parso\python\parser.py", line 151, in error_recovery
    return super(Parser, self).error_recovery(token)
  File "C:\Python38\lib\site-packages\parso\parser.py", line 151, in error_recovery
    raise ParserSyntaxError('SyntaxError: invalid syntax', error_leaf)
parso.parser.ParserSyntaxError: ('SyntaxError: invalid syntax', <ErrorLeaf: TokenType(OP):':=', (1, 4)>)

I am using parso 0.5.1. This issue only happens on Python 3.8.

AssertionError thrown on autocomplete keyword `from`

Running from a pipenv managed virtualenv (Python 3.6.5).

I'm using Neovim v0.3.1 with deoplete-jedi. Trying to autocomplete after typing any keyword throws this exception:

[deoplete] Traceback (most recent call last):
  File "/home/sett/.vim/plugged/deoplete.nvim/rplugin/python3/deoplete/child.py", line 179, in _gather_results
    result = self._get_result(context, source)
  File "/home/sett/.vim/plugged/deoplete.nvim/rplugin/python3/deoplete/child.py", line 234, in _get_result
    ctx['candidates'] = source.gather_candidates(ctx)
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/sources/deoplete_jedi/profiler.py", line 37, in wrapper
    return func(self, *args, **kwargs)
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/sources/deoplete_jedi.py", line 167, in gather_candidates
    environment=self._env)
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/sources/deoplete_jedi/profiler.py", line 37, in wrapper
    return func(self, *args, **kwargs)
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/sources/deoplete_jedi.py", line 119, in get_script
    return jedi.Script(source, line, col, filename, environment=self._env)
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/jedi/jedi/api/__init__.py", line 118, in __init__
    cache_path=settings.cache_directory,
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/jedi/jedi/evaluate/__init__.py", line 388, in parse_and_get_code
    return self.grammar.parse(code=code, path=path, **kwargs), code
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/parso/parso/grammar.py", line 76, in parse
    return self._parse(code=code, **kwargs)
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/parso/parso/grammar.py", line 126, in _parse
    new_lines=lines
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/parso/parso/python/diff.py", line 236, in update
    self._copy_from_old_parser(line_offset, i2, j2)
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/parso/parso/python/diff.py", line 289, in _copy_from_old_parser
    line_offset
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/parso/parso/python/diff.py", line 598, in copy_nodes
    self.prefix,
  File "/home/sett/.vim/plugged/deoplete-jedi/rplugin/python3/deoplete/vendored/parso/parso/python/diff.py", line 686, in _copy_nodes
    assert last_line_offset_leaf == ':'
AssertionError
Error from jedi: AssertionError().  Use :messages / see above for error details.

Simply removing/commenting offending line (line 686, diff.py) makes it work normally.

YAML parsing

@davidhalter mentioned to me on Twitter that parso could potentially be used to create a round-trippable YAML parser, if one has a tokenizer and a grammar.

So I thought I'd open this issue to discuss the possibility. How hard is it to write a tokenizer?

It looks like the grammar is context sensitive, which is expressed in the BNF grammar as production parameters (http://www.yaml.org/spec/1.2/spec.html, section 4.1).

Is this something that parso can handle?

I'm also happy to have only a subset of YAML supported, so long as it's a commonly used one. My personal use-case is programmatically modifying .travis.yml files.

tests fail under Python 3.8

I noticed that Python 3.8 is at least on your mind due to #47 and #52. However I discovered that the tests do not pass on that version. Using the python:3.8.0a2 container image from Docker Hub, I cloned this repo, ran pip install -e .[testing], then pytest, and got many failures.

pytest output (click to expand)
root@tassadar:~/parso# pytest 
========================================================= test session starts ==========================================================
platform linux -- Python 3.8.0a2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
rootdir: /root/parso, inifile: pytest.ini
collected 937 items                                                                                                                    

parso/__init__.py .                                                                                                              [  0%]
parso/python/tree.py .                                                                                                           [  0%]
test/test_absolute_import.py ...                                                                                                 [  0%]
test/test_cache.py ..                                                                                                            [  0%]
test/test_diff_parser.py ............................................................                                            [  7%]
test/test_error_recovery.py ............                                                                                         [  8%]
test/test_file_python_errors.py .......                                                                                          [  9%]
test/test_fstring.py .................................                                                                           [ 12%]
test/test_get_code.py ......                                                                                                     [ 13%]
test/test_grammar.py .                                                                                                           [ 13%]
test/test_load_grammar.py ............                                                                                           [ 14%]
test/test_normalizer_issues_files.py ..................................                                                          [ 18%]
test/test_old_fast_parser.py ...............                                                                                     [ 19%]
test/test_param_splitting.py .........                                                                                           [ 20%]
test/test_parser.py ............................................................................................................ [ 32%]
.............                                                                                                                    [ 33%]
test/test_parser_tree.py ...................................................                                                     [ 39%]
test/test_pep8.py ...                                                                                                            [ 39%]
test/test_pgen2.py .......................x..................................................................................... [ 51%]
................................................................................................................................ [ 64%]
.................................................                                                                                [ 70%]
test/test_prefix.py .....................                                                                                        [ 72%]
test/test_python_errors.py ..F....F......FF..........FF........FFFFFFFFFFFFFFFFFFFFF.FFFFFFFFFFFFF................F............. [ 83%]
..................................FF.FFF.......F................................................                                 [ 93%]
test/test_tokenize.py ........................................                                                                   [ 97%]
test/test_utils.py ......................                                                                                        [100%]

=============================================================== FAILURES ===============================================================
________________ test_python_exception_matches[for a in [1]:\n    try:\n        pass\n    finally:\n        continue\n] ________________

code = 'for a in [1]:\n    try:\n        pass\n    finally:\n        continue\n'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
>       wanted, line_nr = _get_actual_exception(code)

test/test_python_errors.py:32: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

code = 'for a in [1]:\n    try:\n        pass\n    finally:\n        continue\n'

    def _get_actual_exception(code):
        with warnings.catch_warnings():
            # We don't care about warnings where locals/globals misbehave here.
            # It's as simple as either an error or not.
            warnings.filterwarnings('ignore', category=SyntaxWarning)
            try:
                compile(code, '<unknown>', 'exec')
            except (SyntaxError, IndentationError) as e:
                wanted = e.__class__.__name__ + ': ' + e.msg
                line_nr = e.lineno
            except ValueError as e:
                # The ValueError comes from byte literals in Python 2 like '\x'
                # that are oddly enough not SyntaxErrors.
                wanted = 'SyntaxError: (value error) ' + str(e)
                line_nr = None
            else:
>               assert False, "The piece of code should raise an exception."
E               AssertionError: The piece of code should raise an exception.
E               assert False

test/test_python_errors.py:95: AssertionError
_________________________________________ test_python_exception_matches[f(x for x in bar, 1)] __________________________________________

code = 'f(x for x in bar, 1)'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       AssertionError: assert 'SyntaxError: positional argument follows keyword argument' in ['SyntaxError: Generator expression must be parenthesized']

test/test_python_errors.py:39: AssertionError
_____________________________________________ test_python_exception_matches[__debug__ = 1] _____________________________________________

code = '__debug__ = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       AssertionError: assert 'SyntaxError: assignment to keyword' in ['SyntaxError: cannot assign to __debug__']

test/test_python_errors.py:39: AssertionError
______________________________________ test_python_exception_matches[with x() as __debug__: pass] ______________________________________

code = 'with x() as __debug__: pass'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       AssertionError: assert 'SyntaxError: assignment to keyword' in ['SyntaxError: cannot assign to __debug__']

test/test_python_errors.py:39: AssertionError
_______________________________________________ test_python_exception_matches[foo(+a=3)] _______________________________________________

code = 'foo(+a=3)'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: keyword can't be an expression" in ['SyntaxError: expression cannot contain assignment, perhaps you meant "=="?']

test/test_python_errors.py:39: AssertionError
____________________________________________ test_python_exception_matches[f(lambda: 1=1)] _____________________________________________

code = 'f(lambda: 1=1)'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert 'SyntaxError: lambda cannot contain assignment' in ['SyntaxError: expression cannot contain assignment, perhaps you meant "=="?']

test/test_python_errors.py:39: AssertionError
____________________________________________ test_python_exception_matches[lambda a: 1 = 1] ____________________________________________

code = 'lambda a: 1 = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to lambda" in ['SyntaxError: cannot assign to lambda']

test/test_python_errors.py:39: AssertionError
__________________________________________ test_python_exception_matches[[x for x in y] = 1] ___________________________________________

code = '[x for x in y] = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       AssertionError: assert None in ['SyntaxError: cannot assign to list comprehension']

test/test_python_errors.py:39: AssertionError
__________________________________________ test_python_exception_matches[{x for x in y} = 1] ___________________________________________

code = '{x for x in y} = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to literal" in ['SyntaxError: cannot assign to set comprehension']

test/test_python_errors.py:39: AssertionError
_________________________________________ test_python_exception_matches[{x:x for x in y} = 1] __________________________________________

code = '{x:x for x in y} = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to literal" in ['SyntaxError: cannot assign to dict comprehension']

test/test_python_errors.py:39: AssertionError
__________________________________________ test_python_exception_matches[(x for x in y) = 1] ___________________________________________

code = '(x for x in y) = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       AssertionError: assert None in ['SyntaxError: cannot assign to generator expression']

test/test_python_errors.py:39: AssertionError
_______________________________________________ test_python_exception_matches[None = 1] ________________________________________________

code = 'None = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to keyword" in ['SyntaxError: cannot assign to None']

test/test_python_errors.py:39: AssertionError
________________________________________________ test_python_exception_matches[... = 1] ________________________________________________

code = '... = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to Ellipsis" in ['SyntaxError: cannot assign to Ellipsis']

test/test_python_errors.py:39: AssertionError
______________________________________________ test_python_exception_matches[a == b = 1] _______________________________________________

code = 'a == b = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to comparison" in ['SyntaxError: cannot assign to comparison']

test/test_python_errors.py:39: AssertionError
______________________________________________ test_python_exception_matches[{a, b} = 1] _______________________________________________

code = '{a, b} = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to literal" in ['SyntaxError: cannot assign to set display']

test/test_python_errors.py:39: AssertionError
______________________________________________ test_python_exception_matches[{a: b} = 1] _______________________________________________

code = '{a: b} = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to literal" in ['SyntaxError: cannot assign to dict display']

test/test_python_errors.py:39: AssertionError
_________________________________________________ test_python_exception_matches[1 = 1] _________________________________________________

code = '1 = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to literal" in ['SyntaxError: cannot assign to literal']

test/test_python_errors.py:39: AssertionError
________________________________________________ test_python_exception_matches["" = 1] _________________________________________________

code = '"" = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to literal" in ['SyntaxError: cannot assign to literal']

test/test_python_errors.py:39: AssertionError
_______________________________________________ test_python_exception_matches[b"" = 10] ________________________________________________

code = 'b"" = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to literal" in ['SyntaxError: cannot assign to literal']

test/test_python_errors.py:39: AssertionError
_______________________________________________ test_python_exception_matches[b"" = 11] ________________________________________________

code = 'b"" = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to literal" in ['SyntaxError: cannot assign to literal']

test/test_python_errors.py:39: AssertionError
_______________________________________________ test_python_exception_matches["" "" = 1] _______________________________________________

code = '"" "" = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to literal" in ['SyntaxError: cannot assign to literal']

test/test_python_errors.py:39: AssertionError
_______________________________________________ test_python_exception_matches[1 | 1 = 3] _______________________________________________

code = '1 | 1 = 3'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to operator" in ['SyntaxError: cannot assign to operator']

test/test_python_errors.py:39: AssertionError
_______________________________________________ test_python_exception_matches[1**1 = 3] ________________________________________________

code = '1**1 = 3'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to operator" in ['SyntaxError: cannot assign to operator']

test/test_python_errors.py:39: AssertionError
________________________________________________ test_python_exception_matches[~ 1 = 3] ________________________________________________

code = '~ 1 = 3'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to operator" in ['SyntaxError: cannot assign to operator']

test/test_python_errors.py:39: AssertionError
_______________________________________________ test_python_exception_matches[not 1 = 3] _______________________________________________

code = 'not 1 = 3'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to operator" in ['SyntaxError: cannot assign to operator']

test/test_python_errors.py:39: AssertionError
______________________________________________ test_python_exception_matches[1 and 1 = 3] ______________________________________________

code = '1 and 1 = 3'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to operator" in ['SyntaxError: cannot assign to operator']

test/test_python_errors.py:39: AssertionError
_______________________________________ test_python_exception_matches[def foo(): (yield 1) = 3] ________________________________________

code = 'def foo(): (yield 1) = 3'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to yield expression" in ['SyntaxError: cannot assign to yield expression']

test/test_python_errors.py:39: AssertionError
_____________________________________ test_python_exception_matches[async def foo(): await x = 3] ______________________________________

code = 'async def foo(): await x = 3'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to await expression" in ['SyntaxError: cannot assign to await expression']

test/test_python_errors.py:39: AssertionError
__________________________________________ test_python_exception_matches[(a if a else a) = a] __________________________________________

code = '(a if a else a) = a'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to conditional expression" in ['SyntaxError: cannot assign to conditional expression']

test/test_python_errors.py:39: AssertionError
_______________________________________________ test_python_exception_matches[a, 1 = x] ________________________________________________

code = 'a, 1 = x'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to literal" in ['SyntaxError: cannot assign to literal']

test/test_python_errors.py:39: AssertionError
_______________________________________________ test_python_exception_matches[foo() = 1] _______________________________________________

code = 'foo() = 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to function call" in ['SyntaxError: cannot assign to function call']

test/test_python_errors.py:39: AssertionError
_________________________________________ test_python_exception_matches[with x as foo(): pass] _________________________________________

code = 'with x as foo(): pass'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to function call" in ['SyntaxError: cannot assign to function call']

test/test_python_errors.py:39: AssertionError
______________________________________________ test_python_exception_matches[del bar, 1] _______________________________________________

code = 'del bar, 1'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't delete literal" in ['SyntaxError: cannot delete literal']

test/test_python_errors.py:39: AssertionError
_________________________________________ test_python_exception_matches[for x, 1 in []: pass] __________________________________________

code = 'for x, 1 in []: pass'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to literal" in ['SyntaxError: cannot assign to literal']

test/test_python_errors.py:39: AssertionError
________________________________________ test_python_exception_matches[for (not 1) in []: pass] ________________________________________

code = 'for (not 1) in []: pass'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to operator" in ['SyntaxError: cannot assign to operator']

test/test_python_errors.py:39: AssertionError
____________________________________________ test_python_exception_matches[[x for 1 in y]] _____________________________________________

code = '[x for 1 in y]'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       AssertionError: assert None in ['SyntaxError: cannot assign to literal']

test/test_python_errors.py:39: AssertionError
___________________________________________ test_python_exception_matches[[x for a, 3 in y]] ___________________________________________

code = '[x for a, 3 in y]'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       AssertionError: assert None in ['SyntaxError: cannot assign to literal']

test/test_python_errors.py:39: AssertionError
____________________________________________ test_python_exception_matches[(x for 1 in y)] _____________________________________________

code = '(x for 1 in y)'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       AssertionError: assert None in ['SyntaxError: cannot assign to literal']

test/test_python_errors.py:39: AssertionError
____________________________________________ test_python_exception_matches[{x for 1 in y}] _____________________________________________

code = '{x for 1 in y}'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       AssertionError: assert None in ['SyntaxError: cannot assign to literal']

test/test_python_errors.py:39: AssertionError
___________________________________________ test_python_exception_matches[{x:x for 1 in y}] ____________________________________________

code = '{x:x for 1 in y}'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       AssertionError: assert None in ['SyntaxError: cannot assign to literal']

test/test_python_errors.py:39: AssertionError
__________________________________________ test_python_exception_matches[{**{} for a in [1]}] __________________________________________

code = '{**{} for a in [1]}'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       AssertionError: assert None in ['SyntaxError: dict unpacking cannot be used in dict comprehension']

test/test_python_errors.py:39: AssertionError
_________________________ test_python_exception_matches[async def foo():\n def nofoo():[x async for x in []]] __________________________

code = 'async def foo():\n def nofoo():[x async for x in []]'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
>       errors = _get_error_list(code)

test/test_python_errors.py:34: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test/test_python_errors.py:22: in _get_error_list
    return list(grammar.iter_errors(tree))
parso/grammar.py:165: in iter_errors
    return self._get_normalizer_issues(node, self._error_normalizer_config)
parso/grammar.py:185: in _get_normalizer_issues
    normalizer.walk(node)
parso/normalizer.py:34: in walk
    value = self.visit(node)
parso/python/errors.py:280: in visit
    return super(ErrorFinder, self).visit(node)
parso/normalizer.py:45: in visit
    return ''.join(self.visit(child) for child in children)
parso/normalizer.py:45: in <genexpr>
    return ''.join(self.visit(child) for child in children)
parso/python/errors.py:280: in visit
    return super(ErrorFinder, self).visit(node)
parso/normalizer.py:45: in visit
    return ''.join(self.visit(child) for child in children)
parso/normalizer.py:45: in <genexpr>
    return ''.join(self.visit(child) for child in children)
parso/python/errors.py:280: in visit
    return super(ErrorFinder, self).visit(node)
parso/normalizer.py:45: in visit
    return ''.join(self.visit(child) for child in children)
parso/normalizer.py:45: in <genexpr>
    return ''.join(self.visit(child) for child in children)
parso/python/errors.py:280: in visit
    return super(ErrorFinder, self).visit(node)
parso/normalizer.py:45: in visit
    return ''.join(self.visit(child) for child in children)
parso/normalizer.py:45: in <genexpr>
    return ''.join(self.visit(child) for child in children)
parso/python/errors.py:280: in visit
    return super(ErrorFinder, self).visit(node)
parso/normalizer.py:45: in visit
    return ''.join(self.visit(child) for child in children)
parso/normalizer.py:45: in <genexpr>
    return ''.join(self.visit(child) for child in children)
parso/python/errors.py:280: in visit
    return super(ErrorFinder, self).visit(node)
parso/normalizer.py:45: in visit
    return ''.join(self.visit(child) for child in children)
parso/normalizer.py:45: in <genexpr>
    return ''.join(self.visit(child) for child in children)
parso/python/errors.py:280: in visit
    return super(ErrorFinder, self).visit(node)
parso/normalizer.py:45: in visit
    return ''.join(self.visit(child) for child in children)
parso/normalizer.py:45: in <genexpr>
    return ''.join(self.visit(child) for child in children)
parso/python/errors.py:280: in visit
    return super(ErrorFinder, self).visit(node)
parso/normalizer.py:44: in visit
    with self.visit_node(node):
/usr/local/lib/python3.8/contextlib.py:113: in __enter__
    return next(self.gen)
parso/python/errors.py:285: in visit_node
    self._check_type_rules(node)
parso/normalizer.py:54: in _check_type_rules
    rule.feed_node(node)
parso/normalizer.py:182: in feed_node
    if self.is_issue(node):
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <parso.python.errors._CompForRule object at 0x7f2dfe14cb50>, node = <CompFor: async for x in []@2,16>

    def is_issue(self, node):
        # Some of the nodes here are already used, so no else if
>       expr_list = node.children[1 + int(node.children[0] == 'async')]
E       IndexError: list index out of range

parso/python/errors.py:948: IndexError
__________________________________________ test_python_exception_matches[[*[] for a in [1]]] ___________________________________________

code = '[*[] for a in [1]]'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       AssertionError: assert None in ['SyntaxError: iterable unpacking cannot be used in comprehension']

test/test_python_errors.py:39: AssertionError
_______________________________________________ test_python_exception_matches[del None] ________________________________________________

code = 'del None'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't delete keyword" in ['SyntaxError: cannot delete None']

test/test_python_errors.py:39: AssertionError
______________________________________________ test_python_exception_matches[(True,) = x] ______________________________________________

code = '(True,) = x'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to keyword" in ['SyntaxError: cannot assign to True']

test/test_python_errors.py:39: AssertionError
___________________________________________ test_python_exception_matches[([False], a) = x] ____________________________________________

code = '([False], a) = x'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to keyword" in ['SyntaxError: cannot assign to False']

test/test_python_errors.py:39: AssertionError
______________________________________________ test_python_exception_matches[[a, 1] += 3] ______________________________________________

code = '[a, 1] += 3'

    @pytest.mark.parametrize('code', FAILING_EXAMPLES)
    def test_python_exception_matches(code):
        wanted, line_nr = _get_actual_exception(code)
    
        errors = _get_error_list(code)
        actual = None
        if errors:
            error, = errors
            actual = error.message
>       assert actual in wanted
E       assert "SyntaxError: can't assign to literal" in ['SyntaxError: cannot assign to literal']

test/test_python_errors.py:39: AssertionError
========================================== 47 failed, 889 passed, 1 xfailed in 11.89 seconds ===========================================

I was able to reproduce the same errors using the python38 package in Fedora.

Python 3.7.2 and diff parser

Issue is consistently reproducible with python 3.7.2 and doesn't exist with python 3.5.6.

I'm using parso (used from anaconda-mode/jedi, but that should not matter - I'm pretty sure it's easy to reproduce with 2 direct parso invokes, but sadly I don't have time for this right now) and encountering following error time to time:

Traceback (most recent call last):
  File "/root/.emacs.d/var/anaconda-mode/0.1.13/service_factory-0.1.5-py3.7.egg/service_factory/service.py", line 101, in apply
    result = method(**params)
  File "<string>", line 97, in wrapper
  File "/root/.emacs.d/var/anaconda-mode/0.1.13/jedi-0.13.2-py3.7.egg/jedi/api/__init__.py", line 118, in __init__
    cache_path=settings.cache_directory,
  File "/root/.emacs.d/var/anaconda-mode/0.1.13/jedi-0.13.2-py3.7.egg/jedi/evaluate/__init__.py", line 388, in parse_and_get_code
    return self.grammar.parse(code=code, path=path, **kwargs), code
  File "/root/.emacs.d/var/anaconda-mode/0.1.13/parso-0.3.2-py3.7.egg/parso/grammar.py", line 76, in parse
    return self._parse(code=code, **kwargs)
  File "/root/.emacs.d/var/anaconda-mode/0.1.13/parso-0.3.2-py3.7.egg/parso/grammar.py", line 126, in _parse
    new_lines=lines
  File "/root/.emacs.d/var/anaconda-mode/0.1.13/parso-0.3.2-py3.7.egg/parso/python/diff.py", line 236, in update
    self._copy_from_old_parser(line_offset, i2, j2)
  File "/root/.emacs.d/var/anaconda-mode/0.1.13/parso-0.3.2-py3.7.egg/parso/python/diff.py", line 289, in _copy_from_old_parser
    line_offset
  File "/root/.emacs.d/var/anaconda-mode/0.1.13/parso-0.3.2-py3.7.egg/parso/python/diff.py", line 598, in copy_nodes
    self.prefix,
  File "/root/.emacs.d/var/anaconda-mode/0.1.13/parso-0.3.2-py3.7.egg/parso/python/diff.py", line 686, in _copy_nodes
    assert last_line_offset_leaf == ':'
AssertionError

Apparently it got something to do with parser results caching/diffing. After some code reading/debugging of diff.py I decided to turn it off in jedi (jedi.settings.fast_parser = False) and problem disappeared for 3.7.2 as well.

I don't think it's high priority issue but it'd be nice if it'll be fixed at some point.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.