Decorators have the same line number as the functions they decorate about python-driver

[juanjux]

The token of a function is the name, but the start and ending positions are the ones of the function node itself with body and all, not the token and obviously could overlap (if a node is inside a function it will overlap with it). This is the same for all nodes. The current UAST specification doesn't give a position for the tokens except when the token match the length of the node (variable names, etc.) so that part is a wontfix.

I'll take a look at the decorators, the native AST gives them the same position as the functions but maybe they can be fixed with the tokenizer.

[EgorBu]

but at least start position should be the same as in code - don't you think so?

[juanjux]

No, because in def somefunc or class SomeClass the start position is the position of def or class, not of somefunc.

[EgorBu]

so it will be position of def, class, self?
what about a.b? is the position of b == position of a?

[juanjux]

The code a.b produces this native AST. As you can see the col_offsets are different (1 and 3):

{'ast_type': 'Module',
 'body': [{'ast_type': 'Expr',
           'col_offset': 1,
           'end_col_offset': 3,
           'end_lineno': 1,
           'lineno': 1,
           'value': {'ast_type': 'Attribute',
                     'attr': 'b',
                     'col_offset': 3,
                     'ctx': 'Load',
                     'end_col_offset': 3,
                     'end_lineno': 1,
                     'lineno': 1,
                     'value': {'ast_type': 'Name',
                               'col_offset': 1,
                               'ctx': 'Load',
                               'end_col_offset': 1,
                               'end_lineno': 1,
                               'id': 'a',
                               'lineno': 1}}}],
 'col_offset': 1,
 'end_col_offset': 3,
 'end_lineno': 1,
 'lineno': 1}

[EgorBu]

can you explain about line number, col number, col offset, end col offset? Do we have it somewhere in bblfsh docs?

[juanjux]

This is an example of the native AST, you don't have to worry about it, it would be translated to StartPosition and EndPosition in the UAST (with some corrections).

[EgorBu]

for key, val in some_dict.items():
...

key & val has the same positions according to image.
Is it because of start position right after for?

[EgorBu]

In Repo2nBOW class definition:

MODEL_CLASS = NBOW

and it was converted to

MODEL_CLANBOW0m = NBOW

0m from color definition WHITE = "\033[0m", so it looks like that in this case position somehow is shifted.

[EgorBu]

btw - what is the right way to estimate a position of the token in code?

[juanjux]

Code:

class C():
    MODEL_CLANBOW = NBOW

Gives me an UAST with the columns 5 and 17 for MODEL_CLANBOW which is Vim is not lying to me are correct.

[juanjux]

Code:

for key, val in some_dict.items(): pass

Gives me this (reduced and sorted) UAST:

                     {
                        "Token" : "key",
                        "StartPosition" : {
                           "Line" : 1,
                           "Col" : 5,
                           "Offset" : 4
                        },
                        "EndPosition" : {
                           "Line" : 1,
                           "Col" : 7,
                           "Offset" : 6
                        }
                     },
                     {
                        "Token" : "val",
                        "StartPosition" : {
                           "Line" : 1,
                           "Col" : 10,
                           "Offset" : 9
                        },
                        "EndPosition" : {
                           "Line" : 1,
                           "Col" : 12,
                           "Offset" : 11,
                        },
                     }

Please check the UAST produced.

[EgorBu]

So initial code:

class Repo2nBOW(Repo2Base):
    """
    Implements the step repository -> :class:`ast2vec.nbow.NBOW`.
    """
    MODEL_CLASS = NBOW

and after processing:

class Repo2nBOW(Repo2Base):!Wrong length of token 'Repo2nBOW' (expected 22 - 7 + 1 == 9)!!token 'Repo2nBOW' start & end line: 14, 18!
    """
    Implements the step repository -> :class:`ast2vec.nbow.NBOW`.
    """
    MODEL_CLANBOW0m = NBOW

[juanjux]

@EgorBu I don't know where that 0m comes from but in my test the positions in the UAST are correct - check it on your own directly without the uast_playground and tell me if you have different positions in the UAST.

[juanjux]

I've tested that the decorators are correct in their line numbers. It's the FunctionDef parent object that starts in the previous line... but this is correct at least with regard to the native Python AST since the decorators are part of the function AST, so for the language the function starts where the decorator (instead of in the def where undecorated functions start).

[EgorBu]

It will be really nice to mention it in the documentation.
Because these details are not obvious and can make a lot of people confused.

[juanjux]

If you check the UAST (or the native AST), the decorator is a child of the function, and since is an AST parser and not a tokenizer it's logical that the start and end locations go from the first subnode node to the last.

Still, a list of things to consideer and caveats for each driver is something that could be good to have (we've already spoken about this in the last meeting but we still haven't had time for it).

[juanjux]

Reopening since the documentation clearly says that the StartPosition of a node with a Token is the start position of the Token.

[juanjux]

Should be fixed in v0.7.1 (update the client-python too), thanks for the report!