incorrect ignore behavior about lark

[gerases]

Cool! Confirming the fix! Thanks much.

[erezsh]

Hi Gerases,
You are not doing anything wrong! Thanks to your simple example, I fixed a long-standing bug related to %ignore.
Check out the latest master branch and let me know if it works.
( using %ignore /^#.*$/m should work just fine as well )

[gerases]

I think something got broken with the fix though.

Because of the amount of the input it's hard to pinpoint what exactly is happening but the symptom is that what is a Cmnd_Alias in the file somehow became a host alias. I would need to reduce the input data to an absolute minimum to see exactly what is happening but what is definitely helping is going back to the previous commit :(

[gerases]

After quite a bit of reducing, I've established the minimum input to trigger the issue:

Cmnd_Alias C_AL1 = CMD1, \
  CMD2
#COMMENT1
Cmnd_Alias C_AL2 = CMD3
# ======> BLAH
# COMMENT2
Host_Alias H_AL = SOME_HOSTS
User_Alias U_AL = me, you, him

Removing the last line, for example, hides the issue. And the issue is that C_AL2 starts being part of the user_spec rule instead of a Command_Alias. Removing one of the comment lines also fixes the issue. Here's the grammar for reference one more time:

sudo_parser = Lark(r"""
    sudo_item : (alias | user_spec)*

    alias : "User_Alias"    user_alias  (":" user_alias)*
            | "Runas_Alias" runas_alias (":" runas_alias)*
            | "Host_Alias"  host_alias  (":" host_alias)*
            | "Cmnd_Alias"  cmnd_alias  (":" cmnd_alias)*

    user_alias  : ALIAS_NAME "=" user_list
    host_alias  : ALIAS_NAME "=" host_list
    runas_alias : ALIAS_NAME "=" runas_list
    cmnd_alias  : ALIAS_NAME "=" cmnd_list

    user_spec      : user_list host_list "=" cmnd_spec_list (":" host_list "=" cmnd_spec_list)*
    cmnd_spec_list : cmnd_spec ("," cmnd_spec)*
    cmnd_spec      : runas_spec? tag_spec? negator? command
    runas_spec     : "(" runas_list? (":" runas_list)? ")"
    tag_spec       : (tag_nopwd
                       | tag_pwd
                       | tag_noexec
                       | tag_exec
                       | tag_setenv
                       | tag_nosetenv
                       | tag_log_output
                       | tag_nolog_output) ":"
    ?command         : command_name | cmnd_alias_name
    command_name    : all | file_name (args)*
    all             : "ALL"
    file_name       : FILENAME
    args            : (arg)* | star | no_arg_string
    arg             : ARG
    star            : "*"
    no_arg_string   : "\"\""
    negator         : NEGATOR
    cmnd_alias_name : ALIAS_NAME

    host_list  : host      ("," host)*
    user_list  : user      ("," user)*
    runas_list : user      ("," user)*
    cmnd_list  : command   ("," command)*

    host : HOST_NAME

    user :    "!"*       USER_NAME
            | "!"* "%"   GROUP_NAME
            | "!"* "#"   UID
            | "!"* "%#"  GID
            | "!"* "+"   NETGROUP_NAME
            | "!"* "%:"  NONUNIX_GROUP_NAME
            | "!"* "%:#" NONUNIX_GID
            | "!"* user_alias

    tag_pwd          : "PASSWD"
    tag_nopwd        : "NOPASSWD"
    tag_exec         : "EXEC"
    tag_noexec       : "NOEXEC"
    tag_setenv       : "SETENV"
    tag_nosetenv     : "NOSETENV"
    tag_log_output   : "LOG_OUTPUT"
    tag_nolog_output : "NOLOG_OUTPUT"

    NEGATOR            : "!"
    UID                : /[0-9]+/
    GID                : /[0-9]+/
    NONUNIX_GID        : /[0-9]+/
    USER_NAME          : /[-_.a-z0-9A-Z]+/
    GROUP_NAME         : CNAME
    NETGROUP_NAME      : CNAME
    NONUNIX_GROUP_NAME : CNAME
    ALIAS_NAME         : CNAME
    HOST_NAME          : /[-_.a-z0-9A-Z\[\]*]+/
    FILENAME           : /\/[-._a-zA-Z0-9\/]+/
    ARG                : /[-{}\\\\:@;_.\/a-zA-Z0-9]+/

    %import common.CNAME
    %import common.WS
    %ignore /[\\\\]$/m
    %ignore /^#.*$/m
    %ignore /^Defaults.*$/m
    %ignore WS
""", start='sudo_item')

[erezsh]

When I run it, it seems just fine.
If you're encountering a problem, it's probably because your grammar is ambiguous (it can be parsed in more than one correct way), and the latest commit has shifted the default interpretation.

Try instanciating Lark with Lark(..., ambiguity='explicit') to see all the possible interpretations. If one of them is incorrect (or undesirable), you have two options:

1. Correct the grammar to allow only the desirable interpretation (the recommended way)
2. Get the full tree, and choose the correct one yourself.

Let me know if this helps.