Post 1388 changes

[MegaIng]

This is a list of TODOs I would like to see/do after #1388 is merged (or independently, but reviewing that PR is why I am now making this list):

Doc changes

• Explain that _rules are inlined after parsing, not before parsing, preparing for potentially addition of __rule syntax, and preempting confusion about this (since this has semantic differences, especially with respect to lalr)
• Similar thing for templates: They are instantiated, not expanded. (and can be recursive)
• Check if we can move over to proper lark syntax highlighting instead of misusing perl.

Code changes

• Add terminal templates
• Add ability to %declare rules so that they can later be implemented with either %override or more importantly %extend, useful for plugin features.
• long term: Turn LarkValidator into something that can be used by load_grammar.py
• Move find_grammar_errors into lark_validator.py, and probably make it use lark.lark
• Extend find_grammar_errors to do semantic checks using LarkValidator (and therefore add the ability to LarkValidator to report multiple issues)
• longer term: Add a build step that turns lark.lark into a BNF representation that can be used by load_grammar.py - For example, a lark.bnf.json file loaded by the stdlib module. (We could potentially also "cache" the lalr parse tables in a json file. If we expose this build tool inside of lark.tools, together with a load function, this could be a more lightweight alternative to standalone)

[erezsh]

Is there anything I can do to get 1388 merged faster?

[MegaIng]

You currently still have an outstanding "Request Changing" review that I think Ross thinks is fully resolved. If you have changes other than the latest I suggested, you can already suggest them, otherwise you could add a review "Comment" to clear the "Requesting Changes" review status.

[erezsh]

I added a few comments.

[RossPatterson]

And I'm working through those comments.