Add multiline comment support to .pt grammar (#42)

Support multiline comments using python-style docstring syntax

Add this to
- lark parsing
- vscode highlighting
- documentation
- unit tests (written by copilot, thanks)

Also add 2 orthogonal changes
- Extra debug for `VisitError`
- Remove unused block comments from syntax highlighting

Author: ahizzey

docs/syntax/constant.md

@@ -27,6 +27,11 @@ custom grammar:
             "Comments may be attached to values with a string following the definition"
         VALUE_B : constant[16] = 234
             "These are attached to the constant definitions"
+        VALUE_C: constant[2] = 3
+            """
+            Multiline comments can be used for long descriptions.
+            Use triple quotes for these like with Python docstrings.
+            """
     }
     ```
 

docs/syntax/docstrings.md

@@ -0,0 +1,66 @@
+Descriptions can be added to packtype definitions as shown below.
+The documentation is attached to the code in a way that will allow automated generation of documentation,
+as with Python docstrings.
+
+## Example
+
+Descriptions can be added with normal [Python docstrings]((https://peps.python.org/pep-0257/)) or with the Packtype custom grammar:
+
+=== "Python (.py)"
+
+    ```python linenums="1"
+    import packtype
+    from packtype import Constant
+
+    @packtype.package()
+    class MyPackage:
+        """
+        Package decription,
+        using normal Python docstrings
+        """
+        ...
+
+    @MyPackage.enum()
+    class Fruit:
+        """
+        Class description,
+        using normal Python docstrings
+        """
+        APPLE  : Constant
+        ORANGE : Constant
+        PEAR   : Constant
+        BANANA : Constant
+    ```
+
+=== "Packtype (.pt)"
+
+    ```sv linenums="1"
+    package my_package {
+        """
+        Package description
+        using multiline docstring syntax
+        """
+        enum fruit_t {
+            """
+            Class description
+            using multiline docstring syntax
+            """
+            @prefix=FRUIT
+            APPLE  : constant
+                "This is an apple"
+            ORANGE : constant
+                """
+                Member descriptions can also be multiline.
+                Use triple quotes for these.
+                """
+            PEAR   : constant
+                "This is a pear"
+            BANANA : constant
+                "This is a banana"
+        }
+
+    }
+    ```
+
+
+```

docs/syntax/package.md

@@ -22,7 +22,7 @@ custom grammar:
 
     ```sv linenums="1"
     package my_package {
-        "Description of what purpose this package serves"
+        """Description of what purpose this package serves"""
         // ...
     }
     ```

docs/syntax/scalar.md

@@ -39,6 +39,10 @@ custom grammar:
         TypeB : Scalar[TYPE_B_W]
             "They can be queried from the digested result"
         TypeC : Scalar[7]
+            """
+            Multiline comments can be used for long descriptions.
+            Use triple quotes for these like with Python docstrings.
+            """
     }
     ```
 

mkdocs.yml

@@ -43,6 +43,7 @@ nav:
     - Scalars: syntax/scalar.md
     - Structs: syntax/struct.md
     - Unions: syntax/union.md
+    - Docstrings: syntax/docstrings.md
   - Utilities:
     - Basic: utilities/basic.md
     - Constants: utilities/constant.md

packtype/grammar/grammar.py

@@ -8,7 +8,7 @@
 from pathlib import Path
 
 from lark import Lark
-from lark.exceptions import UnexpectedToken
+from lark.exceptions import UnexpectedToken, VisitError
 
 from ..common.logging import get_log
 from ..types.base import Base
@@ -93,6 +93,9 @@ def parse_string(
             f"Failed to parse {source.name if source else 'input'} on line {exc.line}: "
             f"\n\n{exc.get_context(definition)}\n{exc}"
         ) from exc
+    except VisitError as exc:
+        raise exc.orig_exc from exc
+
     # Gather declarations
     known_entities: dict[str, tuple[type[Base] | Constant, Position]] = {}
 

packtype/grammar/packtype.lark

@@ -18,9 +18,11 @@
 
 %import common.CNAME
 %import common.ESCAPED_STRING
+%import common._STRING_ESC_INNER
 %import common.INT
 %import common.SIGNED_INT
 %import common.WS
+%import common.NEWLINE
 %ignore WS
 
 // =============================================================================
@@ -42,7 +44,12 @@ dimension: "[" expr "]"
 dimensions: dimension+
 
 ?name: CNAME
-descr: ESCAPED_STRING
+
+// Multiline docstrings as with Python - start and end with three double-quotes
+_DOCSTRING_QUOTES: "\"\"\""
+ESCAPED_MULTILINE_DOCSTRING: _DOCSTRING_QUOTES (_STRING_ESC_INNER|NEWLINE)* _DOCSTRING_QUOTES
+
+descr: (ESCAPED_STRING | ESCAPED_MULTILINE_DOCSTRING)
 
 modifier: "@" name "=" (name | ESCAPED_STRING | NUMERIC)
 

packtype/grammar/transformer.py

@@ -3,6 +3,7 @@
 #
 
 import math
+import textwrap
 
 from lark import Transformer, v_args
 
@@ -85,7 +86,11 @@ def CNAME(self, body):  # noqa: N802
         return str(body)
 
     def descr(self, body):
-        return Description(str(body[0]).strip('"'))
+        """
+        Take description and trim surrounding quotes,
+        then remove common indentation and remove leading and trailing newlines.
+        """
+        return Description(textwrap.dedent(str(body[0]).strip('"')).strip("\n"))
 
     def modifier(self, body):
         return Modifier(*body)