UPGRADE: myst-parser 1.0 (#479)

Co-authored-by: Chris Holdgraf <choldgraf@berkeley.edu>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Angus Hollands <goosey15@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Holdgraf <choldgraf@gmail.com>
Co-authored-by: Josh Mitchell <yoshanuikabundi@gmail.com>
Co-authored-by: Michael Aye <michaelaye@users.noreply.github.com>
Co-authored-by: Jan-Hendrik Müller <44469195+kolibril13@users.noreply.github.com>
Co-authored-by: Oriol Abril-Pla <oriol.abril.pla@gmail.com>
Co-authored-by: mmcky <mamckay@gmail.com>
Co-authored-by: mmcky <mmcky@users.noreply.github.com>

Author: 10 people

.pre-commit-config.yaml

@@ -48,7 +48,7 @@ repos:
       args: [--config-file=pyproject.toml]
       additional_dependencies:
       - importlib_metadata
-      - myst-parser~=0.18.0
+      - myst-parser~=1.0.0
       - "sphinx~=5.0"
       - nbclient
       - types-PyYAML

CHANGELOG.md

@@ -92,7 +92,7 @@ WARNING: 'jupyter_execute_notebooks' is deprecated for 'nb_execution_mode' [myst
 
 `nb_render_priority` has been removed and replaced by `nb_mime_priority_overrides`, which has a different format and is more flexible. See [Outputs MIME priority](docs/render/format_code_cells.md) for more information.
 
-As per the changes in [`myst_parser`](myst:develop/_changelog), the `dollarmath` syntax extension is no longer included by default.
+As per the changes in [`myst_parser`](inv:myst#develop/_changelog), the `dollarmath` syntax extension is no longer included by default.
 To re-add this extension, ensure that it is specified in your `conf.py`: `myst_enable_extensions = ["dollarmath"]`.
 
 For cell-level configuration the top-level key `render` has now been deprecated for `mystnb`.
@@ -168,7 +168,7 @@ See [Embedding outputs as variables](docs/render/glue.md) for more details.
   - `nbconvert`
 - Updated:
   - `Python`: `3.6+ -> 3.7+`
-  - `myst_parser`: [`0.15 -> 0.17`](myst:develop/_changelog)
+  - `myst_parser`: [`0.15 -> 0.17`](inv:myst#develop/_changelog)
   - `jupyter-cache`: [`0.4 -> 0.5`](https://github.com/executablebooks/jupyter-cache/blob/master/CHANGELOG.md)
   - `sphinx-togglebutton`: [`0.1 -> 0.3`](https://sphinx-togglebutton.readthedocs.io/en/latest/changelog.html)
 

docs/authoring/custom-formats.Rmd

@@ -21,7 +21,7 @@ nb_custom_formats = {
 ```
 
 - The string should be a Python function that will be loaded by `import mylibrary.converter_function`
-- The function should take a file's contents (as a `str`) and return an [nbformat.NotebookNode](nbformat:api)
+- The function should take a file's contents (as a `str`) and return an [nbformat.NotebookNode](inv:nbformat#api)
 
 If the function takes additional keyword arguments, then you can specify these as dictionary in a second argument.
 For example this is what the default conversion would look like:

docs/authoring/jupyter-notebooks.md

@@ -16,13 +16,13 @@ Sphinx using the MyST parser.[^download]
 
 :::{seealso}
 For more information about what you can write with MyST Markdown, see the
-[MyST Parser documentation](myst:intro/get-started).
+[MyST Parser documentation](inv:myst#intro/get-started).
 :::
 
 ### Configuration
 
-The MyST-NB parser derives from [the base MyST-Parser](myst:intro/get-started), and so all the same configuration options are available.
-See the [MyST configuration options](myst:sphinx/config-options) for the full set of options, and [MyST syntax guide](myst:syntax/core) for all the syntax options.
+The MyST-NB parser derives from [the base MyST-Parser](inv:myst#intro/get-started), and so all the same configuration options are available.
+See the [MyST configuration options](inv:myst#sphinx/config-options) for the full set of options, and [MyST syntax guide](inv:myst#syntax/core) for all the syntax options.
 
 To build documentation from this notebook, the following options are set:
 
@@ -38,7 +38,7 @@ myst_url_schemes = ("http", "https", "mailto")
 ```
 
 :::{note}
-Loading the `myst_nb` extension also activates the [`myst_parser`](myst:index) extension, for enabling the MyST flavour of Markdown.
+Loading the `myst_nb` extension also activates the [`myst_parser`](inv:myst#index) extension, for enabling the MyST flavour of Markdown.
 It is not required to add this explicitly in the list of `extensions`.
 :::
 
@@ -53,7 +53,7 @@ For example, here's the MyST-NB logo:
 
 ![myst-nb logo](../_static/logo-wide.svg)
 
-By adding `"html_image"` to the `myst_enable_extensions` list in the sphinx configuration ([see here](myst:syntax/images)), you can even add HTML `img` tags with attributes:
+By adding `"html_image"` to the `myst_enable_extensions` list in the sphinx configuration ([see here](inv:myst#syntax/images)), you can even add HTML `img` tags with attributes:
 
 ```html
 <img src="../_static/logo-wide.svg" alt="logo" width="200px" class="shadow mb-2">
@@ -66,7 +66,7 @@ For example, here's a note admonition block:
 
 :::::{note}
 **Wow**, a note!
-It was generated with this code ([as explained here](myst:syntax/admonitions)):
+It was generated with this code ([as explained here](inv:myst:std:label#syntax/admonitions)):
 
 ````md
 :::{note}
@@ -77,7 +77,7 @@ It was generated with this code ([as explained here](myst:syntax/admonitions)):
 :::::
 
 If you wish to use "bare" LaTeX equations, then you should add `"amsmath"` to the `myst_enable_extensions` list in the sphinx configuration.
-This is [explained here](myst:syntax/amsmath), and works as such:
+This is [explained here](inv:myst:std:label#syntax/amsmath), and works as such:
 
 ```latex
 \begin{equation}
@@ -110,7 +110,7 @@ $$e^{i\pi} + 1 = 0$$ (euler)
 Euler's identity, equation {math:numref}`euler`, was elected one of the
 most beautiful mathematical formulas.
 
-You can see the syntax used for this example [here in the MyST documentation](myst:syntax/math).
+You can see the syntax used for this example [here in the MyST documentation](inv:myst:std:label#syntax/math).
 
 ## Code cells and outputs
 

docs/index.md

@@ -86,7 +86,7 @@ Build single or collections of documents into multiple formats (HTML, PDF, ...).
 
 MyST-NB is a module within the [Executable Books Project](https://executablebooks.org),
 an international collaboration to build open source tools that facilitate publishing computational narratives using the Jupyter ecosystem.
-It is also a core component of [Jupyter Book](jb:intro).
+It is also a core component of [Jupyter Book](inv:jb#intro).
 
 Check out the [Gallery of Jupyter Books](https://executablebooks.org/en/latest/gallery),
 for inspiration from across the community.

myst_nb/__init__.py

@@ -1,5 +1,5 @@
 """A docutils/sphinx parser for Jupyter Notebooks."""
-__version__ = "0.17.2"
+__version__ = "0.18.0"
 
 
 def setup(app):

myst_nb/core/config.py

@@ -1,6 +1,7 @@
 """Configuration for myst-nb."""
 import dataclasses as dc
 from enum import Enum
+import sys
 from typing import Any, Callable, Dict, Iterable, Optional, Sequence, Tuple
 
 from myst_parser.config.dc_validators import (
@@ -12,7 +13,13 @@
     optional,
     validate_fields,
 )
-from typing_extensions import Literal
+
+if sys.version_info >= (3, 8):
+    from typing import Literal
+else:
+    from typing_extensions import Literal  # noqa: F401
+
+from myst_nb.warnings_ import MystNBWarnings
 
 
 def custom_formats_converter(value: dict) -> Dict[str, Tuple[str, dict, bool]]:
@@ -130,7 +137,7 @@ def __post_init__(self):
         default_factory=dict,
         metadata={
             "help": "Custom formats for reading notebook; suffix -> reader",
-            "docutils_exclude": True,
+            "omit": ["docutils"],
             "sections": (Section.global_lvl, Section.read),
         },
     )
@@ -184,7 +191,7 @@ def __post_init__(self):
             "validator": deep_mapping(instance_of(str), instance_of(str)),
             "help": "Mapping of kernel name regex to replacement kernel name"
             "(applied before execution)",
-            "docutils_exclude": True,
+            "omit": ["docutils"],
             "sections": (Section.global_lvl, Section.execute),
         },
     )
@@ -228,7 +235,7 @@ def __post_init__(self):
             "validator": deep_iterable(instance_of(str)),
             "help": "Exclude (POSIX) glob patterns for notebooks",
             "legacy_name": "execution_excludepatterns",
-            "docutils_exclude": True,
+            "omit": ["docutils"],
             "sections": (Section.global_lvl, Section.execute),
         },
     )
@@ -395,7 +402,7 @@ def __post_init__(self):
             "help": "Overrides for the base render priority of mime types: "
             "list of (builder name, mime type, priority)",
             # TODO how to allow this in docutils?
-            "docutils_exclude": True,
+            "omit": ["docutils"],
             "sections": (Section.global_lvl, Section.file_lvl, Section.render),
         },
         repr=False,
@@ -462,7 +469,7 @@ def __post_init__(self):
         metadata={
             "validator": deep_mapping(instance_of(str), instance_of((str, int))),
             "help": "Options for image outputs (class|alt|height|width|scale|align)",
-            "docutils_exclude": True,
+            "omit": ["docutils"],
             # TODO backward-compatible change to "image_options"?
             "cell_key": "image",
             "sections": (
@@ -479,7 +486,7 @@ def __post_init__(self):
         metadata={
             "validator": deep_mapping(instance_of(str), instance_of((str, int))),
             "help": "Options for figure outputs (classes|name|caption|caption_before)",
-            "docutils_exclude": True,
+            "omit": ["docutils"],
             "cell_key": "figure",
             "sections": (
                 Section.global_lvl,
@@ -513,7 +520,7 @@ def __post_init__(self):
                 instance_of(str), deep_mapping(instance_of(str), instance_of(str))
             ),
             "help": "Javascript to be loaded on pages containing ipywidgets",
-            "docutils_exclude": True,
+            "omit": ["docutils"],
             "sections": (Section.global_lvl, Section.render),
         },
         repr=False,
@@ -575,7 +582,7 @@ def get_cell_level_config(
         self,
         field_name: str,
         cell_metadata: Dict[str, Any],
-        warning_callback: Callable[[str, str], Any],
+        warning_callback: Callable[[str, MystNBWarnings], Any],
     ) -> Any:
         """Get a configuration value at the cell level.
 
@@ -601,7 +608,7 @@ def get_cell_level_config(
             warning_callback(
                 f"Deprecated `cell_metadata_key` 'render' "
                 f"found, replace with {self.cell_metadata_key!r}",
-                "cell_metadata_key",
+                MystNBWarnings.CELL_METADATA_KEY,
             )
             cell_meta = cell_metadata["render"]
         else:
@@ -619,7 +626,10 @@ def get_cell_level_config(
                             field.metadata["validator"](self, field, value)
                     return value
             except Exception as exc:
-                warning_callback(f"Cell metadata invalid: {exc}", "cell_config")
+                warning_callback(
+                    f"Cell metadata invalid: {exc}",
+                    MystNBWarnings.CELL_CONFIG,
+                )
 
         # default/global/file level should have already been merged
         return getattr(self, field.name)

myst_nb/core/read.py

@@ -312,25 +312,22 @@ class _MockDirective:
 
 
 def _read_fenced_cell(token, cell_index, cell_type):
-    from myst_parser.parsers.directives import (
-        DirectiveParsingError,
-        parse_directive_text,
-    )
+    from myst_parser.parsers.directives import parse_directive_text
 
-    try:
-        _, options, body_lines, _ = parse_directive_text(
-            directive_class=_MockDirective,
-            first_line="",
-            content=token.content,
-            validate_options=False,
-        )
-    except DirectiveParsingError as err:
+    result = parse_directive_text(
+        directive_class=_MockDirective,
+        first_line="",
+        content=token.content,
+        validate_options=False,
+    )
+    if result.warnings:
         raise MystMetadataParsingError(
             "{} cell {} at line {} could not be read: {}".format(
-                cell_type, cell_index, token.map[0] + 1, err
+                cell_type, cell_index, token.map[0] + 1, result.warnings[0]
             )
         )
-    return options, body_lines
+
+    return result.options, result.body
 
 
 def _read_cell_metadata(token, cell_index):

myst_nb/core/render.py

@@ -28,8 +28,9 @@
 
 from myst_nb.core.config import NbParserConfig
 from myst_nb.core.execute import NotebookClientBase
-from myst_nb.core.loggers import DEFAULT_LOG_TYPE, LoggerType
+from myst_nb.core.loggers import LoggerType  # DEFAULT_LOG_TYPE,
 from myst_nb.core.utils import coalesce_streams
+from myst_nb.warnings_ import MystNBWarnings, create_warning
 
 if TYPE_CHECKING:
     from markdown_it.tree import SyntaxTreeNode
@@ -57,7 +58,6 @@ class MditRenderMixin:
     # required by mypy
     md_options: dict[str, Any]
     document: nodes.document
-    create_warning: Any
     render_children: Any
     add_line_and_source_path: Any
     add_line_and_source_path_r: Any
@@ -95,8 +95,8 @@ def get_cell_level_config(
         :param cell_metadata: the metadata for the cell
         """
 
-        def _callback(msg: str, subtype: str):
-            self.create_warning(msg, line=line, subtype=subtype)
+        def _callback(msg: str, subtype: MystNBWarnings):
+            create_warning(self.document, msg, line=line, subtype=subtype)
 
         return self.nb_config.get_cell_level_config(field, cell_metadata, _callback)
 
@@ -222,10 +222,11 @@ def _get_nb_source_code_lexer(
             # TODO this will create a warning for every cell, but perhaps
             # it should only be a single warning for the notebook (as previously)
             # TODO allow user to set default lexer?
-            self.create_warning(
+            create_warning(
+                self.document,
                 f"No source code lexer found for notebook cell {cell_index + 1}",
-                wtype=DEFAULT_LOG_TYPE,
-                subtype="lexer",
+                # wtype=DEFAULT_LOG_TYPE,
+                subtype=MystNBWarnings.LEXER,
                 line=line,
                 append_to=self.current_node,
             )
@@ -310,11 +311,6 @@ class MimeData:
     """Index of the output in the cell"""
     line: int | None = None
     """Source line of the cell"""
-    md_headings: bool = False
-    """Whether to render headings in text/markdown blocks."""
-    # we can only do this if know the content will be rendered into the main body
-    # of the document, e.g. not inside a container node
-    # (otherwise it will break the structure of the AST)
 
     @property
     def string(self) -> str:
@@ -598,9 +594,7 @@ def render_markdown(self, data: MimeData) -> list[nodes.Element]:
         fmt = self.renderer.get_cell_level_config(
             "render_markdown_format", data.cell_metadata, line=data.line
         )
-        return self._render_markdown_base(
-            data, fmt=fmt, inline=False, allow_headings=data.md_headings
-        )
+        return self._render_markdown_base(data, fmt=fmt, inline=False)
 
     def render_text_plain(self, data: MimeData) -> list[nodes.Element]:
         """Render a notebook text/plain mime data output."""
@@ -753,9 +747,7 @@ def render_markdown_inline(self, data: MimeData) -> list[nodes.Element]:
         fmt = self.renderer.get_cell_level_config(
             "render_markdown_format", data.cell_metadata, line=data.line
         )
-        return self._render_markdown_base(
-            data, fmt=fmt, inline=True, allow_headings=data.md_headings
-        )
+        return self._render_markdown_base(data, fmt=fmt, inline=True)
 
     def render_text_plain_inline(self, data: MimeData) -> list[nodes.Element]:
         """Render a notebook text/plain mime data output."""
@@ -796,7 +788,7 @@ def render_widget_view_inline(self, data: MimeData) -> list[nodes.Element]:
         return self.render_widget_view(data)
 
     def _render_markdown_base(
-        self, data: MimeData, *, fmt: str, inline: bool, allow_headings: bool
+        self, data: MimeData, *, fmt: str, inline: bool
     ) -> list[nodes.Element]:
         """Base render for a notebook markdown mime output (block or inline)."""
         psuedo_element = nodes.Element()  # element to hold the parsed markdown
@@ -832,7 +824,6 @@ def _render_markdown_base(
                     data.string,
                     data.line or 0,
                     inline=inline,
-                    allow_headings=allow_headings,
                 )
         finally:
             # restore the parser
@@ -986,11 +977,12 @@ def create_figure_context(
             caption.source = self.document["source"]
             caption.line = line
         elif not (isinstance(first_node, nodes.comment) and len(first_node) == 0):
-            self.create_warning(
+            create_warning(
+                self.document,
                 "Figure caption must be a paragraph or empty comment.",
                 line=line,
-                wtype=DEFAULT_LOG_TYPE,
-                subtype="fig_caption",
+                # wtype=DEFAULT_LOG_TYPE,
+                subtype=MystNBWarnings.FIG_CAPTION,
             )
 
     self.current_node.append(figure_node)