✨ Add the eval-rst directive (#226)

This directive parses its contents as ReStructuredText, which integrates back into the rest of the document, e.g. for cross-referencing.

It provides a solution for both including rST files into Markdown, and utilising the sphinx autodoc directives (which have hard-coded rST aspects).

Co-authored-by: Chris Sewell <chrisj_sewell@hotmail.com>

Author: stephenroller

docs/conf.py

@@ -69,6 +69,7 @@
 myst_admonition_enable = True
 myst_html_img_enable = True
 myst_dmath_enable = True
+myst_url_schemes = ("http", "https", "mailto")
 
 
 def run_apidoc(app):
@@ -127,6 +128,7 @@ def run_apidoc(app):
     ("py:class", "docutils.nodes.Element"),
     ("py:class", "docutils.parsers.rst.directives.misc.Include"),
     ("py:class", "docutils.nodes.document"),
+    ("py:class", "docutils.parsers.rst.Parser"),
 ]
 
 

docs/index.md

@@ -15,7 +15,7 @@ and extensibility of Sphinx with the simplicity and readability of Markdown.
 MyST has the following main features:
 
 * **[A markdown parser for Sphinx](parse-with-sphinx)**. You can write your entire
-  {doc}`Sphinx documentation <sphinx:usage/quickstart>` in markdown.
+  {doc}`Sphinx documentation <sphinx:usage/quickstart>` in Markdown.
 * **[Call Sphinx directives and roles from within Markdown](syntax/directives)**,
   allowing you to extend your document via Sphinx extensions.
 * **[Extended Markdown syntax for useful rST features](extended-block-tokens)**, such

docs/using/howto.md

@@ -2,6 +2,49 @@
 
 This page describes several common uses of MyST parser and how to accomplish them.
 
+(howto/include-rst)=
+## Include rST files into a Markdown file
+
+As explained in [this section](syntax/directives/parsing), all MyST directives will parse their content as Markdown.
+Therefore, using the conventional `include` directive, will parse the file contents as Markdown:
+
+````md
+```{include} snippets/include-md.md
+```
+````
+
+```{include} snippets/include-md.md
+```
+
+To include rST, we must first "wrap" the directive in the [eval-rst directive](syntax/directives/parsing):
+
+````md
+```{eval-rst}
+.. include:: snippets/include-rst.rst
+```
+````
+
+```{eval-rst}
+.. include:: snippets/include-rst.rst
+```
+
+(howto/autodoc)=
+## Use `sphinx.ext.autodoc` in Markdown files
+
+The [sphinx.ext.autodoc](sphinx:sphinx.ext.autodoc) is currently hard-coded to write rST, and so can not be used as a conventional MyST directive.
+Instead the special [eval-rst directive](syntax/directives/parsing) can be used to "wrap" the autodoc directives:
+
+```{eval-rst}
+.. autoclass:: myst_parser.mocking.MockRSTParser
+    :show-inheritance:
+    :members: parse
+```
+
+As with other objects in MyST, this can then be referenced:
+
+- Using the role `` {py:class}`myst_parser.mocking.MockRSTParser` ``: {py:class}`myst_parser.mocking.MockRSTParser`
+- Using the Markdown syntax `[MockRSTParser](myst_parser.mocking.MockRSTParser)`: [MockRSTParser](myst_parser.mocking.MockRSTParser)
+
 ## Show backticks inside raw markdown blocks
 
 If you'd like to show backticks inside of your markdown, you can do so by nesting them
@@ -28,7 +71,7 @@ hi
 ```
 ````
 
-(autosectionlabel)=
+(howto/autosectionlabel)=
 ## Automatically create targets for section headers
 
 If you'd like to *automatically* generate targets for each of your section headers,

docs/using/intro.md

@@ -12,7 +12,7 @@ it in the Sphinx documentation engine.
 Installing the MyST parser provides access to two tools:
 
 * A MyST-to-docutils parser and renderer.
-* A Sphinx parser that utilizes the above tool in building your documenation.
+* A Sphinx parser that utilizes the above tool in building your documentation.
 
 To install the MyST parser, run the following in a
 [Conda environment](https://docs.conda.io) (recommended):
@@ -43,11 +43,11 @@ Naturally this site is generated with Sphinx and MyST!
 
 :::{admonition,tip} You can use both MyST and reStructuredText
 
-Activating the MyST parser will simply *enable* parsing markdown files with MyST, and the rST
-parser that ships with Sphinx by default will still work the same way. You can have
-combinations of both markdown and rST files in your documentation, and Sphinx will
-choose the right parser based on each file's extension. Sphinx features
-like cross-references will work just fine between the pages.
+Activating the MyST parser will simply *enable* parsing markdown files with MyST, and the rST parser that ships with Sphinx by default will still work the same way.
+You can have combinations of both markdown and rST files in your documentation, and Sphinx will choose the right parser based on each file's extension.
+Sphinx features like cross-references will work just fine between the pages.
+
+You can even inject raw rST into Markdown files! (see [this explanation](syntax/directives/parsing))
 :::
 
 :::{admonition,seealso} Want to add Jupyter Notebooks to your documentation?
@@ -146,9 +146,8 @@ For those who are familiar with reStructuredText, here is the equivalent in rST:
 ```
 
 Note that almost all documentation in the Sphinx ecosystem is written with
-reStructuredText (MyST is only a few months old). That means you'll likely see examples
-that have rST structure. You can modify any rST to work with MyST. Use this page,
-and [the syntax page](syntax) to help guide you.
+reStructuredText (MyST is only a few months old).
+That means you'll likely see examples that have rST structure. You can modify any rST to work with MyST. Use this page, and [the syntax page](syntax) to help guide you.
 ````
 
 As seen above, there are four main parts to consider when writing directives.

docs/using/snippets/include-md.md

@@ -0,0 +1 @@
+Hallo I'm from a Markdown file, [with a reference](howto/autodoc).

docs/using/snippets/include-rst.rst

@@ -0,0 +1 @@
+Hallo I'm from an rST file, :ref:`with a reference <howto/autodoc>`.

docs/using/syntax.md

@@ -18,10 +18,6 @@ and further details of a few major extensions from the CommonMark flavor of mark
 For an introduction to writing Directives and Roles with MyST markdown, see {ref}`intro/writing`.
 :::
 
-% ```{seealso}
-% {ref}`MyST Extended AST Tokens API <api/tokens>`
-% ```
-
 MyST builds on the tokens defined by markdown-it, to extend the syntax
 described in the [CommonMark Spec](https://spec.commonmark.org/0.29/), which the parser is tested against.
 
@@ -389,11 +385,13 @@ print(f'my {a}nd line')
 ```
 ````
 
+(syntax/directives/parsing)=
 ### How directives parse content
 
-Some directives parse the content that is in their content block. This
-means that MyST markdown can be written in the content areas of any directives written
-in MyST markdown. For example:
+Some directives parse the content that is in their content block.
+MyST parses this content **as Markdown**.
+
+This means that MyST markdown can be written in the content areas of any directives written in MyST markdown. For example:
 
 ````md
 ```{admonition} My markdown link
@@ -405,7 +403,7 @@ Here is [markdown link syntax](https://jupyter.org)
 Here is [markdown link syntax](https://jupyter.org)
 ```
 
-As a short-hand for directives that require no arguments, and when no paramter options are used (see below),
+As a short-hand for directives that require no arguments, and when no parameter options are used (see below),
 you may start the content directly after the directive name.
 
 ````md
@@ -416,10 +414,41 @@ you may start the content directly after the directive name.
 ```{note} Notes require **no** arguments, so content can start here.
 ```
 
+For special cases, MySt also offers the `eval-rst` directive.
+This will parse the content **as ReStructuredText**:
+
+````md
+```{eval-rst}
+.. figure:: img/fun-fish.png
+  :width: 100px
+  :name: rst-fun-fish
+
+  Party time!
+
+A reference from inside: ref:`rst-fun-fish`
+
+A reference from outside: :ref:`syntax/directives/parsing`
+```
+````
+
+```{eval-rst}
+.. figure:: img/fun-fish.png
+  :width: 100px
+  :name: rst-fun-fish
+
+  Party time!
+
+A reference from inside: ref:`rst-fun-fish`
+
+A reference from outside: :ref:`syntax/directives/parsing`
+```
+
+Note how the text is integrated into the rest of the document, so we can also reference [party fish](rst-fun-fish) anywhere else in the documentation.
+
 ### Nesting directives
 
-You can nest directives by ensuring that the ticklines corresponding to the
-outermost directive are longer than the ticklines for the inner directives.
+You can nest directives by ensuring that the tick-lines corresponding to the
+outermost directive are longer than the tick-lines for the inner directives.
 For example, nest a warning inside a note block like so:
 
 `````md
@@ -912,7 +941,7 @@ to them.
 ```{tip}
 If you'd like to *automatically* generate targets for each of your section headers,
 check out the [`autosectionlabel`](https://www.sphinx-doc.org/en/master/usage/extensions/autosectionlabel.html)
-sphinx feature. See {ref}`autosectionlabel` for more details.
+sphinx feature. See {ref}`howto/autosectionlabel` for more details.
 ```
 
 Target headers are defined with this syntax:

myst_parser/docutils_renderer.py

@@ -31,6 +31,7 @@
     MockStateMachine,
     MockingError,
     MockIncludeDirective,
+    MockRSTParser,
 )
 from .parse_directives import parse_directive_text, DirectiveParsingError
 from .parse_html import HTMLImgParser
@@ -363,7 +364,22 @@ def render_fence(self, token):
             token.info = token.info.strip()
         language = token.info.split()[0] if token.info else ""
 
-        if (
+        if not self.config.get("commonmark_only", False) and language == "{eval-rst}":
+            # copy necessary elements (source, line no, env, reporter)
+            newdoc = make_document()
+            newdoc["source"] = self.document["source"]
+            newdoc.settings = self.document.settings
+            newdoc.reporter = self.reporter
+            # pad the line numbers artificially so they offset with the fence block
+            pseudosource = ("\n" * token.map[0]) + token.content
+            # actually parse the rst into our document
+            MockRSTParser().parse(pseudosource, newdoc)
+            for node in newdoc:
+                if node["names"]:
+                    self.document.note_explicit_target(node, node)
+            self.current_node.extend(newdoc[:])
+            return
+        elif (
             not self.config.get("commonmark_only", False)
             and language.startswith("{")
             and language.endswith("}")

myst_parser/mocking.py

@@ -4,7 +4,7 @@
 from typing import List, Optional, Tuple, Type
 
 from docutils import nodes
-from docutils.parsers.rst import Directive, DirectiveError
+from docutils.parsers.rst import Directive, DirectiveError, Parser as RSTParser
 from docutils.parsers.rst.directives.misc import Include
 from docutils.parsers.rst.languages import get_language
 from docutils.parsers.rst.states import Inliner, RSTStateMachine, Body
@@ -438,3 +438,21 @@ def add_name(self, node):
                 del node["name"]
             node["names"].append(name)
             self.renderer.document.note_explicit_target(node, node)
+
+
+class MockRSTParser(RSTParser):
+    """RSTParser which avoids a negative side effect."""
+
+    def parse(self, inputstring: str, document: nodes.document):
+        """Parse the input to populate the document AST."""
+        from docutils.parsers.rst import roles
+
+        should_restore = False
+        if "" in roles._roles:
+            should_restore = True
+            blankrole = roles._roles[""]
+
+        super().parse(inputstring, document)
+
+        if should_restore:
+            roles._roles[""] = blankrole

setup.py

@@ -53,7 +53,12 @@
             "beautifulsoup4",
         ],
         # Note: This is only required for internal use
-        "rtd": ["sphinxcontrib-bibtex", "ipython", "sphinx-book-theme", "sphinx_tabs"],
+        "rtd": [
+            "sphinxcontrib-bibtex",
+            "ipython",
+            "sphinx-book-theme>=0.0.36",
+            "sphinx_tabs",
+        ],
     },
     zip_safe=True,
 )