docs: execute examples via myst-nb; native tables and validated refs

Removes the last RST-syntax islands from the converted MyST markdown so
the docs are markdown-native for both human and LLM authors.

Executable examples (A): replace IPython.sphinxext.ipython_directive with
myst-nb. The 83 `{eval-rst}` + `.. ipython:: python` blocks become native
`{code-cell} ipython3` blocks, and the 14 pages that carry them gain
jupytext/kernelspec front matter so myst-nb runs them. conf.py routes .md
through myst-nb with nb_execution_mode="force" and
nb_execution_raise_on_error=True, so a failing example now fails the build.

myst-nb gives each page its own kernel instead of the IPython directive's
single namespace shared across all documents in build order. That isolation
surfaced expressions.md, which only ever worked by inheriting `col`/`lit`
from an earlier-built page — it now imports them itself. It also changes the
execution working directory to each page's own folder, so build.sh symlinks
the example data next to every page that reads it by relative name and
registers the python3 kernel; CI now calls build.sh so it matches local.

Tables (B): the 3 `.. list-table::` directives become GFM markdown tables.

Cross-references (C): the two intra-page links in distributing-work.md that
the conversion left as undefined markdown references (and that built green
while rendering literal brackets) become `{ref}` roles backed by explicit
`(label)=` targets, so a future break fails the build instead of shipping
silently.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: timsaucer

.github/workflows/build.yml

@@ -552,9 +552,10 @@ jobs:
         run: |
           set -x
           cd docs
-          curl -O https://gist.githubusercontent.com/ritchie46/cac6b337ea52281aa23c049250a4ff03/raw/89a957ff3919d90e6ef2d34235e6bf22304f3366/pokemon.csv
-          curl -O https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2021-01.parquet
-          uv run --no-project make html
+          # build.sh downloads the example data, registers the Jupyter kernel
+          # myst-nb needs, symlinks the data next to each executed page, and
+          # runs sphinx. Using it here keeps CI identical to a local build.
+          uv run --no-project bash ./build.sh
 
       - name: Copy & push the generated HTML
         if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref_type == 'tag')

docs/build.sh

@@ -36,6 +36,21 @@ rm -rf build 2> /dev/null
 rm -rf temp 2> /dev/null
 mkdir temp
 cp -rf source/* temp/
+
+# myst-nb executes each page as a notebook from the directory that page
+# lives in, so the example data files must sit alongside every page that
+# loads them by relative name (e.g. `ctx.read_csv("pokemon.csv")`). Symlink
+# them into each directory that has such a page rather than copying the
+# 20 MB parquet repeatedly.
+for d in temp temp/user-guide temp/user-guide/common-operations; do
+    ln -sf "$script_dir/pokemon.csv" "$d/pokemon.csv"
+    ln -sf "$script_dir/yellow_tripdata_2021-01.parquet" "$d/yellow_tripdata_2021-01.parquet"
+done
+
+# myst-nb runs `{code-cell}` blocks against a Jupyter kernel named "python3".
+# Register the active environment's interpreter as that kernel (idempotent).
+python -m ipykernel install --sys-prefix --name python3 --display-name "Python 3"
+
 make SOURCEDIR=`pwd`/temp html
 
 cd "$original_dir" || exit

docs/source/conf.py

@@ -48,20 +48,33 @@
 extensions = [
     "sphinx.ext.mathjax",
     "sphinx.ext.napoleon",
-    "myst_parser",
-    "IPython.sphinxext.ipython_directive",
+    # myst_nb is a superset of myst_parser: it provides the MyST markdown
+    # parser plus executable `{code-cell}` notebook directives. Do NOT also
+    # list "myst_parser" — myst_nb activates it internally and listing both
+    # raises an extension conflict.
+    "myst_nb",
     "autoapi.extension",
 ]
 
 # NOTE: .rst stays alongside .md because sphinx-autoapi generates RST
 # under autoapi/ and Sphinx needs the suffix to parse it. The human-
-# authored docs are all MyST .md now; the .rst entry is only for the
-# autoapi build artifacts.
+# authored docs are all MyST .md now. ".md" is routed through myst-nb so
+# pages carrying jupytext/kernelspec front matter execute their
+# `{code-cell}` blocks; pages without that front matter render as plain
+# MyST markdown. The ".rst" entry is only for the autoapi build artifacts.
 source_suffix = {
     ".rst": "restructuredtext",
-    ".md": "markdown",
+    ".md": "myst-nb",
 }
 
+# Execute notebook code cells at build time and fail the build if any cell
+# raises — this replaces the old IPython sphinx directive, whose executed
+# examples are now `{code-cell}` blocks. "force" re-executes every build so
+# stale cached output can never ship.
+nb_execution_mode = "force"
+nb_execution_timeout = 120
+nb_execution_raise_on_error = True
+
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ["_templates"]
 

docs/source/index.md

@@ -1,3 +1,12 @@
+---
+jupytext:
+  text_representation:
+    extension: .md
+    format_name: myst
+kernelspec:
+  name: python3
+  display_name: Python 3
+---
 <!---
   Licensed to the Apache Software Foundation (ASF) under one
   or more contributor license agreements.  See the NOTICE file
@@ -39,16 +48,14 @@ pip install datafusion
 
 ## Example
 
-```{eval-rst}
-.. ipython:: python
+```{code-cell} ipython3
+from datafusion import SessionContext
 
-    from datafusion import SessionContext
+ctx = SessionContext()
 
-    ctx = SessionContext()
+df = ctx.read_csv("pokemon.csv")
 
-    df = ctx.read_csv("pokemon.csv")
-
-    df.show()
+df.show()
 
 ```
 

docs/source/user-guide/basics.md

@@ -1,3 +1,12 @@
+---
+jupytext:
+  text_representation:
+    extension: .md
+    format_name: myst
+kernelspec:
+  name: python3
+  display_name: Python 3
+---
 <!---
   Licensed to the Apache Software Foundation (ASF) under one
   or more contributor license agreements.  See the NOTICE file
@@ -25,22 +34,20 @@ In this section, we will cover a basic example to introduce a few key concepts.
 2021 Yellow Taxi Trip Records ([download](https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2021-01.parquet)),
 from the [TLC Trip Record Data](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page).
 
-```{eval-rst}
-.. ipython:: python
+```{code-cell} ipython3
+from datafusion import SessionContext, col, lit, functions as f
 
-    from datafusion import SessionContext, col, lit, functions as f
+ctx = SessionContext()
 
-    ctx = SessionContext()
-
-    df = ctx.read_parquet("yellow_tripdata_2021-01.parquet")
+df = ctx.read_parquet("yellow_tripdata_2021-01.parquet")
 
-    df = df.select(
-        "trip_distance",
-        col("total_amount").alias("total"),
-        (f.round(lit(100.0) * col("tip_amount") / col("total_amount"), lit(1))).alias("tip_percent"),
-    )
+df = df.select(
+    "trip_distance",
+    col("total_amount").alias("total"),
+    (f.round(lit(100.0) * col("tip_amount") / col("total_amount"), lit(1))).alias("tip_percent"),
+)
 
-    df.show()
+df.show()
 ```
 
 ## Session Context