Skip to content

Simplify docstring pipeline: python emits the <Docstring> component directly - #797

Merged
mishig25 merged 2 commits into
mainfrom
simplify-docstring-pipeline
Jul 4, 2026
Merged

Simplify docstring pipeline: python emits the <Docstring> component directly#797
mishig25 merged 2 commits into
mainfrom
simplify-docstring-pipeline

Conversation

@mishig25

@mishig25 mishig25 commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

What

Docstring rendering used to go through two regex transformation layers: autodoc.py serialized metadata into custom tags (<docstring><name>…</name><anchor>…</anchor><parameters>…</parameters>…<paramgroups>N</paramgroups>), and kit/preprocessors/docstring.js regex-parsed them all back out to build a <Docstring …/> component.

Now python emits the component directly: the <Docstring> open tag carries all metadata props as final values (name/anchor/source/parameters/isGetSetDescriptor, with MDX escaping undone since JS string/JSON props need none), and the component body carries only the markdown-bearing sections (parameter descriptions, return/yield/raise types & descriptions). The kit preprocessor's only remaining job is rendering those sections with mdsvex into the remaining props — it no longer parses names, anchors, sources, signatures, or group counts.

Why the JS stage still exists at all

The prop values are rendered markdown — they must go through the same mdsvex pipeline (remark + custom hljs highlighter + KaTeX) as the rest of the page, and that renderer lives in JS. Likewise the rendered-<ul> parsing stays (now documented in a dedicated helper): parameter lists must be rendered as one markdown list to preserve exact tight/loose list semantics, then parsed back into per-parameter structured data for the tooltips/anchors.

Also fixes a latent bug

Python emits <yielddesc>, but the old regex matched <yieldesc> — yield descriptions have been silently dropped forever. The tag names now match, so docstrings with Yields: sections will (correctly) start showing their description. This doesn't affect accelerate (no rendered yields), but other libraries may gain previously-missing content.

Compatibility note

The tag format is an internal contract between the python package and kit/; the doc-build workflows always build both from the same repo checkout, so the change is atomic. Mixing an old PyPI hf-doc-builder with a newer kit checkout (or vice versa) would break docstring rendering — same caveat as any change to this format.

Verification

  • Byte-identical output: full accelerate e2e build compared against the pre-change build — 48/48 pages byte-identical, normalizing only hashed asset paths, the kit build timestamp, and nondeterministic python <object object at 0x…> addresses. (accelerate is docstring-dense: 15 package_reference pages.)
  • pytest tests/ passing (expectations updated for the new format; the one test_style_doc failure is pre-existing on main in my env and unrelated).
  • eslint/prettier clean, including fixing the no-shadow violations the old file carried.

Second commit: a one-hunk svelte.config.js change (state_referenced_locally warning suppression) that was verified as part of #795 but accidentally left out of that PR (git add kit/src missed the config at kit/ root).

🤖 Generated with Claude Code

mishig25 and others added 2 commits July 4, 2026 16:02
autodoc.py previously serialized docstring metadata into a zoo of custom
tags (<docstring><name>...<paramgroups>N) that kit/preprocessors/
docstring.js regex-parsed back into a <Docstring .../> component. Python
now emits the component open tag with all metadata props inline
(name/anchor/source/parameters/isGetSetDescriptor as final JSON values,
MDX escaping undone since JS strings need none); the component body
carries only the markdown-bearing sections, which the kit preprocessor
renders with mdsvex into the remaining props and closes the component.

- parameter groups become nested <paramsgroup> blocks instead of numbered
  tags + a count
- the rendered-<ul> parsing stays (documented): parameter lists must be
  rendered as one markdown list to keep exact tight/loose list semantics
- fixes a long-standing tag mismatch: python emits <yielddesc> but the
  old regex matched <yieldesc>, silently dropping yield descriptions

Verified byte-identical output (48/48 pages, modulo hashed asset names
and nondeterministic python object addresses) on the accelerate e2e
build; python test suite updated and passing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This hunk was part of the runes migration verification but was left out
of #795 (staged with `git add kit/src`, which misses kit/svelte.config.js).
The flagged initial-value captures have the same semantics as the
pre-runes svelte 4 code — doc pages pass static props — and the warning
floods every downstream doc build log otherwise.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mishig25
mishig25 merged commit 2f7376e into main Jul 4, 2026
5 checks passed
@mishig25
mishig25 deleted the simplify-docstring-pipeline branch July 4, 2026 14:25
mishig25 added a commit that referenced this pull request Jul 31, 2026
`strip_html_from_markdown` still looked for the legacy
`<div class="docstring">...<docstring>...</docstring>` markup, which
`autodoc` stopped emitting in #797 (it now emits the `<Docstring>`
component with metadata as props). Nothing matched, so the whole opening
tag was deleted by the generic tag cleanup -- taking the name, anchor and
signature with it -- and the `**Parameters:**` / `**Returns:**` labels
vanished too, gluing the return type onto the last parameter line.

Parse the current form instead: props from the opening tag (whose
attribute values are single-line JSON, decoded with `raw_decode` so `}`
and `>` inside type annotations can't truncate it), sections from the
component body. The signature is rendered from the `parameters` prop, and
the parameter groups / yields / raises sections -- previously dropped
entirely -- are emitted as well. Types are only wrapped in backticks when
they carry no markup, since that would break a resolved `[Name](url)` doc
link.

Section order (signature, parameters, returns, then the description that
follows the component) matches how `Docstring.svelte` renders the page.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant