Rollup merge of #148183 - fmease:rustdoc-test-n-doc-doctest-test-harness, r=GuillaumeGomez

rustdoc: Test & document `test_harness` code block attribute

They were fully untested and undocumented previously despite being stable.
Context: [#t-rustdoc > `test_harness` langstr @ 💬](https://rust-lang.zulipchat.com/#narrow/channel/266220-t-rustdoc/topic/.60test_harness.60.20langstr/near/546748038).

While at it, I've also improved the documentation for code block attributes in general.
I was only inspired to do so because there was no good place for `test_harness` and because it got hard to skim due to a lack of subsubsections.

Author: jhpratt

src/doc/rustdoc/src/write-documentation/documentation-tests.md

@@ -1,7 +1,7 @@
 # Documentation tests
 
-`rustdoc` supports executing your documentation examples as tests. This makes sure
-that examples within your documentation are up to date and working.
+`rustdoc` supports executing your documentation examples as tests.
+This makes sure that examples within your documentation are up to date and working.
 
 The basic idea is this:
 
@@ -14,11 +14,13 @@ The basic idea is this:
 # fn f() {}
 ```
 
-The triple backticks start and end code blocks. If this were in a file named `foo.rs`,
-running `rustdoc --test foo.rs` will extract this example, and then run it as a test.
+Here, the triple backticks start and end the code block.
+If this were in a file named `foo.rs`, running `rustdoc --test foo.rs` will extract this example,
+and then run it as a test.
 
-Please note that by default, if no language is set for the block code, rustdoc
-assumes it is Rust code. So the following:
+Please note that by default,
+if no language is set for the block code, rustdoc assumes it is Rust code.
+So the following:
 
 ``````markdown
 ```rust
@@ -321,21 +323,43 @@ we can add the `#[macro_use]` attribute. Second, we’ll need to add our own
 
 ## Attributes
 
-Code blocks can be annotated with attributes that help `rustdoc` do the right
-thing when testing your code:
+Code blocks can be annotated with attributes that tell `rustdoc` how to build and interpret the test.
+They follow the [code fence] in the opening line.
+As such, they share the place with language strings like `rust` or `text`.
+Multiple attributes can be provided by separating them with commas, spaces or tabs.
+You can also write comments which are enclosed in parentheses `(…)`.
 
-The `ignore` attribute tells Rust to ignore your code. This is almost never
-what you want as it's the most generic. Instead, consider annotating it
-with `text` if it's not code or using `#`s to get a working example that
-only shows the part you care about.
+As alluded to in the introduction at the very top,
+unless you specify `rust` or something that isn't an attribute (except for `custom`),
+the code block is assumed to be Rust source code (and is syntax highlighted as such).
+
+You can of course add `rust` explicitly (like `rust,ignore`) if the Markdown is also consumed by
+other tools (e.g., if it's contained inside of a `README.md` that's included via `include_str`).
+
+### `ignore`
+
+The `ignore` attribute tells `rustdoc` to ignore your code. This is useful if you would like to
+have Rust syntax highlighting but the snippet is incomplete or pseudocode.
+It is customary to add the reason why it should be ignored in a `(…)` comment.
 
 ```rust
 /// ```ignore
 /// fn foo() {
 /// ```
+///
+/// ```ignore (needs extra dependency)
+/// use dependency::functionality;
+/// functionality();
+/// ```
 # fn foo() {}
 ```
 
+Do note that this is almost never what you want as it's the most generic.
+Instead, consider annotating it with `text` if it's not code or
+using `#`s to get a working example that only shows the part you care about.
+
+### `should_panic`
+
 `should_panic` tells `rustdoc` that the code should compile correctly but
 panic during execution. If the code doesn't panic, the test will fail.
 
@@ -346,6 +370,8 @@ panic during execution. If the code doesn't panic, the test will fail.
 # fn foo() {}
 ```
 
+### `no_run`
+
 The `no_run` attribute will compile your code but not run it. This is
 important for examples such as "Here's how to retrieve a web page,"
 which you would want to ensure compiles, but might be run in a test
@@ -361,10 +387,10 @@ used to demonstrate code snippets that can cause Undefined Behavior.
 # fn foo() {}
 ```
 
+### `compile_fail`
+
 `compile_fail` tells `rustdoc` that the compilation should fail. If it
-compiles, then the test will fail. However, please note that code failing
-with the current Rust release may work in a future release, as new features
-are added.
+compiles, then the test will fail.
 
 ```rust
 /// ```compile_fail
@@ -374,6 +400,13 @@ are added.
 # fn foo() {}
 ```
 
+<div class="warning">
+However, please note that code failing with the current Rust release may work in a future release,
+as new features are added!
+</div>
+
+### `edition…`
+
 `edition2015`, `edition2018`, `edition2021`, and `edition2024` tell `rustdoc`
 that the code sample should be compiled using the respective edition of Rust.
 
@@ -390,6 +423,8 @@ that the code sample should be compiled using the respective edition of Rust.
 # fn foo() {}
 ```
 
+### `standalone_crate`
+
 Starting in the 2024 edition[^edition-note], compatible doctests are merged as one before being
 run. We combine doctests for performance reasons: the slowest part of doctests is to compile them.
 Merging all of them into one file and compiling this new file, then running the doctests is much
@@ -441,7 +476,7 @@ should not be merged with the others. So the previous code should use it:
 In this case, it means that the line information will not change if you add/remove other
 doctests.
 
-### Ignoring targets
+### `ignore-…`: Ignoring targets
 
 Attributes starting with `ignore-` can be used to ignore doctests for specific
 targets. For example, `ignore-x86_64` will avoid building doctests when the
@@ -478,7 +513,7 @@ struct Foo;
 In older versions, this will be ignored on all targets, but starting with
 version 1.88.0, `ignore-x86_64` will override `ignore`.
 
-### Custom CSS classes for code blocks
+### `{…}` & `custom`: Custom CSS classes for code blocks
 
 ```rust
 /// ```custom,{class=language-c}
@@ -504,8 +539,9 @@ To be noted that you can replace `class=` with `.` to achieve the same result:
 pub struct Bar;
 ```
 
-To be noted, `rust` and `.rust`/`class=rust` have different effects: `rust` indicates that this is
-a Rust code block whereas the two others add a "rust" CSS class on the code block.
+To be noted, `rust` and `{.rust}` / `{class=rust}` have different effects:
+`rust` indicates that this is a Rust code block whereas
+the two others add a "rust" CSS class on the code block in the generated HTML.
 
 You can also use double quotes:
 
@@ -516,24 +552,42 @@ You can also use double quotes:
 pub struct Bar;
 ```
 
+### `test_harness`
+
+With `test_harness` applied, `rustdoc` will run any contained *test functions*
+instead of the (potentially implicit) `main` function.
+
+```rust
+//! ```test_harness
+//! #[test]
+//! #[should_panic]
+//! fn abc() { assert!(false); }
+//!
+//! #[test]
+//! fn xyz() { assert!(true); }
+//! ```
+```
+
+You can read more about *test functions* in [the Book][testing-book] or in [the Rust Reference][testing-ref].
+
+[testing-book]: ../../book/ch11-01-writing-tests.html
+[testing-ref]: ../../reference/attributes/testing.html
+
 ## Syntax reference
 
-The *exact* syntax for code blocks, including the edge cases, can be found
-in the [Fenced Code Blocks](https://spec.commonmark.org/0.29/#fenced-code-blocks)
-section of the CommonMark specification.
+The *exact* syntax for code blocks, including the edge cases,
+can be found in the [Fenced Code Blocks] section of the CommonMark specification.
 
-Rustdoc also accepts *indented* code blocks as an alternative to fenced
-code blocks: instead of surrounding your code with three backticks, you
-can indent each line by four or more spaces.
+Rustdoc also accepts *indented* code blocks as an alternative to fenced code blocks:
+Instead of surrounding your code with a [code fence] (e.g., three backticks),
+you can indent each line by four or more spaces.
 
 ``````markdown
     let foo = "foo";
     assert_eq!(foo, "foo");
 ``````
 
-These, too, are documented in the CommonMark specification, in the
-[Indented Code Blocks](https://spec.commonmark.org/0.29/#indented-code-blocks)
-section.
+These, too, are documented in the CommonMark specification, in the [Indented Code Blocks] section.
 
 However, it's preferable to use fenced code blocks over indented code blocks.
 Not only are fenced code blocks considered more idiomatic for Rust code,
@@ -595,3 +649,8 @@ operations within documentation test examples, such as `std::fs::read_to_string`
 The `--test-run-directory` flag allows controlling the run directory separately from the compilation directory.
 This is particularly useful in workspaces, where compiler invocations and thus diagnostics should be
 relative to the workspace directory, but documentation test examples should run relative to the crate directory.
+
+
+[code fence]: https://spec.commonmark.org/0.29/#code-fence
+[Fenced Code Blocks]: https://spec.commonmark.org/0.29/#fenced-code-blocks
+[Indented Code Blocks]: https://spec.commonmark.org/0.29/#indented-code-blocks

src/librustdoc/html/markdown.rs

@@ -846,11 +846,12 @@ pub(crate) enum Ignore {
     Some(Vec<String>),
 }
 
-/// This is the parser for fenced codeblocks attributes. It implements the following eBNF:
+/// This is the parser for fenced codeblocks attributes.
 ///
-/// ```eBNF
-/// lang-string = *(token-list / delimited-attribute-list / comment)
+/// It implements the following grammar as expressed in ABNF:
 ///
+/// ```ABNF
+/// lang-string = *(token-list / delimited-attribute-list / comment)
 /// bareword = LEADINGCHAR *(CHAR)
 /// bareword-without-leading-char = CHAR *(CHAR)
 /// quoted-string = QUOTE *(NONQUOTE) QUOTE
@@ -861,7 +862,7 @@ pub(crate) enum Ignore {
 /// attribute-list = [sep] attribute *(sep attribute) [sep]
 /// delimited-attribute-list = OPEN-CURLY-BRACKET attribute-list CLOSE-CURLY-BRACKET
 /// token-list = [sep] token *(sep token) [sep]
-/// comment = OPEN_PAREN *(all characters) CLOSE_PAREN
+/// comment = OPEN_PAREN *<all characters except closing parentheses> CLOSE_PAREN
 ///
 /// OPEN_PAREN = "("
 /// CLOSE_PARENT = ")"

tests/run-make/doctests-test_harness/doctests.rs

@@ -0,0 +1,26 @@
+// Test that we can successfully run two separate test suites.
+// Check that we run all four tests even though `ill` and `bad` both fail.
+
+//! ```test_harness
+//! #[test]
+//! fn well() {
+//!     assert!(true);
+//! }
+//!
+//! #[test]
+//! fn ill() {
+//!      assert!(false);
+//! }
+//! ```
+
+//! ```test_harness
+//! #[test]
+//! fn bad() {
+//!     assert!(false);
+//! }
+//!
+//! #[test]
+//! fn good() {
+//!     assert!(true);
+//! }
+//! ```

tests/run-make/doctests-test_harness/doctests.stdout

@@ -0,0 +1,64 @@
+
+running 2 tests
+test doctests.rs - (line 15) ... FAILED
+test doctests.rs - (line 4) ... FAILED
+
+failures:
+
+---- doctests.rs - (line 15) stdout ----
+Test executable failed ($STATUS).
+
+stdout:
+
+running 2 tests
+test bad ... FAILED
+test good ... ok
+
+failures:
+
+---- bad stdout ----
+
+thread 'bad' ($TID) panicked at doctests.rs:4:5:
+assertion failed: false
+note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
+
+
+failures:
+    bad
+
+test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in $TIME
+
+
+
+---- doctests.rs - (line 4) stdout ----
+Test executable failed ($STATUS).
+
+stdout:
+
+running 2 tests
+test ill ... FAILED
+test well ... ok
+
+failures:
+
+---- ill stdout ----
+
+thread 'ill' ($TID) panicked at doctests.rs:9:6:
+assertion failed: false
+note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
+
+
+failures:
+    ill
+
+test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in $TIME
+
+
+
+
+failures:
+    doctests.rs - (line 15)
+    doctests.rs - (line 4)
+
+test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in $TIME
+

tests/run-make/doctests-test_harness/rmake.rs

@@ -0,0 +1,41 @@
+//@ ignore-cross-compile (needs to run host tool binary)
+
+// Test the behavior of doctests that are marked `test_harness` and that contain multiple `#[test]`
+// functions. Sadly this needs to be a run-make test instead of rustdoc-UI one because at the time
+// of writing we can only pass `--test-threads=1` to the inner test suite using a runtool (needed to
+// guarantee deterministic test output) which we'd like to be written in Rust to be cross-platform.
+//
+// See also #157511 and `tests/rustdoc-ui/doctest/test_harness.rs`.
+
+use std::path::Path;
+
+use run_make_support::{diff, rustc, rustdoc};
+
+fn main() {
+    let doctests_path = Path::new("doctests.rs");
+    let runtool_path = Path::new("runtool.rs");
+
+    rustc().input(doctests_path).crate_type("lib").run();
+    rustc().input(runtool_path).run();
+
+    let output = rustdoc()
+        .input(doctests_path)
+        .arg("--test")
+        // for the outer test suite
+        .arg("--test-args=--test-threads=1")
+        .arg("--test-runtool")
+        .arg(Path::new(".").join(runtool_path).with_extension(std::env::consts::EXE_EXTENSION))
+        .arg("-L.")
+        .env("RUST_BACKTRACE", "0")
+        .run_fail();
+    output.assert_exit_code(101);
+    output.assert_stderr_equals("");
+
+    diff()
+        .expected_file(doctests_path.with_extension("stdout"))
+        .actual_text("stdout", output.stdout_utf8())
+        .normalize(r#"finished in \d+\.\d+s"#, "finished in $$TIME")
+        .normalize(r"thread '(?P<name>.*?)' \(\d+\) panicked", "thread '$name' ($$TID) panicked")
+        .normalize(r"Test executable failed \(.+?\)", "Test executable failed ($$STATUS)")
+        .run();
+}

tests/run-make/doctests-test_harness/runtool.rs

@@ -0,0 +1,11 @@
+// The whole purpose of this runtool is to pass `--test-threads=1`
+// to the inner testsuite to guarantee deterministic output.
+// See also #157511.
+
+fn main() {
+    let status = std::process::Command::new(std::env::args().nth(1).unwrap())
+        .arg("--test-threads=1")
+        .status()
+        .unwrap();
+    std::process::exit(status.code().unwrap())
+}