Skip to content

Handle external links during mdbook-linkcheck better during CI #21

@sshine

Description

@sshine

Since the age of AI, https://developer.hashicorp.com/ and https://www.terraform.io/ have decided to consistently throttle our linkchecker. That means running nix run .#serve fails every time because of 429 Too Many Requests. The short-term solution is to disable linkcheck so that mdbook serve works. The medium-term solution is to decouple linkchecking and serving so that you can still run linkcheck, but separate from serve. The long-term solution is to keep linkchecking in CI, but handle 429s as "pass" in CI, since we cannot confirm or deny that it's a valid link; perhaps look into another crawler, since the site in question doesn't explicitly deny crawlers.

$ nix run .#serve
2025-09-11 18:05:50 [INFO] (mdbook::book): Book building has started
2025-09-11 18:05:50 [INFO] (mdbook::book): Running the html backend
2025-09-11 18:05:50 [INFO] (mdbook::book): Running the linkcheck backend
2025-09-11 18:05:50 [INFO] (mdbook::renderer): Invoking the "linkcheck" renderer
error: Server returned 429 Too Many Requests for https://developer.hashicorp.com/terraform/language/syntax/json
  ┌─ index.md:3:55
  │
3 │ Terranix works as a replacement for HCL by generating [Terraform JSON][tf-json].
  │                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^ Server returned 429 Too Many Requests for https://developer.hashicorp.com/terraform/language/syntax/json

error: Server returned 429 Too Many Requests for https://developer.hashicorp.com/terraform/language/providers
   ┌─ index.md:12:3
   │
12 │   [Terraform reference material](https://www.terraform.io/docs/providers/index.html)
   │   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Server returned 429 Too Many Requests for https://developer.hashicorp.com/terraform/language/providers

error: Server returned 429 Too Many Requests for https://developer.hashicorp.com/terraform
  ┌─ documentation/what-is-terranix.md:6:1
  │
6 │ [Terraform](https://terraform.io) is a tool to interact with APIs via declarative files.
  │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Server returned 429 Too Many Requests for https://developer.hashicorp.com/terraform

error: Server returned 429 Too Many Requests for https://developer.hashicorp.com/terraform/language/providers
   ┌─ documentation/what-is-terranix.md:21:1
   │
21 │ [the Terraform website](https://www.terraform.io/docs/providers/index.html).
   │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Server returned 429 Too Many Requests for https://developer.hashicorp.com/terraform/language/providers

error: Server returned 429 Too Many Requests for https://developer.hashicorp.com/terraform/language/syntax/json
   ┌─ documentation/what-is-terranix.md:25:37
   │
25 │ Here is an example of how you would [define multiple providers of the same kind](https://www.terraform.io/language/syntax/json#provider-blocks)
   │                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Server returned 429 Too Many Requests for https://developer.hashicorp.com/terraform/language/syntax/json

error: Server returned 429 Too Many Requests for https://developer.hashicorp.com/terraform/language/expressions
   ┌─ documentation/terranix-vs-hcl.md:82:1
   │
82 │ [multi line strings](https://www.terraform.io/docs/configuration/expressions.html#string-literals)
   │ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Server returned 429 Too Many Requests for https://developer.hashicorp.com/terraform/language/expressions

Error: One or more incorrect links
2025-09-11 18:05:54 [ERROR] (mdbook::renderer): Renderer exited with non-zero return code.
2025-09-11 18:05:54 [ERROR] (mdbook::utils): Error: Rendering failed
2025-09-11 18:05:54 [ERROR] (mdbook::utils):    Caused By: The "linkcheck" renderer failed

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions