ci: check that API references do not cause Docusaurus build failures

[anakin87]

Related Issues

• fixes API reference sync sometimes break Docusaurus builds #2816

Proposed Changes:

I started from the idea that the simplest way to check whether generated Markdown API references are compatible with Docusaurus is to check out the Docs Website and try building it. Then I found better tools, and this idea evolved into something simpler.

• Created a new "Core / Check API reference changes" workflow
  • detect integrations with API reference changes (if pydoc config or docstring checksum have changed)
  • for each one of the changed integrations:
    • generates Markdown API reference
    • moves it to a fresh website folder for testing
  • runs docusaurus-mdx-checker, as suggested in Docusaurus docs: this does not build the website but just checks generated Markdown files.
    • if it fails, the informative build error is available on CI logs

How did you test it?

Temporarily added some changes:

• valid config change for Pgvector + invalid Docstring in Qdrant -> clear error: https://github.com/deepset-ai/haystack-core-integrations/actions/runs/21941003098/job/63366427299?pr=2829
• new integration (pgvector2) -> triggers the check, which is OK -> https://github.com/deepset-ai/haystack-core-integrations/actions/runs/21941146095/job/63366923997?pr=2829

Notes for the reviewer

1. By running the checker but not building the website, we only get MD/MDX errors: for example, we don't get errors for duplicate page ids or broken links. In general, it seems OK since our previous errors were caused by invalid MD/MDX. Plus, the check is much faster than building the website.

2. With this change, we no longer need to run hatch run docs in each integration workflow:
   

   haystack-core-integrations/.github/workflows/aimlapi.yml

   Lines 57 to 59 in d964201

   	- name: Generate docs
   	if: matrix.python-version == '3.10' && runner.os == 'Linux'
   	run: hatch run docs

   
   I prefer to remove this step in another PR to keep this focused and minimal.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

[anakin87]

I found some simplification opportunities. Putting this in draft.

[julian-risch]

LGTM!