API reference sync sometimes break Docusaurus builds

[anakin87]

Whenever a core integration is released, the API reference is generated and a PR targeting Haystack is automatically created.

Problem

Sometimes the generated API reference causes the Docusaurus build to fail, even if the generated Markdown itself is valid. This happens with characters like { in docstrings and in other situations.

When this happens, the maintainer has to:

• Open another PR targeting core-integrations (and manually verify that Docusaurus can build successfully)
• Get the PR approved
• Manually re-run the "Core / Sync API reference with Docusaurus" workflow or re-release the integration
• Approve the new PR generated on Haystack

This process is manual and time-consuming, and mainly comes from the fact that core integrations API reference is hosted in the separate Haystack repo.

Ideas

I don't expect it to be easy to solve, but it would already help to offer maintainers a way to check if Docusaurus can build the website correctly.

One possible idea could be to automatically open a draft PR on Haystack for each PR on core integrations that changes the API reference and inform contributors they can inspect the preview there. Needs investigation.

[anakin87]

Another idea we can evaluate: clone Haystack docs folder and try to build the website in a workflow for the core-integrations PR. This would skip Vercel, opening draft PRs, etc.