Skip to content

Commit c9ed49c

Browse files
anakin87anakin87
authored
docs: sync API reference updates with Docusaurus on release (#385)
* docusaurus configs * wip * api ref generation * retry * righ dir * progress * btter script * try creating a PR * fix acorn error * simplify script * new improvements * clean up * try again * clean again
1 parent ce0a09c commit c9ed49c

12 files changed

Lines changed: 376 additions & 2 deletions

File tree

‎.github/utils/pydoc-markdown.sh‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,24 @@
11
#!/bin/bash
22

3+
# Usage: ./pydoc-markdown.sh [CONFIG_PATH]
4+
#
5+
# Generate documentation from pydoc-markdown config files.
6+
#
7+
# Examples:
8+
# ./pydoc-markdown.sh # Uses default path: ../config/*
9+
# ./pydoc-markdown.sh ../config/api/* # Uses custom path
10+
# ./pydoc-markdown.sh /path/to/configs/* # Uses absolute path
11+
312
set -e # Fails on any error in the following loop
13+
14+
# Set default config path or use provided parameter
15+
CONFIG_PATH="${1:-../config/*}"
16+
417
cd docs/pydoc
518
rm -rf temp && mkdir temp
619
cd temp
7-
for file in ../config/* ; do
20+
echo "Processing config files in $CONFIG_PATH"
21+
for file in $CONFIG_PATH ; do
822
echo "Converting $file..."
923
pydoc-markdown "$file"
1024
done

‎.github/workflows/docusaurus_sync.yml‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
name: Sync API reference with Docusaurus
2+
3+
on:
4+
push:
5+
tags:
6+
- "v[0-9]+.[0-9]+.[0-9]+*"
7+
8+
# Manual trigger: Use this ONLY if the automatic API reference sync failed during a release.
9+
# This will regenerate and sync the API reference to the Haystack repository.
10+
# WARNING: Running this workflow when this repository has evolved significantly since the release
11+
# will sync an incorrect API reference to Haystack.
12+
workflow_dispatch:
13+
14+
jobs:
15+
generate-api-reference:
16+
runs-on: ubuntu-latest
17+
18+
steps:
19+
- name: Checkout this repo
20+
uses: actions/checkout@v5
21+
22+
- name: Set up Python
23+
uses: actions/setup-python@v6
24+
with:
25+
python-version: "3.10"
26+
27+
- name: Install dependencies
28+
run: |
29+
python -m pip install --upgrade pip
30+
pip install -U haystack-pydoc-tools
31+
32+
- name: Generate API reference
33+
run: ./.github/utils/pydoc-markdown.sh "../config_docusaurus/*"
34+
35+
- name: Upload API reference artifact
36+
uses: actions/upload-artifact@v4
37+
with:
38+
name: experimental-api-reference
39+
path: docs/pydoc/temp
40+
if-no-files-found: error
41+
retention-days: 1
42+
overwrite: true
43+
44+
45+
sync-api-reference:
46+
runs-on: ubuntu-latest
47+
needs: generate-api-reference
48+
49+
steps:
50+
- name: Checkout Haystack repo
51+
uses: actions/checkout@v5
52+
with:
53+
repository: deepset-ai/haystack
54+
ref: main
55+
token: ${{ secrets.HAYSTACK_BOT_TOKEN }}
56+
57+
- name: Set up Python
58+
uses: actions/setup-python@v6
59+
with:
60+
python-version: "3.10"
61+
62+
- name: Download API reference artifact
63+
uses: actions/download-artifact@v4
64+
with:
65+
name: experimental-api-reference
66+
path: temp_api_reference
67+
68+
- name: Sync API reference
69+
run: |
70+
# Function to sync generated API reference to a destination
71+
sync_to_dest() {
72+
echo "Syncing to $1"
73+
mkdir -p "$1"
74+
rsync -av --delete --exclude='.git/' "temp_api_reference/" "$1/"
75+
}
76+
77+
# Sync to main reference
78+
sync_to_dest "docs-website/reference/experiments-api"
79+
80+
# Sync to all versioned directories
81+
if [ -d "docs-website/reference_versioned_docs" ]; then
82+
for version_dir in "docs-website/reference_versioned_docs"/version-*; do
83+
[ -d "$version_dir" ] && sync_to_dest "$version_dir/experiments-api"
84+
done
85+
fi
86+
87+
- name: Create Pull Request
88+
uses: peter-evans/create-pull-request@v7
89+
with:
90+
token: ${{ secrets.HAYSTACK_BOT_TOKEN }}
91+
commit-message: "Sync Haystack Experimental API reference on Docusaurus"
92+
branch: sync-docusaurus-api-reference-experimental
93+
base: main
94+
title: "docs: sync Haystack Experimental API reference on Docusaurus"
95+
add-paths: |
96+
docs-website
97+
body: |
98+
This PR syncs the Haystack Experimental API reference on Docusaurus. Just approve and merge it.

‎docs/pydoc/config_docusaurus/agents_api.yml‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
loaders:
2+
- ignore_when_discovered:
3+
- __init__
4+
modules:
5+
- haystack_experimental.components.agents.agent
6+
- haystack_experimental.components.agents.human_in_the_loop.breakpoint
7+
- haystack_experimental.components.agents.human_in_the_loop.dataclasses
8+
- haystack_experimental.components.agents.human_in_the_loop.errors
9+
- haystack_experimental.components.agents.human_in_the_loop.policies
10+
- haystack_experimental.components.agents.human_in_the_loop.strategies
11+
- haystack_experimental.components.agents.human_in_the_loop.types
12+
- haystack_experimental.components.agents.human_in_the_loop.user_interfaces
13+
search_path:
14+
- ../../../
15+
type: haystack_pydoc_tools.loaders.CustomPythonLoader
16+
processors:
17+
- do_not_filter_modules: false
18+
documented_only: true
19+
expression: null
20+
skip_empty_modules: true
21+
type: filter
22+
- type: smart
23+
- type: crossref
24+
renderer:
25+
description: Tool-using agents with provider-agnostic chat model support.
26+
id: experimental-agents-api
27+
markdown:
28+
add_member_class_prefix: false
29+
add_method_class_prefix: true
30+
classdef_code_block: false
31+
descriptive_class_title: false
32+
descriptive_module_title: true
33+
filename: experimental_agents_api.md
34+
title: Agents
35+
type: haystack_pydoc_tools.renderers.DocusaurusRenderer

‎docs/pydoc/config_docusaurus/chat_message_stores_api.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
loaders:
2+
- ignore_when_discovered:
3+
- __init__
4+
modules:
5+
- haystack_experimental.chat_message_stores.in_memory
6+
search_path:
7+
- ../../../
8+
type: haystack_pydoc_tools.loaders.CustomPythonLoader
9+
processors:
10+
- do_not_filter_modules: false
11+
documented_only: true
12+
expression: null
13+
skip_empty_modules: true
14+
type: filter
15+
- type: smart
16+
- type: crossref
17+
renderer:
18+
description: Storage for the chat messages.
19+
id: experimental-chatmessage-store-api
20+
markdown:
21+
add_member_class_prefix: false
22+
add_method_class_prefix: true
23+
classdef_code_block: false
24+
descriptive_class_title: false
25+
descriptive_module_title: true
26+
filename: experimental_chatmessage_store_api.md
27+
title: ChatMessage Store
28+
type: haystack_pydoc_tools.renderers.DocusaurusRenderer

‎docs/pydoc/config_docusaurus/generators_api.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
loaders:
2+
- ignore_when_discovered:
3+
- __init__
4+
modules:
5+
- haystack_experimental.components.generators.chat.openai
6+
search_path:
7+
- ../../../
8+
type: haystack_pydoc_tools.loaders.CustomPythonLoader
9+
processors:
10+
- do_not_filter_modules: false
11+
documented_only: true
12+
expression: null
13+
skip_empty_modules: true
14+
type: filter
15+
- type: smart
16+
- type: crossref
17+
renderer:
18+
description: Enables text generation using LLMs.
19+
id: experimental-generators-api
20+
markdown:
21+
add_member_class_prefix: false
22+
add_method_class_prefix: true
23+
classdef_code_block: false
24+
descriptive_class_title: false
25+
descriptive_module_title: true
26+
filename: experimental_generators_api.md
27+
title: Generators
28+
type: haystack_pydoc_tools.renderers.DocusaurusRenderer

‎docs/pydoc/config_docusaurus/preprocessors_api.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
loaders:
2+
- ignore_when_discovered:
3+
- __init__
4+
modules:
5+
- haystack_experimental.components.preprocessors.embedding_based_document_splitter
6+
search_path:
7+
- ../../../
8+
type: haystack_pydoc_tools.loaders.CustomPythonLoader
9+
processors:
10+
- do_not_filter_modules: false
11+
documented_only: true
12+
expression: null
13+
skip_empty_modules: true
14+
type: filter
15+
- type: smart
16+
- type: crossref
17+
renderer:
18+
description: Pipelines wrapped as components.
19+
id: experimental-preprocessors-api
20+
markdown:
21+
add_member_class_prefix: false
22+
add_method_class_prefix: true
23+
classdef_code_block: false
24+
descriptive_class_title: false
25+
descriptive_module_title: true
26+
filename: experimental_preprocessors_api.md
27+
title: Preprocessors
28+
type: haystack_pydoc_tools.renderers.DocusaurusRenderer

‎docs/pydoc/config_docusaurus/query_api.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
loaders:
2+
- ignore_when_discovered:
3+
- __init__
4+
modules:
5+
- haystack_experimental.components.query.query_expander
6+
search_path:
7+
- ../../../
8+
type: haystack_pydoc_tools.loaders.CustomPythonLoader
9+
processors:
10+
- do_not_filter_modules: false
11+
documented_only: true
12+
expression: null
13+
skip_empty_modules: true
14+
type: filter
15+
- type: smart
16+
- type: crossref
17+
renderer:
18+
description: Components for query processing and expansion.
19+
id: experimental-query-api
20+
markdown:
21+
add_member_class_prefix: false
22+
add_method_class_prefix: true
23+
classdef_code_block: false
24+
descriptive_class_title: false
25+
descriptive_module_title: true
26+
filename: experimental_query_api.md
27+
title: Query
28+
type: haystack_pydoc_tools.renderers.DocusaurusRenderer

‎docs/pydoc/config_docusaurus/retrievers_api.yml‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
loaders:
2+
- ignore_when_discovered:
3+
- __init__
4+
modules:
5+
- haystack_experimental.components.retrievers.chat_message_retriever
6+
- haystack_experimental.components.retrievers.multi_query_embedding_retriever
7+
- haystack_experimental.components.retrievers.multi_query_text_retriever
8+
search_path:
9+
- ../../../
10+
type: haystack_pydoc_tools.loaders.CustomPythonLoader
11+
processors:
12+
- do_not_filter_modules: false
13+
documented_only: true
14+
expression: null
15+
skip_empty_modules: true
16+
type: filter
17+
- type: smart
18+
- type: crossref
19+
renderer:
20+
description: Sweep through Document Stores and return a set of candidate documents
21+
that are relevant to the query.
22+
id: experimental-retrievers-api
23+
markdown:
24+
add_member_class_prefix: false
25+
add_method_class_prefix: true
26+
classdef_code_block: false
27+
descriptive_class_title: false
28+
descriptive_module_title: true
29+
filename: experimental_retrievers_api.md
30+
title: Retrievers
31+
type: haystack_pydoc_tools.renderers.DocusaurusRenderer

‎docs/pydoc/config_docusaurus/summarizers_api.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
loaders:
2+
- ignore_when_discovered:
3+
- __init__
4+
modules:
5+
- haystack_experimental.components.summarizers.llm_summarizer
6+
search_path:
7+
- ../../../
8+
type: haystack_pydoc_tools.loaders.CustomPythonLoader
9+
processors:
10+
- do_not_filter_modules: false
11+
documented_only: true
12+
expression: null
13+
skip_empty_modules: true
14+
type: filter
15+
- type: smart
16+
- type: crossref
17+
renderer:
18+
description: Components that summarize texts into concise versions.
19+
id: experimental-summarizers-api
20+
markdown:
21+
add_member_class_prefix: false
22+
add_method_class_prefix: true
23+
classdef_code_block: false
24+
descriptive_class_title: false
25+
descriptive_module_title: true
26+
filename: experimental_summarizer_api.md
27+
title: Summarizers
28+
type: haystack_pydoc_tools.renderers.DocusaurusRenderer

‎docs/pydoc/config_docusaurus/supercomponents_api.yml‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
loaders:
2+
- ignore_when_discovered:
3+
- __init__
4+
modules:
5+
- haystack_experimental.super_components.indexers.sentence_transformers_document_indexer
6+
search_path:
7+
- ../../../
8+
type: haystack_pydoc_tools.loaders.CustomPythonLoader
9+
processors:
10+
- do_not_filter_modules: false
11+
documented_only: true
12+
expression: null
13+
skip_empty_modules: true
14+
type: filter
15+
- type: smart
16+
- type: crossref
17+
renderer:
18+
description: Pipelines wrapped as components.
19+
id: experimental-supercomponents-api
20+
markdown:
21+
add_member_class_prefix: false
22+
add_method_class_prefix: true
23+
classdef_code_block: false
24+
descriptive_class_title: false
25+
descriptive_module_title: true
26+
filename: experimental_supercomponents_api.md
27+
title: SuperComponents
28+
type: haystack_pydoc_tools.renderers.DocusaurusRenderer

0 commit comments

Comments
 (0)