feat: Add GitHub integration with agent_prompts and github_components#1637
feat: Add GitHub integration with agent_prompts and github_components#1637julian-risch merged 53 commits intomainfrom
Conversation
|
@sjrl Finally ready for another review! We're using What do you think about the parameter name |
| :param name: Optional name for the tool. | ||
| :param description: Optional description. |
There was a problem hiding this comment.
I do realize this is a bit confusing in our tools, but it seems that if we define a __init__ then these docstrings are put under the __init__ def. If there is no __init__ defined like in Tool then we put it in the class description.
There was a problem hiding this comment.
Ah, of course! Do you think we should a usage example here then (in addition to moving the param docstrings to the init)? I realized that's missing too.
There was a problem hiding this comment.
Yeah if it's not too much to ask, a usage example would be great!
sjrl
left a comment
There was a problem hiding this comment.
Looks good! Only some minor comments left
* add agent_prompts and github_components * rename to github_haystack * remove github-haystack * renamed integration, added components dir * add tests, pydoc, update pyproject.toml * add workflow * fmt * fmt * lint * ruff * fmt * lint:all * replace StrEnum for py 3.9+ compatibility * move files * fix tests * lint * fix pydoc and extend init files * Add integration:github to labeler.yml * unify how we set GITHUB_TOKEN in tests * fix 3 usage examples. 3 remaining * remove empty lines from prompts * GitHub capitalization * add license header * all caps for prompts * add GitHubFileEditorTool * enforce kwargs instead of positional args * use _get_request_headers and base_headers consistently * lint * rename GitHubRepositoryViewer to GitHubRepoViewer * lint * add pipeline serialization test * extend pipeline to_dict test * set default branch of repo viewer * lint * add four more tools * lint * rename prompts * add tests for four more tools * rename context prompt * add outputs_to_state as param to GitHubFileEditorTool * add outputs_to_state as param to GitHubRepoViewerTool * set default outputs_to_state for GitHubRepoViewerTool * extract serialize_handlers to utils; don't use mutable defaults * replace init_parameters with data for serde in FileEditor, RepoViewer * add outputs_to_state to GitHubIssueCommenterTool; replace init_parameters with data * add outputs_to_state to GitHubIssueViewerTool; replace init_parameters with data * add outputs_to_state to GitHubPRCreatorTool; replace init_parameters with data * move param docstrings to init methods * use generate_qualified_class_name instead of hardcoded name * test with lowest supported version * don't test http_client_kwargs for compatibility with Haystack 2.12
* feat(azure-ai-search): Allow full metadata field customization So far, the `metadata_fields` init parameter only allowed a few custom simple value types to be mapped (e.g., no nested metadata) and also hardcoded the fields to be only `filterable` (but not `searchable` or `facetable`, for instance). For full flexibility, allow an Azure AI Search `SearchField` instance to be passed as mapping instead of a Python type. * PR comments * feat: Add OpenRouter integration (#1723) * Add openrouter integration * Add tests for chat generator and support extra headers * Add async tests * Fix config files * Add example * Fixes * Fix read me * PR comments * Small fixes * Updated labeler and README * Update docstrings * Add user agent to Azure AI Search(#1743) * docs: update changelog for integrations/azure_ai_search (#1745) * Update changelog for integrations/azure_ai_search * Update CHANGELOG.md --------- Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com> * docs: ChatMessage examples (#1752) * feat: Support Llama API as a Chat Generator (#1742) * init: llama-api chat generator * docs: update comments for LlamaChatGenerator * feat: add keyword only * * fix: replace streaming_callback type * fix: add Toolset for tools * fix: rm unused typing * docs: add meta header * docs: fix comments to llama api * docs: add meta header * docs: add meta header * fix: rename LlamaChat to MetaLlamaChat * docs: add meta header * docs: align doc format * add workflow for nightly tests * add meta_llama to labeler * add new integration to repo readme overview table * replace .llama.chat. with .meta_llama.chat. * fmt * replace llama with meta_llama in pydocs --------- Co-authored-by: Julian Risch <julian.risch@deepset.ai> * Update changelog for integrations/meta_llama (#1754) Co-authored-by: julian-risch <4181769+julian-risch@users.noreply.github.com> * chore(deps): bump fossas/fossa-action from 1.6.0 to 1.7.0 (#1750) Bumps [fossas/fossa-action](https://github.com/fossas/fossa-action) from 1.6.0 to 1.7.0. - [Release notes](https://github.com/fossas/fossa-action/releases) - [Commits](fossas/fossa-action@v1.6.0...v1.7.0) --- updated-dependencies: - dependency-name: fossas/fossa-action dependency-version: 1.7.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update how skipping works (#1756) * test: Ollama - make test_run_with_response_format more robust (#1757) * feat: adapt `OllamaGenerator` metadata to OpenAI format (#1753) * feat: adapt Ollama metadata to OpenAI format in `OllamaGenerator` * Add: `OllamaGenerator` support in Langfuse * Ran Linters * Revert "Add: `OllamaGenerator` support in Langfuse" This reverts commit 1f399e0. --------- Co-authored-by: Sebastian Husch Lee <sjrl@users.noreply.github.com> * Add: `OllamaGenerator` support in Langfuse (#1759) * Update changelog for integrations/ollama (#1761) Co-authored-by: sjrl <10526848+sjrl@users.noreply.github.com> * Update changelog for integrations/langfuse (#1762) Co-authored-by: sjrl <10526848+sjrl@users.noreply.github.com> * docs: update changelog for integrations/openrouter (#1763) * Update changelog for integrations/openrouter --------- Co-authored-by: Amnah199 <13835656+Amnah199@users.noreply.github.com> Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com> * chore: fix README for meta-llama (#1766) * chore(deps): bump aws-actions/configure-aws-credentials (#1751) Bumps [aws-actions/configure-aws-credentials](https://github.com/aws-actions/configure-aws-credentials) from 4.2.0 to 4.2.1. - [Release notes](https://github.com/aws-actions/configure-aws-credentials/releases) - [Changelog](https://github.com/aws-actions/configure-aws-credentials/blob/main/CHANGELOG.md) - [Commits](aws-actions/configure-aws-credentials@f24d719...b475783) --- updated-dependencies: - dependency-name: aws-actions/configure-aws-credentials dependency-version: 4.2.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * ci: Bedrock - improve worfklow; skip tests from CI (#1773) * feat: OllamaChatGenerator - add Toolset support (#1765) * Add Toolset support to OllamaChatGenerator * Lint * Lambdas are not serializable * Lint * Generate tool call id if not available * Lint * Revert back to not using ToolCall id * Lint * Update changelog for integrations/ollama (#1775) Co-authored-by: vblagoje <458335+vblagoje@users.noreply.github.com> * feat: MCPTool and MCPToolset async resource management improvements (#1758) * Add MCPClientSessionManager to connect/close mcp clients * Update and refactor mcp tests * More descriptive connection error raising * Proper test cleanup * Testing CI windows * linting * Improve connection error raise * PR feedback * Proper naming, and more precise cleanup sequence --------- Co-authored-by: Michele Pangrazzi <xmikex83@gmail.com> * test: add service_tier to test_convert_anthropic_chunk_to_streaming_chunk (#1778) * fix: Bring Mistral integration up to date with changes made to OpenAIChatGenerator and OpenAI Embedders (#1774) * Bringing Mistral up to date * Fix Mistral Embedders to be deserializable * Fix lint * Fix lint * Bump minimum haystack version * Update changelog for integrations/mistral (#1781) Co-authored-by: sjrl <10526848+sjrl@users.noreply.github.com> * feat: Add `to_dict` to `STACKITDocumentEmbedder` and `STACKITTextEmbedder` and more init parameters from underlying OpenAI classes (#1779) * Add to_dicts and more tests * Bumpy haystack version * Add changes to chat generator as well * Update changelog for integrations/stackit (#1782) Co-authored-by: sjrl <10526848+sjrl@users.noreply.github.com> * feat: add run_async for CohereChatGenerator (#1689) * CohereChatGenerator async support * Tests and linter fixes * fix * refinements * refactor + tests reorgani | ^^^^^ T201 * rename test * remove markers * reformat * fix * minor fixes * Trigger CI --------- Co-authored-by: anakin87 <stefanofiorucci@gmail.com> * Update changelog for integrations/cohere (#1784) Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> * docs: update changelog for integrations/google_ai (#1812) * Update changelog for integrations/google_ai * Update CHANGELOG.md --------- Co-authored-by: wochinge <7667273+wochinge@users.noreply.github.com> Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com> * fix: Fix exposing Qdrant api-key in `metadata` field when running `to_dict` (#1813) * Add to_dict test * Add more type hints * More type hints * Add fix for exposing api key in metadata when running to_dict * Add unit test * PR comments * Update changelog for integrations/qdrant (#1814) Co-authored-by: sjrl <10526848+sjrl@users.noreply.github.com> * ci: check lowest direct dependencies (#1788) * ci: check lowest direct dependencies * try single quotes * debug * debugging * try chroma * no bedrock * retry * explicit option * don't run tests * debug 1 * try output file * more * no deepeval --------- Co-authored-by: David S. Batista <dsbatista@gmail.com> * build: add pins for Anthropic (#1811) * build: add pins for Anthropic * rm file incorrectly added * Update changelog for integrations/anthropic (#1815) Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> * build: add pins for Vertex (#1810) * Update changelog for integrations/google_vertex (#1816) Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> * build: add pins for Cohere (#1817) * Update changelog for integrations/cohere (#1829) Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> * build: remove pin for Deepeval (#1826) * Update changelog for integrations/deepeval (#1830) Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> * feat: Add streamable-http transport MCP support (#1777) * Add streamable-http transport * Improve error message for tool invocation * Add streamable MCPTool example, update examples * Improve examples * Add unit tests * Update integrations/mcp/examples/mcp_client.py Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com> * initialize vars outside try block * Small fix * Fix linting --------- Co-authored-by: Amna Mubashar <amnahkhan.ak@gmail.com> * Update changelog for integrations/mcp (#1831) Co-authored-by: vblagoje <458335+vblagoje@users.noreply.github.com> * build: pining lower versions of haystack and `aiohttp` for `ElasticSearch` (#1827) * pining lower versions * adding missing comma * pinning to >=2.4.0 * pinning to >=2.3.0 * pinning aiohttp to >=3.0.0 * pinning aiohttp to >=2.0.0 * pinning aiohttp to >=2.5.0 * pinning aiohttp to >=2.6.0 * pinning aiohttp to >=3.0.0 * pinning aiohttp to >=3.1.0 * pinning aiohttp to >=3.2.0 * pinning aiohttp to >=3.3.0 * pinning aiohttp to >=3.10.0 * pinning aiohttp to >=3.9.0 * pinning aiohttp to >=3.8.0 * reverting back aiohttp to 3.9.0 * Update changelog for integrations/elasticsearch (#1834) Co-authored-by: davidsbatista <7937824+davidsbatista@users.noreply.github.com> * build: add Jina pins (#1836) * Update changelog for integrations/jina (#1838) Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> * build: add Langfuse pins (#1837) * Update changelog for integrations/langfuse (#1839) Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> * build: pin version for `pymongo` and `haystack` in MongoDB integration (#1832) * pinning to older version o haystack and mongodb * pinining haystack and pymongo * wip * fixing format * adding missing CI job * making sure lowest version of pymongo has the async client * making sure lowest version of pymongo has the async client * versioning * haysack 2.9 * haysack 2.10 * haysack 2.11 * Remove failing test. No need to have it here since it's already tested in haystack main. (#1842) * ci: Missing labels for stackit and anthropic (#1844) * Missing labels for stackit and anthropic * PR comments * build: app pins for MCP (#1845) * Update changelog for integrations/mongodb_atlas (#1840) Co-authored-by: davidsbatista <7937824+davidsbatista@users.noreply.github.com> Co-authored-by: David S. Batista <dsbatista@gmail.com> * docs: update changelog for integrations/mcp (#1848) * Update changelog for integrations/mcp * Update CHANGELOG.md --------- Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com> * build: add pins for Pgvector (#1849) * Update changelog for integrations/pgvector (#1850) Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> * build: add pins for Optimum (#1847) * build: add pins for Optimum * try with python 3.13 * don't call HF on unit tests * Update changelog for integrations/optimum (#1852) Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> * build: add pins for Qdrant (#1853) * build: add pins for Pinecone (#1851) Co-authored-by: David S. Batista <dsbatista@gmail.com> * Update changelog for integrations/pinecone (#1855) Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> * docs: update changelog for integrations/qdrant (#1856) * Update changelog for integrations/qdrant * Update CHANGELOG.md --------- Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com> * chore: review license compliance workflow (#1843) * chore: review license compliance workflow * refactor * deepeval * build: add pins for Ragas (#1854) * feat: Add GitHub integration with components, tools, and prompts (#1637) * add agent_prompts and github_components * rename to github_haystack * remove github-haystack * renamed integration, added components dir * add tests, pydoc, update pyproject.toml * add workflow * fmt * fmt * lint * ruff * fmt * lint:all * replace StrEnum for py 3.9+ compatibility * move files * fix tests * lint * fix pydoc and extend init files * Add integration:github to labeler.yml * unify how we set GITHUB_TOKEN in tests * fix 3 usage examples. 3 remaining * remove empty lines from prompts * GitHub capitalization * add license header * all caps for prompts * add GitHubFileEditorTool * enforce kwargs instead of positional args * use _get_request_headers and base_headers consistently * lint * rename GitHubRepositoryViewer to GitHubRepoViewer * lint * add pipeline serialization test * extend pipeline to_dict test * set default branch of repo viewer * lint * add four more tools * lint * rename prompts * add tests for four more tools * rename context prompt * add outputs_to_state as param to GitHubFileEditorTool * add outputs_to_state as param to GitHubRepoViewerTool * set default outputs_to_state for GitHubRepoViewerTool * extract serialize_handlers to utils; don't use mutable defaults * replace init_parameters with data for serde in FileEditor, RepoViewer * add outputs_to_state to GitHubIssueCommenterTool; replace init_parameters with data * add outputs_to_state to GitHubIssueViewerTool; replace init_parameters with data * add outputs_to_state to GitHubPRCreatorTool; replace init_parameters with data * move param docstrings to init methods * use generate_qualified_class_name instead of hardcoded name * test with lowest supported version * don't test http_client_kwargs for compatibility with Haystack 2.12 * build: pinning `unstructured` to lowest working versions (#1841) * finding lowest working versions * adding missing CI job * adding missing limitation * feat: AnthropicChatGenerator - add Toolset support (#1787) * AnthropicChatGenerator - add Toolset support * Use new serialization method for tools * Update haystack dep to 2.13.1 which includes Toolset * Small update * buid: add pins for Snowflake + small refactoring (#1860) * Update changelog for integrations/snowflake (#1862) Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> * Update changelog for integrations/ragas (#1857) Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> * Update changelog for integrations/unstructured (#1861) Co-authored-by: davidsbatista <7937824+davidsbatista@users.noreply.github.com> * build: add pins for Nvidia (#1846) * Update changelog for integrations/nvidia (#1863) Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> * build: add pins for Google AI (#1828) * docs: update changelog for integrations/google_ai (#1864) * Update changelog for integrations/google_ai * Update CHANGELOG.md --------- Co-authored-by: anakin87 <44616784+anakin87@users.noreply.github.com> Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com> * Update changelog for integrations/anthropic (#1865) Co-authored-by: vblagoje <458335+vblagoje@users.noreply.github.com> * docs: Update changelog for integrations/github (#1858) Co-authored-by: julian-risch <4181769+julian-risch@users.noreply.github.com> * feat: adding an `HybridRetriever` as a `Supercomponent` having `OpenSearch` as the document store (#1701) * adding tests * linting and typing * adding env variable * env variable * extending docstring * removing generation part * updating tests * adding a run test with mocked sentence_transformers * fixing format * refactor: use `component_to_dict` in OpenSearchHybridRetriever (#1866) * Update changelog for integrations/opensearch (#1867) Co-authored-by: davidsbatista <7937824+davidsbatista@users.noreply.github.com> * oshr-docs (#1868) * refactor: OpenSearchHybridRetriever use `deserialize_chatgenerator_inplace` (#1870) * test to use deserialize_chatgenerator_inplace * removing unused imports * using deserialize_chatgenerator_inplace * Update integrations/opensearch/src/haystack_integrations/components/retrievers/opensearch/open_search_hybrid_retriever.py * Update changelog for integrations/opensearch (#1874) Co-authored-by: davidsbatista <7937824+davidsbatista@users.noreply.github.com> * feat: add run_async support for CohereTextEmbedder (#1873) * feat: add run_async support for CohereTextEmbedder * fix: review comments * feat: Add Google GenAI GoogleGenAIChatGenerator (#1875) * Initial work * Remove utils * Add async support * Async test issue * Simplify async tests * Linting * Improve comment * Linting * Improve pyproject.toml * Add new google genai integration to workflow * Add labeler * Add pydoc * Pin deps * Pin google-genai dep * Update integrations/google_genai/src/haystack_integrations/components/generators/google_genai/chat/chat_generator.py Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com> * Update integrations/google_genai/src/haystack_integrations/components/generators/google_genai/chat/chat_generator.py Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com> * PR feedback * Add system message comment * Leave only minimal working examples in README * Update integrations/google_genai/src/haystack_integrations/components/generators/google_genai/chat/chat_generator.py Co-authored-by: Julian Risch <julian.risch@deepset.ai> * Update integrations/google_genai/src/haystack_integrations/components/generators/google_genai/chat/chat_generator.py Co-authored-by: Julian Risch <julian.risch@deepset.ai> * Linting --------- Co-authored-by: Sebastian Husch Lee <10526848+sjrl@users.noreply.github.com> Co-authored-by: Julian Risch <julian.risch@deepset.ai> * Update changelog for integrations/google_genai (#1886) Co-authored-by: vblagoje <458335+vblagoje@users.noreply.github.com> * feat: Use Langfuse local to_openai_dict_format function to serialize messages (#1885) * Use Langfuse local to_openai_dict_format function to serialize messages * Linting * PR feedback * Add detailed tracing for GoogleGenAIChatGenerator (#1887) * docs: update changelog for integrations/langfuse (#1888) * Update changelog for integrations/langfuse * Update CHANGELOG.md --------- Co-authored-by: vblagoje <458335+vblagoje@users.noreply.github.com> Co-authored-by: Vladimir Blagojevic <dovlex@gmail.com> * try reenabling pinecone tests (#1871) * PR comments * Small updates --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Denis Washington <denis.washington@futurice.com> Co-authored-by: Denis Washington <denis@denisw.de>
Related Issues
Related to Add GitHub integration with agent_prompts and github_components #1650
Related to Move Agent from experimental to main haystack-experimental#250
@sjrl brought up that one way to keep example files from the experimental Agent around is an integration chore: Remove Agent after Haystack 2.12 release haystack-experimental#263 (comment)
Proposed Changes:
The idea is to enable users to run the example notebook (or a version with updated imports) after having installed this new integration.
How did you test it?
New unit tests and I ran all usage examples successfully with a test repo.
I have tested it with the updated notebook. I'll update the cookbook PR once the integration is released: deepset-ai/haystack-cookbook#183
Notes for the reviewer
github_tokenparameter toapi_keyfor consistency with many other integrations.GithubRepositoryViewer has abranchparameter in the run method, which could also be named ref to make more clear it can also be a tag or commit hash. I prefer keeping the parameter namebranch.Some components havegithub_token: Optional[Secret] = None,because they can work without any token while others useSecret.from_env_var("GITHUB_TOKEN"). I suggest we useSecret.from_env_var("GITHUB_TOKEN", strict=False)where we currently haveNoneas the default.The internal implementation of the components differs in how they use_get_headersor_get_request_headersor define headers inline. We could refactor that.Checklist
fix:,feat:,build:,chore:,ci:,docs:,style:,refactor:,perf:,test:.