Skip to content

feat: add olostep-haystack integration (WebSearch + Fetcher)#3296

Closed
umerkay wants to merge 4 commits into
deepset-ai:mainfrom
umerkay:main
Closed

feat: add olostep-haystack integration (WebSearch + Fetcher)#3296
umerkay wants to merge 4 commits into
deepset-ai:mainfrom
umerkay:main

Conversation

@umerkay
Copy link
Copy Markdown

@umerkay umerkay commented May 12, 2026

Related Issues

  • No related issue this is a new integration contribution

Proposed Changes:

Adds olostep-haystack, a new integration providing two components:

OlostepWebSearch searches the web using Olostep's /v1/searches
endpoint and returns results as Haystack Document objects (with title
and link in metadata) plus a list of URLs. Works as a drop-in alternative
to BraveSearchWebSearch and SerperDevWebSearch. Calls the REST endpoint
directly via requests (the Olostep Python SDK does not expose a searches
namespace).

OlostepFetcher fetches and converts web pages to clean Markdown (or
HTML) using the official olostep Python SDK (SyncOlostepClient). Returns
content as Haystack Document objects. Works as a drop-in alternative to
LinkContentFetcher + Firecrawl in web RAG pipelines.

Both components:

  • Read the API key via Haystack's Secret (env var OLOSTEP_API_KEY)
  • Support to_dict() / from_dict() for pipeline serialization
  • Follow the same structure as brave-haystack

How did you test it?

  • Unit tests for OlostepWebSearch mock requests.post and verify correct
    Document output, top_k limiting, allowed_domains filtering, missing
    API key error handling, and to_dict/from_dict round-trip
  • Unit tests for OlostepFetcher mock SyncOlostepClient and verify correct
    Document output, None content handling, auth error propagation, and
    to_dict/from_dict round-trip
  • Ran hatch run fmt, hatch run test:types, hatch run test:unit — all pass

Notes for the reviewer

  • /v1/searches has no SDK wrapper requests.post is intentional, not an
    oversight
  • SyncOlostepClient is imported inside run() rather than at module level
    to avoid import errors for users who haven't installed olostep
  • The SDK requires Python 3.11+ which is reflected in pyproject.toml
  • Olostep docs: https://docs.olostep.com

Checklist

  • I have read the contributors guidelines and the code of conduct
  • I have updated the related issue with new insights and changes
  • I added unit tests and updated the docstrings
  • I've used one of the conventional commit types for my PR title: feat:

@umerkay umerkay requested a review from a team as a code owner May 12, 2026 07:59
@umerkay umerkay requested review from julian-risch and removed request for a team May 12, 2026 07:59
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented May 12, 2026

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions Bot added topic:CI type:documentation Improvements or additions to documentation labels May 12, 2026
@socket-security
Copy link
Copy Markdown

socket-security Bot commented May 12, 2026

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedolostep@​1.1.0100100100100100

View full report

@julian-risch
Copy link
Copy Markdown
Member

Thank you for your suggestion and opening this issue. We recommend to publish and release this integration yourself using our repository template if you believe it benefits a larger group of users. https://github.com/deepset-ai/custom-component
Once you released it, your integration can be added to https://github.com/deepset-ai/haystack-integrations so that it gets listed on our web page https://haystack.deepset.ai/integrations

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic:CI type:documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants