Skip to content

[DOC] Add documentation page for opensearch_api source in Data Prepper #12460

@divakarsingh

Description

@divakarsingh

What do you want to do?

  • Request a change to existing documentation
  • Add new documentation
  • Report a technical problem with the documentation
  • Other

Tell us about your request.

The opensearch_api source plugin for Data Prepper has no documentation page on the docs portal. The URL https://docs.opensearch.org/latest/data-prepper/pipelines/configuration/sources/opensearch-api-source/ returns 404.

Every other Data Prepper source plugin (http_source, otel_trace_source, kafka, etc.) has a corresponding page on the docs website.

Additionally, the existing plugin README (https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch-api-source/README.md) is missing several important details:

  • Supported bulk actions (index, create, update, delete) and which require a document body
  • Event metadata attributes set per action (opensearch_action, opensearch_index, opensearch_id, opensearch_routing, opensearch_pipeline)
  • Response body format ({took, errors, items[]}) — clients depend on this for error handling
  • Integration examples showing usage with Flink OpenSearch Connector, opensearch-py, and Logstash

A new page needs to be created at opensearch-project/documentation-website/_data-prepper/pipelines/configuration/sources/opensearch-api-source.md covering:

  • Overview and use cases (accepting OpenSearch Bulk API traffic from standard clients like Flink, Logstash, opensearch-py)
  • Configuration parameters (port, path, authentication, SSL, compression)
  • Supported bulk actions (index, create, update, delete)
  • Response format ({took, errors, items[]})
  • Event metadata attributes (opensearch_action, opensearch_index, opensearch_id, opensearch_routing, opensearch_pipeline)
  • Metrics reference
  • Example pipeline configurations

Version: 2.15+

What other resources are available?

Metadata

Metadata

Assignees

Labels

Backlog - DEVDeveloper assigned to issue is responsible for creating PR.data-prepper

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions