Skip to content

[BUG] opensearch_api source does not work with Kafka buffer - no ByteDecoder registered #6876

Description

@divakarsingh

Describe the bug
The opensearch_api source does not work when configured with a kafka buffer. Traffic stops flowing — events are written to Kafka but never consumed/reconstructed on the other side. It works correctly with bounded_blocking buffer.

To Reproduce

  1. Configure Data Prepper with opensearch_api source and kafka buffer
  2. Send a _bulk request to Data Prepper
  3. Observe: events are written to Kafka topic but never reach the sink
  4. Switch buffer to bounded_blocking — traffic flows correctly

Expected behavior
The opensearch_api source should register a ByteDecoder that can parse NDJSON bulk format back into Data Prepper events with correct metadata attributes (opensearch_action, opensearch_index, opensearch_id, etc.) when using Kafka buffer.

Screenshots
N/A

Environment (please complete the following information):

  • OS: Linux
  • Version: Data Prepper 2.15.1 / main branch

Additional context
Forum report: https://forum.opensearch.org/t/opensearch-api-source-doesnt-work-with-kafka-buffer/28059

Root cause: When buffer.isByteBuffer() == true (Kafka buffer), the source writes raw bytes via buffer.writeBytes(content.array(), ...). On the consumer side, the buffer needs a ByteDecoder to reconstruct Record<Event> objects from those bytes. OpenSearchAPISource does not override getDecoder(), so no decoder is registered. The Kafka buffer consumer cannot parse the NDJSON bulk format back into events.

The bounded_blocking buffer works because it stores Record<Event> objects directly in a BlockingQueue — no serialization/deserialization needed.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Fields

No fields configured for issues without a type.

Projects

Status
In review

Relationships

None yet

Development

No branches or pull requests

Issue actions