Describe the bug
The opensearch_api source does not work when configured with a kafka buffer. Traffic stops flowing — events are written to Kafka but never consumed/reconstructed on the other side. It works correctly with bounded_blocking buffer.
To Reproduce
- Configure Data Prepper with
opensearch_api source and kafka buffer
- Send a
_bulk request to Data Prepper
- Observe: events are written to Kafka topic but never reach the sink
- Switch buffer to
bounded_blocking — traffic flows correctly
Expected behavior
The opensearch_api source should register a ByteDecoder that can parse NDJSON bulk format back into Data Prepper events with correct metadata attributes (opensearch_action, opensearch_index, opensearch_id, etc.) when using Kafka buffer.
Screenshots
N/A
Environment (please complete the following information):
- OS: Linux
- Version: Data Prepper 2.15.1 / main branch
Additional context
Forum report: https://forum.opensearch.org/t/opensearch-api-source-doesnt-work-with-kafka-buffer/28059
Root cause: When buffer.isByteBuffer() == true (Kafka buffer), the source writes raw bytes via buffer.writeBytes(content.array(), ...). On the consumer side, the buffer needs a ByteDecoder to reconstruct Record<Event> objects from those bytes. OpenSearchAPISource does not override getDecoder(), so no decoder is registered. The Kafka buffer consumer cannot parse the NDJSON bulk format back into events.
The bounded_blocking buffer works because it stores Record<Event> objects directly in a BlockingQueue — no serialization/deserialization needed.
Describe the bug
The
opensearch_apisource does not work when configured with akafkabuffer. Traffic stops flowing — events are written to Kafka but never consumed/reconstructed on the other side. It works correctly withbounded_blockingbuffer.To Reproduce
opensearch_apisource andkafkabuffer_bulkrequest to Data Prepperbounded_blocking— traffic flows correctlyExpected behavior
The
opensearch_apisource should register aByteDecoderthat can parse NDJSON bulk format back into Data Prepper events with correct metadata attributes (opensearch_action,opensearch_index,opensearch_id, etc.) when using Kafka buffer.Screenshots
N/A
Environment (please complete the following information):
Additional context
Forum report: https://forum.opensearch.org/t/opensearch-api-source-doesnt-work-with-kafka-buffer/28059
Root cause: When
buffer.isByteBuffer() == true(Kafka buffer), the source writes raw bytes viabuffer.writeBytes(content.array(), ...). On the consumer side, the buffer needs aByteDecoderto reconstructRecord<Event>objects from those bytes.OpenSearchAPISourcedoes not overridegetDecoder(), so no decoder is registered. The Kafka buffer consumer cannot parse the NDJSON bulk format back into events.The
bounded_blockingbuffer works because it storesRecord<Event>objects directly in aBlockingQueue— no serialization/deserialization needed.