You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello,
I have a pipeline with a single input (MQTT) and I need to send the same events to two different outputs with different formats:
• VictoriaMetrics using http_client with Prometheus text format
• PostgreSQL using sql_insert with relational rows
In Benthos OSS this works using broker + fan_out, because each output can have its own processors and transform the message independently.
In Redpanda Connect 4.x, outputs cannot have processors and all outputs must consume the same payload, so this pattern does not seem possible in a single pipeline. The only option I found is to split the solution into two pipelines, duplicating the input and parsing logic.
My questions are:
Is this limitation intentional in Redpanda Connect?
Is there a recommended way to support multiple outputs with different formats from the same input?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
-
Hello,
I have a pipeline with a single input (MQTT) and I need to send the same events to two different outputs with different formats:
• VictoriaMetrics using http_client with Prometheus text format
• PostgreSQL using sql_insert with relational rows
In Benthos OSS this works using broker + fan_out, because each output can have its own processors and transform the message independently.
In Redpanda Connect 4.x, outputs cannot have processors and all outputs must consume the same payload, so this pattern does not seem possible in a single pipeline. The only option I found is to split the solution into two pipelines, duplicating the input and parsing logic.
My questions are:
Thanks for any clarification.
Gustavo
Beta Was this translation helpful? Give feedback.
All reactions