HAProxy performance degradation after applying telemetry

[asllop]

I'm evaluating haproxy-otel before using it in a production environment. I set up a Docker environment with:

• HAProxy 3.3
• 4 simple web services (hello worlds) used as backends for the load balancer.
• An OpenTelemetry collector that accepts traces and sends them to stdout.

I performed basic benchmarking before applying the telemetry. I used plow with the following options:

plow http://localhost -c 500 -d 2m

I got the following results:

• Average latency: 27ms
• Average requests per second: 18.8k

Then I installed the haproxy-otel following the instructions in this repo. The solution works as expected and reports traces to the OpenTelemetry collector. Then I ran the same benchmark and the results are like one order of magnitude worse:

• Average latency: 180ms
• Average requests per second: 2.7k

Questions:

• Are these results expected? Maybe I'm doing something wrong?
• Have you detected it before?
• If yes, is this caused by the overhead of the HAProxy filters, or is caused by the opentelemetry rust sdk?