in_opentelemetry: reduce protobuf decode allocation churn

[edsiper]

Problem

Decoding a large OTLP protobuf logs request asks protobuf-c to allocate every
message, repeated field, string, and byte field independently. These objects
share the request lifetime, so individual heap allocations and frees add CPU
and allocator churn without providing useful lifetime granularity.

Change

• Decode protobuf log payloads of at least 64 KiB with a request-local CFL
  arena.
• Keep protobuf-c's default allocator for smaller requests, avoiding arena
  setup on the latency-sensitive small-payload path.
• Fall back to the default allocator if arena creation fails.
• Destroy the arena after conversion, including malformed and partially
  decoded requests.
• Add integration coverage for a valid 128 KiB request and malformed input at
  the 64 KiB boundary.

This is most beneficial for high-volume OTLP log ingestion with large batched
protobuf requests containing many records and nested attributes. JSON, metrics,
traces, and small protobuf log requests are unchanged.

Performance

Release build with GCC 13.3, pinned to CPU 2. The same 2,978,046-byte,
2,048-record request was decoded 50 times. Each perf stat value is the mean
of five runs.

Counter	Before	After	Change
task-clock	985.86 ms	821.58 ms	-16.66%
cycles	4.947B	4.161B	-15.88%
instructions	15.491B	12.558B	-18.94%

Massif measured the same 49,758,368-byte process peak before and after. Its
cumulative allocation-time proxy fell 0.59%. This optimization reduces CPU and
allocation-call churn; this workload does not demonstrate lower peak memory.

Verification

ctest --test-dir /tmp/flb-proto-input-test \
  -R '^(flb-rt-in_opentelemetry|flb-rt-in_opentelemetry_routing|flb-it-opentelemetry|flb-it-http_server)$' \
  --output-on-failure

Passed 4/4.

The following focused cases passed 3/3 normally and 3/3 under strict Valgrind:

FLUENT_BIT_BINARY=/tmp/flb-proto-input-test/bin/fluent-bit \
tests/integration/.venv/bin/python -m pytest -q \
tests/integration/scenarios/in_opentelemetry/tests/test_in_opentelemetry_001.py::test_opentelemetry_to_opentelemetry_basic_log \
tests/integration/scenarios/in_opentelemetry/tests/test_in_opentelemetry_001.py::test_in_opentelemetry_large_protobuf_logs \
tests/integration/scenarios/in_opentelemetry/tests/test_in_opentelemetry_001.py::test_in_opentelemetry_rejects_invalid_logs_payload

VALGRIND=1 VALGRIND_STRICT=1 \
FLUENT_BIT_BINARY=/tmp/flb-proto-input-test/bin/fluent-bit \
tests/integration/.venv/bin/python -m pytest -q \
tests/integration/scenarios/in_opentelemetry/tests/test_in_opentelemetry_001.py::test_opentelemetry_to_opentelemetry_basic_log \
tests/integration/scenarios/in_opentelemetry/tests/test_in_opentelemetry_001.py::test_in_opentelemetry_large_protobuf_logs \
tests/integration/scenarios/in_opentelemetry/tests/test_in_opentelemetry_001.py::test_in_opentelemetry_rejects_invalid_logs_payload

Commit-prefix validation passed for the full master..HEAD range.

Compatibility

There are no configuration or wire-format changes. The arena is local to one
decode operation and is not shared between requests or threads.

Summary by CodeRabbit

• Bug Fixes

  • Improved handling of large OpenTelemetry Logs protobuf payloads to support reliable processing and cleanup.
  • Continued rejecting invalid or oversized log payloads with appropriate error responses.

• Tests

  • Added coverage for large log batches.
  • Expanded invalid-payload testing to include oversized requests.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: fd02d322-8419-4721-93b2-0adb53b62f86

📥 Commits

Reviewing files that changed from the base of the PR and between 0896e9c and a012257.

📒 Files selected for processing (2)

• plugins/in_opentelemetry/opentelemetry_logs.c
• tests/integration/scenarios/in_opentelemetry/tests/test_in_opentelemetry_001.py

---

📝 Walkthrough

Walkthrough

The OpenTelemetry input plugin adds arena-backed protobuf decoding for large Logs payloads and matching cleanup. Integration tests cover large protobuf ingestion and rejection of invalid or oversized payloads.

Changes

OpenTelemetry protobuf arena decoding

Layer / File(s)	Summary
Arena allocator and protobuf wrappers plugins/in_opentelemetry/opentelemetry_logs.c	Adds cfl_arena allocation hooks and protobuf unpack/free wrappers using a configurable allocator.
Large-payload decoding and validation plugins/in_opentelemetry/opentelemetry_logs.c, tests/integration/scenarios/in_opentelemetry/tests/test_in_opentelemetry_001.py	Uses arena-backed decoding above the payload threshold, updates cleanup, and tests large and invalid protobuf payloads.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• fluent/fluent-bit#11421: Modifies the same binary_payload_to_msgpack packing and cleanup flow.

Suggested reviewers: cosmo0920

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: reducing protobuf decode allocation churn in in_opentelemetry.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch in_opentelemetry-protobuf-input-arena

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}