Skip to content

external-libraries.md

Latest commit

 

History

History
102 lines (76 loc) · 7.18 KB

external-libraries.md

File metadata and controls

102 lines (76 loc) · 7.18 KB
description External libraries embedded in Fluent Bit

External libraries

Fluent Bit embeds several external libraries in its /lib directory. These libraries are included to minimize external dependencies and maximize cross-platform compatibility.

Core Fluent Bit libraries

These libraries are developed and maintained by the Fluent Bit team:

Library Purpose License
CFL Common data structures (strings, lists, key-value pairs, arrays, variants, time utilities, hashing) used across Fluent Bit and related projects. Apache 2.0
Chunk I/O Manages chunks of data in the file system for buffering. Handles memory-mapped I/O with optional CRC32 checksums for data integrity. Powers filesystem buffering for Fluent Bit. Apache 2.0
CMetrics Creates and manages metrics (counters, gauges, histograms, summaries) with label support. Based on Go Prometheus Client API design. Apache 2.0
CTraces Creates and manages trace contexts with encoding/decoding support for OpenTelemetry and other formats. Apache 2.0
CProfiles Creates and manages profiling data based on the OpenTelemetry Profiles schema. Apache 2.0
Fluent OTel Proto Provides C interfaces for OpenTelemetry protocol buffer definitions (common, resource, trace, logs, metrics). Apache 2.0
flb_libco Fork of libco for cooperative multithreading (coroutines). Includes ARMv8 fixes and macOS support. Internet Systems Consortium

HTTP and networking

Library Purpose License
Monkey Fast, lightweight HTTP server providing the embedded web server HTTP endpoints (health checks, metrics, and so on) for Fluent Bit. Apache 2.0
c-ares Modern asynchronous DNS resolver library. Enables non-blocking DNS queries for network operations. Massachusetts Institute of Technology (MIT)
nghttp2 HTTP/2 C library implementing the framing layer. Enables HTTP/2 client and server capabilities. Massachusetts Institute of Technology (MIT)

Data serialization and parsing

Library Purpose License
msgpack-c MessagePack serialization library. Used for internal data format for records in Fluent Bit. Boost 1.0
Jansson JSON encoding, decoding, and manipulation with full Unicode support. Massachusetts Institute of Technology (MIT)
yyjson High-performance JSON library capable of processing gigabytes per second. Used for fast JSON operations. Massachusetts Institute of Technology (MIT)
jsmn Minimalistic JSON parser (~200 lines of code). Zero-copy, no dynamic allocation. Massachusetts Institute of Technology (MIT)

Text processing

Library Purpose License
Onigmo Regular expression library (fork of Oniguruma). Supports Perl 5.10+ expressions like \K, \R, and conditional patterns. Default regular expression engine for Ruby 2.0+. BSD
tutf8e Tiny UTF-8 encoder supporting ISO-8859-x and Windows-125x character set conversions. Massachusetts Institute of Technology (MIT)
simdutf SIMD-accelerated Unicode validation and transcoding (UTF-8, UTF-16, UTF-32). Apache 2.0

Compression

Library Purpose License
miniz Single-file deflate/inflate, zlib-subset implementation. Provides gzip compression. Public Domain
Snappy Fast compression/decompression library. Prioritizes speed over compression ratio. BSD
zstd Zstandard compression providing high compression ratios at fast speeds. BSD

Scripting and WebAssembly

Library Purpose License
LuaJIT Just-in-time (JIT) compiler for Lua. Powers the Lua filter plugin for custom record processing. Massachusetts Institute of Technology (MIT)
WAMR WebAssembly Micro Runtime from the Bytecode Alliance. Enables WASM filter plugins and WASM input plugins. Supports interpreter, AOT, and JIT execution modes. Apache 2.0

Integration libraries

Library Purpose License
librdkafka Apache Kafka C/C++ client library. Enables the Kafka input and Kafka output plugins. Supports producer, consumer, and admin clients with high performance (millions of messages/second). BSD-2
Avro Apache Avro data serialization system. Apache 2.0

Storage and utilities

Library Purpose License
SQLite Self-contained SQL database engine. Used for persistent storage operations. Public Domain
jemalloc General-purpose memory allocator emphasizing fragmentation avoidance and scalable concurrency. Optional alternative to system malloc. BSD-2
lwrb Lightweight ring buffer implementation. Thread-safe FIFO buffer with zero-copy DMA support. Massachusetts Institute of Technology (MIT)
mpack Simple and fast MessagePack encoder/decoder. Massachusetts Institute of Technology (MIT)
libbacktrace Library for producing symbolic back traces. Used for debugging and error reporting. BSD

Version information

When Fluent Bit starts, it logs the versions of key libraries. For example, as of Fluent Bit 4.2:

[2025/01/15 10:30:00] [ info] [chunkio ] version=1.5.4
[2025/01/15 10:30:00] [ info] [ctraces ] version=1.0.5
[2025/01/15 10:30:00] [ info] [monkey  ] version=1.8.5
[2025/01/15 10:30:00] [ info] [wamr    ] version=2.4.1

Build options

Some libraries are optional and can be enabled or disabled at build time:

CMake option Library Default
FLB_WAMRC WAMR AOT compiler Off
FLB_LUAJIT LuaJIT On
FLB_OUT_KAFKA librdkafka Off
FLB_JEMALLOC jemalloc Off (platform dependent)

For more information on build options, see Build and install.