Draft
Conversation
…eberg tables Adds a new log4j-iceberg module that provides an IcebergAppender plugin for writing log events as Parquet-backed rows in an Iceberg table. The appender buffers events and periodically flushes them as Parquet data files committed to the table. Commit failures are retried with exponential backoff and table metadata refresh; persistent failures crash rather than silently dropping data. Uses Apache Iceberg 1.10.1 and Parquet 1.16.0. Supports any Iceberg catalog implementation (Hadoop, REST, etc.) via configuration.
Covers all code paths in IcebergAppender and IcebergManager including: - Table creation, loading existing tables, idempotent startup - Event write, batch-triggered flush, empty flush, manual flush - Events with thrown exceptions, null messages - Commit retry on transient failure, crash after retry exhaustion - Parquet write failure, interrupted retry - Stop lifecycle (closeable catalog, IOException on close, null scheduler) - Builder setters and factory method JaCoCo enforces 100% line coverage at build time.
Writes 1M log events through both a local FileAppender and the IcebergAppender, verifying correctness and reporting a side-by-side comparison of write throughput, read throughput, disk size, and Parquet compression ratio.
Writes 1K log events to a local Hadoop-catalog Iceberg table at /tmp/iceberg-inspect so the Parquet data files and Iceberg metadata can be examined directly on disk.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
INSERT HERE a clear and concise description of what the pull request is for along with a reference to the associated issue IDs, if they exist.
Checklist
2.xbranch if you are targeting Log4j 2; usemainotherwise./mvnw verifysucceeds (the build instructions)src/changelog/.2.x.xdirectory