Skip to content

out_s3: fix prefix match in log_key extraction#12008

Open
Vaibhav-C-S wants to merge 1 commit into
fluent:masterfrom
Vaibhav-C-S:fix-s3-log-key-exact-match
Open

out_s3: fix prefix match in log_key extraction#12008
Vaibhav-C-S wants to merge 1 commit into
fluent:masterfrom
Vaibhav-C-S:fix-s3-log-key-exact-match

Conversation

@Vaibhav-C-S

@Vaibhav-C-S Vaibhav-C-S commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

This fixes log_key extraction in the S3 output to require an exact key match.

Previously, flb_pack_msgpack_extract_log_key() compared the configured log_key against each record key using the record key length, but did not verify that both keys had the same length. This allowed shorter prefix keys to match longer configured keys.

For example, with log_key log_processed, a record containing both log and log_processed could upload the value of log if it appeared first in the record.

This change checks that the configured log_key length matches the record key length before comparing the key bytes.

For example, with this config:

log_key log_processed

and this record:

{"log":"wrong","log_processed":"correct"}

the record key log could match the configured key log_processed if log appeared first in the record, because only the first 3 characters are compared:

resulting output:

wrong

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
[SERVICE]
    Flush        1
    Log_Level    debug

[INPUT]
    Name    dummy
    Tag     test.s3
    Dummy   {"log":"wrong","log_processed":"correct"}
    Samples 1

[OUTPUT]
    Name                         s3
    Match                        test.s3
    bucket                       fluent-bit-test
    region                       us-east-1
    endpoint                     http://127.0.0.1:9000
    total_file_size            1M
    upload_timeout         5s
    store_dir                    /tmp/flb-s3-store
    s3_key_format          /logs/$TAG/%Y/%m/%d/%H/%M/%S-$UUID
    log_key                      log_processed

Verified uploaded object content locally with MinIO.
Before this change, the output uploadis:
wrong
After this change, with the same config, the uploaded object content is:
correct

  • Debug log output from testing the change
[2026/06/27 07:22:30.686] [debug] [upstream] KA connection #45 to 127.0.0.1:9000 is now available
[2026/06/27 07:22:30.686] [debug] [output:s3:s3.0] PutObject http status=200
[2026/06/27 07:22:30.686] [ info] [output:s3:s3.0] Successfully uploaded object /logs/test.s3/2026/06/27/01/52/23-qEfrOjVi
  • [N/A] Attached Valgrind output that shows no leaks or memory corruption was found
    This change only tightens the existing log_key comparison logic and does not add new allocations or frees.
    If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [N/A] Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

  • Bug Fixes
    • Improved log key matching for structured records so only exact key names are treated as matches.
    • Prevented partial prefix matches from being accepted when a shorter field name appears in the data.

Signed-off-by: Vaibhav-C-S <vaibhavsharma.c@gmail.com>
@coderabbitai

coderabbitai Bot commented Jun 27, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: f02e53d3-7119-458d-9b59-75c7ce65b2eb

📥 Commits

Reviewing files that changed from the base of the PR and between f317143 and 6d77e71.

📒 Files selected for processing (1)
  • plugins/out_s3/s3.c

📝 Walkthrough

Walkthrough

flb_pack_msgpack_extract_log_key now requires the configured log key to have the same length as the msgpack key before comparing bytes, preventing shorter prefix matches from passing.

Changes

S3 log-key matching

Layer / File(s) Summary
Exact-length log key match
plugins/out_s3/s3.c
flb_pack_msgpack_extract_log_key now requires strlen(ctx->log_key) == key_str_size before strncmp, so prefix-only matches no longer pass.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

Poem

🐇 I hopped through S3 with a tiny fix in tow,
A prefix-shaped matching bug had to go.
Now lengths agree before keys can align,
And the log-key check is neat, clear, and fine.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the exact log_key prefix-matching bug fixed in the S3 output plugin.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant