Skip to content

DataPrepper S3-SQS source config should support data_selection similar to S3 Scan configs #6149

Description

@BhattacharyaSumit

Is your feature request related to a problem? Please describe.
S3-SQS based collector in DataPrepper injects an extra field for s3 metadata into the dictionary it sends to processors.

{
  ... # event dictionary
  "s3": {
    "bucket": "test-1758100462",
    "key": "json-debug-1758100510.json"
}

We want to have more control over enabling/disabling this injection of s3 metadata into the dictionary.

Describe the solution you'd like
We could use data_selection: data_only to control the injection for S3 Scan based configs. We ideally should have a similar field for S3-SQS based configs as well.
It will give us the ability to control whether to inject only data or only metadata or both data & metadata.

While having the metadata injected can be a valid use case for customers, but there may be cases when not having it is better suited.

Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

For the given input event on S3 :

{
  "Records": [
    {
      "eventVersion": "1.08",
      "userIdentity": {"type": "IAMUser"},
      "eventName": "CreateBucket"
    }
  ]
}

Output (sink as CW Logs)

Raw CloudWatch Logs JSON

{
  "timestamp": 1758100525190,
  "message": "{\"eventVersion\":\"1.08\",\"userIdentity\":{\"type\":\"IAMUser\"},\"eventName\":\"CreateBucket\",\"s3\":{\"bucket\":\"test-1758100462\",\"key\":\"json-debug-1758100510.json\"}}",
  "ingestionTime": 1758100526104
}

With added injection of metadata

{
  "eventVersion": "1.08",
  "userIdentity": {"type": "IAMUser"},
  "eventName": "CreateBucket",
  "s3": {
    "bucket": "test-1758100462",
    "key": "json-debug-1758100510.json"
  }
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Done

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions