Is your feature request related to a problem? Please describe.
S3-SQS based collector in DataPrepper injects an extra field for s3 metadata into the dictionary it sends to processors.
{
... # event dictionary
"s3": {
"bucket": "test-1758100462",
"key": "json-debug-1758100510.json"
}
We want to have more control over enabling/disabling this injection of s3 metadata into the dictionary.
Describe the solution you'd like
We could use data_selection: data_only to control the injection for S3 Scan based configs. We ideally should have a similar field for S3-SQS based configs as well.
It will give us the ability to control whether to inject only data or only metadata or both data & metadata.
While having the metadata injected can be a valid use case for customers, but there may be cases when not having it is better suited.
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
For the given input event on S3 :
{
"Records": [
{
"eventVersion": "1.08",
"userIdentity": {"type": "IAMUser"},
"eventName": "CreateBucket"
}
]
}
Output (sink as CW Logs)
Raw CloudWatch Logs JSON
{
"timestamp": 1758100525190,
"message": "{\"eventVersion\":\"1.08\",\"userIdentity\":{\"type\":\"IAMUser\"},\"eventName\":\"CreateBucket\",\"s3\":{\"bucket\":\"test-1758100462\",\"key\":\"json-debug-1758100510.json\"}}",
"ingestionTime": 1758100526104
}
With added injection of metadata
{
"eventVersion": "1.08",
"userIdentity": {"type": "IAMUser"},
"eventName": "CreateBucket",
"s3": {
"bucket": "test-1758100462",
"key": "json-debug-1758100510.json"
}
}
Is your feature request related to a problem? Please describe.
S3-SQS based collector in DataPrepper injects an extra field for s3 metadata into the dictionary it sends to processors.
We want to have more control over enabling/disabling this injection of s3 metadata into the dictionary.
Describe the solution you'd like
We could use
data_selection: data_onlyto control the injection for S3 Scan based configs. We ideally should have a similar field for S3-SQS based configs as well.It will give us the ability to control whether to inject only data or only metadata or both data & metadata.
While having the metadata injected can be a valid use case for customers, but there may be cases when not having it is better suited.
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
For the given input event on S3 :
Output (sink as CW Logs)
Raw CloudWatch Logs JSON
With added injection of metadata