Add additional logging for number of records read from an S3 object, along with which object corresponds to a message for S3-SQS #6908
Conversation
graytaylor0
requested review from
KarstenSchnitter,
Zhangxunmt,
dinujoh,
divbok,
dlvenable,
kkondaka,
oeyh,
san81,
sb2k16,
srikanthjg and
srikanthpadakanti
as code owners
graytaylor0
force-pushed
the
S3LogAdditions
branch
from
b656b7d to
4107135
Compare
| parsedMessageVisibilityTimesMap.remove(parsedMessage); | ||
| } | ||
| if (result == true) { | ||
| LOG.info("Acknowledgement received, deleting SQS message for S3 object s3://{}/{}, sqsMessageId={}", |
There was a problem hiding this comment.
Is there not already a log for deleting messages? Maybe it's a debug log. If so, can we merge these into one?
There was a problem hiding this comment.
There is yeah. It is not DEBUG. I do agree having two logs when acknowledgments are enabled is not ideal. I considered downgrading the other Delete message log to DEBUG. That would mean that you would not see any info log signaling deletion when acknowledgments are not enabled though. Does this seem ok? Re-using the existing log requires that method to get s3 object context and maintain a mapping which is more invasive than I would prefer. I could also just put an if statement around the "Delete messages" log to not log that if acknowledgments are enabled. It's not consolidating but it doesn't change logging behavior for users without acknowledgments.
…bject was completed when messages are deleted Signed-off-by: Taylor Gray <tylgry@amazon.com>
graytaylor0
force-pushed
the
S3LogAdditions
branch
from
4107135 to
4e23ba7
Compare
3 participants
Description
This change adds an additional log for S3-SQS as a source that clearly shows which object corresponds to each SQS message on delete, and an additional log for the S3 source that shows how many records were read/written to the buffer after the object is processed.
Issues Resolved
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.