Skip to content

Commit 26395a8

Browse files
estherk15claudemaycmlee
authored
Clarify JSON array ingestion behavior for S3 log files (#1086)
* Clarify JSON array ingestion behavior for S3 log files Adds a troubleshooting section explaining that JSON arrays in S3 log files are ingested as a single event and that NDJSON format is required for separate log entries. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update aws/logs_monitoring/README.md Co-authored-by: May Lee <mayl@alumni.cmu.edu> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: May Lee <mayl@alumni.cmu.edu>
1 parent fc050d7 commit 26395a8

1 file changed

Lines changed: 11 additions & 0 deletions

File tree

aws/logs_monitoring/README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -282,6 +282,17 @@ If you still couldn't figure out, please create a ticket for [Datadog Support][1
282282

283283
If your logs contain an attribute that Datadog parses as a timestamp, you need to make sure that the timestamp is both current and in the correct format. See [Log Date Remapper][24] to learn about which attributes are parsed as timestamps and how to make sure that the timestamp is valid.
284284

285+
### S3 log files containing JSON arrays are ingested as a single event
286+
287+
The Forwarder ingests each line of a log file as a separate event. If your S3 log file contains a JSON array (for example, `[{...}, {...}]`), the entire array is treated as a single log event. Datadog log pipelines and processors cannot split a JSON array into multiple individual events after ingestion.
288+
289+
To ensure each log entry is ingested as a separate event, format your log files as newline-delimited JSON (NDJSON), where each JSON object is on its own line:
290+
291+
```json
292+
{"key": "value1"}
293+
{"key": "value2"}
294+
```
295+
285296
### Issue creating S3 triggers
286297

287298
In case you encounter the following error when creating S3 triggers, we recommend considering following a fanout architecture proposed by AWS [in this article](https://aws.amazon.com/blogs/compute/fanout-s3-event-notifications-to-multiple-endpoints/)

0 commit comments

Comments
 (0)