Losing Logs record fields

[natepage]

Hi there! 👋

We've recently managed to make our PHP based Lambda function work with:

• ddtrace-php sidecar trace sender feature on
• next-gen v75 lambda extension

Everything seems to be working fine, we get both traces and logs in Datadog, however they are not correlated. After investigation, I believe it is something to do with "losing logs data" somewhere in the process.

Our application is a PHP Symfony app, using Monolog as logger and have logs injection turned on, and we do log as JSON, so here is a typical log record in Cloudwatch:

{
    "message": "Read existing security token from the session.",
    "context": {
        "key": "_security_admin",
        "token_class": "Symfony\\Component\\Security\\Http\\Authenticator\\Token\\PostAuthenticationToken",
        "dd.trace_id": "13204366237012237818",
        "dd.span_id": "15763279168591393237",
        "dd.service": "notifications-api",
        "dd.version": "f8f423a934f5893992413072f2527cd6ff63bf11",
        "dd.env": "dev"
    },
    "level": 100,
    "level_name": "DEBUG",
    "channel": "security",
    "datetime": "2025-03-25T01:48:57.102952+00:00",
    "extra": {
        "X-CORRELATION-ID": "1f0091b5-0ac1-6a6e-9e44-c9dbdb3c4b5e",
        "X-REQUEST-ID": "1f0091b5-0ac1-6b36-8acf-c9dbdb3c4b5e"
    }
}

However when looking at this same record in Datadog console, information is missing:

I would expect to see the context at least in Datadog console,
Because we are using PHP which isn't natively supported on Lambda, do we need an additional setup?
Or are we simply missing something? 😄

[JordanGreissman]

I'm also seeing this in a Node-based lambda. Seems like this worked fine in 74-next but upgrading to 75 has broken our logs.

[astuyve]

Thanks folks it's likely related to #610 used to handle ddtags.

I'm taking a look

[astuyve]

Hey @JordanGreissman @natepage can you share your logging setup? I've modified my handler to log out Nate's log using Node like Jordan (I'm not a php expert). I'm using the JSON format option in Lambda and then logging:

  console.log({
    message: "Read existing security token from the session.",
    context: {
        key: "_security_admin",
        token_class: "Symfony\\Component\\Security\\Http\\Authenticator\\Token\\PostAuthenticationToken",
        "dd.trace_id": "13204366237012237818",
        "dd.span_id": "15763279168591393237",
        "dd.service": "notifications-api",
        "dd.version": "f8f423a934f5893992413072f2527cd6ff63bf11",
        "dd.env": "dev"
    },
    level: 100,
    level_name: "DEBUG",
    channel: "security",
    datetime: "2025-03-25T01:48:57.102952+00:00",
    extra: {
        "X-CORRELATION-ID": "1f0091b5-0ac1-6a6e-9e44-c9dbdb3c4b5e",
        "X-REQUEST-ID": "1f0091b5-0ac1-6b36-8acf-c9dbdb3c4b5e"
    }
  });

And I'm able to see the additional fields in Datadog:

The main difference is that in cloudwatch, the entire log is wrapped under the message attribute:

{
    "timestamp": "2025-03-26T17:38:46.101Z",
    "level": "INFO",
    "requestId": "4c93e027-eb17-4806-8934-edaba38f0e37",
    "message": {
        "message": "Read existing security token from the session.",
        "context": {
            "key": "_security_admin",
            "token_class": "Symfony\\Component\\Security\\Http\\Authenticator\\Token\\PostAuthenticationToken",
            "dd.trace_id": "13204366237012237818",
            "dd.span_id": "15763279168591393237",
            "dd.service": "notifications-api",
            "dd.version": "f8f423a934f5893992413072f2527cd6ff63bf11",
            "dd.env": "dev"
        },
        "level": 100,
        "level_name": "DEBUG",
        "channel": "security",
        "datetime": "2025-03-25T01:48:57.102952+00:00",
        "extra": {
            "X-CORRELATION-ID": "1f0091b5-0ac1-6a6e-9e44-c9dbdb3c4b5e",
            "X-REQUEST-ID": "1f0091b5-0ac1-6b36-8acf-c9dbdb3c4b5e"
        }
    }
}

so we parse/send the full message automatically.

Can you share your setup?

Thanks!

[astuyve]

Oh and @natepage we usually perform log/trace correlation using the lambda request ID which is present in your message so that should still work

[ericmbarnard]

Can confirm that I'm seeing the same in our node apps as well.

Currently

CloudWatch:

{
    "body": "",
    "level": "info",
    "message": "[Sendgrid] email response to: notifications-staging@harvestiq.com - Login Link | HarvestIQ {\n  result: { sendGridMessageId: '7WHU8VGGTXi75rBG-rU45w', statusCode: 202 },\n  body: ''\n}",
    "module": "mailClient",
    "result": {
        "sendGridMessageId": "7WHU8VGGTXi75rBG-rU45w",
        "statusCode": 202
    },
    "sqs_message_id": "7076595a-2681-404a-92a1-48c1184525e5"
}

DD:

Prior to v75

Same event type

CloudWatch:

{
    "body": "",
    "level": "info",
    "message": "[Sendgrid] email response to: notifications-staging@harvestiq.com - Login Link | HarvestIQ {\n  result: { sendGridMessageId: '2AyPAnPKQ4yvv05C-vxYyw', statusCode: 202 },\n  body: ''\n}",
    "module": "mailClient",
    "result": {
        "sendGridMessageId": "2AyPAnPKQ4yvv05C-vxYyw",
        "statusCode": 202
    },
    "sqs_message_id": "7fd9ef02-be02-4478-b8b7-b116801b6cac"
}

DD:

My hunch is that something isnt working with the serialization. I definitely have " wrapping quotes showing up in the Log Explorer that were not there last week.

Glad to provide any further configs/diagnostics - just let me know.

[tusharf5]

We're experiencing the same issue with our Node.js 22 Lambda APIs starting from extension version v75. Reverting to v69 resolved it. Let us know if debug logs or configurations would be helpful, we're happy to provide them.

[ValeriyaIvanovaPolestar]

We have the same issue after upgrade from v74 to v75, we use .NET 8 lambda over ARM64 arch. And a lot of the event attributes fields are missing now in Datadog console, even if the log in cloudwatch looks the same before and after upgrade.

[astuyve]

Thanks folks! An example configuration and code snippet would be helpful as I haven't been able to replicate this yet (see my above post), which makes me cautious to approach this as I don't want to break other workflows.

[ValeriyaIvanovaPolestar]

Thanks folks! An example configuration and code snippet would be helpful as I haven't been able to replicate this yet (see my above post), which makes me cautious to approach this as I don't want to break other workflows.

Sure, here is the lambda env vars we use :

The layer versions:

Here is how our log looks in Cloudwatch :
{ "timestamp": "2025-03-27T11:05:46.7028137Z", "message": "pno34: something", "level": "Information", "metadata": { "accountId": null, "region": "eu-north-1", "runtime": "AWS_Lambda_dotnet8", "functionName": "function", "functionMemorySize": "512", "functionVersion": "$LATEST", "isColdStart": null, "user": null, "resource": null, "timestampRequest": null, "viewerCountry": null, "version": "12345", "owner": "Bla", "hostPlatform": "aws", "domain": "lifecycle", "system": "az", "service": "management", "team": "app", "tags": [ "backend", ".net" ], "dataSensitivity": "proprietary", "jurisdiction": "EU", "stage": "prod", "id": "2025-03-27T11:05:46.702Z-07aef902-b4dc-449a-a8fc-b80f8187074e", "correlationId": null, "timestampEpoch": 1743073546, "isError": false, "httpStatusCode": null }, "Pno34": " something ", "XRayTraceId": "Root=12b0c;Parent=6f982;Sampled=0;Lineage=4:53bc803b:0" }

And here is visualization in datadog how these logs look like before and after the upgrade: ( sensitive data is covered)

Let me know if some other config is vital to share to replicate the problem .

[astuyve]

Thanks @ValeriyaIvanovaPolestar!
Could you share the logging code as well? If you're using a custom logging library, please include the version.

Thanks!

[tusharf5]

I've attached debug logs from a recent invocation where the logs fields were missing in this file, with all sensitive information obfuscated. logs-redacted.csv

We are using the serverless plugin with the following config.

  datadog:
    enabled: true
    enableXrayTracing: false
    enableDDTracing: true
    service: xyz-service
    env: dev
    tags: datacenter:aws
    enableDDLogs: true
    captureLambdaPayload: true
    subscribeToAccessLogs: false
    version: git-sha
    monitors: []

and the following environment variables.

    DD_ROLE: api
    DD_SERVICE: xyz-service
    DD_TECHNOLOGY: nodejs
    DD_LOGS_CONFIG_PROCESSING_RULES: '[{"type": "exclude_at_match", "name": "exclude_start_logs", "pattern": "^START RequestId"},{"type": "exclude_at_match", "name": "exclude_end_logs", "pattern": "^END RequestId"},{"type": "exclude_at_match", "name": "exclude_metric_logs", "pattern": "Embedded Metric"}]'

Packages:

"serverless-plugin-datadog": "~5.79.0",
"serverless": "~4.9.1"

[astuyve]

Thanks @tusharf5 - can you include the snippet of logging code? Are you using Lambda's JSON structured logs or text logs?

[tusharf5]

We are using powertools logger typescript (latest) with text logs

import { Logger } from '@aws-lambda-powertools/logger';

const logger = new Logger();

logger.info("some log", { key: value });

[ValeriyaIvanovaPolestar]

Thanks @ValeriyaIvanovaPolestar! Could you share the logging code as well? If you're using a custom logging library, please include the version.

Thanks!

Unfortunately we use custom internal logger based on Serilog , but I cannot share it with you since is private internal package :/.

[astuyve]

Ok thanks @ValeriyaIvanovaPolestar and @tusharf5. I'll try to run this down using Winston and text logs

Can folks please share if they are using Lambda's JSON structured logging format or if they're using text logs with a custom logger?

Thanks!