Make it possible to send very large span envelopes with >1000 spans & >200kb

[msonnb]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/aws-serverless

SDK Version

9.38.0

Framework Version

No response

Link to Sentry event

No response

Reproduction Example/SDK Setup

• followed https://docs.sentry.io/platforms/javascript/guides/aws-lambda/
• set SENTRY_TRACES_SAMPLE_RATE=1.0

Steps to Reproduce

Invoke Lambda function n times while it is still warm

Edit: After some more investigation I found out that this does not happen when disabling the OTel AwsLambdaInstrumentation. However, it also does not happen when running the OTel instrumention standalone without Sentry, so the issue seems to be constrained to the combination of Sentry SDK + Otel AwsLambdaInstrumentation

Expected Result

Have n traces show up in Sentry

Actual Result

n-1 traces, last one missing

following errors from @opentelemetry/instrumentation-aws-lambda in the Lambda logs:

Sentry Logger [error]: Spans may not be exported for the lambda function because we are not force flushing before callback
Sentry Logger [error]: Metrics may not be exported for the lambda function because we are not force flushing before callback

[linear]

JS-733 AWS lambda SDK is not sending all spans

[devdoshi]

To add to this issue, I'm using SDK 9.40.0 and I encounter an issue where normally traces, spans, logs, etc. are ingested but once I pass some limit of around 2000 spans in one lambda invocation, no trace or span shows up. Logs I believe all or mostly show up, and reference a trace id. So, it should be detectable on the backend when there are logs for a trace but no trace. I've increased the Sentry.wrapHandler option flushTimeout to 60000 but I haven't seen any change in behavior and I haven't looked any further into the code. I set up a function that makes n fetch and n aws s3 requests given a query parameter, and I doubled it until I stopped getting traces.

[andreiborza]

@devdoshi could you also double check your lambda's timeout threshold? I believe setting a high flushTimeout might not be enough if that value exceeds the value your lambda has been set up with.

We'll investigate this, thanks for chiming in! cc @msonnb

[msonnb]

Hey @devdoshi, there is a size limit of 200KB (compressed) per event, which you're probably running into. Depending on your actual use case, you might want to filter out spans that you don't need, for example using the beforeSendTransaction hook. The AwsInstrumentation integration also has an option suppressInternalInstrumentation which disables tracing for internal http requests made by the AWS SDK.

Sorry we don't have a better solution for now, we are planning on improving this in the future though.

[devdoshi]

Thanks both, my lambda timeout has been set to 15 minutes as part of debugging this. It makes sense that if there is a 200kb compressed upload size limit, that I could be reaching that with 2000+ spans. It is odd that I didn't see any mention of that in the logs though.

The traffic volume isn't very high, so I want all the spans, and I definitely don't want Sentry to silently drop them. As I mentioned, Sentry shows logs for the trace, but no spans, so I would expect Sentry to be able to tell me it is missing data, not for me to need to manually tune sampling or tracers or anything. I would also expect Sentry to be able to periodically flush spans, not wait until the end of my execution, if I understand what the current behavior is. Is that part of the improvements on the roadmap? If log ingestion is able to handle more, maybe spans can go to logs?

Anyway I'll revert to my aws-native otel instrumentation and I can try again if there are updates to this library/issue.

[andreiborza]

We can't currently periodically flush spans because they're tied to the transaction, but yes that's on the horizon. We're working on making it possible to stream spans.