Possible memory leak? After cold start, lambda keeps consuming memory

[nicoan]

Hello,

Working on some lambdas I noticed that the memory consumption keeps incrementing and never goes down. I made several tests all of them threw more or less the same results. Here's the code I used for testing:

use lambda_http::{service_fn, Error, Request};

pub async fn function_handler(_: Request) -> Result<&'static str, Error> {
    Ok("")
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    lambda_http::run(service_fn(function_handler)).await
}

Cargo.toml

[package]
name = "test_leak"
version = "0.1.0"
edition = "2021"

[dependencies]
lambda_http = "0.14.0"
tokio = { version = "1", features = ["macros"] }

To run the tests, I used an script that calls an API Gateway endpoint that calls the lambda:

#!/bin/bash
for ((i=1; i<=5000; i++)); do
    curl -i -X POST <api_g_url> -H 'Content-Type: application/json' --header 'Content-Length: 6' -d '"test"'
done

Here are the graphs of the memory consumption from the tests:

[jlizen]

A few questions:

1. Is the memory growing without bounds to the point where the lambda is crashing? IE, are we sure this isn't just the allocator holding onto memory and not releasing it to the system, even if isn't actively used? Since, these metrics would reflect all memory held by the process, not just memory that is in active use.

2. Does the same behavior appear locally when run with cargo lambda? Presumably it should, since the runtime is unchanged, just the outside orchestrator is different? Heap profiling would be much simpler in your local than cargo lambda.

3. What happens when you switch the allocator from system allocator to jemalloc?

[nicoan]

Hello @jlizen

1. I never got to the point where the lambda crashed. I made two kind of tests:

• Hitting it continously for 5, 10 and 15 min without pause: here the memeory continously grew
• Do a little pause and hit it again continously: this was to prevent a cold start, the memory started grewing again from the last point, the lambda did not free the memory.

Unfortunately I no longer have access to an AWS account to continue this kind of tests.

Regarding 2 I did not try that. I'll try to profile it with jemalloc to try to figure out what is happening

[jlizen]

the memory continuously grew

Ah, I see, in the charts it looked like it was plateauing after an initial increase. If it continues growing further with subsequent executions, that is indeed suspicious.

[nicoan]

the memory continuously grew

Ah, I see, in the charts it looked like it was plateauing after an initial increase. If it continues growing further with subsequent executions, that is indeed suspicious.

Yes! It plateaus, then spike, then plateau, then spike, but it always goes up. The plateau intervals are not regular, that is also strange

Hi folks,

We've investigated this, and done tests both locally and on production.
So the reason for max-memory-used is fragmentation, with the culprit being serde.

We run thousands of invokes both overriding the global alloc with tracking allocator, and did a flamegraph, and we see that the active memory stays fixed after a few invokes, in both cases there's some continuous allocation due to serde parsing payloads and then deallocating.

Many thanks for prompting this investigation!

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.

[jlizen]

@PartiallyUntyped is there a recommendation for users for reducing fragmentation? Conventional wisdom is usually to use jemalloc, though these days I reach for mimalloc first. Did you try those in profiling / did they behave differently?

Also, do we have a sense of scale wrt extent of continuous allocation pressure from serde usage and how it affects invocation latency? Needing to request more memory from the os with some frequency seems like it could impact tail latencies. Ie, is this something we care to keep an open issue for around optimization?

[nicoan]

Awesome news that you found the cause! Sorry I couldn’t do the profiling myself, I’ve been short on time.

So our doc is pending internal review and we will share our findings here. We have been pretty rigorous on this and @mpindaru has done a great job!

[indiv0]

@PartiallyUntyped hello! Seems like I might be running into the same issue. Are there any updates on the doc? Would like to know the recommended strategy to address this problem.

As a more specific sub-question: it seems like ApiGatewayProxyRequestContext<T> defaults to T = serde_json::Value. Is there any way to replace that with ijson::IValue (or some other, more memory-efficient type) via the lambda_runtime/lambda_http public API?

Hi @indiv0 I am looking at publishing our report in the next few days. What we recommend for now is tuning your allocator, for example using mimalloc, enabling background thread in jemalloc, or reducing arenas to 1 in glibc malloc.

From our investigation we found that memory allocated to the runtime reaches a ceiling, and stops growing.

Due to the nature of the problem we can't provide a prescriptive solution that fits all.

What we are looking at is:

• enabling users at using more compact types,
• using events with a lifetime,
• enabling users to use a bumping allocator like bumpallo for the deserialization.

[jlizen]

@PartiallyUntyped these sound very promising:

What we are looking at is

• enabling users at using more compact types,
• using events with a lifetime,
• enabling users to use a bumping allocator like bumpallo for the deserialization.

Is there an issue open already for users to follow along? Are you interested in community contributions or are developers with Lambda already planning to pick this up?

Our goal here is to provide as much flexibility as possible and not impose what we believe is the best. Rust thrives because of its community and we want to embrace that.

The plan for now is to draft a few ideas, iterate on them internally, and then request feedback from the community.

During work hours tomorrow I will write a separate issue that the community can use to follow along.