RuntimeDependenciesLayer rebuilding on every API request

[Patrick-Erichsen]

First of all, thank you @Envek and others for both the great writeup at https://evilmartians.com/chronicles/serverless-typescript-a-complete-setup-for-aws-sam-lambda and for this template repo.

This is my first time using AWS SAM in general, and it's been tricky with Typescript.

Problem

When running sam local start-api, every request I make to http://127.0.0.1:3000/ triggers a rebuild of the dependency layer.

Steps to reproduce

• Clone fresh copy of repo and walk through the setup
• sam local start-api
• Visit http://127.0.0.1:3000/

The first visit will build the image as expected.

When refreshing the browse, based on this comment, I would expect the image not to rebuild since no source code would have changed.

However, on each request, the image appears to be rebuilt with the following log line:

RuntimeDependenciesLayer is a local Layer in the template

The time it takes to rebuild this image makes it difficult to develop my frontend.

The workaround I have right now is to just point my frontend at my dev environment in AWS, but I'd prefer to be able to run both front & backend locally.

[Envek]

That's weird! I don't remember such nasty behavior.

What is your SAM CLI version? It may be some SAM CLI bug…

[Patrick-Erichsen]

Hey @Envek sorry for the delay, I didn't see your reply come through.

$ sam --version

SAM CLI, version 1.29.0

[kurkle]

+1

sam --version  
SAM CLI, version 1.23.0

Edit: No change after updating to 1.31.0
Edit2: aws/aws-sam-cli#1630 (comment)

[tmbs]

+1

~ sam --version
SAM CLI, version 1.31.0

~ sw_vers
ProductName:	macOS
ProductVersion:	11.6
BuildVersion:	20G165

[jettandres]

+1 also. I'm kinda new to AWS SAM. Thank you sir @Envek for the ts template

~ sam --version        
SAM CLI, version 1.33.0

[semmgeorge]

+1
SAM CLI, version 1.29.0

[moretalk]

+1
SAM CLI, version 1.35.0

[justintilson]

One way I found that changes this behavior is to include the --warm-containers=EAGER flag when starting your local SAM API. Example:

sam local start-api --docker-network your-network --env-vars=env.json --warm-containers=EAGER

From: https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-cli-command-reference-sam-local-start-api.html with some more relevant tidbits here: https://aws.amazon.com/blogs/compute/working-with-aws-lambda-and-lambda-layers-in-aws-sam/

[ahurlburt]

Seems like the workaround for

--warm-containers=EAGER

Does stop the containers rebuilding the layer each time. This is interesting as the docs (cli ref) imply there would be no difference between EAGER and LAZY with regards to layer rebuilding.

While this is helpful it's not great since a lot of times I am only testing a subset of lambdas and don't need the others to invoke at all (or build). Warming all of them still slows down dev process which is what sam local is intended to help with.

This related ticket implies to solve this one would need to publish the lambda and then reference via ARN: aws/aws-sam-cli#1630

IMO this isn't really a great solution since i want SAM to build the layer for me. I use sam so i can have everything in one place and a reproducible build.

I believe the reason to rebuild the layer is because sam needs to check if anything changed. However, i think practically speaking it would be a better solution to assume the layers do not change like the templates and in the event of a layer update sam local would need to be restarted. I think realistically most people want watch type behaviour for their lambda code and not so much the layers which is either shared libs or dependencies.

I also get a lot of errors using the EAGER warm flag, once I start making changes to the code some of the containers don't seem to exit properly and it causes sam local to crash. Then i need to start everything up again which leads me back to why I don't like having to warm everything to avoid the lambda layer rebuild...

[Envek]

This is great research, @ahurlburt!

Please open issue in https://github.com/aws/aws-sam-cli with this text so SAM CLI developers can be aware of it and hopefully will fix it some day.

[ahurlburt]

@Envek done: aws/aws-sam-cli#3725

Maybe people coming across this can upvote :)

[eejlunt]

@Envek really liked your blog post that pointed me to this repo. I didn't use this template as I was updating an existing project (from 4 years ago) to have better developer experience. I moulded the project to be like this, but I ran into the same issue (sam cli 1.121.0) that every code change (using --warm-containers LAZY) killed off the warm container and rebuilt the layer, which would normally take over 1 minute.

I think the issue is that both ContentUri (Layer) and CodeUri (Lambda) are both looking the project root folder, sam cli was picking up TS files changes (and potentially the dist folder too) and deciding that the layer needed rebuilding. I couldn't change either of these, otherwise it would involve moving the Makefile around, which is not ideal.

My solution was to update the sam template with a new parameter:

AttachLayer:
  Type: String
  Default: "true"
  AllowedValues:
    - "true"
    - "false"
  Description: Controls whether to attach the Lambda layer to functions

Then a condition:

Conditions:
  AttachLayerCondition: !Equals [!Ref AttachLayer, "true"]

And then finally in globals/function section, use this condition as to whether the layer should be attached to the function:

Globals:
  Function:
    CodeUri: ./
    Layers:
      - !If
        - AttachLayerCondition
        - !Ref RuntimeDependenciesLayer
        - !Ref "AWS::NoValue"

Then use the command sam local start-api --warm-containers LAZY --parameter-overrides AttachLayer=false to start it all locally. As the docker image running the lambda function(s) mounts the project root to /var/task in the container, it already includes the node_modules folder, just as it does for local developing/testing.