Support adding events to existing functions

[jlhood]

Update: This top-level comment contains the finalized spec based on feedback from the community. Next step is implementation. We're looking for contributors who can help implement this feature. Thanks!

Description:

Per #349 (comment), creating a separate RFC for a SAM syntax that would allow functions provided by nested applications (AWS::Serverless::Application) to be attached to event triggers via the same Events syntax as AWS::Serverless::Function resources.

High-Level Approach: Add a new SAM resource type: AWS::Serverless::FunctionReference. It doesn't create a new AWS::Lambda::Function resource. Instead, it allows you to specify the name and ARN of an existing function and then attach event sources to it using the exact same Events syntax as AWS::Serverless::Function.

The nice thing about this approach is it's not strictly tied to nested apps. It could be used with Fn::ImportValue for example, if you want to add triggers to a function created in a separate stack in your account.

However, we will offer a simplified syntax for the nested application use case specifically.

Example 1: Nested stack with function reference

  # Example native CloudFormation nested stack that creates a function and outputs the function's name as template output variable NestedFunctionName
  NestedStack:
    Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: <url>

  NestedFunctionExample:
    Type: AWS::Serverless::FunctionReference
    Properties:
      FunctionName: !GetAtt NestedStack.Outputs.NestedFunctionName
      Events: 
        GetRoot:
          Type: Api
          Properties:
            Path: /
            Method: GET

Example 2: Nested application with function reference

  # Example nested SAR application that creates a function and outputs the function's name as template output variable NestedFunctionName
  NestedApp:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: <app id>
        SemanticVersion: 1.0.0

  NestedFunctionExample:
    Type: AWS::Serverless::FunctionReference
    Properties:
      FunctionName: !GetAtt NestedApp.Outputs.NestedFunctionName
      Events: 
        GetRoot:
          Type: Api
          Properties:
            Path: /
            Method: GET

Example 3: More compact syntax for nested application use case

  # Example nested SAR application that creates a function and outputs the function's name as template output variable NestedFunctionName
  NestedApp:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: <app id>
        SemanticVersion: 1.0.0
      Events:
        NestedFunctionName: # This tells SAM to use !GetAtt NestedApp.Outputs.NestedFunctionName whenever we need the function name
          - Type: SQS
            Properties:
              Queue: !GetAtt MyQueue.Arn
              BatchSize: 10
          - Type: Api
            Properties:
              Path: /
              Method: GET

Example 4: Attaching events to function whose name is exported by a different stack

  # Assumes other stack exports function name as ExportedFunctionName
  ImportValueExample:
    Type: AWS::Serverless::FunctionReference
    Properties:
      FunctionName: !ImportValue ExportedFunctionName
      Events: 
        SQSTrigger:
          Type: SQS
          Properties:
            Queue: !GetAtt MyQueue.Arn
            BatchSize: 10

Notes/Questions:

1. Does SAM need both name and ARN specified or is one or the other sufficient? Might be event-specific... I know API events + implicit swagger require the function name and not ARN and AWS::Lambda::Permission can take name, ARN or alias ARN so it might be safer to require both? Alternatively, we could try to parse the ARN for the function name, but that's error-prone and also the ARN could be an intrinsic function dict and not the actual value itself.
   1. Decided to go with just FunctionName and construct the ARN assuming the function is in the same account.
2. Implementation needs to support Conditions.

[sgates]

I believe this would solve the major issue we're currently having, that is, sharing a single API Gateway across multiple AWS::Serverless::Application's. This would allow us to define groups of lambdas and endpoints in a modular way (in a sub stack) and then bring them all together in a unified API front end. This would be a huge win for those experiencing the pain of 200 resource limit, max policy size limits, or juggling multiple gateways for a "single" API.

[mikebroberts]

Definitely useful. I think this is a good first pass. Vague things on my mind:

• Try to allow a simple model if possible where only one of Name and ARN is necessary
• Do versions / aliases play into this more than just ARNs? E.g. would traffic shifting impact "upstream" function references?
• How would permissions work, e.g. if target Lambda in a different account - can SAM do anything to smooth that out?

[jlhood]

Thanks for the feedback, @mikebroberts!

Try to allow a simple model if possible where only one of Name and ARN is necessary

I think we can simplify it to only require FunctionName and then construct the ARN if necessary assuming the function is owned in the same account (see below for why cross-account wouldn't be supported).

Do versions / aliases play into this more than just ARNs? E.g. would traffic shifting impact "upstream" function references?

Good question. In cases where I've seen traffic shifting used, you end up doing something like !Sub ${MyLambda}:live wherever you need to specify a function name and it all "just works." So I think you could do the same for AWS::Serverless::FunctionReference specifying the alias for the function name and SAM should do the right thing, but I might be missing something. Is there a specific use case you can think of where this could be an issue?

How would permissions work, e.g. if target Lambda in a different account - can SAM do anything to smooth that out?

This is an interesting use case. After playing around with this, I don't see a way SAM can help in the cross-account case, because when you use SAM to add an event trigger to a function, it creates an AWS::Lambda::Permission resource in the template to add permissions allowing the relevant service principal to invoke the function. Lambda doesn't allow you to add a permission to a Lambda function owned by a different account so this wouldn't deploy.

[jlhood]

We were discussing this today and while we agreed AWS::Serverless::FunctionReference is a more general solution (works with AWS::Serverless::Application, AWS::CloudFormation::Stack and Fn::ImportValue), we should try to provide an even more concise experience for the specific use case of adding triggers to functions defined within an AWS::Serverless::Application resource. We came up with this syntax:

  NestedApp:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: <app id>
        SemanticVersion: 1.0.0
      FunctionReferences:
        NestedFunctionName: # the key is the nested app output that should be used as the FunctionName property of the underlying FunctionReference
          MyApiTrigger:
            Type: Api
            Properties:
              Path: /
              Method: GET
          MySQSTrigger:
            Type: SQS
            Properties:
              # ...

Concerns:

1. Syntax is slightly different from function events which could be confusing.
2. FunctionName output as object key could be confusing/hard to read for users.
3. If we find out during implementation (or with future event types) that we need more than just the function name, we won't be able to easily expand this syntax since the function name is an object key instead of a property.

Wanted to put this out to the community for feedback. @sgates @mikebroberts Any strong objections or feedback?

[sgates]

I think this makes sense - In this scenario, the API Gateway still lives in the main template, and then groups of lambdas (the NestedApps) would effectively be given access to it through the FunctionReferences in the main template? If I'm understanding that correctly I think that would be fine.

Would this mean in the nested stack, the lambdas defined there don't need Event properties because those "trickle down" from the main template?

1. This is true, could the syntax not just be the the same - like add an "Events" level that contains MyApiTrigger and MySQSTrigger?
2. Would this be implicit, or would it require the user to explicitly mark these as exports in the nested stack?
3. True... not sure what the alternative would be, this seems pretty simple, anything more general might be more complicated

[jlhood]

Thanks for the quick response, @sgates! In general, this is more a developer experience/syntax discussion, not changing the proposed functionality described in the original AWS::Serverless::FunctionReference RFC. Just adding extra convenience for the nested app use case.

This is true, could the syntax not just be the the same - like add an "Events" level that contains MyApiTrigger and MySQSTrigger?

I'll play around with the syntax. You're right that we might be able to get it even closer to the Events syntax. The only caveat is that we have to map the function name somewhere, although the Events object keys could do that as long as we're willing to accept the risk of my 3rd concern.

Would this be implicit, or would it require the user to explicitly mark these as exports in the nested stack?

The nested stack template would have to include the function name as a template output, but would not have to export the output.

True... not sure what the alternative would be, this seems pretty simple, anything more general might be more complicated

Thanks, that's helpful feedback!

[jlhood]

I think this is the closest I can get to looking like the Events property of AWS::Serverless::Function:

  # Example nested SAR application that creates a function and outputs the function's name and ARN as template output variables: NestedFunctionName
  NestedApp:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: <app id>
        SemanticVersion: 1.0.0
      Events:
        NestedFunctionName: # This tells SAM to use !GetAtt NestedApp.Outputs.NestedFunctionName whenever we need the function name
          - Type: SQS
            Properties:
              Queue: !GetAtt MyQueue.Arn
              BatchSize: 10
          - Type: Api
            Properties:
              Path: /
              Method: GET

I had to turn it into an array of event triggers, because normally SAM uses different keys for different event triggers, but if we're using the key for the function name, we need a way to support multiple event triggers on the same function.

[sgates]

This makes sense to me, and the syntax isn't as confusing as the previous version because it explicitly calls out "Events" - more intuitive IMO. Thanks for all the effort on this!

[TeddyHandleman]

@jlhood To me this looks like it's going in the right direction. The only thing I see that may be a potential issue is integration with an OpenAPI document via the DefinitionBody parameter in an AWS::Serverless::Api resource.

For example:

ApiResource:
      Type: AWS::Serverless::Api
      Properties:
        Name: MyApi
        StageName: foo
        DefinitionBody:
          Fn::Transform:
            Name: AWS::Include
            Parameters:
              Location: ./openapi.yaml

In the OpenAPI document you might have an APIG integration like so:

 x-amazon-apigateway-integration:
        uri:
          Fn::Sub: "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${MyFunction.Arn}/invocations"

However, maybe your solution just means a change of MyFunction.Arn -> NestedApp.Outputs.MyFunctionArn

[TeddyHandleman]

@jlhood any further updates?