2 Minute Tip: Dead Letter Configs in Serverless

Wed, 15 Nov 2017

Dead letter configs are a pretty slick part of the AWS Serverless ecosystem. The basic idea is this: There are some events which trigger Lambda functions synchronously (such as API Gateway) and others which trigger Lambda asynchronously (such as an S3 event). For the latter group, AWS automatically retries if Lambda fails. This means that you can usually ignore transient Lambda errors. But what if it fails permanently? That’s where dead letter queues come in. You can assign a DeadLetterConfig to your Lambda function and it will push this failed event to an SQS queue or an SNS topic. From here you can set up an automated system to process these or simply alert on them for manual inspection.

The good folks at Serverless Framework added an OnError attribute so you could define a dead letter config easily in your serverless.yml. However, this requires that you create the SNS topic upfront and it does not support SQS. Creating the resources separately may make sense if you want a centralized queue, but we typically prefer that the dead letter resources (SNS or SQS) are part of the same stack to reduce dependencies and increase automation and portability.

Now for the good news!

There is a straightforward way to create a full dead letter config, including SQS/SNS resources, right in your Serverless project. It uses a less-well-known feature of Serverless Framework: the ability to define your function in the main part of serverless.yml and then add custom attributes to that in the Resources: (a.k.a. arbitrary Cloudformation) section of serverless.yml. It looks something like this:

iamRoleStatements: 
  - Effect: Allow 
    Action: 
    - sns:Publish 
    Resource: - { "Fn::Sub" : "${DeadLetterSNSTopic}"}
....
functions: 
  myFunction:
....
resources: 
  Resources: 
    MyFunctionLogGroup: 
      DependsOn: IamRoleLambdaExecution 
    MyFunctionLambdaFunction: 
      Type: AWS::Lambda::Function 
      Properties: 
        DeadLetterConfig: 
          TargetArn: 
            Ref: DeadLetterSNSTopic 
    DeadLetterSNSTopic: 
      Type: AWS::SNS::Topic

Note a couple tricks that are used here:

• In iamRoleStatements we give the function permission to publish to the SNS topic that is created as part of the stack.
• When you declare the function myFunction, Serverless will actually create a Cloudformation resource with the logical name MyFunctionLambdaFunction… note both the upper-casing of the first letter and the suffix added. This is the function name you need to use when adding attributes in the Resources section.
• We add DependsOn so that the IAM policy is created first, which allows the dead letter config to be successfully applied later.

Have fun dead-lettering!

Questions/comments? Feel free to reach us at info@trek10.com.