Lambda Destinations: What We Learned the Hard Way

AWS Lambda destinations, recently introduced, are a new way of efficiently directing events from AWS Lambda functions to various services in AWS. We've spent the past week banging around on the feature here at Trek10, and there were some surprises and hard lessons learned along the way that I think are useful to share.

As noted by the release posts and docs, Lambda destinations are only a feature of asynchronous (Event) Lambda invocations and those from stream sources.

Along with destinations, we were also given a whole host of other configurations to manage retries. In combination with other introduced knobs for streams, the capabilities to make your streaming scenarios more robust as well.

You can target the following services with destinations, making it easy to shuttle around events without having to write all that glue code.

(source: AWS Blog)

What does it look like

On any asynchronous function invocation (not stream sources, more on this later), you can set a destination for onSuccess and onFailure.

If your AWS Lambda function fully executes without any error, your responsePayload and some additional details are shipped off to your destination. If you function errors for any reason, we'll see that come through as an error.

Let's pretend we have a simple AWS Lambda function with the following code.

module.exports.handler = async event => {
 return event;
}

We want to ship this to an SQS queue onSuccess. We need to give the Lambda Execution Role rights to sqs:SendMessage to that queue (precisely as if we were doing this in code), and then set the onSuccess destination to the SQS queue ARN.

Next we invoke our function. aws lambda invoke --function-name test-destinations --invocation-type Event --payload '{"my": "event"}'.

If we check our queue, we see the following.

{
    "version": "1.0",
    "timestamp": "2019-12-13T20:04:08.088Z",
    "requestContext": {
        "requestId": "f14f5ab1-410f-4162-8c7b-c3f6b276a28c",
        "functionArn": "arn:aws:lambda:us-east-1:454679818906:function:sfn-lab-test-Stream-1UKZ0V094MA7T-StreamProcessor-12O2ALY0Z59LC:$LATEST",
        "condition": "Success",
        "approximateInvokeCount": 1
    },
    "requestPayload": {
        "my": "event"
    },
    "responseContext": {
        "statusCode": 200,
        "executedVersion": "$LATEST"
    },
    "responsePayload": {
        "my": "event"
    }
}

Now, about learning things the hard way. You CANNOT test destinations with the "Test" button in the console. It simply does not trigger to the configured destinations, I believe it is because console invocations are synchronous and not of the Event type (Asynchronous). We spent a bit banging our heads against this one, clicking test, seeing the execution successful in our console, and then nothing happening at our destinations. This is obviously looking back, but easy to misunderstand.

You'll also have noticed we get a ton of wrapper information and data. This can be particularly useful if a destination gets lots of events from different sources, or in the case of an error, we can investigate the requestPayload much closer and see what we can do to recover that data or from a bad state.

For example, given the following code.

module.exports.handler = async event => {
 throw new Error("failure example");
 return event;
}

We get the following result in our onFailure SQS Queue.

{
    "version": "1.0",
    "timestamp": "2019-12-13T20:06:50.820Z",
    "requestContext": {
        "requestId": "0972f748-a94c-4902-9ddb-8479e915c0b2",
        "functionArn": "arn:aws:lambda:us-east-1:454679818906:function:sfn-lab-test-Stream-1UKZ0V094MA7T-StreamProcessor-12O2ALY0Z59LC:$LATEST",
        "condition": "RetriesExhausted",
        "approximateInvokeCount": 3
    },
    "requestPayload": {
        "my": "event"
    },
    "responseContext": {
        "statusCode": 200,
        "executedVersion": "$LATEST",
        "functionError": "Unhandled"
    },
    "responsePayload": {
        "errorType": "Error",
        "errorMessage": "failure example",
        "trace": [
            "Error: failure example",
            " at Runtime.module.exports.handler (/var/task/app.js:5:9)",
            " at Runtime.handleOnce (/var/runtime/Runtime.js:66:25)"
        ]
    }
}

This makes for some pretty powerful paradigms and robust architectures. We now know that our payload of {"my": "event"} failed to process, the error message, etc. This makes handling and debugging pretty straight forward. Push all these into a queue and automatically process later, or queue for developer review.

Streams event sources are a special kind of beast

There are some big surprises you have to realize when using destinations, primarily if you are used to CLI or console invocations of lambda functions to test/simulate your streams.

Stream-based lambdas have an entirely different way of creating the onFailure destination. You do not configure them on the function destination configuration, rather you do so in the EventSourceMapping. In fact, you cannot even set an onSuccess destination. Presumably, because of scale reasons, AWS doesn't want to give you the ability to run over the destination services with massively scaled stream infrastructure.

Also note, you can set both EventSourceMapping onFailure and the async onSuccess and onFailure for the same function. Your destinations will be routed to based on the invocation type.

The AWS Lambda EventSourceMapping onFailure destination can only be one of SNS or SQS. No EventBridge, no other AWS Lambda functions. Once again, make sure that you are giving your function execution roles proper permissions to publish to your destination!

Now, another hard lesson learned. Even if you get smart from learning you can't use the console to test your destinations and use the CLI to invoke as an Event type, it will not get routed to your stream based onFailure configuration. It must be an event coming from the actual stream source. Just be aware of this, and know that for testing you'll need to change or create records in DynamoDB or push stuff through Kinesis.

Streams also support some pretty neat additional capabilities like defining the number of events to batch to each invocation, how long to wait in a window while batching events, bisecting a batch before retry (really useful for pinpointing "poison pill" data.), etc. You can read more about these features on the AWS Blog.

{
 "Type" : "AWS::Lambda::EventSourceMapping",
 "Properties" : {
 "BatchSize" : Integer,
 "BisectBatchOnFunctionError" : Boolean,
 "DestinationConfig" : DestinationConfig,
 "Enabled" : Boolean,
 "EventSourceArn" : String,
 "FunctionName" : String,
 "MaximumBatchingWindowInSeconds" : Integer,
 "MaximumRecordAgeInSeconds" : Integer,
 "MaximumRetryAttempts" : Integer,
 "ParallelizationFactor" : Integer,
 "StartingPosition" : String
 }
}

I would also recommend checking out the docs (CloudFormation ends up being the most helpful) for differences in asynchronous destinations config vs. the EventSourceMapping destinations configs.

Important Notes and Gotchas

We've run into a few other gotchas and lessons learned, some mentioned previously but here is a list. In true Festivus fashion...

We will try to keep this up to date as things change or get fixed or updated.

• Your execution policy of the lambda must have permission to put, publish, or invoke your destination source. The same way as if you were doing things in your business logic code.
• Streams (Kinesis, DynamoDB) EventSourceMappings only support onFailure. If you want to do something with the events on success, you do it in your code.
• The console invocations of lambda functions do not test destinations, only CLI calls of the Event type.
• When "adding" a destination in the console, if a destination already exists for the Async onFailure or onSuccess and you try to point it to another destination, it silently overwrites the previous one. It's more of an "update" than an "add" in those cases. Semantics, but worth knowing.
• You cannot update Stream EventSourceMappings in the console. If you want to edit batch sizes, or enable bisect, decrease retries, etc. You must delete and recreate it entirely.
• CloudFormation doesn't seem to correctly recognize stream EventSourceMapping updates as of this time (we have created a support ticket).
• SQS as a source to your function going to destinations is suspiciously missing from all the infographics and docs. We are still experimenting, but this one is odd.
• The docs around particular the difference between stream destinations and async could be much more precise.

A helpful chart

Many thanks to Forrest Brazeal, who contributed research to this post.