AWS Lambda Functions: Return Response and Continue Executing

A feature I’ve always wanted from AWS Lambda functions is the ability to return a response and continue executing. One use case for this is Slack Slash command handler which must return an initial response within 3 seconds. Of course, you could do something similar to this by invoking a second Lambda function (or the same one) or an AWS step function, but both of these significantly increase the complexity. For simple use cases, it would be very convenient to be able to return a response and keep executing.

When AWS Lambda extensions were released I hoped that they would update the internal Lambda API to allow this. Unfortunately, they did not. I had the same hope when Lambda streaming responses were released and this time I was not disappointed. While it’s not documented you can in fact return a response and continue executing in certain circumstances.

In this post, I’ll explain how to do this using the Node.js Lambda runtime.

But first, I’d like to explain how Lambda streaming responses work. According to the documentation, to enable streaming responses you have to turn on function urls and enable the streaming response. In your code, you have to wrap your handler function with a wrapper provided by the AWS Lambda Node.js runtime. This wrapper is accessible on the Javascript global object.

Here are the CloudFormation resource definitions I used to create a test function:

Resources:
  Trek10LambdaTestFunctionWithUrl:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      FunctionName: Trek10-Lambda-Test-Function-With-Url
      CodeUri: ./src
      Runtime: nodejs18.x
      Timeout: 30
  Trek10LambdaTestUrl:
    Type: AWS::Lambda::Url
    Properties:
      AuthType: NONE
      InvokeMode: RESPONSE_STREAM
      TargetFunctionArn: !GetAtt Trek10LambdaTestFunctionWithUrl.Arn
  Trek10LambdaTestUrlPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunctionUrl
      FunctionUrlAuthType: NONE
      Principal: '*'
      FunctionName: !GetAtt Trek10LambdaTestFunctionWithUrl.Arn

And here is the handler code:

const util = require('util')
const sleep = util.promisify(setTimeout)

// awslambda is patched into the global object by the lambda runtime
module.exports.handler = awslambda.streamifyResponse(async function(event, responseStream, context) {
  responseStream.write('Hello my request id is: ' + context.awsRequestId)
  responseStream.end();
  console.log('response sent')
  // the response has been sent and the request to lambda should return very quickly.
  // now we simulate doing some work by sleeping
  await sleep(5000)
  console.log('done waiting')
})

As you can see from the code above all you have to do to continue executing after returning is simply end the stream but don’t return from your function.

Here is the output from calling the lambda URL as well as the corresponding execution logs from CloudWatch:

$ time curl https://wrwtktb7psotxknjhvwnllkpuu0cxvtb.lambda-url.us-east-1.on.aws/
Hello my request id is: 654c7282-1216-496c-956b-e9ad124752c7
real	0m0.240s
user	0m0.015s
sys	0m0.015s

$ aws logs filter-log-events --log-group-name /aws/lambda/Trek10-Lambda-Test-Function-With-Url --filter-pattern '"654c7282-1216-496c-956b-e9ad124752c7"' --output text --query 'events[*].message'
START RequestId: 654c7282-1216-496c-956b-e9ad124752c7 Version: $LATEST
2023-11-05T03:01:34.355Z	654c7282-1216-496c-956b-e9ad124752c7	INFO	response sent
2023-11-05T03:01:39.356Z	654c7282-1216-496c-956b-e9ad124752c7	INFO	done waiting
END RequestId: 654c7282-1216-496c-956b-e9ad124752c7
REPORT RequestId: 654c7282-1216-496c-956b-e9ad124752c7	Duration: 5003.61 ms	Billed Duration: 5004 ms	Memory Size: 128 MB	Max Memory Used: 69 MB

As you can see from the time output the Lambda URL invocation took 240ms, but you can see that at the whole Lambda duration was 5004ms.

Now you may be asking “What if I want to continue executing after returning a response, but also want to invoke via the standard API (or via an ApiGateway proxy integration) and not via the Function URL?”

No problem. Just do it. In fact, you don’t even need to have the Function URL enabled for this to work.

Here is another CloudFormation resource definition, notice there is no URL configuration or permission:

Trek10LambdaTestFunctionWithNoUrl:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      FunctionName: Trek10-Lambda-Test-Function-With-No-Url
      CodeUri: ./src
      Runtime: nodejs18.x
      Timeout: 30

This function uses the same handler code as the previous function. Here is the output when calling via the API/CLI:

$ time aws lambda invoke --function-name Trek10-Lambda-Test-Function-With-No-Url result_no_url.txt
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

real	0m0.690s
user	0m0.221s
sys	0m0.072s

$ cat result_no_url.txt
Hello my request id is: 145b188e-dc16-49fb-9ec5-1b2b270d350c

$ aws logs filter-log-events --log-group-name /aws/lambda/Trek10-Lambda-Test-Function-With-No-Url --filter-pattern '"145b188e-dc16-49fb-9ec5-1b2b270d350c"' --output text --query 'events[*].message'
START RequestId: 145b188e-dc16-49fb-9ec5-1b2b270d350c Version: $LATEST
2023-11-05T03:12:56.715Z	145b188e-dc16-49fb-9ec5-1b2b270d350c	INFO	response sent
2023-11-05T03:13:01.720Z	145b188e-dc16-49fb-9ec5-1b2b270d350c	INFO	done waiting
END RequestId: 145b188e-dc16-49fb-9ec5-1b2b270d350c
REPORT RequestId: 145b188e-dc16-49fb-9ec5-1b2b270d350c	Duration: 5084.14 ms	Billed Duration: 5085 ms	Memory Size: 128 MB	Max Memory Used: 68 MB

Again you can see that the API call took much less time than the Lambda invocation. (Also note you can return non-JSON results with the streaming response).

Conclusion

To summarize, your Lambda can return a response yet continue executing by simply wrapping the handler with the streaming response and then just doing more things after ending the stream.

There is one caveat here: if you invoke your Lambda via the API and request the last 4kb of logs via the --log-type=Tail option, you don’t get a response until the Lambda execution ends.

Is this possible without using the streaming response wrapper? Not as far as I can tell if you are using the AWS-provided Node.js runtime. I hope to be able to do it in a custom runtime or with an internal Lambda extension. I’m hoping to test that in the future, and I’ll post what I find here if it’s interesting.

Templates and function code can be found in this repo.