Review of "Testing Serverless Architectures" by Yan Cui

I will be the first to admit that I do not put as much effort into writing automated tests as I do into writing the code/systems they are supposed to validate. Test cases are boring to me generally, but I am trying to learn to love them because I understand their importance. With this goal in mind, I am always searching for valuable resources to learn more about the current state of the art with regard to testing the serverless applications that I create. One such resource that was mentioned to me recently is “Testing Serverless Architectures” by Yan Cui which seeks to address common concerns with testing serverless apps such as:

• Uncertainty on how to write any integration or end-to-end tests
• Speed of tests
• Testing direct service integrations such as API Gateway’s direct integration with DynamoDB
• Uncertainty on whether or not to use LocalStack

“Testing Serverless Architectures” is a paid course available on Cui’s own site that is designed to teach you about efficient design patterns and methodologies for unit/integration/end-to-end tests; I will review this course here in order to help raise awareness on the pros/cons of its content. Note that the course comes in two flavors: “basic” and “pro” -- in this post, I am reviewing the “pro” variant. Specifically, I’ll provide an overview of what is actually inside of the “pro” course as well as short highlights of what I found to be interesting, as well as any areas for improvement.

The course is divided into a short introduction section as well as six chapters. The introduction, much like this blog post itself, describes the contents of Cui’s course as well as provides some explanation of what common issues developers face when testing serverless applications. The course claims that it assumes that you are a “newcomer to serverless.” In the event that you are not a newcomer to serverless, then the section explaining common issues associated with testing serverless apps will not be particularly useful, as you will likely already understand the situation.

The first chapter aims to address the “common issues” faced surrounding testing serverless apps mentioned in the introduction via describing practical, actionable ideas to the viewer. For example, the introduction describes serverless apps as a paradigm shift in application development that lacks enough training materials or established best practices. Cui resolves this issue in chapter one by presenting ideas such as a “testing honeycomb” which assumedly serves as being a potential new “best practice” in the serverless space. Essentially the “testing honeycomb” is a framework in which greater emphasis is placed upon integration tests rather than unit tests. This idea, along with other various ideas in the first chapter such as the suggestion to use temporary environments for testing purposes, does provide legitimate value to the viewer. While you may have at least heard of [or used] concepts such as using a temporary environment for end-to-end testing, it is likely that this first chapter will contain some new information and will leave you wanting to learn about more specific details on how to test various types of serverless applications.

Luckily, the next four chapters of this course (chapters 2-5) of this course dive into highly specific details on how to test:

• API Gateway APIs
• AppSync APIs
• Step Functions
• Event-Driven Architectures

These four chapters contain the bulk of the content of the course, as they provide a thorough explanation of the testing methodology for each specific type of application as well as a full walkthrough of a sample application that represents each type. The sample application, which is provided as a downloadable zip archive, includes complete examples of what a project looks like when you implement unit, integration, and end-to-end tests according to Cui’s ideas. I believe there is tremendous value in Cui providing these downloadable projects as it is far easier to walkthrough a sample project myself rather than trying to carefully observe the code in the video. Because Yan Cui provides full example projects to the viewer, it is possible to directly observe what his testing patterns look like. Moreover, in the event that the patterns do not make sense to you, Cui does a relatively good job of describing the purpose and intent behind them.

One testing pattern I would like to highlight from chapters 2-5 would be how Cui breaks apart the pieces of the testing logic into three general categories: “given,” “when,” and “then.” I feel that organizing the test logic into modules dedicated to each category is tremendously useful as it makes the tests far easier to understand. Specifically, in the case of the “given” category, we would create a module purely dedicated to logic that performs test setup steps such as writing an item into DynamoDB that an integration test needs to perform its validation. The “when” and “then” categories correspond to the last two steps of testing, assuming that your tests follow the general flow of: 1) setup, 2) do something, and 3) validate something is correct.

Another highlight, which also pertains to integration testing, is that this course will repeatedly demonstrate what integration testing looks like for serverless applications. Cui draws attention to the fact that generally serverless applications involve a great deal of interplay between numerous AWS services, so some developers may appreciate seeing concrete examples of how integration tests should be configured to test those integrations. Indeed, if you are truly starting this course as a “newcomer to serverless,” then in all likelihood Yan Cui is saving you countless hours of time here by demonstrating what your test cases should look like. The examples of integration tests, as well as unit/e2e tests, are provided in a format that should be easily understood even by someone unfamiliar with serverless applications.

With regard to examples of tests in Cui’s style, I will provide below a single example of an e2e test that I have written:

# e2e test file
from tests.steps.when import when_we_get_availability


def test_get_availability_several_skill(given_forecast_csv_uploaded, given_sme_csv_uploaded):
   # todo refine this test. it doesn't cover much currently.
   query = {
       "skills": {"active directory": 2, "airbase": 2, "angular js": 2, "AWS DevOps Engineer Professional": 1},
       "hours_available": 10
   }
   result = when_we_get_availability(query).json()
   skills_desired = query["skills"]
   for person_name, person_skills in result.items():
       for desired_skill_name, desired_skill_value in skills_desired.items():
           try:
               assert desired_skill_name in person_skills
               # assert float(person_skills[desired_skill_name]) >= desired_skill_value
           except AssertionError as e:
               print(person_name, person_skills, desired_skill_value, desired_skill_name)
               raise e

Unlike Yan Cui’s examples, which are written in JS, this example is written in Python 3. Moreover, this example lacks a mention of a “then” step; rather than breaking apart the “then” logic into a particular function, I just validate the results inline. You may also notice that the “given” step is performed via the Pytest fixtures “given_forecast_csv_uploaded” and “given_sme_csv_uploaded.” The implementation of both the “given” steps and the “when” steps are provided below:

# “given” step
import pytest
import logging
import boto3
from botocore.exceptions import ClientError
import os
from os import getenv
from typing import IO

if not getenv('AWS_ACCESS_KEY_ID'):
   raise RuntimeError('set AWS credentials for test environment')
STAGE = os.environ['STAGE']
S3_BUCKET_NAME = f'sme-availability-test-sme-forecastapp-{STAGE}'


def _clear_bucket(bucket_name: str) -> None:
   s3 = boto3.resource('s3')
   bucket = s3.Bucket(bucket_name)
   # suggested by Jordon Philips
   bucket.objects.all().delete()


def _upload_file(file_like_obj: IO, bucket: str, object_name: str):
   """Upload a file to an S3 bucket
   :param file_like_obj: File to upload
   :param bucket: Bucket to upload to
   :param object_name: S3 object name. If not specified then file_name is used
   :return: True if file was uploaded, else False
   """
   assert bucket
   assert object_name

   # Upload the file
   s3_client = boto3.client('s3')
   try:
       response = s3_client.upload_fileobj(file_like_obj, bucket, object_name)
       logging.info(response)
   except ClientError as e:
       logging.error(e)
       return False
   return True


@pytest.fixture(scope="session")
def given_forecast_csv_uploaded():
   with open('tests/steps/availablev3.csv', 'rb') as forecast_file:
       s3_object_path = 'forecastapp/available/availablev3.csv'
       _upload_file(
           forecast_file,
           S3_BUCKET_NAME,
           s3_object_path
       )
       yield
       # teardown
       _clear_bucket(S3_BUCKET_NAME)


@pytest.fixture(scope="session")
def given_sme_csv_uploaded():
   with open('tests/steps/sme_matrix_file.csv', 'rb') as forecast_file:
       s3_object_path = 'sme/SME Skill Matrix - Employees.csv'
       _upload_file(
           forecast_file,
           S3_BUCKET_NAME,
           s3_object_path
       )
       yield
       # teardown
       _clear_bucket(S3_BUCKET_NAME)


# “when” step
import requests
import boto3
import os
from functools import cache

STAGE = os.environ['STAGE']

ssm = boto3.client('ssm')
parameter = ssm.get_parameter(Name=f'/sme-availability-test/{STAGE}/api-gw-url', WithDecryption=True)
apig_url = parameter['Parameter']['Value']
api_id = apig_url.split('//')[1].split('.')[0]

apig_boto3_client = boto3.client('apigateway')


@cache
def obtain_api_key_value() -> str:
   kwargs = {
       "limit": 20,
   }
   api_key_id = None

   def find_matching_key(key_info_response: dict) -> None:
       nonlocal api_key_id
       desired_stage_key = f'{api_id}/{STAGE}'
       for item in key_info_response["items"]:
           for stage_key in item["stageKeys"]:
               if stage_key == desired_stage_key:
                   api_key_id = item["id"]

   get_keys_response = apig_boto3_client.get_api_keys(
       **kwargs
   )
   find_matching_key(get_keys_response)
   while get_keys_response and not api_key_id:
       kwargs["position"] = get_keys_response.get("position")
       get_keys_response = apig_boto3_client.get_api_keys(
           **kwargs
       )
       find_matching_key(get_keys_response)
   assert api_key_id
   result = apig_boto3_client.get_api_key(
       apiKey=api_key_id,
       includeValue=True
   )
   return result["value"]


def when_we_get_skills():
   api_key = obtain_api_key_value()
   return requests.get(
       url=f'{apig_url}/skills',
       headers={
           "x-api-key": api_key
       }
   )


def when_we_get_availability(query: dict):
   api_key = obtain_api_key_value()
   return requests.post(
       url=f'{apig_url}/availability',
       json=query,
       headers={
           "x-api-key": api_key,
           "Content-Type": "application/json"
       }
   )

Despite the accessibility of the learning materials within the course, I do have some criticisms both of the course as a whole as well as the final chapter. Regarding the final chapter of the course, it is in my opinion simply an overview of basic concepts that could have easily been found elsewhere online for free. These topics include things such as smoke testing and load testing. The final chapter does not include any downloadable examples of projects configured to implement these testing practices, which is also disappointing. While you may find Yan Cui’s instructional style to be preferable to watching similar videos on YouTube, I would argue that his videos in this case do not describe the topics in enough detail (or provide any meaningful examples) to be considered comprehensive educational material on the subjects. The final chapter should be seen only as a guide to what testing concepts you should familiarize yourself with if you are not aware of them already. With regard to the course as a whole, it would have been great to at least have some sort of summary or conclusion that could serve as a reminder of all the specific ideas that were covered in the course. This would have made it a lot easier to identify which areas I had failed to properly understand so that I could then either revisit those sections or seek other content online to learn more.

Considering the vast amount of free learning material available which covers testing serverless applications, I think the most important question to answer in reviewing Yan Cui’s course would be: is it worth the price? The answer is, as usual, “it depends.” I think the content in this course is more comprehensive and useful than any other resource I have encountered. That being said, other content creators have covered the fundamentals of testing serverless functions and have made their content available for free, so if price is a concern then I don’t think it is necessary at all that you take this particular course. Several hundred dollars is quite a bit of money, so whether or not the course is worth it for you personally will depend on how deeply invested you are in serverless in general. The course is not targeted towards beginners; anyone new to serverless should start with similar free content available elsewhere.

In short, the course is, in my humble opinion, a fantastic resource to serve as a primer or refresher on best practices surrounding testing serverless applications. Assuming you are not unfamiliar with serverless technologies, I think you will definitely find the ideas within the course to be helpful, and perhaps the course may help you to appreciate the beauty/importance of testing serverless apps. For me, this course certainly did provide some much-needed encouragement to give my test cases the consideration they deserve.