The ideas in this post were developed in collaboration with our experts, Tab Kenney & James Bowyer
Backstory
It may be easy to forget about notices on your AWS account's Personal Health Dashboard, though you may want to check out one particular alert. In particular, if you use AWS Config, then upcoming changes on September 15th, 2021 may deprecate components of your cloud architecture. Let's check out AWS' statement below.
As part of our ongoing efforts to optimize costs associated with recording changes related to certain ephemeral workloads, AWS Config is scheduled to release an update to relationships modeled within Configuration Items (CI) for 7 EC2 resource types on September 15, 2021. Examples of ephemeral workloads include changes to Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances, Amazon Elastic MapReduce jobs, and Amazon EC2 Autoscaling. This update will optimize CI models for EC2 Instance, SecurityGroup, Network Interface, Subnet, VPC, VPN Gateway, and Customer Gateway resource types to record direct relationships and deprecate indirect relationships.
While you can find a breakdown of the announcement here, we’ll cover the fundamentals of AWS Config, dive into the upcoming changes, and offer examples of workarounds for your organization to implement over the course of this post.
Config Concepts
As you may know, AWS Config lets you monitor both current and historical resource configurations in your AWS account. AWS Config works its magic by taking a snapshot of your deployed resources’ settings and storing them in configuration item, which is what AWS refers to as a recording of your specific resources’ settings at a specific time. These can be created instantaneously as your resources are updated or on a schedule, such as every six hours.
A configuration item represents a point-in-time view of the various attributes of a supported AWS resource that exists in your account. The components of a configuration item include metadata, attributes, relationships, current configuration, and related events. AWS Config creates a configuration item whenever it detects a change to a resource type that it is recording. For example, if AWS Config is recording Amazon S3 buckets, AWS Config creates a configuration item whenever a bucket is created, updated, or deleted.
Configuration items play a major role in AWS Config by serving as the source of truth for historical changes. AWS Config bases many things off of these items; for example, when AWS Config provides you a timeline of your resource states, it is using configuration items in the background. Below, we can see an example timeline for an IAM user’s configuration history. Each configuration item is represented by a grey rectangle that is highlighted in blue. By selecting one of these items, we can drill down into the relationships & changes associated with the AWS resource for the item’s timestamp.
Using the example above, we can dive into the changes of the IAM user on November 12th, 2020 by selecting the 'Changes' dropdown, which we find in the pink box above.
AWS Config builds configuration items with the same API calls as the AWS CLI. Specifically, they use list and describe CLI commands, such as aws ec2 describe_instances --filters Name=tag-name, Values=example-tag. For each configuration item built, AWS Config may make multiple API calls. Here’s a concise explanation about this, direct from AWS Support:
There are two types of relationships: direct and indirect relationships.
A direct relationship is one that is returned when the Describe API call is made. For example, when AWS Config [is] making a DescribeInstances call, the security groups attached to the instance are return[ed].
An indirect relationship is the one that is not returned when the Describe API call is made. This requires multiple Describe calls. For example, when AWS Config call[s] DescribeSecurityGroups , the API call does *not* return a list of instances to which the security group is attached. To find this information, Config has to make additional API calls.
In short, direct relationships of a resource's configuration item can be created and documented with the describe CLI command of that same resource’s service. On the other hand, indirect relationships for a resource are relationships that AWS Config creates with an API call to a different service’s describe CLI command.
It can get confusing on what is considered indirect vs direct, so let's dive into a concrete example. Looking at an example timeline of an AWS Config configuration item below, we have highlighted the information for indirect relationships in pink and the direct relationships in green; however, a helpful rule-of-thumb is that if information you want doesn’t come from the CLI command that is tied to the resource that you are referencing, then the information comes from an indirect relationship. Kindly note that when calling describe-security-groups, we get information on the ingress/egress rules and the VPC the security lives in.
What is Changing
In order to increase the scalability of AWS Config, AWS is reducing the amount of API calls that AWS Config makes on the backend. Because of these changes, AWS will be deprecating indirect relationship calls for seven different resource types. Here is the list of the affected resources' indirect relationships:
CustomerGateway’s configuration items will no longer include VPN connections.
EC2 instance’s configuration items will no longer include EIPs nor the route tables associated with the EC2 instance.
Network interface’s configuration items will no longer include the EIPs nor the route tables associated with the network interface.
Security group’s configuration items will not include the attached EC2 instances nor the network interfaces.
Subnet’s configuration items will no longer include the EC2 instances inside the subnet, NACLs, network interfaces, and the route tables associated with the subnet.
VPC’s configuration items will no longer include the EC2 instances inside that VPC, internet gateways, the NACLs, network interfaces, route tables, subnets, security groups, and lastly VPN gateways.
Finally VPN gateway’s configuration items will no longer include the associated route tables nor VPN connections.
To be clear, AWS has confirmed there will be no impact to any current AWS Config rules, only the configuration items associated with resources. This will impact you if you are utilizing AWS Config notifications on configuration item changes. Specifically, if you are using notifications for any of the above resources’ indirect relationship changes for your alerts or runbooks, then your current setup will be deprecated on September 15th, 2021.
The Workaround
In the event that we need the information for a rule that is currently configured, we can use the semi-new AWS Config advanced queries feature to query for this information with SQL. Using the example with security group sg-a1a1a1, we could see which EC2 instances are attached by either going to the EC2 console and manually checking out each instance, or we could use the following query in the Config console:
Furthermore, we’ve expanded upon the SQL queries that AWS offers in its documentation and included additional queries for returning information on indirect relationships for the seven primary resources affected in the upcoming AWS Config changes.
Unfortunately, if you are using SNS notifications for configuration item changes, this is a breaking change with no easy workaround, so you will need to create a custom solution, such as spinning up a Lambda and generating SNS notifications yourself.
How We Can Help
Here at Trek10, we love getting deep into AWS and solving problems like this for our clients. If this change requires you to build more functionality such as generating SNS notifications for changes to resources’ indirect relationships, or if you are looking for an AWS Premier Tier Partner to manage your AWS account, then get in touch with us at our contact page or email our team at info@trek10.com
