Today, Trek10 is releasing AWSets, a new open source utility for crawling an AWS account and exporting all its resources for further analysis.
The Problem
Every day, Trek10 interfaces with numerous environments across multiple clients. This presents unique problems often without known solutions, which leads us to build our own. One such example of this is Awsume.
In this instance, Trek10 frequently gets pulled into existing AWS accounts that lack documentation, don’t practice proper tagging, don’t use infrastructure as code, or just contain so many resources that it’s difficult to get an understanding of what we’re working with. Unfortunately, there is no single AWS call or service that can provide a complete assessment of everything in an account so that we can start to piece together a map of what is going on.
Existing approaches
When we started searching for solutions, we came across many tools that were written to address this need. They varied greatly in resource coverage, approaches, and contributor activity. While there are some useful implementations in this space, we were unable to identify any at the time that met the same goals we were aiming for, and the ones that were the closest were not easy to extend. Some tools stopped at supporting a few dozen resources. Others took an automated approach to calling functions from the SDKs and were difficult to override when necessary. Most importantly though, none of them included one of most important aspects, which is how all the resources relate to each other.
New Solution
After falling short on existing solutions, we decided to explore what it would take to implement something new. We had learned quite a few things from our research and also from the pain points we’ve had with working in the AWS ecosystem for so long. Some goals that we wanted to accomplish include:
Not doing too much - it should focus primarily on querying the data
Provide thorough coverage of the AWS resources
Build relationships between resources
Be easy to use - no complicated installation, available on all platforms
Normalize data for further processing - extract things like Account, Region, Id, Type, Tags, Relationships to a top-level object
The result of this work is AWSets, a CLI tool written in Go. The GitHub page includes a full README on installation and usage, but a few highlights include:
Querying and relationship building for over 200 AWS resources
Ability to query multiple regions simultaneously
Ability to selectively include or exclude AWS resource types
Functions within a readonly role
The output data is a JSON array, with each object in the following format:
{
"Account": "123456789", // account resource is in
"Region": "us-east-1", // region resource is in
"Id": "12345", // resource id
"Version": "", // resource version
"Type": "ec2/instance", // resource type
"Name": "test-instance", // resource name
"Attributes": {}, // full dump of resource attributes
"Tags": {}, // normalized tags for resource
"Relations": [ // array of the identifiers of related resources
{
"Account": "123456789",
"Region": "us-east-1",
"Id": "vpc-123abc123",
"Version": "",
"Type": "ec2/vpc"
}
]
}
The intention of this design is to make it easy to use with specialized JSON processing tools such as jq or jmespath.
AWSets also includes a rudimentary DOT file generator, though it may eventually be split out as it increases in complexity. Resources that are referenced in a relationship but not present in the account show up as red, making it easy to find orphaned resources. For example, this image shows a network stack that was created to support Redshift testing. The Redshift cluster was removed (the expensive part), but the supporting infrastructure was not. We can also see these supporting resources are not used by other things.
Each region is rendered in it’s own subgraph, making it easy to get a quick visual of which regions are being used. In this image, items in yellow are resources without known relations, and are usually defaults, such as default roles, RDS parameter groups, etc.
Next Steps
While this initial release is quite powerful and useful, there is a lot more work to be done, including:
Supporting more AWS resources and relationships - 200+ is a good start, but there are many more to go
In addition to supporting more resources, existing resources may have some gaps. For example, some resources require secondary calls to get Tags
Improve relationship building - AWSets should be able to match a DynamoDB table to a Lambda Function when the DDB table is passed in via environment variable
Build additional post-processors for the data. Some ideas include:
IaC score card - find resources NOT managed by IaC
Automatic clustering of related resources to remediate tagging
Taking daily or weekly snapshots of an account
“Smart” resource cleanup - delete a target resource and any related resources only used by it
Getting started
As mentioned before, you can head over to the project on GitHub for information on installing and using AWSets. Try it out and file issues or PRs for any issues you encounter or additional resources you need supported. At Trek10, AWSets is being used as a base utility for powering several internal tooling - by open sourcing this, we're excited to see what the community can come up with!
