Wed, 14 Aug 2019
Here at Trek10, several clients have asked us what they can do to help avoid a breach similar to the very public and recent Capital One breach. After researching the topic and gathering the data points over the last couple of weeks, we have pulled together some recommendations. Before jumping in, note that this blog post is not a deep dive on what happened in the actual attack. We will do a quick review, but this post is intended to serve as a guide on what actions enterprises can take, in their AWS environments, to mitigate the risk of this type of attack.
As is often the case with a breach, there are two components to the attack — there is the initial attack on the perimeter, followed by lateral movement across the environment for the data exfiltration. In this case, the initial attack occurred when a public-facing EC2 instance, hosting the open source Modsec WAF, was breached. The vulnerability that allowed this initial breach is unknown, which means a virtually infinite number of possible vulnerabilities could have allowed the attacker in. Based on the coverage, it’s very unlikely that this was a zero day vulnerability, which means it probably could have been avoided with appropriate security controls.
Once the attacker has breached the EC2 instance, all AWS services that the EC2 instance has access to via the IAM policy of the IAM role that is attached to the EC2 instance profile (known as “*****-WAF-Role” in the indictment) are now available to the attacker via AWS’s metadata service.
These AWS services are accessible both from the terminal of the EC2 instance, as well as from the attacker’s local environment outside the instance… but the keys are temporary, rotated automatically multiple times per day, and will quickly expire if used locally).
Interestingly, the data exfiltration was completed from a VPN provider’s IP address using the S3 Sync command. GuardDuty, AWS’s threat detection service, provides a high severity notification if the credentials associated with an EC2 instance profile are used outside of the EC2 instance. At Trek10, we integrate GuardDuty into our ticketing system (both internally and for clients) such that high severity alerts create a ticket for review.
One of the key discussion points related to the breach is around whether or not the IAM policy that was attached to *-WAF-Role was overly permissive. It is not clear why the Modsec instance required access to S3. We have not found any public details related to this, but given that we know the attacker ran the ListBuckets command and then the Sync command, we can assume that the IAM policy of the role looks something like this (though it could have very well been scoped down to specific S3 buckets):
{ "Effect": "Allow", "Actions": [ "s3:ListBuckets", "s3:ListAllMyBuckets", "s3:ListBuckets", "s3:GetObject" ], "Resource": "*"}
How to Mitigate Risk from this Attack
This section outlines an overview of recommendations for your AWS environment, based on the attack, as well as some details on how Trek10 implements best security practices into its CloudOps offering. The recommendations are broken down based on whether the suggestion mitigates risk from the initial attack on the perimeter or on the lateral attack that allowed the S3 data exfiltration. We will start with the lateral attack, as that is far more interesting due to the public information available!
Preventing Lateral Attacks on S3 Data
Remember that AWS IAM is an extremely powerful service (one that we take very seriously at Trek10 — just check out our approach to IAM escalation to understand how seriously!). The key takeaway from this breach is to understand that the power of AWS IAM means that organizations need to take the principle of least privilege very seriously and ensure that all IAM entities are scoped appropriately. Below are some recommendations based on the breach:
- Conduct an IAM audit.
- Review the IAM policies attached to all IAM entities
- Trek10 pulls a wide variety of data with its custom IAM enumeration scripts to help our clients implement the principle of least privilege across all IAM entities.
- We are also always developing additional tooling and integrations to support security, like our Excess Access Exorcist.
Preventing Initial Attack on EC2 Instance
Because the public knows so little about that the initial attack on the EC2 instance, these “mitigations” are just general good EC2 security practices — with a particular focus on how an organization can utilize AWS’s vast arsenal of security services to protect EC2 instances:
- Conduct a firewall audit
- Configure hardened security groups / network ACLs to close all unnecessarily opened ports.
- Trek10 uses its custom scripting to enumerate security groups into a report for validation.
- Better yet, use infrastructure as code everywhere with security testing integrated into your CI/CD pipeline to detect any drift. With our AWS DevOps Competency, Trek10 has world renowned expertise in building CI/CD pipelines.
- Regularly scan your EC2 instances with AWS Inspector to detect any vulnerabilities and conduct remediation.
- Trek10 integrates AWS Inspector reports into its ticketing system for remediation.
- Enable AWS GuardDuty, already mentioned previously, for intrusion detection.
- Trek10 integrates GuardDuty alerts into its ticketing system and even automates the blocking of attacking IP addresses with AWS WAF.
- Ensure your systems are up to date with AWS Patch Manager via Systems Manager.
- Trek10 applies patching best practices to Patch Manager, which integrates with its ticketing system.
Trek10’s CloudOps service integrates a wide variety of AWS security services into its ticketing system and automates the deployment with pre-built templates. Additionally, Trek10 is in the process of building an advanced, cloud native security detection and prevention service. We are combining AWS native security services and the absolute best 3rd party commercial and open source security tools for full protection of EC2 instances and cloud native AWS services. We are partnering with Sophos Cloud Optix for this offering, and we are excited to release this service later this year. Be on the look out for a future blog post!
