Getting into AWS cloud security research as a n00bcake

I haven’t done any AWS security research for about 6 years so I have no idea what I’m doing. Which is perfect because I’m grinding the same quest as you, my humble reader. Luckily, I’ve made many friends in the AWS community over the years who haven’t realized yet that I’m a total n00bcake, and they agreed to help. Their advice hides within.

Today, AWS security research can feel impenetrable, like understanding the latest meme that’s already gone through three ironic revivals. But if I’m being honest, I might suggest AWS security research is far more accessible than the other insane research in our industry. That’s why I attempt it. I’m just too dumb to write shellcode or disassemble a binary. So don’t be scared, let’s do it together!

This post is organized in two sections:

How to go about it? This is the general process stuff, like what you need to learn, who you should learn from, and what you should think about along the way while not actually doing research. It’s the narrative arc of your adventure.
Where to start looking? This is more specific technical advice on where to start looking for interesting technical threads to pull, if you are stuck for ideas. It’s the compass that points you in the direction of the cave but you still have to find the gremlins yourself.

1. Process

Build stuff

Build it before you break it

– Christophe Tafani-Dereeper

Christophe is on to something. You don’t need to build everything you want to hack but building in AWS is critical to understanding its oddities and complexities. The APIs look so simple and the tutorials so trivial, but they hide weird quirks and annoying rules.

An extension of “build stuff” is the Fix-it Felix principle, also another banger from Christophe:

Spend time troubleshooting when things break

– Christophe Tafani-Dereeper

How else would you know that the same Cloudformation stack executed twice can produce different results? Or that different error messages are returned when an SQS queue exists but you do not have permission, vs doesn’t exist at all?

The toil of debugging brings a deeper appreciation for what might be happening underneath the thin layer of mozzarella hiding the work of a two pizza team. You’ll be surprised by the violently strange and inconsistent behavior you find.

Build low

Writing code to do the things you’ve tried to do manually is very important. For example, trying to figure out what resources are publicly exposed via network access on AWS became CloudMapper. Trying to figure out how to audit IAM policies became Parliament.

– Scott Piper

In thy grand endeavor, steer clear of the treacherous snares of the web console, for it be a labyrinth of digital shadows and ensnaring cobwebs, luring thee into a realm of bewilderment and misfortune… It hides too much complexity and diminishes repeatability.

Instead use the lowest level abstraction you are comfortable with. The cli and cloudformation are both reasonable choices. Make sure to keep a history of what you do and the responses you get. You’ll need them when you stumble onto something fun.

Break stuff

When you are ready to start hacking stuff, there are some amazeballs resources that you can play with without fear of being raided. I call this the Wreck-It Ralph path.

Here are some practice environments you can break:

flAWS - Built by Scott Piper. Through a series of levels you’ll learn about common mistakes and gotchas when using AWS.
flAWS 2 - Another beauty by Scott. flAWS 2 has two paths: Attacker and Defender! In the Attacker path, you’ll exploit your way through misconfigurations in serverless (Lambda) and containers (ECS Fargate). In the Defender path, that target is now viewed as the victim and you’ll work as an incident responder for that same app, understanding how an attack happened. You’ll get access to logs of a previous successful attack.
CloudGoat - CloudGoat is Rhino Security Labs’ Vulnerable by Design AWS deployment tool. It allows you to hone your cloud cybersecurity skills by creating and completing several capture-the-flag style scenarios. Each scenario is composed of AWS resources arranged together to create a structured learning experience.
CloudFoxable - CloudFoxable, an intentionally vulnerable AWS environment created by BishopFox to help you find latent attack paths more effectively. It focuses on creating as many distinct vulnerable resources and flags as possible – rewarding users for finding new entry points, lateral movement paths, and data access strategies.

Write, write, write

The best thing about writing on the internet is that if you write something terrible, no one will notice. If you write something great, people will read it and think you are a genius. It’s an asymmetric return profile.

There’s lots of generic content on the internet about why writing is good for you. It’s not wrong. If you need convincing, type “why writing is important” into your favorite search engine or chatbot. Write every day if you can. It will give you clarity, and help you retain information. However, I’ll be specific about how writing about AWS security has helped me.

In 2016, I published three blog posts on hacking AWS and gave a few talks based largely on that content. I can trace all of the following back to that writing.

My first advisory gig with Cloud Conformity, one of the first Cloud Security Posture Management tools out there. They got in touch because they were under the impression I was an expert 😬 in this space. They were eventually acquired by Trend and I got to participate in the rewards.
The first advisory gig triggered many subsequent gigs. Without that first one, I never would have had the social proof for the others. One of the founders, then CTO, is now a close friend and CEO of Plerion, another awesome cloud security company that I now advise.
When you write well, people assume you know what you are talking about (see 1). That resulted in many speaking opportunities for me, which in turn snowballed into other things. I could never muster the energy to go back to consulting but I’ve been offered many AWS security consulting opportunities over the years, most referencing my blogs.
The absolute best result of writing is the relationships and contacts I’ve forged, the vast majority of them not initiated by me. Even 7 years later, people DM or email me with questions, requests, appreciation, or just to say ‘hi’. I now have an amazing group of friends to exchange Milo with, and collaborate on security research ideas. None of this happens without the writing.

If you are human, as I suspect you are, writing is probably intimidating and painful. Instead of making a big deal out of it, start with the smallest unit and aim for regularity. A tweet here, a LinkedIn post there. Before you know it, writing won’t be so scary and you’ll start accruing the benefits of putting your work out there.

Siphon the big brains

There are loads of incredible people doing brilliant work in AWS Security. Read, listen, and watch their stuff. Past work is a great place to both learn foundations and find interesting threads to pull yourself.

Before you do that though, take a moment to check out the work of Spencer Gietzen. His research and code was a giant leap forward in cloud security. Unfortunately he passed away in 2021, leaving a big hole in the community. I never got to meet Spencer but every interaction I had with him was warm and compelling. Rest in peace Spencer. <3

In no particular order

Scott Piper - On his passport his full name is actually “Distinguished Cloud Historian Scott Piper”. He keeps a thumb drive with every piece of cloud security content ever published in a secret vault in Switzerland. If you publish something, he’ll explain the full ancestry of that thing back to the time of floppy disks. Besides knowing everything, Scott has a sweet blog, leads many community initiatives, and rents his brain to Wiz.
Gafnit Amiga - When Gafnit writes, it’s time to read. As VP of Security Research at Lightspin, she does inspiring technical work, but it’s her ability to distill insane complexity in writing that really separates her.
Nick Frichette - Is Nick even real, or is he an AI? He has been pumping out AWS security research the last few years while working as the Seniorist of Security Researchers at Datadog. Nick also runs Hacking The Cloud, an encyclopedia of the attacks/tactics/techniques, and is deadly at finding hidden APIs.
Christophe Tafani-Dereeper - They must have two instances of the AI running at Datadog because Christophe is also prolific. All of his research is linked from his website but I personally love the Stratus Red Team project the most. It’s an open source tool to emulate cloud attack techniques in a granular and self-contained manner.
Aidan Steele 🦘 - The Distinguished Cloud Historian dropped this line in a chat recently, “I’m noticing that every blog I’ve been writing recently contains some form of the phrase ‘As Aidan pointed out two years ago….’” That feels right. He usually points those things out in his blog but his Github history is also solid green. When I have NFI how to do something, I ask Aidan.
Shir Tamari - When you’re Head of Research at Wiz, I think that means you’re good at this stuff. Shir’s research is legendary but also he’s just a really kind and happy guy who likes to collaborate on stuff.
Ian Mckay 🐨 - Ian believes there is only One True Cloud. He claims to be a “DevOps dude” on LinkedIn but that undersells just how brilliant his work is. Amongst many many cool projects, Ian runs permissions.cloud which I personally visit all the time to untangle IAM policy crud. His blog is legit too even if the damn pictures tiles are annoying as hell.

There are plenty of other resources out there. Here are some of my faves:

fwd:cloudsec - By far the best cloud security conference and it’s not particularly close. They publish all of their talks on their Youtube channel!!!
Cloud Security Forum Slack - If you are truly interested in this space, joining this Slack is a must. I believe it’s a soft open invite. Anyone in the community should be able to get you an invite.
Newsletters with the latest cloudsec hotness: tl;dr sec, CloudSecList, Last Week in AWS.
Cloud Security Podcast - If listening is more your style, they regularly interview cloud security researchers.

Let me know if I missed something important! Apologies in advance.

Make friends with engineers

Every cool trick I know I learned from watching some incredible engineer just trying to get their job done

– Houston Hopkins

At previous employer™️, I regularly worked with Mike Fuller. He is a brilliant cloud engineer who ultimately specialized in FinOps and wrote a few books along the way. It was one of the best learning experiences of my life.

Mike knew allthethings but didn’t necessarily view them with a security lens. When I started asking questions about a topic, we were able to discover security issues really quickly because of our complimentary skills. Eventually we did a security conference talk together called Gremlins in Your Cloud Success, about the unexpected security issues companies face in the cloud. Point being, fostering relationships with SREs and cloud engineers might lead to a big return for both of you!

Stay ethical

This might get a little bit serious. Give me a second while I put on my business suit and tie… Okay, I’m back.

Even though this blog plays a bit loose with offensive techniques and spicy wording, it’s only for effect to make the content interesting. Most security research is dual use, and it’s easy to stray into illegal or unpleasant territory. Don’t be that person. Respect the work of the people at AWS and respect their customers. Make sure you read and understand Amazon vulnerability reporting and research guidelines, which include safe harbor terms but also specifically exclude certain types of activity.

Ultimately AWS doesn’t owe you any kindness, as Shir points out.

AWS is one of the most critical infrastructures out there. Take extra care when poking around it. Avoid doing things likely to cause a denial of service or overload their services.

– Shir Tamari

I’ve run a number of successful bug bounties for various companies now and I can testify that touching customer data, no matter the intent, is not likely to be a positive experience for you, so do as Aidan suggests.

You should never (need to) touch other people’s stuff. if you want to validate a cross-tenant vuln, create two of your own accounts and use one to try access the other

– Aidan Steele

Sometimes when we are slinging sh*t on the internet, we forget that just being nice is not only the right thing to do, but incredibly rewarding.

be nice and ask questions

– Christophe Tafani-Dereeper

I’m with Christophe. Responding, listening, and thoughtfully discussing issues has helped me make friends and break down complexity that I couldn’t have understood by myself. Simply being kind and asking good questions is a great way to learn because the community responds in kind.

Get the reps in

This one is like content theft inception. I’m stealing it from someone, who stole it from Daniel Miessler’s guide on How to Build a Cybersecurity Career:

90% of being successful is simply getting 100,000 chances to do so. You get chances by showing up. By spinning up that VM. By writing that proof of concept. By writing that blog post. And you have to do it consistently over a number of years.

He is 100% right. Ultimately the people I’ve seen succeed in this space are the ones who wake up and just do it. They have a go. They fail. They try again. And they repeat it over and over until research falls out. If you do nothing else, just have a go.

Tips and other stuff

Now for some stuff I thought was useful but ChatGPT forgot to include while writing this article.

Set a useful user-agent in your scripts/tools. it’s much easier to filter through CloudTrail when you’re looking for “userAgent: daniel-hacker-man” rather than “userAgent: boto3”

– Aidan Steele

[email protected] is your friend

– Aidan Steele

After Aidan shared this nugget, I decided to see what would happen if I sent them a message. I can’t say it was a very formal message but they responded in kind, which was refreshing. It got weird pretty quick as the thread devolved into a battle of who could most stylize their response in Renaissance English. Ryan won after he offered to “assist with any discussions on disclosure, filing for CVE if applicable, and assistance and collaboration on blogs and presentations.”

It is useful to learn their leadership principles to better understand why they do what they do, and how to optimize communication.

– Shir Tamari

I always recommend getting to know how IAM works extremely well. The beautiful thing in AWS is that they use their own services to build additional services. So you should expect to encounter popular services such as IAM, EC2, S3, ECR during your research. It makes it a little less “blackbox”.

– Gafnit Amiga

Review prior art:

2. Focus areas

Trawl AWS documentation

Amazon has invested a lot in describing their services and producing documentation. Their documentation mini-site is vast and detailed and thus a really great place to start looking for interesting threads to pull. I

First you’re going to want a local copy of all the docs so you can search them with precision. AWS documentation was previously open source but these days you’ll need a little work to get it all. I’ve written a script to parse the sitemaps and produce a list of URIs to download.

#!/usr/bin/env python3

import requests, argparse
import xml.etree.ElementTree as ET

SITEMAP_URI = 'https://docs.aws.amazon.com/sitemap_index.xml'

def main():
    get_sitemap_and_parse(SITEMAP_URI)


def get_sitemap_and_parse(sitemap_uri):
    # Recurisively get the sitemap and parse it
    try:
        res = requests.get(sitemap_uri, allow_redirects=False)
        parse_sitemap_xml(res.text)
    except Exception as e:
        pass


def parse_sitemap_xml(sitemap_data):
    root = ET.fromstring(sitemap_data)
    namespace = root.tag.split('}')[0].strip('{')
    for sitemap_child in root.findall(f'{{{namespace}}}sitemap/{{{namespace}}}loc'):
        get_sitemap_and_parse(sitemap_child.text)
    for sitemap_child in root.findall(f'{{{namespace}}}url/{{{namespace}}}loc'):
        print(sitemap_child.text)


if __name__ == "__main__":        
    main()

Once you are ready, you can go ahead and wget the lot.

wget --input-file=aws-docs-uris.txt --timestamping -e robots=off

And if you want to exclude non-English versions, first trim the file with:

grep -v "/[a-zA-Z]\{2\}_[a-zA-Z]\{2\}/" aws-docs-uris.txt

Once you read enough of these docs, you might start picking up on some interesting patterns. For example, what is described in one place is sometimes contradicted in another.

Code is often wrapped in a <code></code> block. SQL often has the class attribute set to class=”sql”. If you’re interested in code or SQL or something similar, you can grep directly for it.

It should be obvious that AWS folks have already done a lot of threat modeling and understand many of the potential risks associated with their services. In fact, they have kindly included many in their documentation in notes (<div class="awsdocs-note awsdocs-important">) and warnings (<awsui-icon name="status-warning" variant="error">). These have been a great source of inspiration for me and I would recommend farming them for insights and ideas.

Example note and warning

If you already have a topic that’s interesting to you, grepping can help you enumerate what is known, and diffing that with existing public research. For example, I would suggest that there is a deep and broad problem with confused deputy issues that has never been comprehensively addressed. A simple grep -i “confused deputy” produces a fertile ground for exploration.

There are other benefits to being a documentation nerd, as Shir points out.

Following the documentation helps in understanding how the backend looks. For example, many AWS services are based on open-source technologies mentioned in the official documentation.

– Shir Tamari

Decompose Boto3 (or other SDK)

A lot of good research ideas come from wanting to understand how things work under the hood. If I’m trying to figure out or do something quirky and it’s not documented anywhere, the official AWS SDKs have often implemented it. Moreover, if AWS wants to allow folks to interact with their services, they will update their SDKs and cli first, sometimes even before a service goes live.

Below are some random examples I’ve looked at in the past but you should focus on the pattern rather than the specifics. Find your favorite official AWS SDK and dig through it.

Understanding API versions, inputs, outputs, and exceptions. These are documented in JSON files as part of botocore’s data directory.
Enumerating and identifying headers that might result in different code paths being triggered in various contexts. These can be found by searching the botocore Github repo for x-amz (amongst other tokens).
Creating a cli proxy lambda as a backdoor into an AWS account. The “driver” is implemented in the aws-cli project and can be used directly as a python object.
Understanding what CA’s are trusted by SDKs, and monitoring what that changes. This is documented in the git history of the botocore cacert.pem file.
Converting undocumented APIs to a format that can be used by botocore, and therefore the AWS cli - a tool by Nick Frichette.

You also get the usual gems that pop up in code, like things that maybe shouldn’t be there that can fruitful threads to pull.

# NOTE: This doesn't appear to be used anywhere.
# It's probably safe to remove this.
class NoVersionFound(Boto3Error):
    pass

Filter for discrepancies in the API

Amazon is well known for their approach to building services fast in small teams. It’s great for innovation but also results in inconsistency between teams and sometimes a lack of systematization of key features such as authorization. Look for those inconsistencies, those multiple ways of doing things and explore from there.

There are plenty of examples where these inconsistencies arise:

Request signing - I’m not super clear on how many versions of request signing AWS has implemented, nor how many are still used, but that’s kind of the point. At the time of writing version 4 is the standard but various articles and code reference at least v2, v3, v3https, v4, s3, s3v4.
Call protocols - In this article on AWS API protocols, Nick Frichette writes that there are five call models; ec2, query, json, rest-json, and rest-xml. In the same article he describes various weird interactions between those protocols and a subset of services, that lead to vulnerabilities.
Parameter names - In theory parameter naming should be pretty uniform, after all that’s what ARNs are for, that’s why regions are well defined, etc. In practice it’s all over the place and it should make you wonder why? For example, sqs:create-queue expects a queue-name, sqs:delete-queue expects a queue-url, and sqs:start-message-move-task expects a source-arn. A lot of hacking techniques come down to the ability to identify the existence of a vulnerable resource and these discrepancies often point to such opportunities.
Endpoint FQDNs - Service endpoints are the URI entry points for communicating with APIs. Again, in theory they are pretty uniform. In reality they vary for various reasons. Prod vs non-prod, fips vs non-fips, internal vs external, global vs regional, ipv4 vs ipv6, feature vs core, fqdn vs URI path… Surely there are problems hiding in this complexity.
Function names - Function names are the basis of IAM policies so they are critically important. Most folks make assumptions about what they mean, including those building AWS services and managed policies. What does a Read* function do vs List* vs Describe* vs Get*? Dragons have lived here for decades.

This is but a small sample to whet the appetite. Send me your favorites and I’ll include them.

Find undocumented APIs

This has become the domain of Nick Frichette in recent times. Nick plans to publish the undocumented API models he has discovered, but for now you might have to follow the breadcrumbs in his Twitter posts here and here.

Nick has identified some of the coolest vulnerabilities by first finding undocumented APIs and then abusing them. It’s possible AWS has since approached this systemtically and eliminated this problem completely but until that is confirmed, it will continue to be a solid space to mine for issues.

Target open source

If you do a review of past vulnerabilities, you might start to see rough edges around how open source tech is integrated into AWS services.

AWS (and other CSPs) try to turn beloved and popular solutions/open-sources into managed services - this change is not natural/easy and may introduce new attack surface

– Shir Tamari

Open source provides a number of research angles. There’s a world of prior art on most big projects that you can revisit and validate. My observation is that there’s often gold in the file system and hardening guidelines. A good example of this is the account takeover vulnerability in Jupyter Notebooks found by Gafnit Amiga, which stemmed from the ability to edit an index.html file on the notebook filesystem (plus some CSRF misconfiguration).

And there’s always another Kubernetes issue around the corner.

Poke shared responsibility

The AWS shared responsibility model states that “AWS maintains the configuration of its infrastructure devices, but a customer is responsible for configuring their own guest operating systems, databases, and applications…Controls which are solely the responsibility of the customer based on the application they are deploying within AWS services.” The model is beautiful because the problems are often systemic but somehow strangely trivial.

Shared Responsibility - look for areas where security configuration is complex and many users will probably do it wrong.

– Shir Tamari

In my opinion, the key to poking shared responsibility is identifying mis-matches in expectations between the AWS interface (web console, cli, policies, IAM) and those of the average human.

The canonical example of this is the deprecated “AuthenticatedUsers” group. Many reasonable humans assumed this referred to an authenticated user in their specific organization but in fact it meant “When you grant access to the Authenticated Users group, any AWS authenticated user in the world can access your resource.” Which is actually just shorthand for everyone and anyone.

A more recent example is the Github OIDC integration, which gives Github users the ability to build CI/CD pipelines that assume AWS roles without the need to manage secrets. All the documentation describes how to do this safely but IAM policies are inherently so complex that this is just a footgun for many people. As a result, there was a period during which many people allowed any Github user to assume a role within their account.

Monitor changes

There are a bunch of bots already pumping out notifications when something potentially important changes. You could either make use of those and look for things that pique your interest or build monitoring yourself for changes to other things. Both approaches have potential to find gold.

To get started, here are some existing AWS change monitoring tools:

Enumerate stuff

I’ll leave this one to Scott Piper because he explains it well and I’m getting tired writing this monster blog post.

I think compiling and maintaining public lists has been helpful. Some examples I’m involved in are my list of AWS breaking changes, cloudvulndb, a list of vendor account IDs, and a list of publicly exposable resources I once maintained. Other lists out there include lists of log sources, lists of credential vending APIs, lists of features that don’t work with IPv6.

This does a few things:

Forces you to focus on something. AWS is so big and constantly evolving that you can quickly be overwhelmed and dart randomly in different directions looking into new shiny things, never making any actual progress.

Motivates you to read docs or announcements looking for those prizes you care about. If I read something without a purpose, I can’t focus on it. I like having some side quests available to me while reading so I can read something for one reason, but if it happens to mention an AWS account ID, then I’ve advanced in one of my side quests.

Gives you some notoriety in an easy way. It is not as exciting and popular as finding a zero day cross tenant vuln, but it can be useful and some folks will pay attention. Because you become known for being the person that cares about a random thing, it becomes a bit of a flywheel, where people will start privately messaging you about the thing, and you’ll become the de facto expert on it because folks are telling you more and more about the thing.

– Scott Piper

Tips and other stuff

Bard also failed to include everything when writing this section. Below is an assortment of useful stuff it left out without consulting me.

Rumors say the Web Console API is a good candidate to check for security issues.

– Anonymous

Finding new sources of data is powerful. The thing that really made Parliament possible was realizing I could scrape the docs to convert the docs about privileges to a computer readable json. That created Parliament, and then Kinnaird took that data and created PolicyUniverse, and Ian McKay took that data and made iamlive. iamlive also came about because I noticed in the SDK the feature for CSM and wrote about that, and Ian figured out how to make use of that.

– Scott Piper

I’m trying to target the extensional functions first, and not the ones that are immediate. Because those flows are less likely to be tested.

– Gafnit Amiga

That’s it. That’s the tweet.

Please get in touch and let me know how you go with your AWS security research journey!