I find one of the most annoying features of AWS is the strict divide between different regions. With this post and a new Cloudformation feature from AWS I will show you how to deploy infrastructure across multiple regions with a single “master” template.
Consider a service like GuardDuty. You don’t really care whether the alerts are coming from London or Dublin or Tokyo, you need someone to look at them ASAP. If the US are asleep you don’t want the alert to wait for the morning if someone in Europe could fix the problem.
And then there are a very few services that are truly global. Like IAM. When you create a role or a policy, it is done once centrally and the change (eventually) appears everywhere.
And there are some services that pretend to be global but are in fact in us-east-1. For example if you want to use an ACM certificate with CloudFront, it has to be created in us-east-1.
The Scenario
I would like all GuardDuty alerts to be sent to our Jira instance. Because we only have one Jira the script is the same in each region. So we need a Lambda script to take the JSON GuardDuty sent to the EventBridge and push it into a Jira ticket.
That requires (at least) 2 extra bits of config :
- A Secret that holds the Jira password.
- An IAM role that the Lambda assumes.
Ideally that infrastructure would be “Infrastructure as code” with as little code and as easy to manage as possible (not lots of files and a complicated list of instructions when changes are made).
The Solutions
Regions Duplicate Everything (Simple Cloudformation)
It is quite easy to create a CloudFormation template that will create everything you need and do it repeatedly in each region.
So you end up with a secret in each region and a role per region.
You’re probably thinking “That’s fine”. And then I change the Jira password. Now (because you’re good) you check all the regions and update the template.
If you aren’t so good, you forget to update one region that is rarely used and those alerts get lost until someone notices.
Also, a minor annoyance that your IAM role list will start to get very long! If you only ever use automation, that doesn’t matter much. But if you ever look at the list in the AWS admin console, it will become a nightmare.
You also have a lot of secrets all with the same information in. AWS charge $0.4 /secret/month. That starts to add up if you use many regions and accounts and if your script is more complex than my simple example.
In a different scenario, there likely will be other more expensive resources that you are duplicating because it is “hard” to work across regions.
Regions Duplicate Everything (Stack Sets)
AWS realise you might want to run things multi region and even multi account. And so there is a feature called “Stack Sets” that allows you to define a set of accounts/regions that you will run a template in.
Now, your issue about forgetting to update a region is gone. The stack set will ask you for the parameter once and apply the same value to all regions.
However, you do still have a lot of duplicated roles and secrets. (See previous solution for reasons that is bad!)
And, until recently the only way to define a stack set was in the console or with some tricky CLI commands. So this fails the “easy to manage” test.
One Region For Shared Resources And Only Duplicate Required Resources
This is starting to sound like a more sensible place. Create one Role and Secret and then just create the Lambda in each region.
Sounds easy. But, until recently it wasn’t so easy to do with only using AWS resources. There are 2 ways to do this :
Multiple Templates And Script
Because you’re doing 2 different stacks you’re going to have 2 templates. One to create the shared resources and one to create the Lambda. However, to share the IDs of the shared resources to the other templates, you need an external script. The 2 methods you would normally use to share values : Nested Stacks and Cross Stack Export/Import do not work to different regions.
You could create the shared stack. Then query its exports, and use the values as parameters to deploying the other template. You’re probably running a script to do this, so you could make your own loop to deploy to each region or create a stack set with the regions you want.
But you need this external script as well as your 2 templates. If someone wants to make an update, they cannot simply login to the console and upload a template file, they need to setup CLI access and have your script available which fails the “easy to manage” test again.
One Template, Multiple Regions
As of 17th Sept 2020, you can define a Stack Set within a template.
This new feature means we can define one template that creates some resources, then inline the template to deploy to each region in a stack set definition.
A highly abbreviated example :
--- |
You can deploy this template to us-east-1 and it will create Lambdas in eu-west-1 and 2 that reference the same IAM role and the same Secret (Secret held in us-east-1)
An alternative to the !Sub on the nested template code would be to submit the role and secret values to the nested template as parameters.
If you don’t mind having multiple files or your nested template exceeds the ~50K limit, you could upload it to an S3 bucket. If you use a bucket, you have to specify the values as parameters. But once you’re in the “multiple files” scenario, you need to make sure that whoever does any updates knows which files to update and where they are.
NB To make Stack Sets work, you need to enable the self managed permissions. Follow this guide from AWS :
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/stacksets-prereqs-self-managed.html
Summary:
In any region of your account create the “AWSCloudFormationStackSetAdministrationRole” role by deploying this template :
https://s3.amazonaws.com/cloudformation-stackset-sample-templates-us-east-1/AWSCloudFormationStackSetAdministrationRole.ymlAnd then create the “AWSCloudFormationStackSetExecutionRole” role by deploying this template :
https://s3.amazonaws.com/cloudformation-stackset-sample-templates-us-east-1/AWSCloudFormationStackSetExecutionRole.ymlAnd specify your own account number as the parameter. These only need to be run once per account so don’t include them in your template!