Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.
Hello Grunts,
In the last month, we made a bunch of changes to make it easier to keep your code DRY, work with multiple AWS accounts, and work with multiple Terraform modules. We also created a crowdfunding spreadsheet to fund new infrastructure and fixed a number of bugs.
As always, if you have any questions or need help, email us at support@gruntwork.io!
Terragrunt
Motivation: The Terraform code in the infrastructure-live repo we set up for most customers is not as DRY as it could be. In particular, many variables, such as account ID and region, are repeated multiple times, making it harder to maintain the code.
Solution: Our first step in improving this situation is to add a small tweak to Terragrunt that extends the find_in_parent_files helper so you can specify the filename to search for (e.g.: find_in_parent_files(“account.tfvars”)) and even a fallback value in case that file isn’t found (e.g.: find_in_parent_files("account.tfvars", “fallback.tfvars")). You can combine this with the the extra_arguments helper to pull in common variable files for all of your modules. For example, you could create account.tfvars, region.tfvars, and environment.tfvars files that contain oft-repeated variables such as account ID and AWS region, and pull the variables in those .tfvars files automatically into every module by adding the following to the root terraform.tfvars:
terraform {
extra_arguments "common_vars" {
commands = ["${get_terraform_commands_that_need_vars()}"]optional_var_files = [
"${find_in_parent_folder("account.tfvars", "ignore")}",
"${find_in_parent_folder("region.tfvars", "ignore")}",
"${find_in_parent_folder("environment.tfvars", "ignore")}",
]
}
}The optional_var_files helper adds-var-file arguments to your Terraform commands if the file you specify exists.
What to do about it: Update to Terragrunt v0.13.9 and try out this new approach to make your infrastructure-live repo more DRY! Check out this PR and this PR in the Acme Reference Architecture for an example of the types of changes you’ll need to make and browse the Acme infrastructure-live repo to see the result (note: only customers who purchased the Gruntwork Reference Architecture have access to the Acme repos).
Terragrunt
Motivation: If your IAM users are defined in one AWS account (e.g., the security account) and your infrastructure is defined in several other AWS accounts (e.g., the dev, stage, and prod accounts), to switch between accounts, you’d have to assume an IAM role in those other accounts, which is a tedious process on the CLI. You’d have to run a bunch of commands (albeit, aws-auth made that a little easier) and the credentials you get back are only good for 1 hour!
Solution:
role_arn parameter of the S3 backend config before creating the S3 bucket or DynamoDB table.--terragrunt-iam-role parameter (or the corresponding TERRAGRUNT_IAM_ROLE environment variable) to tell Terragrunt to assume an IAM role before running any Terraform commands. Terragrunt will run all the STS assume-role commands for you every time you execute it, so you no longer have to jump through hoops, and you get new credentials for every single command you execute, so the 1-hour expiration should no longer be a problem.--role-arn parameter (or corresponding ROLE_ARN environment variable) to tell Terragrunt to assume an IAM role before running any KMS commands. This makes it easier to encrypt/decrypt secrets in other AWS accounts.What to do: Install Terragrunt v0.13.11 and gruntkms v0.0.6 and start using the --terragrunt-iam-role and --iam-role parameters to easily switch between accounts!
OpenVPN Package
Motivation: Several customers wanted to use OpenVPN with a multi-account setup where all the IAM users are defined in one AWS account (e.g., the security account) and each environment is defined in other AWS accounts (e.g., the dev, stage, and prod accounts). Our OpenVPN implementation uses IAM groups to manage who has access to the OpenVPN server, but you can only add IAM users from the same account to an IAM group, whereas with multi-account setups, those IAM users may be in a totally separate account.
Solution: We updated package-openvpn to work better with multiple accounts. You can now assume an IAM role in the AWS accounts where your OpenVPN servers are deployed to request/revoke certs.
What to do: Check out package-openvpn, v0.4.0 for upgrade instructions.
Reference Architecture
Motivation: Several customers asked us for a repo that shows examples of how to use the Gruntwork modules in a multi-account configuration, where each environment (e.g., stage, prod, etc) is deployed in a separate AWS account. This gives you much more fine-grained control over security, and complete isolation between accounts.
Solution: We created example repos for a fictional company called Acme that shows how you can use our modules and Terraform with multiple AWS accounts: Acme multi-account reference architecture. This is in addition to the Acme single-account reference architecture examples we added a few months ago.
What to do: Check out the new example code. Note that you’ll only be able to access the Acme repos if you purchased the Reference Architecture!
AWS Monitoring Package
Motivation: A few of our customers were trying to send multiple log files to CloudWatch Logs using the run-cloudwatch-logs-agent.sh script, but only syslog would ever actually show up.
Solution: We fixed a bug in the run-cloudwatch-logs-agent.sh script so it sets a unique name log stream name for each log file, so all log files now show up in the CloudWatch Logs UI!
What to do: We highly recommend updating all your Packer templates to use v0.7.0 of module-aws-monitoring, rebuilding your AMIs, and redeploying.
Terragrunt
Motivation: Terragrunt has had support for xxx-all commands (e.g., apply-all, plan-all, etc) for a while, but they were error prone to use.
Solution: We are starting to improve the xxx-all commands so you can use them on a regular basis to spin up or tear down an entire environment:
validate-all command that makes it easy to check if all of your dependencies are defined correctly and your Terraform code is free of basic errors.xxx-all commands so that they work with the —-terragrunt-source parameter, which makes testing easier by allowing you to deploy multiple modules from a local checkout rather than published code.xxx-all commands.HOME directory instead of tmp. This should hopefully finally fix caching issues!What to do: Grab Terragrunt v0.13.15, starting using the xxx-all commands, and let us know what other improvements could make this better!
Stateful Server, AMI Cluster, Kafka, ZooKeeper
Motivation: We created an attach-eni script that could be used to attach an ENI on Amazon Linux, where Amazon handles all the local ethernet config details automatically, and Ubuntu, where the script handles all the ethernet config details. Unfortunately, it looks like the way we handle those details works on t2 instances, but not other instance types, such as m4 and c4, as it turns out different instance types mount ethernet devices differently.
Solution: We fixed this bug in module-server, v0.2.1. Since this script is used under the hood by our AMI cluster, Kafka, and ZooKeeper modules, we also updated all of those, releasing module-asg, v0.6.3, package-zookeeper, v0.0.10, and package-kafka, v0.0.9.
What to do: Update to the versions above if you’re using instance types other than t2.
New infrastructure
Motivation: We created a simple spreadsheet as a poor-man’s crowdfunding solution for new infrastructure projects. The goal was (a) to be more transparent about our funding process and (b) provide access to better infrastructure at the lowest cost anywhere by splitting the cost amongst many companies. Although customers keep asking us for new infrastructure, it seems like few of you have added your names to that spreadsheet. This makes it very difficult for us to plan our roadmap and commit to deadlines.
Solution: We have added “drop dead dates” for each project. If not enough companies have signed up for a project by the time that date passes, the project is canceled, and we will focus on something else instead. Projects such as API Gateway, ELK, and Kubernetes are all going to expire soon without your help!
What to do: If you need new infrastructure, simply waiting and hoping someone else will pay for it won’t work! Please put your name down in the Crowdfunding Spreadsheet ASAP, or the project won’t happen at all. If you have questions or feedback, please email us at support@gruntwork.io.
dynamodb_table param, auto-creating the DynamoDB table as necessary.asg-xxx-alarms modules now allow you to create alarms for a list of ASGs, rather than just one.run-exhibitor script, as they seem to get lost after a Packer build.function_name output in the lambda module.engine and engine_version parameters so you have more control over what type of Aurora engine you’re running (e.g., you can use the Postgres-compatible one).What happened: AWS has announced that its managed Elasticsearch Service now works within your VPCs.
Why it matters: In the past, to talk to your Elasticsearch cluster, you had to send all requests over the public Internet. With VPC support, all traffic between your apps and the Elasticsearch cluster stay within the AWS network, which makes things more secure by default.
What to do about it: Version 1.2.0 of the Terraform AWS provider has support for this new functionality (see the CHANGELOG for details). Use the vpc options in the aws_elasticsearch_domain resource.
What happened: Amazon ElastiCache for Redis now supports encryption for all data at rest and all data in transit.
Why it matters: You now can have end-to-end encryption with Redis, which improves your security posture, and is required for many types of compliance (e.g., HIPAA, PCI).
What to do about it: The Terraform AWS provider does not yet support this functionality (follow this issue for details). If you need to enable this now, you’ll have to do it manually in the console.
What happened: Docker has announced that Kubernetes will become a “first-class citizen” of Docker Community Edition and Docker Enterprise Edition.
Why it matters: It’s not entirely clear what “first-class citizen” really means, but this is a strong sign that Kubernetes is winning the Docker orchestration wars.
What to do about it: If you’re interested in funding a Kubernetes module (i.e., instead of ECS or Nomad), add your name to the Crowdfunding Spreadsheet! This project will expire if we don’t get enough backers by December 15th.
What happened: Amazon ECR now supports lifecycle policies, which allow you to define rules for cleaning up your ECR repos.
Why it matters: Without these rules, the repos gradually fill up with Docker images, and it’s harder to figure out what versions you should use. You can now add a rule, for example, to expire all Docker images that are more than 14 days old.
What to do about it. Version 1.2.0 of the Terraform AWS provider has support for this new functionality (see the CHANGELOG for details). Use the aws_ecr_lifecycle_policy resource.
What happened: Amazon Aurora announced compatibility with PostgreSQL a while back and now this functionality is generally available.
Why this matters: Amazon claims that Aurora offers up to 3x better performance than PostgeSQL, while exposing a compatible API. It’s a fully managed service, so it has automatic failover, backups, and replication functionality built-in.
What to do about it: Try it out by launching an Aurora RDS instance! We recently released module-data-storage, v0.3.1, which allows you to specify the new engine and engine_version parameters for this purpose.
What happened: Amazon’s Application Load Balancer (ALB) now allows you to attach multiple TLS certificates to each listener.
Why this matters: You can now using a single ALB for all of your load balancing, even across multiple domain names, and it will pick the proper TLS cert for that domain name automatically. This was possible with TLS certs that specified multiple domain names before, but now it’s easy to do with any TLS certs.
What to do about it: Terraform does not yet have support for this new functionality (follow this issue for details), so if you want to use it, you’ll have to enable it manually in the AWS console.
What happened: The video and slides are now available for Jim’s HashiConf talk, Reusable, composable, battle-tested Terraform Modules.
Why it matters: The talk is an overview of why modules will change how you build and manage infrastructure, as well as a tutorial on how to build modules that are highly reusable and tested.
What to do about it: Go watch and share the talk!
Below is a list of security updates that may impact your services. It is up to you to scan this list and decide which of these apply and what to do about them. Note that for critical security updates, we send out emails immediately to the Gruntwork Security Alerts list.
Amazon Linux is subject to all the vulnerabilities listed in the “Linux” section, but Amazon Linux users generally upgrade when Amazon issues new releases. Note that in some cases, the severity is “critical” because Amazon Linux packages install old, outdated version of tools you would normally install on your own, such as Java or NodeJS.
Got questions or feedback on the newsletter? Email us at support@gruntwork.io. Want to learn more about Gruntwork? Check out https://www.gruntwork.io/.