Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.
Hello Grunts,
We’re excited to announce that it’s finally time to upgrade to Terraform 0.12! We’ve updated the entire Infrastructure as Code Library to be compatible with it and have included upgrade instructions in this blog post. In other news, the Early Release of Terraform: Up & Running, 2nd edition is now available (also fully updated for Terraform 0.12), Terragrunt now supports all Terraform built-in functions and has first-class support for GCS, and we’ve made several dozen other improvements and fixes.
As always, if you have any questions or need help, email us at support@gruntwork.io!
Motivation: Terraform 0.12 came out a couple months ago and we’ve all been waiting to use it.
Solution: We’ve updated the entire Infrastructure as Code Library to be compatible with Terraform 0.12! That means you should now upgrade to take advantage of first-class expressions, for and for_each, a more powerful type system, better error messages, and more.
What to do about it:
Prepare for the upgrade:
Do the upgrade:
infrastructure-modules repo). We recommend updating and testing one module at a time. For the rest of this step-by-step guide, let’s assume you have a module ininfrastructure-modules/networking/vpc-app.cd infrastructure-modules/networking/vpc-appinfrastructure-modules/networking/vpc-app most likely makes use of module-vpc, which you’ll want to upgrade to v0.6.0 (or newer).required_version constraint in your code, for the time being, remove it so the Terraform auto-upgrade step doesn’t complain. We’ll bring it back later.terraform init -backend=false.terraform 0.12upgrade.versions.tf file. We recommend specifying a strict required_version constraint in your Terraform code: e.g., required_version = "= 0.12.4".count = var.boolean_variable no longer works, so you have to explicitly change it to count = var.boolean_variable ? 1 : 0. A second gotcha is that 0.12upgrade occasionally duplicates comment blocks, so you’ll need to delete the extra ones.variable declarations and add appropriate type constraints.infrastructure-live, cd into your dev environment, and run terragrunt apply --terragrunt-source /path/to/infrastructure-modules/networking/vpc-app.Motivation: In late 2016, we released the Comprehensive Guide to Terraform blog post series. In early 2017, we turned it into a book, Terraform: Up & Running. In the two years since, Terraform has changed considerably (4 major releases, a change to HCL2, a revamp of Terraform state, and much more), so both the blog post series and book were due for an update.
Solution: We’re happy to announce that (1) we’ve updated the Comprehensive Guide to Terraform blog post series all the way through Terraform 0.12 and (2) the Early Release of the 2nd edition of *Terraform: Up & Running* is now available! The 2nd edition of the book is nearly double the length of the 1st edition (~160 more pages), and it has been fully updated through Terraform 0.12, including two completely new chapters: Production-grade Terraform Code and How to Test Terraform Code.
What to do about it: Check out the Terraform: Up & Running, 2nd edition Early Release blog post to learn about the top 10 things that have changed in Terraform since the 1st edition, read the early release on O’Reilly, and let us know what you think!
Motivation: Last month, we upgraded Terragrunt to support Terraform 0.12 and use HCL2 syntax. This new syntax includes support for first-class functions and expressions, and users wanted to be able to take advantage of that more.
Solution: We’ve updated Terragrunt so that it now supports ALL built-in Terraform functions! You can use them to do math (e.g., ceil, min, max), manipulate strings (e.g., format, split, substr), interact with files (file, dirname), and much more. And since you can use expressions and functions everywhere in terragrunt.hcl files, this gives you a lot of expressive power.
# Examples of using built-in functions throughout terragrunt.hclterraform {
source = "../modules/${basename(get_terragrunt_dir())}"
}remote_state {
backend = "s3"
config = {
bucket = trimspace(" my-terraform-bucket ")
region = join("-", ["us", "east", "1"])
key = format("%s/foo.tfstate", path_relative_to_include())
}
}inputs = merge(
yamldecode(file(find_in_parent_folders("common.yaml"))),
{
override = "some-value"
},
)What to do about it: Upgrade to Terragrunt v0.19.4 and let us know what you think!
Motivation: In May, we announced a set of reusable, production-grade infrastructure modules for GCP. They work great with Terraform, however, we wanted to add support to Terragrunt for use with our GCP Reference Architecture.
Solution: We’ve updated Terragrunt so that it now supports storing remote state in a GCS bucket. It will also automatically create one with versioning enabled if it doesn’t exist already.
What to do about it: Upgrade to Terragrunt v0.19.8 and let us know what you think!
project variable to all the resources.bastion-host module now outputs the instance private_ip address.bastion-host module google_compute_instance resource now uses the supplied project variable.vpc-network module now outputs the correct private subnetwork name.cloud-sql module has now been upgraded to support Terraform v0.12..tf.json files in addition to .tf files in the download dir when fetching code from a source URL.include blocks are parsed so that path_relative_to_include, path_relative_from_include, and get_parent_terragrunt_dir work correctly when used directly in the child terragrunt.hcl.terragrunt.hcl(disable_init) which, when set to "true", will skip the backend initialization step.project and location are optional when using the GCS backend. Also fixes a bug where Terragrunt may create a GCS state bucket in the wrong location.terraform.Output commands (e.g., OutputE, OutputListE, OutputMapE, etc) to only read from stdout when parsing output. Before, they were also accidentally reading stderr. This worked fine with Terraform, as the output command wrote nothing to stderr, but did not work with Terragrunt.terraform. The default scanners in golang have a limit of 64kb per line, which for some terraform modules is not enough. This allows you to extend the buffer so that you can test terraform modules that have long outputs.dep, where the client-go library did not have a locked version of k8s.io/api. This caused problems when resolving dependencies because the latest version of k8s.io/api has breaking changes that are not compatible with version release-9.0 of client-go that we are using. This release locks down the k8s.io/api dependency so that users pulling in terratest will not run into this issue.for_each loop used to add tags to the ASG from the cluster_extra_tagsargument.Type=notify to systemd config for Consul so that any other systemd services that depend on Consul (e.g., via Requires=consul.service) will be started after Consul is fully up and running, rather than as soon as the Consul task is started.org for organization), but it now allows using alternate names. Checkout the release notes for more details.install-openvpn now supports Ubuntu 18.04.openvpn-server module.create_before_destroy, leading to issues when trying to do a rolling update.us-east-1 in the elasticsearch-cluster-backup module. With this change, the elasticsearch-cluster-backup module now takes in a new required input variable region.apt-get upgrade.aws_alb_target_group.health_check.matcher to the jenkins-server module so that it can be configured.setup-minikube script in the kubernetes-circleci-helpers module, caused by the removal of the specific docker version we depended on from the ubuntu apt caches. Also, the install-jenkins module is now verified to work with Ubuntu 18.04. Finally, the jenkins-server module is now using v0.8.1 of module-asg/modules/server-group, which includes a fix for IAM timing issues.var.enable_elastic_ips was not properly used in the conditional logic to control Route 53 records in modules/server-group. This led to syntax errors on terraform 0.12 when you had the right inputs to enable the resource.terraform apply to fail for code using the server-group module. This release addresses that by adding a sleep to wait for IAM resource propagation after creation.nlb module. Specifically, the access_logs subblock requires a valid bucket and prefix to be specified if the block is included, regardless of enabled flag. This release fixes it so that you can still pass in a null or empty bucket and prefix even if the access logs are disabled.ssm-healthchecks-iam-permissions which provides IAM policies that you can attach to instance profiles that grants the EC2 instance the requisite permissions to run SSM healthchecks, which are enabled by default on many base AWS AMIs such as Ubuntu.ssh-grunt install was missing the --force-user-deletion flag.logs/cloudwatch-log-aggregation-scripts worked with Ubuntu 18.04. This release fixes the logs/cloudwatch-log-aggregation-scripts module to also be compatible with Ubuntu 18.04.type constraint on the metrics variable of the cloudwatch-dashboard-metric-widget module to allow non-string types in the inner list, including map values.var.disable_status_yellow_alarm to true.replication_group_size and cluster_modes input variables in the redis module.mount-ebs-volume so it works properly on Amazon Linux.What happened: The AWS Network Load Balancer (NLB) now supports load balancing UDP traffic.
Why it matters: Before, the NLB only supported TCP, but with UDP support, the NLB can now handle many new use cases, such as DNS, logging, and IoT.
What to do about it: Check out the documentation for details.
What happened: Amazon has launched a new service called EventBridge, which aims to be a central event bus for all of your SaaS applications and AWS Services.
Why it matters: You can use EventBridge to trigger one of many targets (e.g., Lambda function, ECS Task, Kinesis Stream, SNS message) from one of many sources (e.g., more than 90 AWS Services, such as EC2 Auto Scaling, CodeBuild, and AWS Step Functions and ~10 SaaS applications, including ZenDesk, DataDog, and PagerDuty). This allows you to connect all of your systems together: e.g., a support ticket in ZenDesk can trigger a Lambda function that issues a response.
What to do about it: Give EventBridge a shot and let us know what you think!
What happened: AWS has launched a new feature called EC2 Instance Connect that allows you to SSH to your EC2 Instances using your IAM credentials for authentication.
Why it matters: The main advantages of EC2 Instance Connect are a) you don’t have to manage SSH keys, b) each developer connects with their IAM credentials, rather than a shared SSH Key, and c) it works from the AWS Web Console, in addition to your terminal. The main disadvantages of EC2 Instance Connect are that (a) it doesn’t use the SSH protocol, so you can’t use all the SSH port forwarding, tunneling, and other features you’re used to and (b) it doesn’t create OS users for you, so you can only connect as OS users you’ve created yourself through other mechanisms.
What to do about it: If you just need an easy way to connect to EC2 Instances, and don’t care much about the OS users or protocol, give EC2 Instance Connect a shot! However, if you want actual SSH connections and to auto sync OS users (and permissions) with your IAM users, you may want to stick with ssh-grunt.
What happened: AWS has launched a new feature called VPC Traffic Mirroring that allows you to replicate the networking traffic to an EC2 Instance or EC2 Instances behind an NLB, filter the traffic, and forward it to other destinations.
Why it matters: This feature can be very useful for troubleshooting (e.g., send the traffic to an environment where devs can analyze it and debug issues), monitoring (e.g., measure and alert on certain types of network traffic), and security (e.g., detect an attack), all without adding any extra complexity or overhead to your EC2 Instances.
What to do about it: Check out the documentation for details on how to use this feature. Note that VPC Traffic Mirroring currently only works with Nitro instances.
What happened: AWS has announced that Aurora Serverless now supports PostgreSQL.
Why it matters: Aurora Serverless allows you to have an on-demand database that would shut down when not in use and spin up quickly when you need it, which was great for saving money on infrequently used apps and pre-production environments. However, the original release only worked with MySQL. This new announcement means you can now take advantage of the serverless functionality with PostgreSQL too.
What to do about it: See the announcement blog post for details.
Below is a list of critical security updates that may impact your services. We notify Gruntwork customers of these vulnerabilities as soon as we know of them via the Gruntwork Security Alerts mailing list. It is up to you to scan this list and decide which of these apply and what to do about them, but most of these are severe vulnerabilities, and we recommend patching them ASAP.
strong_password Ruby gem has been hacked, and replaced with a Gem that silently executes remote code in production. If you are using this Gem, make sure you are not using release v0.0.7!