Gruntwork Newsletter, August 2020

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

Hello Grunts,

In the last month, we revealed the new design of the Infrastructure as Code Library, which now consists of three layers: a Module Catalog, Service Catalog, and Architecture Catalog. This is a new standard for infrastructure code and we believe it’ll make it easier than ever to go to prod on AWS! We also updated Gruntwork Landing Zone to support a dedicated logs account and streamline the deployment process, updated Gruntwork Pipelines with support for executing arbitrary infrastructure code (e.g., building Docker images, AMIs, committing to Git, etc), added new modules for Redshift and EC2 backup using data lifecycle manager, and wrote a guide to securely managing secrets with Terraform. In the wider DevOps world, there have been a bunch of new releases: AWS Provider 3.0, Terraform 0.13, CDK for Terraform, and Kubernetes 1.17 support in EKS.

As always, if you have any questions or need help, email us at support@gruntwork.io!

Gruntwork Updates

undefined

Introducing: The Gruntwork Module, Service, and Architecture Catalogs

Motivation: While most customers loved being able to directly use modules from the Gruntwork Infrastructure as Code Library (IaC Library) to create their own, custom infrastructure, many of our customers asked for a higher level API to work with—something that let’s them go to prod with less work.

Solution: We’ve created a new design for the Gruntwork IaC Library! It now consists of three layers:

What to do about it: Read the introductory blog post to learn more, including code samples and diagrams that show you how each of these catalogs works. The Module Catalog is already available to everyone. The Service Catalog and Architecture Catalog are available now as part of a private, invite-only alpha. If you’re interested, contact us to learn how to get access!

undefined

Landing Zone Updates: Dedicated Logs Account and Better UX

Motivation: Since we released our Landing Zone solution, we’ve had a lot of requests from our customers to to support aggregating in a dedicated logs account. We also uncovered some chicken-and-egg issues related to AWS Config and CloudTrail that required you to run apply multiple times across multiple modules, once with AWS Config and CloudTrail disabled, and once with them enabled.

Solution: We’ve updated our Landing Zone solution with support for a dedicated logs account which can be used to aggregate AWS Config and CloudTrail data from all your other accounts! Moreover, we’ve created workarounds for the chicken-and-egg problems related to AWS Config and CloudTrail, so now instead of having to run apply multiple times across multiple modules, you only need to run apply once per AWS account, as you’d expect.

What to do about it: If you’re upgrading an existing Landing Zone deployment, upgrade to the v0.36.0 releases of module-security, making sure to follow the migration guide in those releases notes and the release notes for v0.34.0 too! If you’re deploying your Landing Zone from scratch, check out our dedicated deployment guide instead.

undefined

Gruntwork Pipelines Update: Deploy arbitrary infrastructure code

Motivation:**** Back in March, we announced the release of Gruntwork Pipelines, our solution for implementing a secure, automated CI/CD pipeline for infrastructure code. The initial release was limited to just Terraform and Terragrunt: if your deployment workflows depended on building docker images, building AMIs with Packer, or committing changes to Git, you had to do those steps directly on the CI server. This meant that you still had to grant some potentially powerful permissions to the CI server directly (e.g., write access to your Git repos), and mixing a CI servers—which, by design, is used by your entire team to execute arbitrary code—with powerful permissions is not the best idea from a security perspective.

Solution: To address this limitation, we’ve updated Gruntwork Pipelines to support invoking arbitrary scripts—so instead of solely letting you run the built-in infrastructure-deploy-script Terraform/Terragrunt deployments, you can now also run scripts for building Docker images, building AMIs with Packer, committing to Git, and your own custom scripts—all in a secure, locked-down manner that does not require you to give your CI servers powerful permissions. The new version of Gruntwork Pipelines enhances the existing modules with the following feature set:

We also updated our example pipeline so that you have an example of how to configure your apps to utilize all this new functionality.

What to do about it: Check out v0.24.0 of the ECS deploy runner, including the migration guide in the release notes to update your existing pipelines and let us know what you think!

undefined

New module: Redshift Cluster

Motivation: Amazon’s Redshift gives you a managed, scalable, data warehouse in the cloud.

Solution: We’ve added a new redshift module to module-data-storage! This makes it easy to use Redshift in just a few lines of code:

module "redshift_example" {
source = "git::git@github.com:gruntwork-io/module-data-storage.git//modules/redshift?ref=v0.15.0"
name = "example-cluster"
port = 5439
master_username = var.master_username
master_password = var.master_password
instance_type   = "dc2.large"
number_of_nodes = 3
vpc_id     = var.vpc_id
subnet_ids = var.subnet_ids
}

What to do about it: Try out the new redshift module and let us know what you think!

undefined

New module: EC2 Backup

Motivation: Amazon’s data lifecycle manager allows you to configure automatic EBS volume backups on flexible schedules.

Solution: We’ve added a new ec2-backup module to module-server! This makes it a snap to configure your own EBS backup policies.

What to do about it: Try out the new ec2-backup module and let us know what you think!

undefined

New blog post: A comprehensive guide to managing secrets in your Terraform code

Motivation: One of the most common questions we get about using Terraform to manage infrastructure as code is how to handle secrets such as passwords, API keys, and other sensitive data. The right approach to use is not obvious and there are many gotchas and stumbling blocks.

Solution: We wrote a blog post called A comprehensive guide to managing secrets in your Terraform code that goes over the most common techniques you can use—including environment variables, encrypted files (e.g., KMS, PGP, SOPS), and secret Stores (e.g., Vault, AWS Secrets manager)—the trade-offs between them, and important pre-requisites, such as secure Terraform state storage.

What to do about it: Check out the blog post and let us know which secrets management solution you like best!

undefined

Open Source Updates

Terragrunt

Terratest

terraform-aws-nomad

terraform-aws-consul

terraform-aws-vault

package-terraform-utilities

undefined

Other updates

module-security

module-ci

package-static-assets

module-data-storage

terraform-aws-eks

module-asg

module-ecs

terraform-aws-vpc

This module was renamed from module-vpc to terraform-aws-vpc. GitHub automatically redirects traffic to the new name, so no action is required as a result of this change.

module-load-balancer

terraform-aws-monitoring

package-openvpn

package-kafka

module-cache

cis-compliance-aws

DevOps News

Terraform AWS Provider v3.0 is out

What happened: HashiCorp and AWS have released v3.0 of the AWS Provider for Terraform.

Why it matters: This is a major new release of the AWS provider, which means that you get new features, such improvements to the Amazon Certificate Manager (ACM) resources, removal of state hashing, and improved authentication ordering. However, major releases also bring breaking changes. In particular, this new version of the AWS Provider only works with Terraform 0.12 and newer and has breaking changes for many individual resources.

What to do about it: We have started work to upgrade the entire Gruntwork IaC Library to be compatible with AWS Provider v3.0. We will announce when this process is complete and how to update. Until then, if you haven’t already, we strongly recommend pinning your AWS Provider version to v2.x by adding code like this:

provider "aws" {
# ... other configuration ...
version = "~> 2.70"
}

Terraform 0.13 is out

What happened: HashiCorp has released Terraform version 0.13.

Why it matters: The new release brings some powerful new features, including the ability to use count and for_each on modules, automatic installation of third-party providers, and custom variable validation! However, the new release also brings some backwards incompatibilities.

What to do about it: As soon as we’re done with the AWS Provider 3.x upgrade, we will move on to upgrade all of our code to work with Terraform 0.13. We will announce when this process is complete and how to update. Until then, if you haven’t already, we strongly recommend pinning your Terraform version to 0.12.x by adding code like this:

terraform {
required_version = "= 0.12.29"
}

CDK for Terraform: Python and TypeScript support for Terraform

What happened: HashiCorp has launched a developer preview of a Cloud Development Kit (CDK) for Terraform.

Why it matters: The CDK allows you to define your infrastructure using a general purpose programming language, such as Python and TypeScript, instead of HCL, while still running Terraform under the hood. For example, you might create a VPC using the following TypeScript code:

class MyStack extends TerraformStack {
constructor(scope: Construct, name: string) {
super(scope, name);

new AwsProvider(this, 'aws', {
region: 'us-east-1'
});

const vpc = new Vpc(this, 'my-vpc', {
cidrBlock: '10.0.0.0/16'
});
new Subnet(this, 'my-subnet', {
vpcId: Token.asString(vpc.id),
cidrBlock: '10.0.0.0/24'
});
}
}

const app = new App();
new MyStack(app, 'vpc-example');
app.synth();

And then deploy as follows:

$ cdktf deploy
Stack: vpcexample
Resources
+ AWS_SUBNET       mysubnet     aws_subnet.vpcexample_mysubnet_3769B309
+ AWS_VPC              myvpc        aws_vpc.vpcexample_myvpc_80A1790F

Diff: 2 to create, 0 to update, 0 to delete.
Do you want to continue (Y/n)?

What to do about it: Give the CDK a shot and let us know what you think! We will be trying it out too, especially as the solution matures more.

EKS / Kubernetes updates

What happened: Amazon has announced that EKS now supports Kubernetes version 1.17 and managed node groups no support launch templates and custom AMIs.

Why it matters: The Kubernetes 1.17 release includes Cloud Provider Labels, ResourceQuotaScopeSelectors, TaintNodesByCondition, Finalizer protection, CSI Topology graduating to generally available, and the Windows containers RunAsUsername feature is now in beta. Support for custom AMIs on managed node groups allows you to install custom software while still having EKS run and manage the worker nodes for you.

What to do about: As of v0.21.0, our terraform-aws-eks repo supports Kubernetes 1.17, so feel free to upgrade. We also updated our Comprehensive Guide to EKS Worker Nodes blog post with the new information about managed worker nodes.

Text Link

Platform

IaC Library
Account Factory
Pipelines
Patcher

Services

Accelerator
OpenTofu Services
Terragrunt Services
Pricing
Support

Open Source

Terragrunt
Terratest
Boilerplate
Cloudnuke
Git Xargs

Resources

Docs
Blog
Books
Community

Company

About
Careers
Merch
Legal
Security