Gruntwork Newsletter, January 2020

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

Hello Grunts,

In the last month, we made a number of improvements to our EKS modules, including adding support for Managed Node Groups and Fargate (serverless worker nodes), launched Gruntwork CIS Compliance for the Reference Architecture, added a new module for configuring AWS Security Hub, added new modules for creating a Terraform-backed AWS Landing Zone, updated the Reference Architecture to enable VPC logs and build Packer templates in custom VPCs, gave a talk on automated testing best practices for infrastructure code, launched a dedicated website for Terragrunt, wrote a blog post on how to manage multiple versions of Terragrunt and Terraform as a team, and shared our 2019 Year in Review.

As always, if you have any questions or need help, email us at support@gruntwork.io!

Gruntwork Updates

undefined

EKS Worker Nodes Updates: Managed Node Groups and Fargate

Motivation: Last Year in November, AWS announced support for Managed Node Groups, a feature that provides a managed ASG that is optimized for use with EKS. Then, in December, AWS announced support for Fargate in EKS, a serverless option for Kubernetes workers where you do not need to deploy and manage any EC2 instances (servers). With these two features, users now had three options for running their workloads on EKS. To many new users of EKS, it isn’t immediately obvious what the trade offs are between the three options, and there is no way easy to see that other than to ramp up on all three options.

Solution: We wrote A Comprehensive Guide to EKS Worker Nodes blog post that introduces each of the options for worker nodes, how they work, and what trade offs you should consider when picking which one to use. We also updated our EKS module, terraform-aws-eks, to support Managed Node Groups and Fargate. Youo can now use either the eks-cluster-managed-worker module to provision AWS-managed worker nodes or the eks-cluster-workers module to provision self-managed worker nodes. Additionally, the Control Plane module (eks-cluster-control-plane) now exposes an input variable fargate_only which can be used to configure the cluster with default Fargate Profiles such that all pods in the kube-system and default Namespaces (the two Namespaces created by default on all EKS clusters) will deploy to Fargate.

Other EKS updates:

What to do about it: Check out the blog post to learn more about the three worker node options, and upgrade your EKS modules to the latest version (v0.13.0) to try them out!

undefined

CIS Compliance Updates

Motivation: Last month, we announced Gruntwork Compliance for the CIS AWS Foundations Benchmark. The AWS Foundations Benchmark is an objective, consensus-driven guideline for establishing secure infrastructure on AWS from the Center for Internet Security. With Gruntwork Compliance, you gain access to a set of modules from the Infrastructure as Code Library that were certified by Gruntwork and CIS to be compliant with the Benchmark. However, you still had to combine the modules together into a coherent architecture to set up CIS compliant accounts.

Solution:**** This month, we are excited to announce a CIS Compliant Reference Architecture! The Reference Architecture is an opinionated end-to-end tech stack built on top of the Infrastructure as Code Library that we deploy into your AWS accounts in about a day. This month we have extended the Reference Architecture to be CIS compliant for customers who have Gruntwork Compliance. This means that not only do you get a full end-to-end, production grade architecture in a day, but it will also be certified to comply with the CIS AWS Foundations Benchmark.

If you are a Gruntwork Compliance customer, you can check out example Reference Architecture code at the following repositories:

AWS Security Hub Module: This month, we also extended our compliance modules with a module that enables AWS Security Hub in all regions of an account. AWS Security Hub will automatically and continuously monitor your AWS account for CIS compliance status. This provides a way to ensure your account stays compliant over time as you apply changes to your infrastructure.

Other Compliance Updates

What to do about it: If you are a Gruntwork Compliance customer, check out the new AWS Security Hub module. If you are interested in a CIS compliant Reference Architecture, contact us!

undefined

Building blocks for a Terraform-backed AWS Landing Zone

Motivation: Create all the AWS accounts in an AWS Organization can be tricky, as you have to configure each account with a security baseline that includes IAM Roles, IAM Users, IAM Groups, IAM Password Policy, AWS CloudTrail, AWS Config, AWS Config Rules, Amazon GuardDuty, a VPC, and more. AWS offers two products to make it easier to set up AWS accounts, AWS Landing Zone and AWS Control Tower, but both have significant limitations: AWS Landing Zone is only available via an AWS Pro Serve consulting engagement, which can be quite expensive and time consuming, and the the quality and flexibility of the (mostly CloudFormation) code is questionable; AWS Control Tower allows you to create new AWS accounts by clicking around the AWS console, but it’s slow, clunky, only supported in new AWS accounts that don’t already have AWS Organizations enabled, and it only supports a limited set of pre-defined account baseline blueprints, so you have little ability to customize the accounts or manage them as code.

Solution: We have released a set of new Terraform modules to help you create AWS accounts with all the features of AWS Landing Zone, but 100% customizable and managed as code. The new modules include:

You can combine these modules to create your own Terraform-backed Landing Zone that matches your exact requirements and is fully managed as code. We have examples, docs, and reusable account baseline modules coming soon!

What to do about it: Check out the new modules in module-security to learn more.

undefined

Reference Architecture Updates

Motivation: The Reference Architecture is an opinionated, end-to-end tech stack built on top of the Infrastructure as Code Library that can be deployed in about a day. However, we noted a few feature gaps in the current Reference Architecture implementation:

Solution: We have updated the Reference Architecture to address these gaps:

Other Ref Arch updates:

What to do about it: Follow the examples in the commits above to update your own Reference Architecture.

undefined

Terragrunt website

Motivation: The amount of documentation for Terragrunt has grown over the last few years, and the README.md for the repo had gotten so big as to be nearly unusable.

Solution: We’ve launched a dedicated website for Terragrunt at https://terragrunt.gruntwork.io/! It breaks the documentation down across multiple pages, has more examples, and an easier-to-use navigation.

What to do about it: We hope the new website makes it easier to get started with Terragrunt. Let us know what you think!

undefined

New Talk: Automated Testing for Terraform, Docker, Packer, Kubernetes, and More

Motivation: Many customers were asking for guidance on the best way to write automated tests for the infrastructure code.

Solution: We did a talk at QCon SF and AWS re:Invent called Automated Testing for Terraform, Docker, Packer, Kubernetes, and More. Topics covered include: unit tests, integration tests, end-to-end tests, dependency injection, test parallelism, retries and error handling, static analysis, and more.

What to do about it: Check out the slides and video on the InfoQ website.

undefined

A year in review, 2019

Motivation: We wanted to take some time to pause, reflect on what we did, and think a little about the future of Gruntwork.

Solution: We wrote a blog post called A Year in Review, 2019, where we shared three lessons that helped us nearly triple our recurring revenue from $1M to $2.7M (with $0 in funding).

What to do about it: Read the blog post and share your feedback, especially on what you’d love to see us do in 2020!

undefined

Blog post on Terraform and Terragrunt version management

Motivation: Terragrunt and Terraform are relatively young projects in the DevOps ecosystem. As such, both projects introduce backwards incompatible changes more often than we like. Oftentimes we find the need to switch between multiple terraform and terragrunt versions. We also want to enforce versions across the team to avoid inadvertent state file upgrades that are backwards incompatible.

Solution: We wrote a blog post describing a workflow around tfenv and tgenv that makes it easier to work with multi version Infrastructure as Code projects!

What to do about it: Check out our blog post and let us know what you think!

undefined

Open source updates

undefined

Other updates

DevOps News

Packer now supports HCL

What happened:**** Packer 1.5.0**** released beta support for configuring Packer templates using HCL. You can read more about it in the official docs.

Why it matters: Prior to 1.5.0, Packer only supported templates that were defined using JSON. JSON has the advantage of being ubiquitous, but has a number of drawbacks, including lack of support for comments, multi-line strings, and code reuse, which made it difficult to write complex templates. Now, with Packer 1.5.0, you have the option to define your Packer templates using HCL (the same language used in Terraform). This makes it easier to:

What to do about it: Check out the new official docs on using HCL2 with Packer, including the migration guides to convert your packer JSON templates to HCL, to give it a try and let us know what you think!

EKS Fargate support

What happened: AWS announced support for serverless Kubernetes in the form of Fargate for EKS.

Why it matters: Up until this point, you always had to run servers (EC2 instances) in your AWS account to act as worker nodes for your EKS cluster. This meant that you still had to manage the lifecycle of the instances, including worrying about SSH access and applying security patches. Now with Fargate, you can get a truly serverless experience with EKS. Fargate allows you to schedule Pods on ephemeral VMs that AWS fully manages for you. You no longer have to worry about all the concerns around scaling, logging, security, etc that comes with managing servers when you use Fargate.

What to do about it: Check out our blog post on EKS worker nodes to learn more about all the trade offs with using Fargate, and try it out using the latest version of our terraform-aws-eks module.

EKS private Kubernetes endpoint DNS support

What happened: AWS announced support for automatic resolution of EKS private Kubernetes endpoints, even over a peered VPC network.

Why it matters: Locking down your Kubernetes API endpoint is an important step to improve your security posture of the EKS cluster. EKS offers the ability to lock down the Kubernetes API endpoint such that it is only available within the VPC (private endpoint). Up until now, you could not access this endpoint over a peered VPC network unless you had enabled Route 53 DNS forwarding rules across the peered network. This was quite expensive to maintain for a trivial functionality. Now, with this update, you no longer need to configure, maintain, and pay for the forwarding rules as AWS automatically sets them up for you.

What to do about it: This feature is available to all EKS clusters. You can safely remove the DNS forwarding rules and still access your EKS endpoints over a peered network.

ECS Cluster Auto Scaling

What happened: AWS has added support for managed auto scaling for ECS.

Why it matters: Auto scaling ECS clusters used to be tricky, even if you used an Auto Scaling Group under the hood, as you had to carefully detect scale down events, identify the EC2 instances that were going to be terminated, and move any Docker containers on those instances to other instances in the cluster. Now, instead of manually managing all of this, managed auto scaling will do it all for you, automatically.

What to do about it: You can try out managed auto scaling using the aws_ecs_capacity_provider resource. However, please note that Terraform currently has some limitations that may make this resource tricky to use: e.g., Terraform doesn’t currently support deleting ECS cluster capacity providers, so terraform destroy might not clean everything up correctly; also, managed auto scaling will automatically add tags to your Auto Scaling Groups that Terraform won’t know how to manage, so you might get spurious diffs on plan.

Provisioned concurrency for AWS Lambda

What happened: AWS has added a provisioned concurrency feature for AWS Lambda, which allows you to automatically keep some number of Lambda functions warm.

Why it matters: One of the biggest drawbacks with AWS Lambda are cold starts, where each concurrent invocation of a Lambda function could take hundreds of milliseconds or even multiple seconds to boot up the first time. This made Lambda very tricky to use when latency mattered: e.g., for a web application. AWS now supports provisioned concurrency for Lambda functions, which allows you to “pre-warm” a configurable number of Lambda function containers to avoid cold starts. You can also use auto scaling policies to dynamically change the level of provisioned concurrency based on load.

What to do about it: If you’re using Lambda in latency-sensitive situations, give provisioned concurrency a try! You can configure it with Terraform by using the aws_lambda_provisioned_concurrency_config resource.

Security Updates

Below is a list of critical security updates that may impact your services. We notify Gruntwork customers of these vulnerabilities as soon as we know of them via the Gruntwork Security Alerts mailing list. It is up to you to scan this list and decide which of these apply and what to do about them, but most of these are severe vulnerabilities, and we recommend patching them ASAP.

NPM Binary Planting Vulnerability

RDS TLS cert update

Multiple vulnerabilities in Mozilla Firefox

Text Link

Platform

IaC Library
Account Factory
Pipelines
Patcher

Services

Accelerator
OpenTofu Services
Terragrunt Services
Pricing
Support

Open Source

Terragrunt
Terratest
Boilerplate
Cloudnuke
Git Xargs

Resources

Docs
Blog
Books
Community

Company

About
Careers
Merch
Legal
Security