Gruntwork Newsletter, May 2020

Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.

In the last couple months, we’ve released a guide to using Gruntwork with Terraform Cloud and Terraform Enterprise, created a new module for Amazon’s Elastic File Store (EFS), added PrivateLink support to keep AWS API calls within your VPC network, added support for managing secrets in Terragrunt using Mozilla SOPS, did a podcast interview with SE Radio, and made a huge number of other fixes and improvements.

As always, if you have any questions or need help, email us at support@gruntwork.io!

• Gruntwork Updates
• DevOps News
• Security Updates

Gruntwork Updates

How to use Gruntwork with Terraform Cloud and Terraform Enterprise

Motivation:**** Gruntwork customers love saving time by using the Gruntwork IaC Library to do the heavy lifting of building production-grade cloud infrastructure. Many in the Terraform community have enjoyed using Terragrunt for its ability to keep your code and variables, CLI args, etc DRY, and for better support when applying changes across multiple accounts/modules/environments. HashiCorp customers love using Terraform Cloud (TFC) and/or Terraform Enterprise (TFE) for its browser-based controls, support for teams, and remote execution features. Fortunately, in this case, you can have your cake, and eat it too! Gruntwork and Terragrunt are compatible with Terraform Cloud and Terraform Enterprise.

Solution: We’ve written a blog post that shows you how to use Gruntwork’s Terrraform modules with TFC/TFE and even how to integrate Terragrunt with TFC/TFE. We’ve also updated our guide, How to Use the Gruntwork Infrastructure as Code Library, with all the details you need to set the integration up for yourself. Take a look and let us know what you think!

New module: Amazon Elastic File System (EFS)

Motivation: Amazon’s Elastic File System (EFS) gives you a managed, scalable, elastic NFS file system in the cloud. It’s an easy way to create a hard-disk that’s shared across many EC2 instances, but we did not have any module to deploy and manage EFS as code.

Solution: We’ve added a new efs module to module-data-storage! This makes it easy to use EFS in just a few lines of code:

module "efs" {
source = "git::git@github.com:gruntwork-io/module-data-storage.git//modules/efs?ref=v0.12.15"

name              = "example-efs"
vpc_id            = data.aws_vpc.default.id
subnet_ids        = data.aws_subnet_ids.default.ids
}

A huge thank you to Jesse Bye for contributing this new module!

What to do about it: Try out the new efs module and let us know what you think!

New module: PrivateLink for VPCs

Motivation: By default, any API calls you make to AWS go over the public Internet, even if you make those API calls from within a VPC in your AWS account. You can keep those API calls more private by routing them entirely within your VPC using VPC Endpoints (which are built into the Gruntwork VPC modules), but VPC Endpoints only support S3 and DynamoDB. For all other AWS services, you need to use PrivateLink, a separate (paid) service, which our VPC module did not support.

Solution: We’ve added a new vpc-interface-endpoint module to module-vpc which you can use to enable PrivateLink for AWS services in your VPC. Here’s an example of how to use the module to enable PrivateLink for all calls to AWS EC2 APIs:

module "vpc" {
source = "git::git@github.com:gruntwork-io/module-vpc.git//modules/vpc-app?ref=v0.8.7"

vpc_name   = "example-vpc"
aws_region = "us-east-1"
cidr_block = "10.0.0.0/16"
}

module "vpc_endpoint_ec2_example" {
source = "git::git@github.com:gruntwork-io/module-vpc.git//modules/vpc-interface-endpoint?ref=v0.8.7"

vpc_id     = module.vpc.vpc_id
subnet_ids = module.vpc.private_app_subnet_ids

# Enable PrivateLink for EC2
enable_ec2_endpoint = true
}

Note that using PrivateLink incurs an extra fee per hour per AZ where PrivateLink is deployed and per GB of data processed, so make sure to check the pricing page before enabling it for all services!

What to do about it: If you want an extra layer of privacy for your AWS API calls, try out the new vpc-interface-endpoint module and let us know what you think!

Manage secrets in Terragrunt using Mozilla SOPS

Motivation: Terraform does not have great native features for managing secrets (see this issue, which has been open for ~6 years).

Solution: We’ve added support for Mozilla SOPS to Terragrunt. SOPS allows you to securely store secrets in JSON or YAML files encrypted via AWS KMS, GCP KMS, Azure Key Vault, or PGP. For example, you might have a secrets.yml with the contents:

db:
user: ENC[AES256_GCM,data:CwE4O1s=,iv:2k=,aad:o=,tag:w==]
password: ENC[AES256_GCM,data:p673w==,iv:YY=,aad:UQ=,tag:A=]

Note that the user and password are encrypted with one of the mechanisms supported by SOPS, which means it is safe to check this file into version control. You use the new sops_decrypt_file helper in your terragrunt.hcl files to automatically read the file and decrypt the contents, allowing you to pass those values to your Terraform code:

locals {
secrets = yamldecode(sops_decrypt_file("secrets.yml"))
}

inputs = {
user     = local.secrets.db.user
password = local.secrets.db.password
}

This approach allows you to avoid putting any plain text secrets directly in version control, while still managing everything as code. Note that any secrets you pass to Terraform may be stored in its state file in plain text, so make sure you store that state file in an encrypted format too (e.g., using S3 encryption)!

What to do about it: Check out the sops_decrypt_file documentation, give the new helper a try, and let us know what you think!

SE Radio podcast interview: Infrastructure as Code Best Practices

Motivation: The team at Software Engineering (SE) Radio wanted to know more about infrastructure as code.

Solution: Gruntwork co-founder Yevgeniy (Jim) Brikman did a podcast interview with SE Radio on Infrastructure as Code Best Practices. The discussion covers similarities and differences between conventional software engineering and code-driven infrastructure; factoring code into modules; layering; terraform code organization for micro-services; releases and tagging; code reviews; unit testing infrastructure; deployment of infrastructure; ownership and code structure models; and open source and re-usable libraries.

What to do about it: Listen to the Infrastructure as Code Best Practices Podcast and leave your thoughts in the comments.

Open Source Updates

Terragrunt

• v0.23.3: hclfmt now supports formatting a single file as opposed to an entire directory tree. Use --terragrunt-hclfmt-file to specify the single file you wish to format.
• v0.23.4: This release upgraded the internal terraform functions to the latest version (v0.12.24). Previously unavailable functions such as trim, try, and can are now available. hclfmt has also been upgraded to handle formatting files with heredoc syntax correctly.
• v0.23.5: This release introduces an optimization for retrieving dependencies, where each dependency will be retrieved concurrently.
• v0.23.6: Fixed formatting in --help text for Terragrunt.
• v0.23.7: This introduces a new command graph-dependencies which can be used to get a graph representation of the dependency relations between all the modules.
• v0.23.8: This fixes a bug where the relative paths used for read_terragrunt_config were broken when used in a module that was pulled in with dependency blocks.
• v0.23.9: This fixes a bug where setting --terragrunt-source on a module with dependencies updated the source of both the current module and the dependency. Starting this version, when using --terragrunt-source on a module with a dependency the dependency's terraform source will be updated to the combined path using // as an anchor.
• v0.23.10: generate blocks now support an optional attribute disable_signature, which, when true, will avoid emitting the signature line at the top of the generated file.
• v0.23.11: Terragrunt will now correctly omit -var and -var-file arguments when you run destroy with a plan file.
• v0.23.12: Terragrunt will now auto retry on the error TooManyUpdates error for SSM param.
• v0.23.13: get_env can now be made to force an environment variable to be defined. When used with a single argument, the function will error if the environment variable returns empty string. This release also introduces a new helper function, get_terraform_command which can be used to see what the current executed command by terragrunt is.
• v0.23.14: The automated S3 bucket created by terragrunt with the remote_state block will now properly handle new accounts that do not have the default s3 KMS key created.
• v0.23.15: This release introduces a new function (get_terraform_cli_args) to introspect terraform args in your terragrunt config. This can be used to conditionally adjust your hooks based on what args are passed in (e.g. plan --destroy vs. plan).
• v0.23.16: This release introduces a new --terragrunt-parallelism CLI argument that you can use to limit parallelism when executing xxx-all commands (e.g. ,apply-all). This is useful if, for example, you are running apply-all on a large number of modules, and starting to hit cloud provider rate limits as a result.
• v0.23.17: Fix indentation in the --help text of Terragrunt.
• v0.23.18: This release introduces a new function sops_decrypt_file, which will decrypt an encrypted file using SOPS.

Terratest

• v0.26.1: Previously IsPodAvailable was not accounting for readiness of the containers. This could lead to returning true when the pod was not ready for accepting connections. This release updates the function to only return true if all the containers in the Pod are ready.
• v0.26.2: The EmptyS3Bucket / EmptyS3BucketE functions now also clean up delete markers.
• v0.26.3: The check made in IsMinikubeE is now more robust to different flavors of Minikube.
• v0.26.4: Flow through OutputMaxLineSize to all terraform calls.
• v0.26.5: Added docker.Inspect and docker.InspectE methods to run docker inspect and return the results in a convenient struct. Added docker.RunAndGetID and docker.RunAndGetIDE methods to run docker run and return the ID of the started container.
• v0.26.6: terratest_log_parser now buffers input lines so that it can support parsing long lines.
• v0.27.0: You no longer need to provide OutputMaxLineSize to respective command options when you have long outputs. As such the option has been removed from the respective structs.
• v0.27.1: The ContainerInspect struct returned by docker.Inspect now includes health check data.
• v0.27.2: You can now pass in additional arguments to the helm.RenderTemplate and helm.RenderTemplateE functions.
• v0.27.3: You can now customize the logger used by the functions in terratest. Any type that implements the function Logf(t testing.TestingT, format string, args ...interface{}) can be used for logging by wrapping it in the Logger type using logger.New. See the provided loggers in the logging package for more info.

cloud-nuke

• v0.1.17: cloud-nuke now supports nuking S3 buckets. Note that you can exclude certain buckets from consideration by tagging them with the tag key cloud-nuke-excluded and value true. This release also introduces a new CLI arg --exclude-resource-type for removing certain resource types from consideration. For example, to avoid nuking s3 buckets entirely, you can pass in cloud-nuke aws --exclude-resource-type s3. Use cloud-nuke aws --list-resource-types to see all the supported resource types.
• v0.1.18: You can now set the log level using the --log-level param.Also, cloud-nuke now fetches S3 bucket info concurrently, which should speed the process up by 5-10x.

kubergrunt

• v0.5.13: Starting this release, we will publish sha256 checksums with our binaries.

terraform-google-gke

• v0.4.3: Added a new optional variable to the gke-cluster module called services_secondary_range_name that allows you to specify the name of the secondary range within the subnetwork for the services to use.

terraform-google-network

• v0.3.0: This release fixes a configuration problem with the VPC NAT routing by referencing the private subnetwork instead of the public one.
• v0.4.0: This release replaces the deprecated enable_flow_logs parameter from the vpc-network module in favor of the new log_config parameter.

terraform-aws-nomad

• v0.6.0: We have switched Nomad to using systemd instead of supervisord, as systemd is now available on all major Linux distros.
• v0.6.1: Updated the example Packer template in nomad-consul.json to use clean_resource_name instead of clean_ami_name, as the latter was deprecated in Packer 1.5+.
• v0.6.2: Updated the example Packer template in nomad-consul.json to configure Docker to start at boot.
• v0.6.3: The nomad-cluster module will now respect a snapshot_id setting passed in via the ebs_block_devices input variable.

fetch

• v0.3.8: You can now specify an optional --progress flag to have fetch show progress during downloads. This is especially useful for large downloads.
• v0.3.9: This rolls back the change from v0.3.6. This means that you can now pass empty values again to --branch, --tag, and --commit, as long as one of those is non-empty, and fetch will handle it correctly.

Gruntwork Installer

• v0.0.25: Switch from local readonly to local -r, which is the proper way to declare local, read-only variables in Bash.
• v0.0.26: Fix bugs in how gruntwork-install parses, passes around, and logs --module-param arguments to ensure whitespace is handled correctly.
• v0.0.27: Minor fixes to ensure scripts pass shellcheck.

package-terraform-utilities

• v0.1.8: Added a new module called executable-dependency that can be used to install an executable if it's not installed already. This is useful if your Terraform code depends on external dependencies, such as terraform-aws-eks, which depends on kubergrunt.
• v0.2.0:enabled-aws-regions has been removed as there is now a new data source in the aws provider that has the same functionality. Replace usage of the module with the aws_regions data source

Other updates

module-data-storage

• v0.12.11: You can now configure the read replicas with separate settings from the primary. In particular, you can use the new parameter_group_name_for_read_replicas input variable to set a separate parameter group for read replicas and allow_connections_from_security_groups_to_read_replicas and allow_connections_from_cidr_blocks_to_read_replicas to configure a separate security group for read replicas.
• v0.12.12: You can now provide an existing DB subnet group to use with the RDS clusters instead of creating a new one.
• v0.12.13: You can now pass in an optional list of IAM roles to attach to the Aurora cluster using the new cluster_iam_roles input variable.

terraform-aws-eks

• v0.17.1: eks-cluster-workers now supports the create_resources parameter, which when false, will turn off all the resources in the module.
• v0.17.2: Fix bug where the cluster autoscaler in IRSA mode was unable to get the necessary IAM permissions to access the ASG.
• v0.18.0: This release removes the fargate_only flag on eks-cluster-control-plane module and replaces it with the more descriptive and accurate schedule_control_plane_services_on_fargate. Refer to the release notes for more details.
• v0.19.0: This release includes an internal implementation change for the fargate profiles to simplify the authentication mechanism when migrating the control plane services to Fargate. Note that if you were using schedule_control_plane_services_on_fargate = true, you will now need to use kubergrunt version 0.5.12 or greater.
• v0.19.1: The EKS cluster creation timeout is now 60 minutes.
• v0.19.2: The outputs for eks-k8s-cluster-autoscaler-iam-policy are now computed in a manner that is more robust to loss of credentials during an apply.
• v0.19.3: When upgrading from Kubernetes version 1.13 to 1.15, the coredns containers get updated to the latest version. In the newer versions of coredns, the configuration has a backwards incompatible change that was previously unhandled in the upgrade scripts. This release fixes that issue such that it will reformat the configuration to match expectations of later coredns versions.
• v0.19.4: Fix eks-cluster-workers to use properly use var.custom_tags_security_group to allow custom tags for SG.
• v0.19.5: You can now specify the docker repository to use for sourcing the cluster-autoscaler. Recent versions of the cluster-autoscaler is now region sensitive and you must pull from the corresponding repository. Refer to the release notes for more information.
• v0.19.6: You can now disable module calls to eks-cluster-managed-workers by setting create_resources = false. This allows you to implement conditional logic to turn on or off a module block in your terraform module.
• v0.19.7: eks-cluster-control-plane module will now automatically download and install kubergrunt if it is not available in the target system. This behavior can be disabled by setting the input variable auto_install_kubergrunt to false.

package-static-assets

• v0.6.2: Resolve source of perpetual diff when using the cloudfront default certificate.
• v0.6.3: You can now control the suffix appended to the access logs S3 bucket using the new optional input variable access_logs_bucket_suffix.

module-cache

• v0.9.2: You can now specify a custom KMS key to encrypt data at rest in redis using the new kms_key_id input variable.

package-lambda

• v0.7.5: You can now specify a permissions boundary on the IAM role created for lambda-edge.
• v0.7.6: Apply tags to all resources created by the lambda, lambda-edge, and scheduled-lambda-job modules.
• v0.7.7: Both lambda and lambda-edge now support setting reserved_concurrent_executions on the underlying Lambda function.
• v0.8.0: The lambda module is now more robust to partial failures in the module. Previously you could end up in a state where you couldn't apply or destroy the module if it only partially applied the resources due to output errors. This release addresses that by changing the output logic.

module-security

• v0.27.2: This release introduces a new module kms-master-key-multi-region, which can be used to manage KMS CMKs across all enabled regions of an account.
• v0.28.0: Account baseline modules now support managing KMS Customer Master Keys. Also, you can now specify multiple IAM roles for managing cross account access IAM groups.
• v0.28.1: Since AWS provider 2.0, setting num_days_after_which_delete_log_data = 0 no longer works and leads to a provider schema error. This meant that there was no way to configure S3 buckets to never delete data. Starting with this release, you can now prevent deletion of data in S3 for aws-config and cloudtrail by setting the respective variables to null.
• v0.28.2: This fixes our automation process so that binaries will be attached to releases.
• v0.28.3: All the parameters passed to the customer_master_keys variable of the kms-master-key module are now optional instead of required. The module will now only add IAM policy statements for the parameters that are actually set.
• v0.28.4: You can now grant Service Principals (e.g., “s3.amazonaws.com”) access to your KMS CMKs by setting the cmk_service_principals parameter and specifying the actions those Service Principals will be allowed to do via a new service_principal_actions input variable.
• v0.28.5: Fix dynamic statement creation in kms-master-key so that the optional released in v0.28.3 works properly.
• v0.28.6: The behavior of ssh-grunt has changed such that IAM users will be sync’d even if there is a duplicate user name. Previously, ssh-grunt would throw an error and stop processing other users.
• v0.28.7: This release includes updates to several modules, including ssh-grunt, cloudtrail, kms-master-key, and the *-multi-region modules. See the release notes for details.
• v0.29.0: Refactor of AWS Config to better support multi-account, multi-region configurations. AWS Config can now be centralized to a single S3 bucket in a central account, and SNS notifications can be centralized to a single SNS topic per region.
• v0.30.0: In v0.29.0, we updated account-baseline-app and account-baseline-security to allow for centralizing Config output in a single bucket. In this release, we take the same approach with account-baseline-root. It now supports using config bucket in security account. Additionally, the iam-policies module now allows sts:TagSession for the automation users.
• v0.31.0: The cloudtrail module now supports reusing an existing KMS key in your account, as opposed to creating a new one. To use an existing key, set the kms_key_already_exists variable to true and provide the ARN of the key to the variable kms_key_arn.

module-load-balancer

• v0.19.0: This release introduces support for requesting, and automatically verifying, multiple ACM certificates.
• v0.20.0: This release includes support for programmatically looking up Route53 hosted zones by name when their IDs are not supplied as input variables. It also introduces 3 new default variables to help keep your input configuration clean.

module-aws-monitoring

• v0.19.1: Fix a bug in the rds_disk_space_available alarm where it would be enabled, incorrectly, for Aurora instances.
• v0.19.2: Fix a bug in the alb-alarms module where for "low" thresholds (e.g., low request count) it was using GreaterThanThreshold instead of LessThanThreshold.
• v0.20.0: The cloudwatch-dashboard module now supports managing multiple dashboards in one module.
• v0.21.0: The install.sh scripts for the cloudwatch-log-aggregation-scripts, syslog, and cloudwatch-memory-disk-metrics-scripts modules were unnecessarily using eval to execute scripts used in the install steps. This led to unexpected behavior, such as --module-param arguments being shell expanded. We've removed the calls to eval and replaced with a straight call to the underlying scripts.

package-openvpn

• v0.9.12: Fixes a bug in the openvpn-server module which if the Elastic IP associated with the OpenVPN Server was deleted, Terraform would throw an invalid index error.

module-vpc

• v0.8.6: New module that allows to create a VPC Interface Endpoint to connect services within your VPC without needing to create NAT Gateways neither private gateway. Previously, only VPC Gateway Endpoints (S3 and DynamoDB) were permitted.

module-ci

• v0.18.5: The ECS Deploy Runner stack now supports passing in a limited selection of command arguments to the underlying terraform/terragrunt commands.
• v0.18.6: Dockerfile for infrastructure-deploy-script now includes bitbucket.org in known_hosts list. Also fix bug where v0.18.5 was incompatible with previous versions of infrastructure-deployer.
• v0.19.0: The CLI arg for setting the log level in infrastructure-deployer and infrastructure-deploy-script has been renamed to --log-level instead of --loglevel. The infrastructure-deploy-script no longer supports passing in the private SSH key via CLI args. You must pass it in with the environment variable DEPLOY_SCRIPT_SSH_PRIVATE_KEY.install-jenkins will automatically disable jenkins so that it won't start on boot. This ensures that jenkins will not be started unless it has been successfully configured with run-jenkins.
• v0.20.0: ecs-deploy-runner now supports specifying multiple container images, and choosing a container image based on a user defined name. This allows you to configure and use different Docker containers for different purposes of your infrastructure pipeline.
• v0.20.1: build-packer-artifact now supports building a packer template from a git repository. See the updated docs for more info.

module-server

• v0.8.2: The single-server module now applies the tags passed in via the tags input variable to the EIP and IAM Role resources it creates.

package-messaging

• v0.3.2: You can now add tags to the SNS topic created by the sns module.

cis-compliance-aws

• v0.4.1: aws-securityhub no longer depends on python to get enabled regions, and instead uses a terraform native data source.

module-ecs

• v0.19.1: You can now configure the platform version of ECS Fargate using the platform_version variable.

module-asg

• v0.8.7: You can now enable encryption in the server-group module for the root block device by using the root_block_device_encrypted input variable.

DevOps News

EKS now supports Kubernetes version 1.16; 1.13 is deprecated

What happened: EKS now supports Kubernetes 1.16 and Kubernetes 1.13 is now considered deprecated.

Why this matters: Kubernetes 1.16 includes support for volume resizing, Windows GMSA, and Finalizer Protection for Service LoadBalancers. Also, custom resource definitions and admission webhooks have both graduated to generally available. As for Kubernetes 1.13, since EKS only supports the 3 most recent releases, 1.13 is now officially deprecated in EKS and will no longer be supported by June 30, 2020, so if you’re still using it, make sure to update ASAP!

What to do about it: We are updating terraform-aws-eks with support for 1.16 as we speak: follow PR #160 for details. Once that PR is merged, you are safe to upgrade. If you’re still on 1.13, you should update ASAP!

EFS updates: faster read operations and ECS/Fargate integration

What happened:**** AWS has made two big improvements to its Elastic File System (EFS) Service: it now supports up to 35,000 read operations per second (a 400% increase) and ECS and Fargate tasks can now mount EFS file systems.

What to do about it: Check out the documentation for EFS limits and mounting EFS Volumes for details. Also, give our new efs module announced earlier in the newsletter a try!

Vault 1.4 has been released

What happened: HashiCorp has released Vault 1.4.

Why it matters: Some of the highlights of Vault 1.4 include:

• Integrated Storage: This feature, which allows you to run Vault without a separate storage engine (e.g., Consul), is now generally available.
• Transform Secrets Engine (Vault Enterprise only): Transform Secrets Engine performs secure data transformation for protecting secrets that reside in untrusted or semi-trusted systems outside of Vault.
• Vault Helm Chart: Run Vault Enterprise via the Helm Chart hosting on Kubernetes.

What to do about it: Try the new version out and let us know what you think!

Security Updates

Below is a list of critical security updates that may impact your services. We notify Gruntwork customers of these vulnerabilities as soon as we know of them via the Gruntwork Security Alerts mailing list. It is up to you to scan this list and decide which of these apply and what to do about them, but most of these are severe vulnerabilities, and we recommend patching them ASAP.

Rails / Ruby

• CVE-2020–8163: Potential remote code execution of user-provided local names in Rails < 5.0.1. There was a vulnerability in versions of Rails prior to 5.0.1 that would allow an attacker who controlled the locals argument of a render call. This vulnerability has been assigned the CVE identifier CVE-2020–8163. We recommend that users of Rails < 5.0.1 upgrade to a version ≥= 5.0.1. More information: https://groups.google.com/forum/#!topic/rubyonrails-security/hWuKcHyoKh0
• CVE-2020–8161 Directory traversal in Rack::Directory. There was a possible directory traversal vulnerability in the Rack::Directory app that is bundled with Rack. We recommend that users of Rack::Directory < 2.2.0 upgrade to a version ≥= 2.2.0. More information: https://groups.google.com/forum/#!msg/rubyonrails-security/IOO1vNZTzPA/Ylzi1UYLAAAJ