Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.
Hello Grunts,
In the last month, we released a collection of detailed, hands-on, step-by-step deployment guides that show you how to go to production on top of AWS and GCP, added support for fine-grained IAM Roles for Kubernetes Service Accounts support in EKS, added a new module to create and manage IAM users as code, updated all our modules to be compatible with Ubuntu 18.04, and made many other improvements and fixes. In other news, HashiCorp has released Terraform Cloud, a web UI for Terraform that’s free for teams of up to 5 users, AWS has released a number of improvements to EKS, and Python 2 will be sunset on January 1, 2020.
As always, if you have any questions or need help, email us at support@gruntwork.io!
Motivation: We’ve often said that DevOps is a bit like death by a thousand cuts. There are just so many little details to get right, across so many different disciplines: network configuration, Docker orchestration, CI, CD, monitoring, alerting, log aggregation, TLS certificates, DNS settings, secrets management, SSH access, VPN connectivity, data backup, performance testing, high availability, scalability, compliance, and so much more. The most common feeling we hear from DevOps newcomers is: “I’m overwhelmed.”
Solution: I’m excited to announce the next step in our battle against feeling overwhelmed: the Gruntwork Production Deployment Guides. This is a collection of detailed, hands-on, step-by-step guides that show you how to go to production on top of AWS and GCP. There are guides that show you how to deploy a production-grade Kuberneter cluster, set up your AWS account structure (landing zone), configure your VPCs, and many more coming soon. These are not surface-level, 5-minute “Hello, World” tutorials, but in-depth guides that walk you through everything you need to do to build, deploy, and manage a piece of infrastructure in production.
What to do about it: Check out Introducing the Gruntwork Production Deployment Guides announcement blog post for an introduction and then dive into the Production Deployment Guides themselves!
Motivation: This month, AWS announced support for fine-grained IAM Roles for Kubernetes Service Accounts. Up until now there was no sane way to assign IAM roles to individual Pods so that only those Pods had access to the IAM credentials in EKS. You either had to deploy a complex application that translates Pod annotations to IAM credentials by hijacking the metadata endpoint (such as kiam), or you had to manually generate and rotate credentials as Kubernetes Secrets injected into the container. With the new IAM Roles for Service Accounts (IRSA for short) feature, there is a EKS native way to set up and assign IAM Roles to individual Kubernetes Service Accounts.
Solution: We’ve updated our EKS module, terraform-aws-eks/eks-cluster-control-plane, to add support for setting up IRSA! Our module will automatically provision the necessary OpenID Connect Provider for IAM that EKS can use to exchange the Kubernetes Service Account credentials for IAM credentials as part of deploying the cluster.
We also created a new module, terraform-aws-eks/eks-iam-role-assume-role-policy-for-service-account, which simplifies the set up of IAM roles such that they can be assumed by the Kubernetes Service Account. For example, suppose you want to create a new IAM role application-iam-role for your application, and only want it to be assumable by the Kubernetes Service Account application-sa in the application Namespace of your EKS cluster. You can do so with the following:
module "eks_cluster" {
# args omitted for brevity
}module "assume_role_policy" {
source = "git::git@github.com:gruntwork-io/terraform-aws-eks.git//modules/eks-iam-role-assume-role-policy-for-service-account?ref=v0.8.0"
eks_openid_connect_provider_arn = module.eks_cluster.eks_iam_openid_connect_provider_arn
eks_openid_connect_provider_url = module.eks_cluster.eks_iam_openid_connect_provider_url
namespaces = []
service_accounts = [{
name = "application-sa"
namespace = "application"
}]
}
resource "aws_iam_role" "example" {
name = "application-iam-role"
assume_role_policy = module.assume_role_policy.assume_role_policy_json
}Note that IRSA is only supported on EKS clusters running Kubernetes version 1.13 and above. Be sure to upgrade your cluster to Kubernetes 1.13 before attempting to use IRSA!
Other EKS updates:
eks-cluster-control-plane module now supports upgrading Kubernetes components to the expected version for the Kubernetes version deployed on EKS. This is handled using a python script that is run everytime the kubernetes version is updated on the cluster. The deployed versions of each component follows what is described in the official upgrade guide.use_upgrade_cluster_script input variable to false.upgrade_cluster script used in the eks-cluster-control-plane module where the script incorrectly redeployed the plugins when using a region other than us-west-2, even though the versions were already up to date.What to do about it: Update to the latest terraform-aws-eks release (v0.8.2) and give the new IRSA features a try!
Motivation: Customers have been asking for a way to define and manage IAM users as code for a long time, but due to a Terraform limitation, we couldn’t build a generic module to do this. That’s because an IAM users module would need to take in a list of users and loop over them, and in Terraform, the only way to loop over resources used to be the count meta parameter. Unfortunately, [count had a significant limitation](https://blog.gruntwork.io/terraform-tips-tricks-loops-if-statements-and-gotchas-f739bbae55f9) where if you removed something from the middle of the list, it would end up deleting and recreating everything after it: e.g., if someone left your company and you deleted their IAM user from the middle of the list, Terraform would delete and recreate all the IAM users that came after that item in the list.
Solution: Terraform 0.12.6 added the ability to use for_each on resources, which does not have this limitation, so we’ve created a new [iam-users module](https://github.com/gruntwork-io/module-security/tree/master/modules/iam-users)! This module can create IAM users, add them to IAM groups, and generate a console password and access keys for the user, encrypting these secrets with the user’s PGP key (which can be fetched automatically from Keybase). Here’s an example usage:
module "iam_users" {
source = "git::git@github.com:gruntwork-io/module-security.git//modules/iam-users?ref=v0.18.3"
users = {
alice = {
groups = ["ops"]
}bob = {
groups = ["developers"]
pgp_key = "keybase:bob"
create_login_profile = true
create_access_keys = true
}cindy = {
groups = ["admin"]
pgp_key = "keybase:cindy"
create_login_profile = true
path = "/foo"
tags {
foo = "bar"
}
}
}
}output "user_arns" {
value = module.iam_users.user_arns
}output "user_access_keys_encrypted" {
value = module.iam_users.user_access_keys
}output "user_passwords_encrypted" {
value = module.iam_users.user_passwords
}When you deploy the code above, it’ll create three IAM users, alice, bob, and cindy, and output each user’s ARN, access key (encrypted with that user’s PGP key), and password (encrypted with that user’s PGP key).
What to do about it: Give the iam-users module a try in module-security, v0.18.3, and let us know what you think!
Motivation: Ubuntu 18.04, the latest LTS version was originally released in April of 2018. Since then, there has been 3 additional releases to the 18.04 line and it should now be considered stable. Up until now all the modules in the Gruntwork Infrastructure as Code Library have defaulted to version 16.04, which has been out for a while. While it is still supported (end of standard support for 16.04 is April 2021), there has been increasing demand for using Ubuntu 18.04 with our modules.
Solution: We’ve updated all our modules to test against Ubuntu 18.04! For most modules, there is no change necessary to use with Ubuntu 18.04 and you can safely bump to Ubuntu 18.04. The following modules have additional steps that are necessary:
systemd-resolved no longer works when using dnsmasq, which is not available on Ubuntu 18.04. We have released a new module that you can use as a replacement. Refer to the release notes for more info.What to do about it: Upgrade each module that you want to use with Ubuntu 18.04 to the latest release. Be sure to take a look at the release notes for information on what to do when crossing backwards incompatible releases! In general, any release that bumps the minor version (X in 0.X.Y) has backwards incompatible changes. Refer to the release notes for the .0(0.X.0) version for information on what to do to migrate to that version.
imagePullSecrets on the k8s-service chart to specify which Secret to use when pulling down container images. This is useful when authenticating the container engine to a private registry. You can now directly append to the Ingress paths, using the additionalPaths and additionalPathsHigherPriority input values. See the README for more information.google.managedCertificate to deploy a Google Managed SSL Certificate.containerCommand input value. Also, the google managed certificate name is now configurable, using the google.managedCertificate.name input value.Ingress tls configuration issue that prevented chart from deploying. Security Context is now configurable, using the securityContext input map.k8s-service chart now supports injecting the projected Service Account token and environment variables into the Pod for the IAM Roles for Service Account feature in AWS EKS.alternative_default_service_account which can be used to specify a Service Account that should be used by the default node pool during cluster creation, that is alternative to the default Service Account. This should be used if the default Service Account is deleted from the Project.service_account_roles on the gke-service-account module, which can be used to add extra roles to the service account.http-helper methods to add HTTP header(s), set a HTTP method and return a response body. Also fixed broken builds (disabled Azure tests).GetVpcById and GetVpcByIdE, which allows you to look up a VPC by ID, and GetVpcsE, a general purpose helper function to look up VPCs using provided filters.HTTPDo, HTTPDoE, HTTPDoWithRetry, HTTPDoWithRetryE, HTTPDoWithValidationRetry, and HTTPDoWithValidationRetryE functions to use []byte for the request body instead of io.Reader. This change fixes a bug in the retry flavors of the functions that allow for reusing the body multiple times when making the HTTP request.cloud-nuke defaults-aws , which can be used to remove the default VPC and default security groups from existing VPCs.hclfmt now runs before anything is decoded in the terragrunt config. This avoids interpolating blocks that can’t be interpolated until the module is applied (e.g dependency blocks).dependency blocks, specifically checking for cyclic dependencies and unapplied dependencies. Additionally, this release implements mock_outputs for dependency blocks, which can be used to run validate on a module before the dependencies are applied. See the updated README for more information.dependency block config_path that were relative were not resolved from the right directory when detecting cycles in the dependency graph, causing Terragrunt to incorrectly crash.tag public-restricted for restricting inbound traffic from public internet.--s3-bucket-path option for the run-vault script. It is useful when you configure S3 as a storage backend and want to store vault data at a specific path in the S3 bucket.kubergrunt eks oidc-thumbprint which can be used to compute the root CA thumbprint for a given OpenID Connect Issuer. This thumbprint is necessary when setting up an OpenID Connect Provider for use with the IAM Roles for Service Accounts feature provided by EKS.eks deploy command will now cordon all the old nodes before starting to drain them so that the evicted pods get scheduled on the new nodes in the cluster.run-pex-as-resource and run-pex-as-data-source modules now exposes a variable (enabled) that can be used to conditionally decide whether or not to execute the pex resource. This is helpful when you want to support disabling script execution in your modules.cluster_resource_id output to the Aurora module.custom-iam-group to create an IAM group and attach a set of policies, optionally with MFA required.read_only policy in iam-policies module.install-open-jdk now supports installing OpenJDK 11 for CentOS and Ubuntu 18.04. See the compatibility information for more info.ecs-fargate module has been updated to only enable ecs deployment check when desired tasks > 0. This allows you to set desired_tasks to 0 to scale down your service. Additionally, the ecs-fargate module has been updated to add task definition ARN as an output, under the name aws_ecs_task_definition_arn.ecs-service-with-alb module from using template_file data sources to localvariables for intermediate variables. This fixes an issue where terraform plan was incorrectly reporting ECS services being recreated.run-ecs-task script where it was not forcing the aws CLI output to be JSON, so the script would fail if a user had overridden the default on their systems to have text output.alb module calculates the ALB ARN to use locals instead of a template_file. This fixes an issue where terraform plan was incorrectly reporting load balancers being recreated.nlb module has been deprecated and removed. When #61 was fixed, the nlb module reduced to being a thin wrapper over the aws_lb resource and thus it no longer made sense to maintain the module. Instead, users of the module should update to using the aws_lb resource directly. Refer to the provided migration guide for information on how to replace your usage of the nlb module with the aws_lb resource, including migrating the state to avoid downtime.vpc-mgmt module now adds a Name tag to its NAT Gateway(s) and allows you to specify custom tags via the optional nat_gateway_custom_tags input variable.rds module now supports storage auto scaling by allowing you to set a new optional input variable called max_allocated_storage.s3-cloudfront module now supports specifying multiple origin groups, which allows you to specify one or more S3 buckets to use as failovers in case the primary one fails. You can specify the failover buckets using the new input variables failover_buckets and failover_bucket_website_endpoints.install-openvpn script was ignoring all command-line args passed to it! Add documentation on how to connect to multiple VPN servers at the same time.What happened: HashiCorp has announced the release of Terraform Cloud, a new web UI for Terraform.
Why it matters: Before Terraform Cloud, if you wanted to run Terraform from a web UI, your only options were Terraform Enterprise, which had a very high start price point that was only accessible to massive organizations, or Atlantis, which was an open source tool that was limited to executing a few commands via comments on GitHub PRs. Now you can give Terraform Cloud a shot, which is free for teams up to 5 users, and includes a number of features, such as remote plans & applies, VCS integration and webhooks, state management, and a provide module registry.
What to do about it: We’ll be trying out Terraform Cloud the next few months to see if it would be a good fit within our tech stack. In the meantime, check out the announcement blog post for the details, give Terraform Cloud a try, and let us know what you think!
What happened: HashiCorp has announced a new cost estimation feature for Terraform Cloud and Terraform Enterprise.
Why it matters: The Terraform Cloud and Terraform Enterprise UIs will now give you an estimate of the cost impact of your Terraform code changes: e.g., if you increase the number of servers in your cluster from X to Y, the monthly cost will go up by ~$Z. This allows you to see roughly how much a Terraform module will cost you before you deploy it, and even enforce rules and policies around pricing using HashiCorp Sentinel (which is only available in Terraform Enterprise).
What to do about it: Check out the announcement blog post for details.
What happened: AWS has announced a number of improvements to EKS, including support for Kubernetes 1.14, fine-grained IAM roles for service accounts, cluster tagging, and support for the EBS CSI Driver.
Why it matters:
What to do about it: Try out the new features above and let us know what you think!
What happened: January 1, 2020, has been set as the official day that the Python maintainers will sunset Python 2.
Why it matters: After January 1, 2020, there will be no more improvements to Python 2—not even security fixes!
What to do about it: If you are still using Python 2, make sure to upgrade to Python 3 ASAP!
What happened: Amazon has announced that using Lambda functions with VPCs and ENIs will now be dramatically faster.
Why it matters: In the past, if you wanted to run your Lambda function within a VPC—e.g., so the function could access private resources within that VPC, such as a database—you would be faced with a massive cold-start overhead that could be as high as 20 seconds! With this update, there will be virtually no additional overhead for running Lambda functions in a VPC (other than a one-time setup per AWS account).
What to do about it: There’s nothing for you to do! This change will gradually roll out to all regions, allowing you to take advantage of much better Lambda function performance in VPCs.
Below is a list of critical security updates that may impact your services. We notify Gruntwork customers of these vulnerabilities as soon as we know of them via the Gruntwork Security Alerts mailing list. It is up to you to scan this list and decide which of these apply and what to do about them, but most of these are severe vulnerabilities, and we recommend patching them ASAP.
apache2, some of which could allow an attacker to expose sensitive information. If you’re using Apache, we strongly recommend updating to the latest version.