Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.
Hello Grunts,
Happy holidays! In the last month, we launched a Gruntwork Community Forum, created a Gruntwork CLI tool, and made important changes and fixes in OpenVPN, MongoDB, Terragrunt, and our code formatting. There were also some very exciting announcements from AWS (including managed Kubernetes, Aurora multi-master and serverless, managed GraphQL, and much more) and a new version of Terraform came out that is backwards incompatible. Read on to learn more!
As always, if you have any questions or need help, email us at support@gruntwork.io!
Motivation: All of our customers are using the same infrastructure, so we see the same questions come up time and again, but did not have a way to expose those repeated discussions to everyone.
Solution: We’ve launched the Gruntwork Community Forum as a place our customers can use for troubleshooting, design discussions, new infrastructure requests, and to discuss the Infrastructure as Code Library, Reference Architecture, Terraform, AWS, Docker, Packer, DevOps, etc.
What to do about it: Please head over to https://community.gruntwork.io/ and try it out for all your future questions! We will gently nudge any support question that looks like it could affect multiple customers towards the forum, but don’t worry, Gruntwork employees will be monitoring the forum regularly, and will respond with the same SLA as as always; it’s just that now, you’ll also be able to find answers from the community and via Google search! And, of course, you can still email us at support@gruntwork.io at any time!
Motivation: Many of our customers had to perform similar actions manually, such as creating the initial AWS accounts and Git repos for the Reference Architecture or granting Gruntwork employees access to your accounts for troubleshooting. These manual actions were tedious and error prone.
Solution: We’ve created a new cross-platform CLI tool to automate these actions. It’s called, not very creatively, gruntwork! It has commands built-in to create the AWS accounts and Git repos for the Reference Architecture, which is useful for new customers, as well as commands to grant Gruntwork employees access to your AWS accounts and Git repos, which is useful for all customers, especially when troubleshooting. Instead of creating an IAM user for each Gruntwork employee (which means juggling lots of usernames, passwords, API keys, etc), you can now give Gruntwork’s AWS account access to your AWS account, and we’ll be able to “switch” into your account with our existing credentials. For example, if you were troubleshooting an issue in your “prod” AWS account, a user in your “master” account could grant us read-only access with a single command:
gruntwork aws grant --managed-policy ReadOnlyAccess --account prodWhen we were done troubleshooting, you could revoke all access just as easily:
gruntwork aws revoke --account prodWhat to do about it:**** Try out the gruntwork CLI next time you need to give us access! Also, if Gruntwork employees already have IAM users in your AWS accounts, we recommend removing those IAM users, and switching to this model, as it’s more secure and easier to use.
parameter_group bugMotivation: A strange bug showed up recently where the parameter_group configuration used in ElastiCache and RDS would always show a diff when you ran plan. Worse yet, if you ran apply, you’d sometimes get an error trying to apply that phantom diff.
Solution: This appears to be a bug in Terraform (terraform-providers/terraform-provider-aws#2468), so for now, we’ve put in a workaround in both module-cache and module-data-storage where we set the default parameter_group value explicitly so there is no phantom diff.
What to do about it: Update to module-cache, v0.3.1 and module-data-storage, v0.4.0.
Motivation: Creating a single module for MongoDB that worked well in both dev and prod was tricky, as you’d want to disable some clusters (e.g., backup) in dev, but not prod, and there’s no way to conditionally disable a module in Terraform. Also, the install-mongodb module broke due to a strange error with pip.
Solution: We’ve released a couple new versions of package-mongodb: v0.2.4 removes an earlierpip workaround, which should fix the install-mongodb module; v0.2.5 allows you to set the cluster_size parameter of mongodb-cluster to 0 to disable that module from creating any resources.
What to do about it: Upgrade to package-mongodb, v0.2.5 to take advantage of the new features/fixes.
Motivation: We discovered an issue where the Certificate Revocation List (CRL) in OpenVPN would expire after 30 days of not performing any revocations. This condition would prevent users from being able to connect to the VPN.
Solution: We’ve released version v0.5.0 of package-openvpn with a fix that sets the default CRL expiration date to 10 years in the future. We also added a new backup-openvpn-pki module and corresponding cron job to regularly backup the entire PKI infrastructure to the S3 bucket.
What to do about it: Upgrade to package-openvpn, v0.5.0 to take advantage of the new features/fixes.
Motivation: We were not formatting our Terraform code in a consistent manner, which made it harder to read.
Solution: We are updating each of our repos to (a) use a pre-commit hook (managed using pre-commit) to automatically run terraform fmt on the code and (b) fail the build if the code isn’t formatted. Best of all, this new functionality is available as part of the new precommit-hooks module, so you can enforce the same formatting on your own code!
What to do about it: Check out the precommit-hooks module for instructions.
Motivation: We want to spend time with our families during the holidays.
Solution: The Gruntwork team will be on vacation December 25 — January 5. We will not be responding to support tickets or Slack inquiries during this time. Please plan accordingly.
What to do about it: We hope you’re able to take some time off and enjoy the holidays too!
Motivation: Prod is down and you need to let your team know why.
Solution: Why Prod is Down.
What to do about it: Share it with your team!
We built Terragrunt to try to fill in gaps and weaknesses in Terraform. It’s a popular open source project, so this section will list the updates that happen each month.
CTRL+C.xxx-all commands (e.g., apply-all).TF_INPUT environment variable to set Terragrunt to non-interactive mode. This is the same environment variable used by Terraform.getAWSAccountId() helper now assumes the IAM role specified via --terragrunt-iam-role, if any.full-access IAM group in the iam-groups module via the should_create_iam_group_full_access parameter. The iam-groups module now also outputs the ARNs and names of the ssh-iam groups.ssh-iam will now do a better job of reporting error messages if it reads an OS user that is missing the "comment field", which ssh-iam uses for storing the IAM user name.https_listener_ports_and_ssl_certs and https_listener_ports_and_acm_ssl_certs input variables on the alb module from maps to lists of maps. The problem with using maps is that in Go—which Terraform uses under the hood—the iteration order for maps is (intentionally) randomized, so with multiple ports and certs in these variables, you would get spurious diffs in the plan command as Terraform would swap their order and assume listeners had to created/destroyed. By using lists, we can make the sort order consistent.source_dest_check param on the single-server module.What happened: Terraform 0.11 has been released. The major changes are:
apply command now automatically runs plan and asks you to interactively approve the plan.TYPE.NAME.ATTR instead of TYPE.NAME.*.ATTR) on a resource with count, you will now get a runtime error if that count parameter is set to anything other than 1.Items (3) and (4) are backwards incompatible (see the upgrade guide)!
Why it matters: If you tried to upgrade to Terraform 0.11 right now, you’d run into two issues:
apply command is now interactive, the automated deployment scripts in your CI servers will now be broken. You’ll have to add the -auto-approve flag to the apply command to make it non-interactive.count parameter, so many of the modules in your codebase and in the gruntwork-io repos do not work with Terraform 0.11! Worse yet, the errors only manifest at runtime and are hard to track down. We filed a bug about this and fortunately, HashiCorp released Terraform 0.11.1, which hasthe ability to issue a warning for all potentially invalid usages of the singleton syntax. This will make it possible for us to find all these errors and fix them.What to do about it: We do NOT recommend upgrading to Terraform 0.11 at this time. We’ll start updating all of our modules next week to be compatible with 0.11, but it will take some time. If you absolutely must use 0.11 in the meantime, as a workaround, you can install Terraform 0.11.1 and set the TF_WARN_OUTPUT_ERRORS environment variable to 1. This will force Terraform to fall back to the old behavior with count (see 0.11.1 release notes for details).
AWS had its annual re:Invent conference and announced a ton of new services, features, and improvements.
There are way more announcements than we can cover in this newsletter, so we’ll write a full blog post about this next week. In the meantime, here are a few of the highlights:
Below is a list of security updates that may impact your services. It is up to you to scan this list and decide which of these apply and what to do about them. Note that for critical security updates, we send out emails immediately to the Gruntwork Security Alerts list.
go get a malicious repository.Got questions or feedback on the newsletter? Email us at support@gruntwork.io. Want to learn more about Gruntwork? Check out https://www.gruntwork.io/.