Update Nov 10, 2021: Updated to recommend [tgswitch(https://warrensbox.github.io/tgswitch/)](https://warrensbox.github.io/tgswitch/) instead of tgenv. Also added a section covering other tools in this domain.
Terragrunt and Terraform are relatively young projects in the DevOps ecosystem. As such, both projects introduce backwards incompatible changes more often than we like. For example, Terraform state is incompatible even at the patch version level (the Y in the semantic version scheme of 0.X.Y) to the extent that you can't read state across different patch versions. In fact, Terraform has features to auto update the state representation (e.g running terraform apply with 0.12.17 can update a state file created with Terraform 0.12.14), but this is a one way change: you can't downgrade to a lower version state file!
This means that it is important to try to restrict your team to use specific versions of Terraform and Terragrunt for a given module, to avoid forcing the entire team to upgrade at the whim of someone downloading the latest version of Terraform.
We previously shared how to manage multiple versions of Terraform using homebrew, but that focused on managing multiple versions on an individual level.
In this post, I want to share several tools and features in the ecosystem that help you manage a multiversion IaC project at the team level.
Here is what this post will cover:
tgswitch and tfenv to install multiple versions of Terragrunt and Terraform.terragrunt-version and .terraform-version to automatically switch based on folderThe first feature I would like to share is a native feature of Terraform that allows you to lock down the compatible versions supported by a given module. In your module code, you can add a terraform block with a required_version property set in it to restrict compatible terraform versions.
For example, you can add the following block to restrict deployments of the module to only allow Terraform version 0.12.17:
terraform {
# NOTE: The second `=` in the string is intentional, and it means
# exactly version 0.12.17. In other words, we are setting the
# required_version property of the terraform block to "= 0.12.17".
required_version = "= 0.12.17"
}This causes Terraform to halt with an error if it detects you are using an incompatible Terraform version, preventing you from deploying using a different version. By adding this block to the top level modules that you are directly deploying, you can ensure your team only deploys those modules using the specified Terraform version.
You can read more about this feature in the official docs.
This kind of version locking in your top level modules is most useful when you have dependencies that are strung together with terraform_remote_state data sources, which are sensitive to the Terraform version.
One problem with this level of version locking is that when you are upgrading Terraform versions, you won’t be able to apply your older environments. For example, you might want to test the version change in a dev environment, forcing you to use a newer version of Terraform that is not compatible with older versions of the module in your other environments (such as prod, which may not be updated until dev is updated). In this situation, you will lose the ability to manage your prod environments unless you install and use multiple versions of terraform.
In the next section, I will introduce tools in the ecosystem that simplify working with multiple versions of terraform.
Previously we talked about using homebrew to manage multiple versions of Terraform. This works great on a Mac, but is not compatible with users of Terraform who are on other Unix based machines like Linux or BSD. In this post I’d like to introduce two tools in the Terraform/Terragrunt ecosystem that work with any Unix machine: tfenv (https://github.com/tfutils/tfenv) and tgswitch (https://github.com/warrensbox/tgswitch) respectively.
tfenv was inspired by a similar tool rbenv which can be used to manage Ruby versions. These are a set of bash scripts that provide a workflow for managing and using multiple versions of the supported tool.
tgswitch is a tool written in golang that offers similar features for versioning terragrunt. It offers the same features as tfenv, including managing the versions to use in a version file.
When you install one of these tools, you can install and switch between versions of the binaries with ease. For example, to install a specific version (0.12.17) of terraform using tfenv, you would do:
tfenv install 0.12.17Then, you can switch between versions using the tfenv use command:
tfenv use 0.12.17
# terraform is now pointing to the 0.12.17 version
terraform version# Switch to 0.12.14
tfenv use 0.12.14
terraform versionThe way these tools work is as follows:
.XXenv, where XX is for whatever tool you are managing, like rb for rbenv, and tf for tfenv) that acts as the working directory for the tool. Note that for tgswitch, the working directory is slightly different (.terragrunt.versions), but it works more or less the same way.tfenv will install different versions of terraform under .tfenv/versions/VERSION_NUMBER..XXenv/bin. The selection of the version is resolved in the following order:XXENV_VERSION (for tfenv, this is TFENV_VERSION and for rbenv, this is RBENV_VERSION ).TOOL-version. TOOL depends on the tool. For example, tgswitch uses .terragrunt-version while tfenv uses .terraform-version. Note that typically each XXenv tool will support walking up the directory tree..XXenv/version.Using tgswitch and tfenv greatly simplify the process of installing and switching between multiple versions, but knowing what version to switch to is still a manual process. For example, you might have a require_version setting on the module and you might be currently using an incompatible version of Terraform. You will only find this out after running terraform apply and seeing the error.
In the next section, I will walk through how you can use the version files to automatically switch versions depending on which folder you are in.
.terragrunt-version and .terraform-version to automatically switch based on folderAs mentioned in the previous section, tgswitch and tfenv support switching versions automatically based on a version file (.terragrunt-version for tgswitch and .terraform-version for tfenv). You can use these to your advantage to automatically use the right version of the binaries for your project.
Suppose you had the following typical Terragrunt folder structure:
infrastructure-live
├── non-prod
│ └── us-east-1
│ ├── qa
│ │ ├── mysql
│ │ └── webserver-cluster
│ └── stage
│ ├── mysql
│ └── webserver-cluster
└── prod
└── us-east-1
└── prod
├── mysql
└── webserver-clusterinfrastructure-modules
├── asg-elb-service
└── mysqlIn this example, let’s assume that we had mixed terraform version requirements such that in non-prod, we are using a version of the infrastructure-modules that have been updated to Terraform 0.12.17, while in prod we are still using Terraform 0.11.14. In turn, we are still using Terragrunt 0.18.7 in prod, but 0.21.0 in non-prod to support Terraform 12.
We can implicitly enforce these version constraints provided that everyone in your team is using tgswitch and tfenv by creating the version files. In this model, you will want to create the two version files (.terragrunt-version and .terraform-version) at the root of the environment folders in infrastructure-live like so:
infrastructure-live
├── non-prod
│ ├── .terragrunt-version
│ ├── .terraform-version
│ └── us-east-1
│ ├── qa
│ │ ├── mysql
│ │ └── webserver-cluster
│ └── stage
│ ├── mysql
│ └── webserver-cluster
└── prod
├── .terragrunt-version
├── .terraform-version
└── us-east-1
└── prod
├── mysql
└── webserver-clusterThe contents of the version file will be the version for that environment:
# infrastructure-live/non-prod/.terragrunt-version
0.21.0# infrastructure-live/non-prod/.terraform-version
0.12.17# infrastructure-live/prod/.terragrunt-version
0.18.7# infrastructure-live/prod/.terraform-version
0.11.14By adding these version files, users of tgswitch and tfenv will automatically use the right version of terragrunt and terraform when they call out to those utilities.
As in, when you run terragrunt apply in infrastructure-live/non-prod/us-east-1/qa/mysql, tgswitch will automatically select 0.21.0 of terragrunt (because it will find .terragrunt-version in the directory tree) and tfenv will automatically select 0.12.17 of terraform (because it will find .terraform-version in the tree), regardless of what the user has currently selected as the default version using the use command.
Similarly, when you run terragrunt apply in infrastructure-live/prod/us-east-1/qa/mysql, tgswitch will automatically select 0.18.7 of terragrunt and tfenv will automatically select 0.11.14 of terraform.
Note that tgswitch and tfenv are not the only tools that offer this functionality. Here I will list a few other tools that are in this category of version management. You may want to use a different tool depending on your style or environment (e.g., tfenv, being a bash based tool, can’t be used in a Windows environment and thus you might want to use tfswitch instead).
[tfswitch (https://tfswitch.warrensbox.com/)](https://tfswitch.warrensbox.com/): works the same way as tgswitch, but manages terraform instead.[asdf-vm (https://asdf-vm.com/)](https://asdf-vm.com/), specifically the hashicorp plugin and the terragrunt plugin: plugin oriented tool version manager. Supports a wide range of tools and languages under a single interface.[tgenv (https://github.com/cunymatthieu/tgenv)](https://github.com/cunymatthieu/tgenv): works the same way as tfenv, but manages terragrunt instead. Originally recommended in this article, but it hasn’t had updates since the original publishing date.To tie it all together:
required_version on the terraform block to restrict modules to only be deployable from specific Terraform versions to ensure state compatibility.tgswitch and tfenv to make it easier to work with multiple versions of terragrunt and terraform respectively..terragrunt-version and .terraform-version) in your projects to automatically switch versions depending on which module you are deploying.Your entire infrastructure. Defined as code. In about a day. Gruntwork.io.
