A comprehensive guide to managing secrets in your Terraform code

One of the most common questions we get about using Terraform to manage infrastructure as code is how to handle secrets such as passwords, API keys, and other sensitive data. For example, here’s a snippet of Terraform code that can be used to deploy MySQL using Amazon RDS:

resource "aws_db_instance" "example" {
engine               = "mysql"
engine_version       = "5.7"
instance_class       = "db.t2.micro"
name                 = "example"
# How should you manage the credentials for the master user?
username = "???"
password = "???"
}

Notice how Terraform requires you to set two secrets, username and password, which are the credentials for the master user of the database. In this blog post, I’ll go over the most common techniques you can use to safely and securely manage such secrets:

  1. Pre-requisite #1: Don’t Store Secrets in Plain Text
  2. Pre-requisite #2: Keep Your Terraform State Secure
  3. Technique #1: Environment Variables
  4. Technique #2: Encrypted Files (e.g., KMS, PGP, SOPS)
  5. Technique #3: Secret Stores (e.g., Vault, AWS Secrets manager)

Pre-requisite #1: Don’t Store Secrets in Plain Text

The first rule of secrets management is:

Do not store secrets in plain text.

The second rule of secrets management is:

DO NOT STORE SECRETS IN PLAIN TEXT.

Seriously, don’t do it. For example, do NOT hard-code your database credentials directly in your Terraform code and check it into version control:

resource "aws_db_instance" "example" {
engine               = "mysql"
engine_version       = "5.7"
instance_class       = "db.t2.micro"
name                 = "example"
# DO NOT DO THIS!!!
username = "admin"
password = "password"
# DO NOT DO THIS!!!
}

Storing secrets in plain text in version control is a BAD IDEA. Here are just a few of the reasons why:

  1. Anyone who has access to the version control system has access to that secret. In the example above, every single developer at your company has access to the master credentials for your database.
  2. Every computer that has access to the version control system keeps a copy of that secret. Every single computer that has ever checked out that repo may still have a copy of that secret on its local hard drive. That includes the computer of every developer on your team, every computer involved in CI (e.g., Jenkins, CircleCi, GitLab, etc.), every computer involved in version control (e.g., GitHub, GitLab, BitBucket), every computer involved in deployment (e.g., all your pre-prod and prod environments), every computer involved in backup (e.g., CrashPlan, Time Machine, etc.), and so on.
  3. Every piece of software you run has access to that secret. Because the secrets are sitting in plain text on so many hard-drives, every single piece of software running on any of those computers can potentially read that secret.
  4. No way to audit or revoke access to that secret. When secrets are sitting on hundreds of hard drives in plain text, you have no way to know who accessed them (there’s no audit log) and no way to revoke access.

In short, if you store secrets in plain text, you are giving malicious actors (e.g., hackers, competitors, disgruntled former employees) countless ways to access your company’s most sensitive data—e.g., by compromising the version control system, or any of the computers you use, or any piece of software on any of those computers, etc—and you’ll have no idea if you were compromised or have any easy way to fix things if you were.

Therefore, I strongly recommend that you always store secrets in an encrypted format—and this applies to all secrets, and not just those used with Terraform! Later in this post, I’ll discuss several different techniques for encrypting and decrypting such secrets.

Pre-requisite #2: Keep Your Terraform State Secure

Hopefully, the previous section has convinced you to not store your secrets in plain text, and the subsequent sections will show you some techniques for encrypting your secrets. However, no matter which technique you use to encrypt the secrets on your end, there is still one place where they will end up in plain text: Terraform state.

Every time you deploy infrastructure with Terraform, it stores lots of data about that infrastructure, including all the parameters you passed in, in a state file. By default, this is a terraform.tfstate file that is automatically generated in the folder where you ran terraform apply. So even if you use one of the techniques mentioned later to safely pass in your secrets, such as the credentials for a database:

resource "aws_db_instance" "example" {
engine               = "mysql"
engine_version       = "5.7"
instance_class       = "db.t2.micro"
name                 = "example"
# Let's assume you found safe way to pass these in
username = 
password = 
}

These secrets will still end up in terraform.tfstate in plain text! This has been an open issue for more than 6 years now, with no clear plans for a first-class solution. There are some workarounds out there that can scrub secrets from your state files, but these are brittle and likely to break with each new Terraform release, so I don’t recommend them.

For the time being, no matter which of the techniques discussed below you end up using to manage secrets, you must also:

  1. Store Terraform state in a backend that supports encryption. Instead of storing your state in a local terraform.tfstate file, Terraform natively supports a variety of backends, such as S3, GCS, and Azure Blob Storage. Many of these backends support encryption, so that instead of your state files being in plain text, they will always be encrypted, both in transit (e.g., via TLS) and on disk (e.g., via AES-256). Most backends also support collaboration features (e.g., automatically pushing and pulling state; locking), so using a backend is a must-have both from a security and teamwork perspective. See How to Manage Terraform State for more info.
  2. Strictly control who can access your Terraform backend. Since Terraform state files may contain secrets, you’ll want to carefully control who has access to the backend you’re using to store your state files. For example, if you’re using S3 as a backend, you’ll want to configure an IAM policy that solely grants access to the S3 bucket for production to a small handful of trusted devs (or perhaps solely just the CI server you use to deploy to prod).

And now, without further ado, let’s discuss the various techniques available to you for managing secrets with Terraform.

Technique #1: Environment Variables

This first technique keeps plain text secrets out of your code by taking advantage of Terraform’s native support for reading environment variables.

Overview

To use this technique, declare variables for the secrets you wish to pass in:

variable "username" {
description = "The username for the DB master user"
type        = string
}
variable "password" {
description = "The password for the DB master user"
type        = string
}

Update, December 3, 2020: Terraform 0.14 has added the ability to mark variables as sensitive, which helps keep them out of your logs, so you should add sensitive = true to both variables above!

Next, pass the variables to the Terraform resources that need those secrets:

resource "aws_db_instance" "example" {
engine               = "mysql"
engine_version       = "5.7"
instance_class       = "db.t2.micro"
name                 = "example"
# Set the secrets from variables
username             = var.username
password             = var.password
}

You can now pass in a value for each variable foo by setting an environment variable called TF_VAR_foo. For example, here’s how you could set username and password via environment on Linux, Unix, or Mac, and run terraform apply to deploy the database:

# Set secrets via environment variables
export TF_VAR_username=(the username)
export TF_VAR_password=(the password)
# When you run Terraform, it'll pick up the secrets automatically
terraform apply

(Pro tip: if you have the HISTCONTROL environment variable set correctly in a Bash terminal, then any command with a leading space will not be stored in Bash history. Use this when setting environment variables with secrets to avoid having those secrets stored on disk.)

This technique helps you avoid storing secrets in plain text in your code, but it leaves the question of how to actually securely store and manage the secrets unanswered. So in a sense, this technique just kicks the can down the road, whereas the other techniques described later in this blog post are more prescriptive.

That said, so as not to leave you entirely hanging, if you do go with environment variables, the most common solution for storing and managing secrets is to use a password manager such as:

These tools solve the “kick the can down the road” problem by relying on human memory: that is, your ability to memorize a password that gives you access to the password manager.

An example using pass

Let’s go through a quick example using pass. First, you’ll need to store your secrets by using the pass insert command:

$ pass insert db_username
Enter password for db_username: admin
$ pass insert db_password
Enter password for db_password: password

You can read a secret out to stdout by running pass <secret>:

$ pass db_username
admin

You can use this functionality in a subshell to set your secrets as environment variables and then call terraform apply:

# Read secrets from pass and set as environment variables
export TF_VAR_username=$(pass db_username)
export TF_VAR_password=$(pass db_password)
# When you run Terraform, it'll pick up the secrets automatically
terraform apply

Advantages of this technique

Drawbacks to this technique

Technique #2: Encrypted Files (e.g., KMS, PGP, SOPS)

The second technique relies on encrypting the secrets, storing the cipher text in a file, and checking that file into version control.

Overview

To encrypt some data, such as some secrets in a file, you need an encryption key. This key is itself a secret! This creates a bit of a conundrum: how do you securely store that key? You can’t check the key into version control as plain text, as then there’s no point of encrypting anything with it. You could encrypt the key with another key, but then you then have to figure out where to store that second key. So you’re back to the “kick the can down the road problem,” as you still have to find a secure way to store your encryption key.

The most common solution to this conundrum is to store the key in a key service provided by your cloud provider, such as:

These key services solve the “kick the can down the road” problem by relying on human memory: in this case, your ability to memorize a password that gives you access to your cloud provider (or perhaps you store that password in a password manager and memorize the password to that instead).

An example using AWS KMS

Here’s an example of how you can use a key managed by AWS KMS to encrypt secrets. First, create a file called db-creds.yml with your secrets:

username: admin
password: password

Note: do NOT check this file into version control!

Next, encrypt this file by using the aws kms encrypt command and writing the resulting cipher text to db-creds.yml.encrypted:

aws kms encrypt \
--key-id  \
--region  \
--plaintext fileb://db-creds.yml \
--output text \
--query CiphertextBlob \
> db-creds.yml.encrypted

You can now safely check db-creds.yml.encrypted into version control.

To decrypt the secrets from this file in your Terraform code, you can use the aws_kms_secrets data source (for GCP KMS or Azure Key Vault, you’d instead use the google_kms_secret or azurerm_key_vault_secret data sources, respectively):

data "aws_kms_secrets" "creds" {
secret {
name    = "db"
payload = file("${path.module}/db-creds.yml.encrypted")
}
}

The code above will read db-creds.yml.encrypted from disk and, assuming you have permissions to access the corresponding key in KMS, decrypt the contents to get back the original YAML. You can parse the YAML as follows:

locals {
db_creds = yamldecode(data.aws_kms_secrets.creds.plaintext["db"])
}

And now you can read the username and password from that YAML and pass them to the aws_db_instance resource:

resource "aws_db_instance" "example" {
engine               = "mysql"
engine_version       = "5.7"
instance_class       = "db.t2.micro"
name                 = "example"
# Set the secrets from the encrypted file
username = local.db_creds.username
password = local.db_creds.password
}

One gotcha with this approach is that working with encrypted files is awkward. To make a change, you have to locally decrypt the file with a long aws kms decrypt command, make some edits, re-encrypt the file with another long aws kms encrypt command, and the whole time, be extremely careful to not accidentally check the plain text data into version control or leave it sitting behind forever on your computer. This is a tedious and error prone process—unless you use a tool like sops.

An example using AWS KMS with sops and Terragrunt

sops is an open source tool designed to make it easier to edit and work with files that are encrypted via AWS KMS, GCP KMS, Azure Key Vault, or PGP. sops can automatically decrypt a file when you open it in your text editor, so you can edit the file in plain text, and when you go to save those files, it automatically encrypts the contents again. This removes the need to run long aws kms commands to encrypt or decrypt data or worry about accidentally checking plain text secrets into version control. Here’s a .gif that shows sops in action:

undefined

Terraform does not yet have native support for decrypting files in the format used by sops. One solution is to install and use the custom provider for sops, terraform-provider-sops. Another option, which I’ll demonstrate here, is to use Terragrunt, which has native sops support built in. Terragrunt is a thin wrapper for Terraform that helps you keep your Terraform code DRY and maintainable (check out the Quick Start guide for an overview).

Let’s say you used sops to create an encrypted YAML file called db-creds.yml, as shown in the .gif above. Now, in your terragrunt.hcl config, you can use the sops_decrypt_file function built into Terragrunt to decrypt that file and yamldecode to parse it as YAML:

locals {
db_creds = yamldecode(sops_decrypt_file(("db-creds.yml")))
}

Next, you can pass username and password as inputs to your Terraform code:

inputs = {
username = local.db_creds.username
password = local.db_creds.password
}

Your Terraform code, in turn, can read these inputs via variables:

variable "username" {
description = "The username for the DB master user"
type        = string
}
variable "password" {
description = "The password for the DB master user"
type        = string
}

Update, December 3, 2020: Terraform 0.14 has added the ability to mark variables as sensitive, which helps keep them out of your logs, so you should add sensitive = true to both variables above!

And pass those variables through to aws_db_instance:

resource "aws_db_instance" "example" {
engine               = "mysql"
engine_version       = "5.7"
instance_class       = "db.t2.micro"
name                 = "example"
# Set the secrets from variables
username = var.username
password = var.password
}

Advantages of this technique

Drawbacks to this technique

Technique #3: Secret Stores (e.g., Vault, AWS Secrets Manager)

The third technique relies on storing your secrets in a dedicated secret store: that is, a database that is designed specifically for securely storing sensitive data and tightly controlling access to it.

Overview

Here a few of the more popular secret stores you can consider:

  1. HashiCorp Vault: Open source, cross-platform secret store.
  2. AWS Secrets Manager: AWS-managed secret store.
  3. AWS Param Store: AWS-managed data store that supports encryption.
  4. GCP Secret Manager: GCP-managed key/value store.

These secret stores solve the “kick the can down the road” problem by relying on human memory: in this case, your ability to memorize a password that gives you access to your cloud provider (or multiple passwords in the case of Vault, as it uses Shamir’s Secret Sharing).

An example using AWS Secrets Manager

First, login to the AWS Secrets Manager UI, click “store a new secret,” and enter the secrets you wish to store:

undefined

The default is to use a JSON format, as you can see in the screenshot above. Next, give the secret a unique name:

undefined

Click “next” and “store” to save the secret.

Now, in your Terraform code, you can use the aws_secretsmanager_secret_version data source to read this secret (for HashiCorp Vault, AWS SSM Param Store, or GCP Secret Store, you’d instead use the vault_generic_secret, aws_ssm_parameter, or google_secret_manager_secret_version data source):

data "aws_secretsmanager_secret_version" "creds" {
# Fill in the name you gave to your secret
secret_id = "db-creds"
}

If you stored the secret data as JSON, you can use jsondecode to parse it:

locals {
db_creds = jsondecode(
data.aws_secretsmanager_secret_version.creds.secret_string
)
}

And now you can use those secrets in the rest of your Terraform code:

resource "aws_db_instance" "example" {
engine               = "mysql"
engine_version       = "5.7"
instance_class       = "db.t2.micro"
name                 = "example"
# Set the secrets from AWS Secrets Manager
username = local.db_creds.username
password = local.db_creds.password
}

Advantages of this technique

undefined

Drawbacks to this technique

Conclusion

Here are your key takeaways from this blog post:

  1. Do not store secrets in plain text.
  2. Use a Terraform backend that supports encryption.
  3. Use environment variables, encrypted files, or a secret store to securely pass secrets into your Terraform code. See the table below for the trade- offs between these options.

undefined

Your entire infrastructure. Defined as code. In about a day. Gruntwork.io.

Text Link

Platform

IaC Library
Account Factory
Pipelines
Patcher

Services

Accelerator
OpenTofu Services
Terragrunt Services
Pricing
Support

Open Source

Terragrunt
Terratest
Boilerplate
Cloudnuke
Git Xargs

Resources

Docs
Blog
Books
Community

Company

About
Careers
Merch
Legal
Security