Once a month, we send out a newsletter to all Gruntwork customers that describes all the updates we’ve made in the last month, news in the DevOps industry, and important security updates. Note that many of the links below go to private repos in the Gruntwork Infrastructure as Code Library and Reference Architecture that are only accessible to customers.
Hello Grunts,
April was a busy month! We refactored and open sourced our infrastructure testing library Terratest, created a new module called ip-lockdown that makes IAM role usage much more secure, created a new module for deploying Docker images using AWS Fargate, added a module for deploying the NLB, open sourced our collection of reusable Bash scripts in a repo called bash-commons, and made many other changes, fixes, and updates. In other news, AWS has launched two new services—Private Certificate Authority and Secrets Manager—and Russia has blocked AWS and GCP. Read on to learn more!
As always, if you have any questions or need help, email us at support@gruntwork.io!
Motivation: At Gruntwork, we maintain a library of over 250,000 lines of infrastructure code that’s used in production by hundreds of customers. To make it possible for our small team to maintain all of this code, we developed a Go library called Terratest, which provides a variety of tools that make it easier to write automated tests for Terraform, Packer, Docker, AWS, etc. Terratest has grown organically over the last few years as we figured out best practices for infrastructure testing, so we wanted to clean up the design and improve the documentation so everyone else could use it.
Solution: We have refactored Terratest, cleaned up the APIs, added lots of example code and documentation, and released it as an open source project on GitHub: https://github.com/gruntwork-io/terratest.
What to do about it: Read the Terratest announcement blog post, check out all the examples and docs, and start using Terratest to create automated tests for your infrastructure!
Motivation:**** Every EC2 Instance has an EC2 Metadata endpoint (169.254.169.254) that gives you the credentials you need to assume any IAM role associated with that EC2 instance, and thereby, get all the permissions available in that IAM role. By default, that endpoint is accessible to anyone on the server, so we wanted a way to lock it down, ensuring that if a hacker gets access to the server, they can only make use of the IAM role if they have access to privileged user accounts.
Solution: We created a new module called [ip-lockdown](https://github.com/gruntwork-io/module-security/tree/master/modules/ip-lockdown) that can lock down IP addresses on a Linux server such that only specific OS users (say, only root) can access them. For example, you can use ip-lockdown to lock down the EC2 metadata endpoint to just the root user with the following command:
ip-lockdown 169.254.169.254 rootNow, if an attacker gets on your server, to be able to use the permissions in the IAM role, the attacker would also need to find a way to get root/sudo privileges, which significantly improves your security posture.
What to do: We strongly recommend using ip-lockdown on ALL of your EC2 Instances to lock down the EC2 metadata endpoint to the minimal set of OS users (e.g., just root) to protect your IAM roles. Check out module-security, v0.8.2 and the example usage for details.
Motivation: AWS recently released a new way to deploy docker images to the cloud called Fargate. Fargate lets you deploy your containerized apps without being responsible for managing the underlying infrastructure; it’s a bit like AWS Lambda, but for Docker images instead of individual functions. You just hand AWS a Docker image, specify the CPU/memory/network/etc requirements, and AWS will automatically maintain and scale the underlying instances for you.
Solution: We added a new ecs-fargate module to allow you deploy your docker images with Fargate. A typical usage will look something like this:
module "fargate_service" {
source = "git::git@github.com:gruntwork-io/module-ecs.git//modules/ecs-fargate?ref=v0.6.5"service_name = "fargate-example"
container_definitions = "${data.template_file.container.rendered}"vpc_id = "${var.vpc_id}"
subnet_ids = "${var.subnet_ids}"desired_number_of_tasks = 3cpu = 1024
memory = 2048
}data "template_file" "container" {
template = <If you run terraform apply on the code above, AWS will automatically provision the hardware to run 3 copies of your Docker image. It doesn’t get much easier than that!
What to do: If you’d like to give Fargate a shot, use module-ecs, v0.6.5 and check out an example usage for details.
Motivation: AWS recently added a new type of Load Balancer called the Network Load Balancer (NLB), which is a TCP (Layer 4) load balancer that supports very high throughput, rapid scaling, and static IP addresses.
Solution: We added a new nlb module to make it easier to deploy the NLB.
What to do: If you’d like to take advantage of the scalability and throughput benefits that an NLB brings, use module-load-balancer, v0.8.0 (example usage).
Motivation: At Gruntwork, we write a lot of Bash code that’s used in install scripts, run scripts, User Data scripts, and other glue code. We found that we were reimplementing the same functionality over and over again throughout our Bash scripts (e.g., basic assertions, logging, looking up data in EC2 metadata, etc.), and wanted a way to keep things more DRY and tested.
Solution: We’ve created a new open source repo called bash-commons! It’s a collection of reusable bash utilities, complete with documentation and automated tests. You can install bash-commons on your servers using the Gruntwork Installer (be sure to replace <VERSION> in the command below with the latest version from the releases page):
gruntwork-install \
--repo https://github.com/gruntwork-io/bash-commons \
--module-name bash-commons \
--tag Now you can use the source command to import the modules and start using them in your code:
source /opt/gruntwork/bash-commons/log.sh
source /opt/gruntwork/bash-commons/assert.sh
source /opt/gruntwork/bash-commons/os.sh
log_info "Hello, World!"
assert_not_empty "--foo" "$foo"
if os_is_ubuntu "16.04"; then
log_info "This script is running on Ubuntu 16.04!"
elif os_is_centos; then
log_info "This script is running on CentOS!"
fiWhat to do: Check out the bash-commons repo for more info and start using it in your Bash scripts!
Motivation: In our April Newsletter, we indicated that CentOS support would be added for Kafka shortly.
Solution: That work is now complete! In addition, we upgraded our Kafka package to support more best best practices.
What to do: If you’d like to use Kafka with CentOS or upgrade to the latest best practices, check out package-kafka, v0.3.0.
after hooks were not running unless run_on_error was set.source URLs so it should be able to handle both URLs with and without a //.terragrunt plan --help).vault-cluster module now sets a Name tag on the Launch Configuration security group.install-consul script now exposes the --ca-file-path, --cert-file-path, and --key-file-path options to make it easier to install TLS certs. The run-consul script now exposes --enable-gossip-encryption and --gossip-encryption-keyoptions to make it easier to enable gossip encryption, and --enable-rpc-encryption, --ca-path, --cert-file-path, and --key-file-path options to make it easier to enable RPC encryption.consul-client-security-group-rules module. This allows you to attach those rules to any Security Group that has Consul clients on it (e.g., Vault).provider and aws_region from examples so you can use them following the instructions in the Terraform Registry.run-vault now configures api_addr instead of redirect_addr.vault-clustermodule using the new additional_security_group_ids parameter.provider and aws_region from examples so you can use them following the instructions in the Terraform Registry.provider and aws_region from examples so you can use them following the instructions in the Terraform Registry.rds and aurora modules now allow you to enable enhanced monitoring and to specify a custom monitoring IAM role to associate with the DB.minor_version_upgrade setting on rds read replicas so they use the same setting as the primary.vpc-mgmt-network-acls module now allows all inbound and outbound traffic within the private subnet and between the public and private subnet. Before, all inbound traffic was allowed, but outbound traffic was limited solely to TCP. The vpc-app-network-acls module now allows all inbound and outbound traffic from/to the mgmt VPC. Before, all inbound traffic was allowed, but outbound traffic was limited solely TCP.count in the alb module that would crop up if allow_inbound_from_cidr_blocks or allow_inbound_from_security_group_ids were empty.jenkins-server module to v0.8.1 to pick up the fixes listed above.What happened: AWS has launched Private Certificate Authority (PCA), which is a new feature for AWS Certificate Manager (ACM) that you can use to manage private certificates (e.g., TLS certificates) within your AWS account.
Why it matters: Before, if you wanted to use private certificates, you’d have to create your own CA, sign the certs, manage renewing them, manage distributing the trusted cert files, manage certificate revocation lists (CRLs), and so on. Now, all of this is managed for you by AWS, and you can create new private certs with a few clicks or a few API calls.
What to do about it: Check out the PCA documentation to get started. We’ll be adding support for PCA to our Ref Arch (e.g., for certs for internal services, such as sample-app-backend and Jenkins) in the near future.
What happened: AWS has launched a new product called the AWS Secrets Manager, which allows you to securely store, retrieve, and rotate secrets in AWS via API calls.
Why it matters: You now have an additional way to securely manage secrets and to grant access to them using IAM. Other options include using KMS to encrypt secrets and store them in files and tools such as HashiCorp Vault, which provide more powerful tools for secrets management, but require you to run your own Vault cluster (and a Consul cluster for HA).
What to do about it: Give the Secrets Manager a shot and let us know what you think!
What happened: Vault 0.10 has been released!
Why it matters: The key new features in Vault 0.10 are: secret versioning and check-and-set operations; the previously enterprise-only UI is now open source; root / admin credentials for DBs can now be securely rotated; Azure auth via Active Directory credentials; Vault can now create dynamic IAM credentials for accessing Google Cloud Platform environments.
What to do about it: Check out the announcement blog post for all the details. We hope to add support for all these features in our open source Vault module soon.
What happened: CloudWatch now allows you to perform mathematical operations on your metrics.
Why it matters: You can now create graphs for derived metrics, rather than being solely limited to the raw data. For example, instead of just seeing the total number of 5xx errors from your load balancer, you could use metric math to compute the error rate, which is the number of 5xx errors divided by the total number of requests.
What to do about it: Check out the metric math documentation and start using it with your CloudWatch dashboards! Unfortunately, it doesn’t seem like you can create alerts from computed metrics, but perhaps that will be available in the future.
What happened: In a battle with encrypted messaging service Telegram, Russia has apparently taken the drastic measure of blocking all of AWS and GCP IPs.
Why it matters: If you are running your infrastructure on AWS or GCP, then users in Russia may not be able to access it.
What to do about it: Other than migrating off of AWS and GCP, there’s not much you can do other than wait it out, and hope Russia finds a less extreme way to deal with this situation.
Below is a list of critical security updates that may impact your services. We notify Gruntwork customers of these vulnerabilities as soon as we know of them via the Gruntwork Security Alerts mailing list. It is up to you to scan this list and decide which of these apply and what to do about them, but most of these are severe vulnerabilities, and we recommend patching them ASAP.