How To Setup a Terraform Remote Backend in Google Cloud

Terraform is an infrastructure provisioning automation tool that allows you to specify resources to run your application — in the cloud or on-premises — using code. This code is written in Hashicorp Configuration Language (HCL) syntax in a declarative manner, within files with a .tf extension. You only have to define the resources you need, Terraform will handle the rest.

Objectives

By the end of this article, you should understand:

• Common Terraform commands
• How Terraform works
• Why you need a remote backend
• How to set up a remote backend in Google Cloud
• Workspaces

Requirements

To get started, you will need:

• A Google Cloud account
• Google Cloud SDK
• Terraform — follow this link to download and install on your PC
• A code editor — VSCode, Sublime Text, etc.

Before we dive into creating the remote backend, let’s take a brief look at some Terraform commands.

Common Terraform commands

The common commands when working with a remote backend are:

• terraform init : This is the first command to initialize Terraform after writing a new Terraform configuration.
• terraform plan : Detects changes by comparing your current configuration and the state file.
• terraform apply : Executes any changes.
• terraform workspace : For managing workspaces.

You can learn more about these commands and others in the Terraform documentation.

What happens under the hood?

When you run Terraform, a state file is created. This state file is a record that maps the infrastructure created on the cloud to the resources defined in your Terraform configuration file. This file is a JSON format with a .tfstate extension.

Whenever you run terraform plan, the state is pulled down from local storage or a remote backend storage and compared with your current configuration to determine the changes to be made.

When you run terraform apply, these changes are executed and the state file is updated.

This is how Terraform can keep track of your infrastructure, making it possible to create, update and destroy infrastructure in an automated manner.

Why do you need a remote backend?

When it’s just you working on a personal Terraform project, the state file is stored locally — on your PC. However, when working on a team, with multiple individuals working on the project, the same state file will have to be shared. This comes with its own challenges:

• You need a shared location for the storage of state files.
• Multiple people running terraform at the same time may result in data loss or file corruption.
• Secure storage of sensitive data.

This is why you need a remote backend—it solves these problems in the following ways:

• It provides a shared location for storing state files.
• It supports state locking. This means that when terraform apply is run by a team member, a lock is placed on the state file to prevent another team member from running terraform apply at the same time. This averts the accidental duplication of deployments or data loss.
• Most remote backends support encryption of state files, in transit and at rest, natively. In addition, there is also support for access control using identity management and permissions, to regulate who has access to state files and secrets.

Set up the remote backend

Terraform supports a number of remote backend platforms — Google Cloud Storage, Amazon S3, Azure Storage and Terraform Cloud. We will be using storage services from Google Cloud — a Cloud Storage Bucket.

However, before we can configure the remote backend, we need to have an existing storage bucket. So let’s create one.

In your code editor, create an empty directory. Within this directory, create a main.tf file. We will define our configuration within this file.

The first step to using Terraform is to configure the cloud provider — so it knows what cloud provider API to connect with. The basic format used to define a resource in Terraform is:

Basic resource definition format in Terraform

So, in order to define GCP as our cloud provider, we use the following block of code:

The first step to using Terraform is to configure the cloud provider

credentials refers to the private key file of a GCP service account — this is needed for Terraform authentication to GCP. To obtain your private key file, navigate to IAM & Admin > Service Accounts

Select the default Compute Engine service account:

Click on the Keys tab and then the Add Key button:

Select Add new key > select key type as JSON. Click Create.

This downloads the private key file to your PC. Include the file in your project directory and then pass its path as the value to the variable credentials .

Next, create a bucket.tf file — to define the storage bucket — and add the following block of code:

Declare storage bucket properties

This tells Terraform to create a storage bucket with the given name and location, in GCP. With object versioning enabled, whenever a state file is deleted — accidentally or otherwise — a version is kept; it does not get deleted permanently. This makes it possible to roll back to an older version if something goes wrong.

For more on object versioning, see the documentation.

Now, that we have the cloud provider and storage bucket declared, navigate to your project directory within your terminal. Run terraform init to initialize Terraform and connect to GCP. You should have an output similar to this:

$ terraform init
Initializing provider plugins...
- Checking for available provider plugins on https://releases...
...
Terraform has been successfully initialized!
...

Then, run terraform plan for a representation of the changes that Terraform will apply to create the storage bucket. The output should be:

$ terraform plan
Terraform will perform the following actions:
# google_storage_bucket.<bucket_name> will be created 
+ resource "google_storage_bucket" "your-bucket-name" {
   ...
  }

Follow the prompt, and run terraform apply to create the bucket.

Once the creation is complete, you should have an output similar to this:

...
google_storage_bucket.<bucket_name>: Creation complete after 12s
...
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Head over to your GCP account to confirm that the storage bucket was created — Navigate to Cloud Storage > Browser

You should see your created bucket on the Browser page:

Now that we have a storage bucket created, let us define our remote backend configuration.

Add the following block of code within main.tf :

This block of code is specific to GCP. It tells Terraform to establish a remote backend in the bucket specified. As for the prefix variable, you can think of it as defining the name of the directory where all the state files will be stored (we will demonstrate this below).

Run terraform init to initialize the backend. You will see an output similar to this:

$ terraform init
Initializing the backend...
Successfully configured the backend "gcs"! Terraform will automatically use this backend unless the backend configuration changes.
Initializing provider plugins...
...
Terraform has been successfully initialized!

Now, when you navigate to Storage > Browser and click on your bucket name, you should see a directory/folder named after the value you passed to prefix.

You have officially configured a Terraform remote backend!

But, there’s more…

Click on the folder to view its content. You will notice that it contains a default.tfstate file:

This is the state file. But why does it have that name and where did it come from?

Cue “workspaces”.

Workspaces

A Terraform workspace is an isolated environment for storing state files. It allows you to run the same configuration file in different/separate environments. The state files are named after the workspace.

There is a default workspace created for you the first time you initialize Terraform, called default. If you do not create a new workspace, the default workspace is the active workspace for all the Terraform actions you run.

To view workspaces, run the command terraform workspace list. The output should be:

$ terraform workspace list
* default

The asterisk indicates the active workspace.

To create a new workspace, run the command terraform workspace new <name> . The output should be:

$ terraform workspace new <name>
Created and switched to workspace "<name>"!
You're now on a new, empty workspace. Workspaces isolate their state, so if you run "terraform plan" Terraform will not see any existing state for this configuration.

Now, when you run terraform workspace list , you will have two workspaces with an asterisk on the just created workspace:

$ terraform workspace list
  default
* <new workspace>

For demonstration purposes, let’s assume the new workspace created is called staging . The output will be:

$ terraform workspace list
  default
* staging

Within this new workspace, run terraform plan :

$ terraform plan
Terraform will perform the following actions:
# google_storage_bucket.<bucket_name> will be created 
+ resource "google_storage_bucket" "your-bucket-name" {
   ...
  }

You’ll notice that Terraform wants to create an entirely new storage bucket from scratch. Each workspace is completely isolated from the other — once within a workspace, Terraform has no knowledge of state files in other workspaces.

NOTE: Although Terraform can use a single configuration file in multiple work spaces, the resources created need to have unique name values. As a result, you will need to specify these values in a separate .tfvars file. But this is outside the scope of this article. Learn more here and here.

Run terraform apply to create the new storage bucket.

Now, when you navigate to Cloud Storage > Browser > Bucket, you will find a new .tfstate state file created within the bucket, as seen in this image:

Back in your project directory, you’ll also notice that you have the.tfstate file created by Terraform. This file should not be committed to Git version control to prevent the state file from being edited. This is because the state file is a private interface meant only for internal use within Terraform.

Also, it is best practice to create workspaces for different environments — development, staging, production — to avoid accidentally breaking the app in production.

Clean up

Run terraform destroy (in both workspaces) to take down the resources created in this demonstration to avoid incurring unplanned charges from the cloud provider.

Done!

Now you can collaborate seamlessly on your project through a remote backend. Explore the Terraform documentation to learn more about remote backends.

To get you started using Terraform, you can check out my repo for some resource configurations.

~ Hope you found this article helpful. Thank you for reading! ~