Infrastructure as Code On Google Cloud Platform using Terraform

In the previous posts, we covered development aspects of projects. In this and the coming posts, we will cover how developers can deploy their projects using various DevOps practices such as IaC and CI/CD.

Terraform is a tool designed for provisiong infrastructure on various platforms using the Infrastructure as Code (IaC) approach. In this approach, you define configuration files to create resources, on cloud platforms, instead of manually creating them either through the UI or through respective CLI.

As always, let's get our hands dirty and see how we can utilize Terraform to create resources on GCP.

Here is a list down of the files that are involved

• main.tf - The file where resources will be defined
• variables.tf - Definition of variables along with defaults and validation constraints
• prd.tfvars - File used to assign values to variables

You can definitely get by using just the main.tf file and hard coding all the values there, but, to keep the deployments dynamic across multiple users/teams, I suggest keeping everything separate.

Step 1: Download Terraform Binary

Let us first create a directory, download terraform binary archive, unzip it and set permissions to be able to use it locally.

mkdir ~/projects/terraform
cd ~/projects/terraform
wget https://releases.hashicorp.com/terraform/0.14.3/terraform_0.14.3_linux_amd64.zip
unzip terraform_0.14.3_linux_amd64.zip
chmod +x terraform

Step 2: The variables.tf file

For the sake of this post, we will create some Pub/Sub topics and subscriptions using Terraform, intention being to show how to work with Terraform.

Below are the contents that go into the file.

variable "projectId" {
  description = "Project ID of the GCP project to create resources in"
  type = string
}
variable "pubsub_topics_and_subscriptions" {
  description = "Topic and subscriptions to create in the project"
  type = list(object({
    topic = string
    subscriptions = list(string)
  }))
  default = []  validation {
    condition = alltrue([
      for topic_and_subscriptions in var.pubsub_topics_and_subscriptions :
         can(regex("^(prd-|stg-|dev-).*(-pst)$", topic_and_subscriptions.topic)) &&
           alltrue([for subscription in topic_and_subscriptions.subscriptions : can(regex("^(prd-|stg-|dev-).*(-pss)$", subscription))])
    ])
    error_message = "All pubsub topics and subscriptions should start with one of \"prd-\", \"stg-\" or \"dev-\", topics should end with \"-pst\" and subscriptions should end with \"-pss\"."
  }
}

As said earlier, this file is all about defining your variables. Here we are first defining the "projectId" variable, setting its type to string.

Next we define the complex variable pubsub_topics_and_subscriptions. This variable is a list type, with list elements being a custom object with "topic" and "subscriptions" as its members. Topic is of string type and the subscriptions we create out of it will be put in a list of string.

We are setting the default value of this variable to be an empty list. We are defining some validation constraints. I have intentionaly made it bit more complex, you can look the the simple ones in the terraform documentation.

In validation, we are using regex to validate that all topic names start with either "prd", "stg" or "dev". This is a best practice, so that people can tell in which environment the resource resides, just by looking at its name.

We also check if all topic names end with "pst", which is an abbreviation for pub-sub-topic. This, again, is a best practice.

We follow similar approach to validate subscription names as well, but those need to end with "pss" to denote pub-sub-subscription.

While you can see how for loop is defined to iterate over the variable, observe use of alltrue() function. We use this to check if all the values in the array is true, for loop returns array of booleans.

Step 3: The main.tf file

The first block of code would be to define provider, projectId and default regions.

provider "google" {
  project = var.projectId
  region = "us-central1"
  zone = "us-central1-a"
}

Observe how we are referring to a variable in line number 2. Let's now define the Pub/Sub topic block to create Pub/Sub topics.

resource "google_pubsub_topic" "local_name" {
  count = length(var.pubsub_topics_and_subscriptions)
  name = var.pubsub_topics_and_subscriptions[count.index].topic
    message_storage_policy {
    allowed_persistence_regions = [
      "us-central1"
    ]
  }
}

In the above block, you can see how multiple topics can be created while looping over a variable of list. "count" is a meta-argument the helps in looping over list and create multiple instances of the resource.

Now let's go ahead and write our Pub/Sub subscription block.

resource "google_pubsub_subscription" "local_name" {
  depends_on = [google_pubsub_topic.local_name]
  for_each = {
    for subscription in local.subsciptions : "${subscription.topic_name}.${subscription.subscription_name}}" => subscription
  }
  name  = each.value.subscription_name
  topic = each.value.topic_name
  # 20 minutes
  message_retention_duration = "600s"
  retain_acked_messages      = false
  ack_deadline_seconds = 10
}

In line 30, we are setting the depends_on flag to let terraform know that it has to wait till the topics are created before trying to create the subscription. Otherwise, terraform qould try to create all resources parallely and the subscriptions would error out.

Notice two things in line 32, the use of local.subscriptions variable instead of var.pubsub_topics_and_subscriptions and what the for loop is doing.

Terraform does not allow nested loops, like two dimensional loops. So, we use the flatten() function to create a map variable out of list variable, expanding our topic -> list(subscriptions) to, multiple entries of topic -> subscription. We store this in the local variable.

locals {
  subsciptions = flatten([
    for obj in var.pubsub_topics_and_subscriptions : [
      for subscription in obj.subscriptions : {
        topic_name = obj.topic
        subscription_name = subscription
      }
    ]
  ])
}

But Terraform needs a key for the map, and that is what we are doing in the line 32 for loop, creating a key of format "topicname.subscriptioname".

The for_each meta-argument used in line 31 is a map() specific one, which helps iterate over the map and create multiple instances of the resource.

Step 4: The prd.tfvars file

The prd.tfvars will be a file which we use to assign production environment resource names to variables. You can name it anything. I prefer to use dev.tfvars for development, stg.tfvars for staging and prd.tfvars for production.

pubsub_topics_and_subscriptions = [
  {
    topic = "prd-telemetry-receiver-pst"
    subscriptions = [
      "prd-telemetry-raw-ingestion-pss",
      "prd-telemetry-raw-analytics-pss",
      "prd-telemetry-raw-archive-pss"
    ]
  },
  {
    topic = "prd-telemetry-enriched-pst"
    subscriptions = [
      "prd-telemetry-enriched-advertising-pss",
      "prd-telemetry-enriched-archive-pss"
    ]
  }
]

Note, we are not setting the projectId variable here. Terraform can also fetch variable values from the environment. So, we will set the TF_VAR_projectId as an environment variable, to use in our deployment.

Step 5: Generate Terraform Deployment Plan

Now that we have created all the files, let execute terraform commands to generate a deployment plan.

First, lets initialize the tool in the directory, this also downloads the "google" provider binary and stores it in the .terraform directory within the working directory.

./terraform init .

Now that it has downloaded all the required provider plugins, let's generate the plan.

But before that, create a service account with necessary priviledges, for terraform to use, to create the resources on GCP. We point the GOOGLE_APPLICATION_CREDENTIALS environment variable to the json key file, which terraform can pick up to use for deployment.

export GOOGLE_APPLICATION_CREDENTIALS="/home/karthik/projects/user-service-account.json"export TF_VAR_projectId="barrelsofdata-dp-prod"./terraform plan -var-file prd.tfvars -out generated_plan.zip .

Note how we are passing the prd.tfvars file. Also note, the -out flag is not mandatory, you need not save the plan, but doing so is a best practice.

This will create a plan zip archive and also display, on the console, a json format of the configuration of resources that will be created.

Step 6: Terraform Deploy

Now that the plan is generated, let's go ahead and deploy it. This will create the defined Pub/Sub topics and subscriptions in the said project.

./terraform apply -auto-approve generated_plan.zip

If you do not want to create a plan file, you can also use below command to deploy the infrastructure.

./terraform apply -auto-approve -var-file prd.tfvars .

You should now be able to see the topics and subscriptions in you GCP project. Terraform would have also creatted a terraform.tfstate file in the working directory. This file is used to keep track of the state of deployments.

Step 7: Teardown Provisioned Resources

You can also delete all the created resources using the destroy command.

./terraform destroy -auto-approve .

That should give you a good idea of how you can use terraform. Go ahead and execute below commands one after another to check what the commands support.

./terraform --help
./terraform init --help
./terraform validate --help
./terraform plan --help
./terraform apply --help
./terraform destroy --help

References and further reading

• Terraform Google Cloud Provider
• Terraform Docs
• Terraform Iterations