Let's learn IaC using Terraform with GitHub Actions to deploy into Microsoft Azure

It has been a busy start to the year and I regret that I haven’t been able to dedicate more time to blogging so I intend on catching up on my backlog in the following months. One of the topics I’ve been meaning to write about is how to use Terraform with GitHub actions to deploy infrastructure in Azure. Both Terraform (IaC) and GitHub Actions (orchestration engine) are technologies I’ve been self-training over the past few months and I am excited to continue building the knowledge I’ve acquired. Those who may not be familiar with IaC can refer to one of my previous posts here:

Infrastructure as Code in 15 Minutes PowerPoint Presentation
http://terenceluk.blogspot.com/2022/04/infrastructure-as-code-in-15-minutes.html

My journey through learning these two technologies has been challenging but very fulfilling, and the purpose of this post is to share the various features I came across and what I’ve been able put together to demo all of them. I have to admit that I am not an expert and there may be better approaches so please feel free to comment on this post.

A few of my colleagues have indicated that they feel it would be beneficial to include more diagrams in my posts rather than just writing so I have taken the time to create a series of diagrams to better illustrate the workflow and the Terraform code that used.

What I’m trying to Demo

The components and features I’m trying to demo are the following:

1. How to use Terraform for IaC
2. How to execute terraform init, format, validate, plan, apply, destroy in a workflow
3. How to use different .tfvars variables to deploy different environments: dev, stg, prod
4. How to use GitHub Actions as an orchestration engine for pipelines
5. How to initiate a workflow manually, on push, on pull request, on complete of another workflow
6. How to have the workflow store the terrafirn .tfstate file to an Azure Storage Account Container
7. How to set a dependency on a previous step
8. How to use and reference custom self-written Terraform modules stored in a different registry and in subfolders (some may not know about the // when accessing a module in a sub directory)
9. How to use and reference a Terraform Registry module
10. How to use the Super-Linter to scan the code
11. How to get a branch name
12. How to pass a branch name a different step
13. How to store and use secrets in GitHub
14. How to configure different environments in GitHub
15. How to configure a protection rule for a GitHub environment
16. How to require an approval before executing a workflow
17. How to configure an Azure Storage Account Container to store the .tfstate file

GitHub Repositories

Let me begin by providing the links to the GitHub repositories I will be using for the demonstration.

GitHub Repository that contains the GitHub Actions workflows, Terraform code for deploying dev, stg, and production environments:

https://github.com/terenceluk/terraform-k8s-acr-psql-vms-demo

GitHub Repository that contains the Terraform modules that are referenced and used for the deployment of Azure Kubernetes Service, Azure Container Registry, and PostgreSQL server and database:

https://github.com/terenceluk/terraform-modules

I’ve added as many comments into the code to explain the purpose in hopes that whoever is reading it will understand the function. Feel free to fork them to your repo and test or modify as you see fit.

GitHub Repository Branches

There will be 3 branches in the GitHub repo:

• Dev
• Stg
• Prod

The Terraform code and workflows will be directly pushed to the dev branch to test, then merged into stg and production.

Terraform and GitHub Actions Code

The GitHub Actions YAML files will be stored in the mandatory .github/workflows directory of the repository.

The Terraform code will be split as follows.

terraform-k8s-acr-psql-vms-demo repository

• The main.tf, output.tf, provider.tf, and variables.tf files are stored in the root
• The .tfvars files containing the variable values for dev, stg, and prod environments are stored in the subfolder Variables
• The main.tf references modules that are stored outside of its repository:
  • Another GitHub public repository named terraform-modules
  • A Terraform Registry module

terraform-modules repository

• This repository contains 3 modules that are used to deploy:
  • Azure Container Registry
  • Azure Kubernetes Service
  • PostgreSQL Server and Database

What we are deploying with Terraform

The resources that will be deployed are the following:

1. Azure Container Registry
2. Azure Kubernetes Service
3. PostgreSQL Server and Database
4. Linux Virtual Machine
5. Windows Virtual Machine
6. VNet with subnets
7. Management lock for the Azure Container Registry

Workflow: terraform-plan.yml and terraform-apply-dev.yml

The terraform-plan.yml and terraform-apply-dev.yml workflows are dispatched whenever there is a push to the dev branch in the GitHub repo. I have included a diagram below that walks through the process and will also list the flow here:

1. User updates Terraform or GitHub Action YAML code and pushes it to the dev branch of the GitHub repo
2. The terraform-plan.yml workflow is started as it is configured to start on push to dev
3. Two steps are now executed in parallel:
   1. Get-Branch-Name to determine what branch this was pushed on
   2. The download and use of the Super-Linter is ran in parallel to scan the code
4. Several Terraform commands are executed:
   1. fmt is ran on the Terraform code to ensure it is formatted properly
   2. validate is ran on the Terraform code
   3. init is ran to initialize and configure an Azure Storage Account Container to store the .tfstate file
   4. plan is ran to generate a plan with the appropriate terraform-dev.tfvars file
5. Once the terraform-plan.yml workflow is complete, initiate the terraform-apply-dev.yml workflow
6. Get-Branch-Name will start to obtain the previous workflow run conclusion
7. If previous workflow was not successful then end the workflow, if it was successful then get the branch path that is currently being worked on
8. If the branch path is not dev then end the workflow, if it is the dev branch then set the environment to GitHub dev-deploy and go to the next step
9. The dev-deploy GitHub environment has a protection rule configured that requires review and approval so the reviewer will receive an email to approve or reject
10. If the reviewer has approved then proceed with the deploy where the following are executed:
    1. fmt is ran on the Terraform code to ensure it is formatted properly
    2. validate is ran on the Terraform code
    3. init is ran to initialize and configure an Azure Storage Account Container to store the .tfstate file
    4. plan is ran to generate a plan with the appropriate terraform-dev.tfvars file and the -out switch is used to create the plan.tfdata file
    5. apply with -auto-approve using the plan.tfdata file is executed
11. Resources will not get deployed to Azure

The following is a screenshot of the jobs in the workflows and the process during the deployment:

What a pending review looks like:

The email a reviewer would receive:

The review prompt in GitHub:

The apply output when deploying infrastructure:

A successfully deployment (note that the duration of 20h 13m 31s is because I left the review pending over a day):

Workflow: terraform-apply.yml

The terraform-apply.yml workflow is executed upon completing a pull request for stg and prod. It is much less complex so I will simply include the two diagrams to describe the process:

Staging:

Production:

Workflow: terraform-destroy.yml

The terraform-destroy.yml workflow is dispatched manually when we want to remove the environment. The following are a few screenshots of manually dispatching the workflow:

The output during a destroy of the infrastructure:

Successfully destroying the infrastructure:

Setting up Azure Storage Account Container and Resource Group

With the walkthrough of the Terraform and GitHub Actions completed, I would like to provide the steps required to set up the Azure Storage Account Container that will be used to store the terraform .tfstate file as none of this would work without it.

We’ll be using the Azure CLI to configure this:

# Log into Azure

az login

# Define variables for subscription ID, Azure Region, storage account, container

subscriptionID = "xxxxxxxx-71c2-40f2-b3d4-xxxxxxxxxx"

resourceGroup = "ca-cn-dev-demo-rg"

azureRegion = "canadacentral"

storageAccount = "cacndevdemost"

containerName = "terraform-state"

# List available subscriptions

az account list

# Specify the subscription to use

az account set -s $subscriptionID

# Create a App Registration and corresponding Enterprise Application / Service Principal and assign it contributor role to the subscription – Ref: https://docs.microsoft.com/en-us/cli/azure/ad/sp?view=azure-cli-latest

az ad sp create-for-rbac --name $servicePrincipalName --role Contributor --scopes /subscriptions/$subscriptionID --sdk-auth

Copy the clientId, clientSecret, tenantId values.

Note that the following App Registration will be configured along with a secret in Azure AD:

The corresponding Enterprise application (Service Principal) will be created:

We’ll need to grant permissions to the Service Principal to the subscription that Terraform will deploy resources to. Contributor typically sufficient but there are some configurations such as Resource Locks that require Owner:

# Create resource group that will store storage account for saving the Terraform State

az group create -g $resourceGroup -l $azureRegion

# Create a new storage account and place it in the resource group

az storage account create -n $storageAccount -g $resourceGroup -l $azureRegion --sku Standard_LRS

The following storage account will be created:

# Create a container in the storage account to store the terraform state

az storage container create -n $containerName --account-name $storageAccount

The following Container will be created and when used a .tfstate will be stored here:

Setting up GitHub Secrets

Various parameters such as service principal attributes, secrets, storage account access keys should not be stored directly in the Terraform .tfvars files and should be stored in the GitHub secrets vault for retrieval.

Proceed to navigate to the previously configured Storage Account’s Access Keys and copy the key1 as we’ll need to configure it in GitHub secrets:

For the purpose of this example, the dev environment will require the following secrets configured as they are referenced in the workflows and terraform code:

• DEV_ARM_CLIENT_ID
• DEV_ARM_CLIENT_SECRET
• DEV_ARM_SUBSCRIPTION_ID
• DEV_ARM_TENANT_ID
• DEV_PSQL_ADMINISTRATOR_LOGIN_PASSWORD
• DEV_PSQL_ADMIN_LOGIN
• DEV_STORAGE_ACCESS_KEY
• DEV_STORAGE_ACCOUNT_NAME
• DEV_STORAGE_CONTAINER_NAME

Note that you will not be able to view the values of these secrets once they are configured in GitHub.

In addition to the DEV secrets, the STG and PROD secrets will also need to be configured for the other branches.

Setting up GitHub Environments

The last requirement for this demo is to set up the different environments in GitHub for the branches. It’s important to note that Environments are NOT available in private repositories for free accounts so you’ll need to use a public repo for it. This demo has the following environments configured:

• dev-deploy
• prod
• dev
• stg

The additional dev-deploy environment is really just a way for me to execute the plan step to verify the code is free of errors and then requiring a review and approve to initiate the deployment of the resources. This method likely isn’t best practice but I thought I’d use this to demonstrate how to set the environment in the workflow to force a review or reject.

With the environments setup, the dev-deploy is configured with the Required reviewers protection rule:

… and that’s it. I hope this was beneficial for anyone who may be trying to learn Terraform and GitHub actions. There are plenty of blog posts available but I’ve noticed that some were not very clear on the steps and I’ve spent countless hours troubleshooting the code from start to finish. The process can be very frustrating at times but it’s also very satisfying when everything starts to work.

I’ll be working on another new project to incorporate an actual application in the future and will be sharing it.