Bootstrapping your Terraform automation with Amazon CodeCatalyst

Terraform is awesome to manage all your infrastructure, but when you have more than one developer trying to make changes to the infrastructure, things can get messy very quickly if there isn't a mechanism (CI/CD pipeline) in place to manage it. Without one, making changes to any infrastructure requires coordination and communication, and the challenge quickly scales the more people that are involved with making these changes. Imagine having to run around shouting "Hey Bob! Hey Jane! You done yet with that DB change? I need to add a new container build job!". As Jeff Bezos said:

"Good intentions never work, you need good mechanisms to make anything happen."

This tutorial will show you how to set up a CI/CD pipeline using Amazon CodeCatalyst and Terraform. The pipeline will utilize pull requests to submit, test, and review any changes requested to the infrastructure. We will cover the following topics in this tutorial:

• Using S3 as a backend for Terraform state files, with DynamoDB for locking, and encrypting the state file at rest with KMS
• CodeCatalyst to run our CI/CD pipelines to create and update all your infrastructure

Table of Contents

Chicken and egg problem

Automating your infrastructure is a great idea, but you need infrastructure to automate your infrastructure. There are three approaches to doing this:

1. Clicking in the console to set everything up, aka "ClickOps"
2. Using a CLI to create the resources for you with scripts, "Procedural"
3. Using Terraform without storing the state file to bootstrap, then add in the state file configurations to store it

We will be using the 3rd option, have a look at the Stack Overflow discussion around approaches for more details on the trade-offs.

Getting started

Let's get started setting this up! Make sure you are logged into your AWS, and CodeCatalyst accounts in the same browser.

Setting up a CodeCatalyst Space, Project, Repo, and Environment

Now, let's set up our CodeCatalyst Space and Project. Create a new space by clicking on Create Space on the CodeCatalyst Dashboard, add a name (we will use Terraform CodeCatalyst), add the AWS Account ID to link to for billing (111122223333 is a placeholder), you can find your account ID in the top right of your AWS Console, and follow the prompts to link your AWS Account with CodeCatalyst.

Next, we need to create a new Project, click on the Create Project button, select Start from scratch, and give your project a name - we will use TerraformCodeCatalyst.

Now we need to create a new repository for our code. Click Code in the left-side navigation menu, then on Source repositories, Add repository and choose Create repository. Set a repository name (we will use bootstrapping-terraform-automation-for-amazon-codecatalyst in this tutorial), add a description, and Terraform for the .gitignore file:

Lastly, we need to set up the AWS environment we will use for our workflow. In the left-side navigation menu, click on CI/CD, then on Environments, and then Create environment. Add the Environment name, Description, choose your AWS account from the dropdown under AWS account connection, and click Create environment.

Setting up a Dev Environment

To start working on our code, we need to set up a development environment, and will be using the built-in ones provided by CodeCatalyst. In the left navigation menu, click on Dev Environment under Code, then Create Dev Environment, select Cloud9 - this tutorial with use Cloud9. Select Clone a repository, select bootstrapping-terraform-automation-for-amazon-code-catalyst in the dropdown for Repository, add an Alias of TerraformBootstrap, and then click the Create button.

It will take 1 - 2 minutes to provision your development environment, and once done, you will be presented with a welcome screen:

The version of Terraform may not be the latest, you can check which version is installed by running terraform --version. This tutorial uses version 1.3.7, to ensure you are using that version, use the following commands:

🚨 NB: If you are using a local development environment instead of one managed by CodeCatalyst, the architecture / operating system may be different, please see the downloads page to download the appropriate version of Terraform.

Lastly, we need to add AWS CLI credentials to our Dev Environment to access resource in our account. It is recommended to not use the root user, if you have not yet set up an IAM user, please do so now by following the instructions, and make sure to copy the Access key ID and Secret access key values, then run aws configure in the terminal of your dev environment (you can leave the last two default values blank, or enter values you prefer):

You can verify that access is set up correctly by running aws sts get-caller-identity in the terminal:

Bootstrapping Terraform

Next, we need to add the required infrastructure to our AWS account using Terraform. We will be creating the following resources:

1. IAM roles: Provides the role for our workflow to assume in the account - one for the main branch, one for any pull requests (PRs).
2. IAM policies: Set the boundaries of what the workflow IAM roles may do in our account - full admin access for main branch allowing creation of infrastructure, ReadOnly for the PR branches to allow validating any changes.
3. S3 bucket: An S3 bucket to store our Terraform state file in.
4. S3 bucket versioning: Allows keeping backup copies of the Terraform state file each time it changes.
5. DynamoDB Table: Used by Terraform to create a lock while running - this prevents multiple CI jobs making changes when run in parallel.
6. KMS Encryption Key: (Optional) While the state file is stored in S3, we want to encrypt it while at rest using a KMS key. For this tutorial, we will use the pre-existing aws/s3 key, if you prefer to use a different KMS key ($1/month/key), there will be a section below to describe how to make changes to do that.

To create all of the required files, you can use the following commands to create the directories, and download the files directly from the sample repository. Run the commands in the root of your cloned git repo via the dev environment terminal:

The files created will have the following content:

Once done, edit the _bootstrap/variable.tf file and update the state_file_bucket_name (S3 bucket names are globally unique), and optionally the state_file_lock_table_name variables with the values for your S3 bucket name for the state file, DynamoDB table name for locks, and optionally change the aws_region if you want to use a different region.

We will now bootstrap our infrastructure (the body of each Terraform resource from the terraform plan command has been omitted using ...):

The output should look like:

The plan command will output a list of resources to create, and you can take a look at exactly what it will create. Once you are satisfied, run terraform apply, and confirm the infrastructure creation.

Next, we will move the state file we just created with all the details of our infrastructure to our S3 bucket. To do this, we need to configure a Terraform backend using S3. Create _bootstrap/terraform.tf with the following, and update the bucket and region values with your values:

It would be easier if we could reference region and state_file_bucket variables in the Terraform backend configuration, but it does not allow any variable / local interpolation.

To migrate the state file to S3, run terraform init -migrate-state, and you should see the following output:

We are now ready to set up our workflows, but first, let's ensure we commit our changes to our git repo. Run git add ., git commit -m "Terraform bootstrapped" and git push:

Setting up workflows

In the previous section, we created two new IAM roles for our workflow, one for the main branch with permissions to create resources, and another for all pull requests with read-only permissions. We need to add these to our CodeCatalyst Space. In the top left of the page, click on the Space dropdown, and then click on your Space name. Navigate to the AWS accounts tab, click your AWS account number, and then on Manage roles from the AWS Management Console. This will open a new tab, select Add an existing role you have created in IAM, and select Main-Branch-Infrastructure from the dropdown. Click Add role:

This will take you to a new page with a green Successfully added IAM role Main-Branch-Infrastructure. banner at the top. Click on Add IAM role, and follow the same process to add the PR-Branch-Infrastructure role. Once done, you can close this window and go back to the CodeCatalyst one.

The base infrastructure is now in place to allow us to start using our workflow for any future changes to our infrastructure. We need to create a similar Terraform backend configuration for all the resource we will create using our workflow - as mentioned, we are intentionally keeping out bootstrapping infrastructure separate from the day-to-day infrastructure. In the root of the repo, create terraform.tf, with the following content - take note that the key for the bucket is different from what we used for the bootstrapping infrastructure, and as before, replace the bucket, region, dynamodb_table, and kms_key_id with your values:

The region set in the above block indicates in which region the S3 bucket was created, not where we will create our resources. We also need to configure the AWS provider, and set the region to use. Will use a variable for this, you could also hard-code it, but it is more manageable to keep all the variables in a single variables.tf file for this purpose. Create providers.tf with the following content:

And the variables.tf file with (you can change the region here if you want to create resources in a different one):

Now we are ready to create our workflow file. First, we need to create the workflow directory and file:

Open .codecatalyst/workflows/main_branch.yml in your IDE, and add the following - remember to replace the placeholder AWS account ID 111122223333 with the value of your account, and the IAM role names if you changed them (you can choose between the standard CodeCatalyst workflow, or to use GitHub Actions with CodeCatalyst):

Let's try out our new workflow! First, we need to stage, commit, and push our changes directly to the main branch - this is needed as only workflows committed to the repo will be run by CodeCatalyst. Use the following commands:

Output:

In your browser, navigate to the CI/CD -> Workflows page. You should see the workflow running:

If you click on Recent runs to expand it, you will see the details of the currently running job. Click on the job ID (Run-XXXXX) to view the different stages of the build:

Pull Request Workflow

Now that we have our main branch workflow done, it is time to set up the pull request one. The workflow will be very similar as the main branch one, with the following difference:

1. A different workflow name - TerraformPRBranch
2. We use the PR-Branch-Infrastructure IAM role to ensure we cannot make any infrastructure changes in the PR workflow
3. We remove the terraform apply step
4. The trigger for the build is for when a PR to the main branch is opened or updated (REVISION)

Create a new file for the PR workflow as .codecatalyst/workflows/pr_branch.yml, and add the following (replacing the placeholder AWS account ID of 111122223333, and the IAM role name if you changed it) - you can choose between the standard CodeCatalyst workflow, or to use GitHub Actions with CodeCatalyst:

This workflow needs to be added to the main branch before it will trigger for a new PR, so let's do that now:

This will trigger the main branch workflow as we added a change, but without adding any additional Terraform resources, it will not make any changes:

We will now add an AWS resource via Terraform via a PR. First, we need to create a new branch:

Next, create a new file in the root of the project vpc.tf - we will create a VPC that has three public subnets, and the required routing tables. Add the following content to the file:

We need to commit the change, and push the branch using --set-upstream origin test-pr-workflow as the remote branch does not yet exist:

The output shows that the remote branch has been created, and we pushed changes from our local branch to that one:

This will not yet trigger a PR branch workflow as we haven't opened the pull request. In CodeCatalyst, navigate to Code, then Pull requests, and click on Create pull request. Select test-pr-workflow as the Source branch, main as the Destination branch, and add in a Pull request title and Pull request description. You can also preview the changes the PR will make on the bottom of the screen:

Click Create, and then navigate to CI/CD -> Workflows, and select All branches from the dropdown in the top of the Workflows menu. After selecting All branches, you will see four workflows, the TerraformMainBranch and TerraformPRBranch ones, and a copy for each of the two branches main and test-pr-workflow. The TerraformMainBranch workflow will have an error with Workflow is inactive, which is expected as we limit that workflow to only run on our main branch. Click on the Recent runs under the TerraformPRBranch workflow for the test-pr-workflow branch, and then on Terraform-PR-Branch-Plan job to see the details.

By clicking on the Terraform Plan step, you will be able to see the proposed infrastructure changes listed in the output. You can now inspect exactly which changes will be made to your infrastructure from this pull request. In you standard day-to-day operations, you would now go back to pull request to decide what action to take. If the proposed changes have been reviewed and approved, you can merge the pull request, or you can start a conversation on the PR to address any issues or concerns. We will now merge this request to roll out this infrastructure in our account by navigating to Code -> Pull requests, clicking on the Title or ID of the PR, and then the Merge button. You are presented with a choice between a Fast forward merge, or a Squash and merge option. Fast forward merge will take all the commits on the branch and add them sequentially to the main branch as if they were done there. For the Squash merge, it will combine all the commits on the test-pr-workflow branch into a single commit before merging that single commit to main. Which one you use will depend on your development approach, for this tutorial, will use the Fast forward merge one. You can also select the option to Delete the source branch after merging this pill request. Source branch: test-pr-workflow, this will help keep your repository clean from too many branches if they are no longer used. Click on Merge, and navigate to CI/CD -> Workflows to see the new VPC being created. Click on the currently running TerraformMainBranch workflow's Recent runs, then on the job ID, and then on the 2nd step to see the progress in the right-hand pane. Once the job completes, we can verify that the VPC was created by navigating to the VPC section of the AWS Console, a clicking on the VPC ID for the VPC with the name CodeCatalyst-Terraform. You should see something similar:

Clean up

We have now reached the end of this tutorial, you can either keep the current setup and expand on it, or delete all the resources created if you are not. If you are planning to manage multiple AWS accounts, we recommend reading the Automating multiple environments with Terraform tutorial - it follows directly from this one, and you can leave the resources created in place.

To remove all the resources we created in this project, follow the following steps in your dev environment:

1. Make sure you are on the main branch by running git checkout main and git pull to ensure you have the latest changes, then run terraform destroy, and type yes to confirm - this will remove the VPC we created

2. To delete all the bootstrapping resourced, first change into the directory by running cd _bootstrap. Before we can delete everything, we need to update our S3 state file bucket. We need to change the lifecycle policy to allow the deletion, and add force_destroy = true to also delete all the objects in the bucket. Edit _bootstrap/state_file_resources.tf, and replace the first aws_s3_bucket resource with:

   1
   2
   3
   4
   5
   6
   7
   8
   9
   

   # Bucket used to store our state file
   resource "aws_s3_bucket" "state_file" {
   bucket = var.state_file_bucket_name
   force_destroy = true
   
   lifecycle {
   prevent_destroy = false
   }
   }
   

3. Run terraform apply, and accept the changes.

4. Now run terraform destroy, and accept the changes. This will result in two errors since we are deleting the S3 bucket where it tries to store the updated state file, and also the DynamoDB table Terraform uses to store the lock to prevent parallel runs. The output will look similar to this:

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12
   13
   14
   15
   16
   17
   18
   19
   20
   21
   22
   23
   24
   25
   26
   27
   28
   29
   30
   31
   32
   33
   34
   

   │ Error: Failed to save state
   │
   │ Error saving state: failed to upload state: NoSuchBucket: The specified bucket does not exist
   │ status code: 404, request id: VJDXS21J9YFQ2J5J, host id: aG3pXy1Kfx2jncT1js0iDL5d+5j/rf3mNDVNzRp7aYpa3bCkAIKKJDh8HJQymS2prphHrazmjmo=
   ╵
   ╷
   │ Error: Failed to persist state to backend
   │
   │ The error shown above has prevented Terraform from writing the updated state to the configured backend. To allow for recovery, the state has been written to the file "errored.tfstate" in the current working directory.
   │
   │ Running "terraform apply" again at this point will create a forked state, making it harder to recover.
   │
   │ To retry writing this state, use the following command:
   │ terraform state push errored.tfstate
   │
   ╵
   ╷
   │ Error: Error releasing the state lock
   │
   │ Error message: failed to retrieve lock info: ResourceNotFoundException: Requested resource not found
   │
   │ Terraform acquires a lock when accessing your state to prevent others
   │ running Terraform to potentially modify the state at the same time. An
   │ error occurred while releasing this lock. This could mean that the lock
   │ did or did not release properly. If the lock didn't release properly,
   │ Terraform may not be able to run future commands since it'll appear as if
   │ the lock is held.
   │
   │ In this scenario, please call the "force-unlock" command to unlock the
   │ state manually. This is a very dangerous operation since if it is done
   │ erroneously it could result in two people modifying state at the same time.
   │ Only call this command if you're certain that the unlock above failed and
   │ that no one else is holding a lock.
   

5. Lastly, we need to delete the project we created in CodeCatalyst. In the left-hand navigation, go to Project settings, click on Delete project, and follow the instructions to delete the project.

Conclusion

Congratulations! You've now bootstrapped Terraform with CodeCatalyst, and can deploy any infrastructure changes using a pull request workflow. If you enjoyed this tutorial, found an issues, or have feedback for us, please send it our way!