Sleep Easy: Is Your Terraform State File Securely Backed Up?

Introduction

So, you are working with terraform, paving your way to the cloud line by line, storing your terraform state file on an Azure Storage Account and feeling pretty good about yourself. When suddenly it dawns upon you that this file (or files) is getting pretty darn important. And if something happens to it, you don’t know what to do and then all of the sudden you stop feeling so good and start worrying. Have no fear! this post has everything you need to rid yourself of these worries. Let’s dive in.

What is a Terraform State File?

Terraform uses a state file usually called terraform.tfstate to track the state of your “real world” infrastructure. It consults the state file every time it needs to assess new changes to your infra and updates it every time your infra changes. Without it, you will not be able to manage your infra with terraform. For more details on the state file check out this link.

Why Should We Protect the State File?

If the state file is lost, Terraform loses its connection with your infrastructure. Without it, Terraform cannot accurately determine what exists and what needs to be created, modified, or deleted. This could lead to an unintentional duplication, deletion of resources or deployment errors, causing significant disruptions and potential downtime.

In summary, the state file is the guardian of your infrastructure’s health and integrity. Backing up and securing it ensures smooth operations, minimized risks, and peace of mind.

Storing Terraform State File in Azure Storage Account

Azure Storage Accounts offer a secure and accessible location for your Terraform state files. When you run Terraform, it interacts with Azure to fetch and update the state file. This ensures a consistent, secure, and centralized storage, while also enabling you to easily configure advanced protections and backup methods.

What Are We Protecting Against?

When securing your Terraform state file, it’s important to understand the potential risks we aim to mitigate:

1. Accidental Blob Deletion: Human errors such as inadvertent deletion of the state file can result in loss of infrastructure mappings, causing inconsistencies and operational disruptions.
2. Corruption: The state file can be corrupted due to software errors or hardware malfunctions, resulting in loss or damage to infrastructure information.
3. Region/AZ Failure: Infrastructure failures such as region or availability zone outages could make the state file inaccessible, disrupting your operations.
4. Accidental / Malicious Blob / Storage Account deletion – Protecting from cases where someone deleted the entire the storage account or the blob along with all of its versions.

By protecting the Terraform state file, you ensure robust resilience against these scenarios, safeguarding the consistency and security of your infrastructure.

Let’s Mitigate:

Risk Factor	Mitigation Option	Mitigation Details
Accidental Blob Deletion	Soft Delete for Blobs and Containers	Azure’s Soft Delete feature provides a safety net against accidental deletions. When Soft Delete is enabled, any deleted data is retained for a set period, allowing easy recovery.
Corruption	Blob Versioning	Azure Blob Versioning retains previous versions of a blob whenever it is modified, allowing the restoration of an earlier state in case of corruption.
Region/AZ Failure	Geo-Redundant Storage (GRS) Replication	GRS / GZRS replication ensures your data is stored in a secondary region that is geographically distant from your primary region. This mitigates the risk of region or AZ outages, providing high availability.
Accidental / Malicious Blob / Storage Account deletion	Azure Backup Vaulted Backup (Preview)	You can configure vaulted backup, a managed offsite data protection solution, to get protection against any accidental or malicious deletion of blobs or storage account. The backup data using vaulted backups is copied and stored in the Backup vault as per the schedule and frequency you define via the backup policy and retained as per the retention configured in the policy.

These features, when combined, provide a comprehensive solution to protect your Terraform state file against common risks, ensuring smooth and secure operations.

Configuring Each Mitigation – Soft Delete & Blob Versioning

In the screenshot above you can see how to configure soft delete for blobs and containers as well as blob versioning. Now, with blob versioning, we have 2 options: first is to keep all versions indefinitely, the other option is to delete versions after a certain number of days. Unfortunately, both of these options are not ideal for Terraform state file backup. Well, keeping all versions indefinitely is actually a good solution for recovery but not so good when it comes to cost savings. On the other hand, deleting versions after a certain number of days can leave us with no versions to fall back to or few versions that are not recoverable.

Configuring Each Mitigation – Custom Blob Versioning

Considering the above, we need something a bit different: a backup solution that will always keep an ‘n’ number of versions. How can we achieve that? well as of today there is no native way to do that on azure but with a simple PowerShell script and Azure Automation, we can do it! Let’s go over the ingredients:

• Azure Automation Account with a system assigned managed identity
• A PowerShell runbook that runs a PowerShell script that deletes old blob versions
• A schedule for running the script.
• A role assignment for the managed identity on the storage account containing the state file

Disclaimer – The code in this post is provided to you “as is” without warranty of any kind. ALWAYS test your code before using it in a production environment

I have already covered how to deploy the first 3 items in a previous post using Terraform. This can easily be done via the portal as well. The only part missing from the last post is the role assignment. We need to provide the Automation Account permissions to be able to read and delete old blob versions on the storage account. The least privileged built-in RBAC role to achieve that is Storage Blob Data Owner. If that’s too much for you, a custom role assignment can also be created to only allow listing and deleting of blob versions. But for the purpose of this demo, we will go with the built-in role.

Deploying the role assignment using Terraform:

data "azurerm_storage_account" "statesa" {
  name = "storageaccount"
  resource_group_name = "sa-rg"
}

resource "azurerm_role_assignment" "blob_owner" {
    scope = data.azurerm_storage_account.statesa.id
    role_definition_name = "Storage Blob Data Owner"
    principal_id = azurerm_automation_account.aa.identity[0].principal_id
}

The above code snippet maps statesa to your storage account then assigns the role to the managed identity of the automation account. If you follow the steps from the previous post, just add this piece of code to your tf file. The full code including the role assignment can be found here

Configuring Each Mitigation – GRS / GZRS Replication

As mentioned above GRS (Geo-redundant storage) and GZRS (Geo-zone-redundant storage) are two options for replicating data in Azure Storage accounts. Both options replicate data to a secondary region that is geographically distant from the primary region, providing protection against regional disasters. GZRS also replicates data synchronously across three Azure availability zones in the primary region, providing maximum consistency, durability, and availability. When dealing with a regional outage, you might need your state file availability to perform changes to your infrastructure. To configure your storage account for replication, checkout this link.

Configuring Each Mitigation – Vaulted Backup (Preview)

To configure vaulted backups, follow this guide. The guide also provides guidance to configure operational backups. make sure you follow the “Vaulted Backups” option.

Conclusion

There are several risk factors when it comes to protecting your data (in this case you terraform state files). By making sure we target each risk factor with the correct mitigation we reduce the risk of losing our progress completely. Let me know what you think in the comments! I hope this has been informative for you. See you in the next one!

Itamar