Terraform sync tool #290

danieldeleo · 2022-06-08T18:29:44Z

No description provided.

google-cla · 2022-06-08T18:29:50Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

tools/terraform_sync_tool/deploy.sh

tools/terraform_sync_tool/modules/terraform-sync-tool/main.tf

tools/terraform_sync_tool/qa/terragrunt.hcl

tools/terraform_sync_tool/state.json

tools/terraform_sync_tool/terraform_sync.py

Co-authored-by: Daniel De Leo <[email protected]>

tools/terraform_sync_tool/README.md

tools/terraform_sync_tool/terraform_sync.py

danieldeleo · 2022-07-27T15:40:39Z

tools/terraform_sync_tool/README.md

+    ├── modules                     # Terraform modules directory
+    │   ├── bigquery                # Example Terraform BigQuery Setup
+    │   └── ...                     # Other modules setup you have
+    ├── qa                          # qa environment directory
+    │   ├── terragrunt.hcl      
+    │   └── terraform-sync-tool     # Tool terraform-sync-tool
+    │           ├── json_schemas    # Terraform schema files 
+    │           ├── terragrunt.hcl
+    │           └── ...
+    ├── cloudbuild.yaml             # Cloud Build configuration file
+    ├── deploy.sh                   # Build Step 0 - contains terragrunt commands


All of these should go inside an example/ directory

Updated in 5496102

danieldeleo · 2022-07-27T15:55:34Z

tools/terraform_sync_tool/README.md

+This directory contains the setup for the Terraform Sync Tool. Terraform Sync Tool was designed to address the schema drifts in BigQuery tables and keep the 
+Terraform schemas up-to-date with the BigQuery table schemas in production environment. Schema drifts occurred when BigQuery Table schemas are updated by newly 
+ingested data while Terraform schema files contain the outdated schemas. Therefore, this tool will detect the schema drifts, trace the origins of the drifts, and alert
+developers/data engineers.


Suggested change

This directory contains the setup for the Terraform Sync Tool. Terraform Sync Tool was designed to address the schema drifts in BigQuery tables and keep the

Terraform schemas up-to-date with the BigQuery table schemas in production environment. Schema drifts occurred when BigQuery Table schemas are updated by newly

ingested data while Terraform schema files contain the outdated schemas. Therefore, this tool will detect the schema drifts, trace the origins of the drifts, and alert

developers/data engineers.

This directory contains the Terraform Sync Tool. This tool intentionally fails your CI/CD pipeline when schema drifts occur between what your BigQuery Terraform resources declare and what's actually present in your BigQuery environment. Theses schema drifts happen when BigQuery tables are updated by processes outside of Terraform (ETL process may dynamically add new columns when loading data into BigQuery). When drifts occur, you end up with outdated BigQuery Terraform resource files. This tool detects the schema drifts, traces the origins of the drifts, and alerts developers/data engineers (by failing the CI/CD pipeline) so they can patch the Terraform in their current commit.

updated in 7f469e4

danieldeleo · 2022-07-27T15:56:05Z

tools/terraform_sync_tool/README.md

+ingested data while Terraform schema files contain the outdated schemas. Therefore, this tool will detect the schema drifts, trace the origins of the drifts, and alert
+developers/data engineers.
+
+The Terraform Schema Sync Tool fails the build attemps if resource drifts are detected and notifies the latest resource information. Developers and data engineers should be able to update the Terraform resources accordingly. 


Suggested change

The Terraform Schema Sync Tool fails the build attemps if resource drifts are detected and notifies the latest resource information. Developers and data engineers should be able to update the Terraform resources accordingly.

Updated in 7f469e4

danieldeleo · 2022-07-27T16:01:23Z

tools/terraform_sync_tool/README.md

+The Terraform Schema Sync Tool fails the build attemps if resource drifts are detected and notifies the latest resource information. Developers and data engineers should be able to update the Terraform resources accordingly. 
+
+Terraform Sync Tool can be integrated into your CI/CD pipeline. You'll need to add two steps to CI/CD pipeline. 
+- Step 0: Use Terraform/erragrunt command to detect resource drifts and write output into a JSON file


Suggested change

- Step 0: Use Terraform/erragrunt command to detect resource drifts and write output into a JSON file

- Step 0: Run the Terraform plan command (using either Terraform/Terragrunt) with the `-json` option and write the output into a JSON file using the caret operator `> output.json`

Updated in 7f469e4

danieldeleo · 2022-07-27T16:05:26Z

tools/terraform_sync_tool/README.md

+
+## How to run Terraform Schema Sync Tool
+
+#### Use Terraform/Terragrunt commands to test if any resources drifts existed


All these should be 3rd level headers (###), not 4 (####)

Updated in 5496102

danieldeleo · 2022-07-27T16:11:46Z

tools/terraform_sync_tool/README.md

+- Step 0: Use Terraform/erragrunt command to detect resource drifts and write output into a JSON file
+- Step 1: Use Python scripts to identify and investigate the drifts
+
+## How to run Terraform Schema Sync Tool


Suggested change

## How to run Terraform Schema Sync Tool

## How to run Terraform Schema Sync Tool

```bash

###############

# Using Terragrunt

###############

terragrunt run-all plan -json --terragrunt-non-interactive > plan_output.json

python3 terraform_sync.py plan_output.json

##############

# Using Terraform

##############

terraform plan -json > plan_output.json

python3 terraform_sync.py plan_output.json

Updated in 5496102 and 34f1461

danieldeleo · 2022-07-27T16:13:35Z

tools/terraform_sync_tool/README.md

+Terragrunt/Terraform commands:
+```
+terragrunt run-all plan -json --terragrunt-non-interactive
+
+# Terraform Command
+terraform plan -json
+```
+
+After running the Terrform plan command, **the event type "resource_drift"("type": "resource_drift") indicates a drift has occurred**.
+If drifts detected, please update your terraform configurations and address the resource drifts based on the event outputs.
+
+
+#### Add Could Build Steps to your configuration file
+
+Please check cloud build steps in `cloudbuild.yaml` file, and add these steps to your Cloud Build Configuration File.
+
+- step 0: run terraform commands in `deploy.sh` to detects drifts
+
+Add `deploy.sh` to your project directory. 
+
+- step 1: run python scripts to investigate terraform output
+
+Add `requirements.txt` and `terraform_sync.py` to your project directory.
+
+#### (Optional if you haven't created Cloud Build Trigger) Create and configure a new Trigger in Cloud Build
+Make sure to indicate your cloud configuration file location correctly.
+
+#### That's all you need! Let's commit and test in CLoud Build!
+


Suggested change

Terragrunt/Terraform commands:

```

terragrunt run-all plan -json --terragrunt-non-interactive

# Terraform Command

terraform plan -json

```

After running the Terrform plan command, **the event type "resource_drift"("type": "resource_drift") indicates a drift has occurred**.

If drifts detected, please update your terraform configurations and address the resource drifts based on the event outputs.

#### Add Could Build Steps to your configuration file

Please check cloud build steps in `cloudbuild.yaml` file, and add these steps to your Cloud Build Configuration File.

- step 0: run terraform commands in `deploy.sh` to detects drifts

Add `deploy.sh` to your project directory.

- step 1: run python scripts to investigate terraform output

Add `requirements.txt` and `terraform_sync.py` to your project directory.

#### (Optional if you haven't created Cloud Build Trigger) Create and configure a new Trigger in Cloud Build

Make sure to indicate your cloud configuration file location correctly.

#### That's all you need! Let's commit and test in CLoud Build!

These are not required to run this tool. This is more related to running the example you provide with this tool.

Updated in 5496102

candicehou07 added 12 commits May 10, 2022 20:53

inital commit

cf9653f

create sample dataset in BigQeury

fbca216

add cloudbuild.yaml

313418f

table json schema

c1ccc83

Delete bq_schema.json

1a69c02

Delete BatchControlLog.json

feb176f

Merge branch 'GoogleCloudPlatform:master' into terraform_sync_tool

7c42756

terraform sync tool

d94c55b

change dir

345ab63

specify dir from yaml file

3cfcbc2

add step1 to run python scripts

8ae5b48

Delete auto generated contents

4d5ac73