adding code for bigquery policy tag extractor

scripts/policy_tag_extractor/README.md

@@ -0,0 +1,23 @@
+# BigQuery Policy Tag Extractor
+
+## Introduction
+This repo contains a bash script for extracting BigQuery policy tag information from a given dataset. The script will review all the tables in a dataset, and output the table name, column, name, and policy tag ID in a CSV format.
+
+## Instructions for use
+The simplest way to execute this script is to run it directly in Cloud Shell, but if needed it can be executed as part of a larger CI/CD pipeline or process.
+
+Before using, make sure to update the bash script with the dataset that needs to be reviewed.
+
+To exceute in Cloud Shell:
+1. Start a new session in the GCP project where your BigQuery data resides
+2. Open Cloud Shell
+3. Upload policy_tag_export.sh to the Cloud Shell environment
+4. Execute the script by running "bash policy_tag_export.sh"
+5. List the resources in Cloud Shell (ls) and verify that a file called "policy_tags.csv" was created
+6. Download the file
+
+## Considerations
+* Ensure either you or the service account executing the bash script has the bigquery.metadataViewer role to access the required level of information.
+* The extractor can identify specific policy tags on columns, but is limited to the information available to the bq command line tool. In it's current state, this is the full policy tag identifier:
+
+projects/<PROJECT_ID>/locations/<LOCATION>/taxonomies/<TAXONOMY_ID>/policyTags/<TAG_ID>

scripts/policy_tag_extractor/policy_tag_export.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+DATASET=""
+
+#write all tables in a dataset to a reference TXT file
+bq ls --max_results=10000 ${DATASET} | awk '{ print $1 }' | sed '1,2d' > table_list.txt
+
+#loop through each table and export policy tags (if any) to a CSV
+echo "Writing to CSV..."
+while IFS= read -r TABLE; do
+    TAG_COUNT="`bq show --schema ${DATASET}.${TABLE} | grep "policyTags" | wc -l`"
+
+    if [ "${TAG_COUNT}" -ge 1 ]
+    then
+        COLUMN=`bq show --format=prettyjson ${DATASET}.${TABLE} | jq '.schema.fields[] | select(.policyTags | length>=1)' | jq '.name'`
+        TAG_ID=`bq show --format=prettyjson ${DATASET}.${TABLE} | jq '.schema.fields[] | select(.policyTags | length>=1)' | jq '.policyTags.names[]'`
+        echo ${TABLE},${COLUMN},${TAG_ID} | tr -d '"'
+    fi
+done < table_list.txt >> policy_tags.csv
+echo "Done."