From 8045ea167533e5c157a1e165ead1c3107542d00f Mon Sep 17 00:00:00 2001 From: stayseesong Date: Mon, 28 Oct 2024 09:32:04 -0700 Subject: [PATCH 1/3] [netlify-build] --- src/_data/sidenav/main.yml | 8 +- .../data-graph/setup-guides/BigQuery-setup.md | 111 ++++++++++++------ .../data-graph/setup-guides/redshift-setup.md | 2 +- .../setup-guides/snowflake-setup.md | 2 +- 4 files changed, 80 insertions(+), 43 deletions(-) diff --git a/src/_data/sidenav/main.yml b/src/_data/sidenav/main.yml index 26bfcfcd9a..8b6fd03183 100644 --- a/src/_data/sidenav/main.yml +++ b/src/_data/sidenav/main.yml @@ -358,10 +358,12 @@ sections: - section_title: Setup Guides slug: /unify/data-graph/setup-guides/ section: - - path: /unify/data-graph/setup-guides/snowflake-setup/ - title: Snowflake Setup + - path: /unify/data-graph/setup-guides/bigquery-setup/ + title: BigQuery Data Graph Setup - path: /unify/data-graph/setup-guides/databricks-setup/ - title: Databricks Setup + title: Databricks Data Graph Setup + - path: /unify/data-graph/setup-guides/snowflake-setup/ + title: Snowflake Data Graph Setup - section_title: Linked Events slug: /unify/data-graph/linked-events/ section: diff --git a/src/unify/data-graph/setup-guides/BigQuery-setup.md b/src/unify/data-graph/setup-guides/BigQuery-setup.md index e9636b7864..d7dd5ade29 100644 --- a/src/unify/data-graph/setup-guides/BigQuery-setup.md +++ b/src/unify/data-graph/setup-guides/BigQuery-setup.md @@ -1,54 +1,89 @@ --- -title: BigQuery Setup +title: BigQuery Data Graph Setup beta: true plan: unify -hidden: true redirect_from: - '/unify/linked-profiles/setup-guides/BigQuery-setup' --- > info "" -> At this time, you can only use BigQuery with Linked Events. +> BigQuery for Data Graph is in beta and Segment is actively working on this feature. Some functionality may change before it becomes generally available. This feature is governed by Segment’s [First Access and Beta Preview Terms](https://www.twilio.com/en-us/legal/tos){:target="_blank"}. -On this page, you'll learn how to connect your BigQuery data warehouse to Segment. +Set up your BigQuery data warehouse to Segment for the [Data Graph](/docs/unify/data-graph/data-graph/). -## Set up BigQuery - +## Step 1: Roles and permissions > warning "" -> You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset. - -To set up the Segment BigQuery connector: +> You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset. -1. Navigate to **IAM & Admin > Service Accounts** in BigQuery. +To set the roles and permissions: +1. Navigate to **IAM & Admin > Service Accounts** in BigQuery. 2. Click **+ Create Service Account** to create a new service account. -3. Enter your **Service account name** and a description of what the account will do. +3. Enter your Service account name and a description of what the account will do. 4. Click **Create and Continue**. -5. In the **Grant this service account access to project** section, select the [*BigQuery User*](https://cloud.google.com/bigquery/docs/access-control#bigquery.user){:target="_blank"} role to add. -6. Click **+ Add another role** and add the *BigQuery Job User* role. -7. Click **+ Add another role** and add the [*BigQuery Metadata Viewer*](https://cloud.google.com/bigquery/docs/access-control#bigquery.metadataViewer){:target="_blank"} role. -8. Click **Continue**, then click **Done**. -9. Search for the service account you've just created. -11. From your service account, click the three dots under **Actions** and select **Manage keys**. -12. Click **Add Key > Create new key**. -13. In the pop-up window, select **JSON** for the key type, and click **Create**. -14. Copy all the content within the file you've created and downloaded. -15. Navigate to Segment and paste all the credentials you've just copied into the **Enter your credentials** section as you connect your warehouse destination. - -## Grant access to datasets and tables for enrichment - -Grant access to datasets and tables so that Segment can list datasets, tables, and columns, and create Linked Events. - -Grant -- [`BigQuery Data Viewer`](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataViewer){:target="_blank"} role
-OR -- Permissions: - - `bigquery.datasets.get` - - `bigquery.tables.list` - - `bigquery.tables.get` - - `bigquery.tables.getData` - -These can be scoped to projects or [datasets](https://cloud.google.com/bigquery/docs/control-access-to-resources-iam#grant_access_to_a_dataset){:target="_blank"}. +5. Click **+ Add another role** and add the *[BigQuery User](https://cloud.google.com/bigquery/docs/access-control#bigquery.user){:target="_blank"}* role. +6. Click **Continue**, then click **Done**. +7. Search for the service account you just created. +8. From your service account, click the three dots under **Actions** and select **Manage keys**. +9. Navigate to **Add Key > Create new key**. +10. In the pop-up window, select **JSON** for the key type, and click **Create**. The file will download. +11. Copy all the content in the JSON file you created in the previous step, and save it for Step 5. + + +## Step 2: Grant read-only access for the Data Graph +Grant the [BigQuery Data Viewer](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataViewer){:target="_blank"} role to the service account at the project level. Make sure to grant read-only access to the Profiles Sync project in case you have a separate project. + +To grant read-only access for the Data Graph: +1. Navigate to **IAM & Admin > IAM** in BigQuery. +2. Search for the service account you just created. +3. From your service account, click the **Edit principals pencil**. +4. Click **ADD ANOTHER ROLE**. +5. Select the **BigQuery Data Viewer role**. +6. Click **Save**. + +## *(Optional)* Step 3: Restrict read-only access +If you want to restrict access to specific datasets, grant the BigQuery Data Viewer role on datasets to the service account. Make sure to grant read-only access to the Profiles Sync dataset. + +To restrict read-only access: +1. In the Explorer pane in BigQuery, expand your project and select a dataset. +2. Navigate to **Sharing > Permissions**. +3. Click **Add Principal**. +4. Enter your service account in the New principals section. +5. Select the **BigQuery Data Viewer** role in the **Select a role** section. +6. Click **Save**. + +You can also run the following command: + +``` +GRANT `roles/bigquery.dataViewer` ON SCHEMA `YOUR_DATASET_NAME` TO "serviceAccount:"; +``` + +## Step 4: Validate permissions +1. Navigate to **IAM & Admin > Service Accounts** in BigQuery. +2. Search for the service account you’ve just created. +3. From your service account, click the three dots under **Actions** and select **Manage permissions**. +4. Click **View Access** and click **Continue**. +5. Select a box with List resources within resource(s) matching your query. +6. Click **Analyze**, then click **Run query**. + +## Step 5: Connect your warehouse to Segment +1. Navigate to **Unify > Data Graph** in Segment. This should be a Unify space with Profiles Sync already set up. +2. Click **Connect warehouse**. +3. Select *BigQuery* as your warehouse type. +4. Enter your warehouse credentials. Segment requires the following settings to connect to your BigQuery warehouse: + * **Service Account Credentials:** JSON credentials for a GCP Service Account that has BigQuery read/write access. This is the credential created in Step 1. + * **Data Location:** This specifies the primary data location. This can be either region or multi-region. +5. Test your connection, then click **Save**. + +## Update user access for Segment Reverse ETL dataset +If you ran Segment Reverse ETL in the project you are configuring as the Segment connection project, a Segment-managed dataset is already created and you need to provide the new Segment user access to the existing dataset. + +Run the following SQL if you run into an error on the Segment app indicating that the user doesn’t have sufficient privileges on an existing `__segment_reverse_etl` dataset: + +``` +GRANT `roles/bigquery.dataEditor` ON SCHEMA `__segment_reverse_etl` TO "serviceAccount:"; +``` + +Grant the [BigQuery Data Editor](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataEditor){:target="_blank"} role on the `__segment_reverse_etl` dataset to the service account. Note that the `__segment_reverse_etl` dataset is hidden in the console. + -> info "" -> To create Linked Events on your listed tables, Segment needs `bigquery.tables.get` and `bigquery.tables.getData` at dataset level. However, you can still scope `bigquery.tables.get` and `bigquery.tables.getData` to specific tables. See BigQuery's [docs](https://cloud.google.com/bigquery/docs/control-access-to-resources-iam#grant_access_to_a_table_or_view){:target="_blank"} for more info. diff --git a/src/unify/data-graph/setup-guides/redshift-setup.md b/src/unify/data-graph/setup-guides/redshift-setup.md index 84fe8db4da..a6da05fd3e 100644 --- a/src/unify/data-graph/setup-guides/redshift-setup.md +++ b/src/unify/data-graph/setup-guides/redshift-setup.md @@ -1,5 +1,5 @@ --- -title: Redshift Setup +title: Redshift Data Graph Setup beta: true plan: unify hidden: true diff --git a/src/unify/data-graph/setup-guides/snowflake-setup.md b/src/unify/data-graph/setup-guides/snowflake-setup.md index 4a69eda7b6..f732e4adc5 100644 --- a/src/unify/data-graph/setup-guides/snowflake-setup.md +++ b/src/unify/data-graph/setup-guides/snowflake-setup.md @@ -1,5 +1,5 @@ --- -title: Snowflake Setup +title: Snowflake Data Graph Setup plan: unify redirect_from: - '/unify/linked-profiles/setup-guides/snowflake-setup' From 35a08d0446907cb9d90af616de7fa0522e3c07ad Mon Sep 17 00:00:00 2001 From: stayseesong Date: Mon, 28 Oct 2024 13:53:43 -0700 Subject: [PATCH 2/3] [netlify-build] --- src/unify/data-graph/setup-guides/BigQuery-setup.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/src/unify/data-graph/setup-guides/BigQuery-setup.md b/src/unify/data-graph/setup-guides/BigQuery-setup.md index d7dd5ade29..f62f07e986 100644 --- a/src/unify/data-graph/setup-guides/BigQuery-setup.md +++ b/src/unify/data-graph/setup-guides/BigQuery-setup.md @@ -78,12 +78,8 @@ GRANT `roles/bigquery.dataViewer` ON SCHEMA `YOUR_DATASET_NAME` TO "serviceAccou ## Update user access for Segment Reverse ETL dataset If you ran Segment Reverse ETL in the project you are configuring as the Segment connection project, a Segment-managed dataset is already created and you need to provide the new Segment user access to the existing dataset. -Run the following SQL if you run into an error on the Segment app indicating that the user doesn’t have sufficient privileges on an existing `__segment_reverse_etl` dataset: +Grant the [BigQuery Data Editor](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataEditor){:target="_blank"} role on the `__segment_reverse_etl` dataset to the service account if you run into an error on the Segment app indicating that the user doesn’t have sufficient privileges on an existing `__segment_reverse_etl` dataset. Note that the `__segment_reverse_etl` dataset is hidden in the console. Run the following SQL command: ``` GRANT `roles/bigquery.dataEditor` ON SCHEMA `__segment_reverse_etl` TO "serviceAccount:"; ``` - -Grant the [BigQuery Data Editor](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataEditor){:target="_blank"} role on the `__segment_reverse_etl` dataset to the service account. Note that the `__segment_reverse_etl` dataset is hidden in the console. - - From 775a4481a01a4c8f25bda6f89a4a295698ecdf87 Mon Sep 17 00:00:00 2001 From: stayseesong <83784848+stayseesong@users.noreply.github.com> Date: Mon, 4 Nov 2024 09:28:11 -0800 Subject: [PATCH 3/3] Update src/unify/data-graph/setup-guides/BigQuery-setup.md --- src/unify/data-graph/setup-guides/BigQuery-setup.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/unify/data-graph/setup-guides/BigQuery-setup.md b/src/unify/data-graph/setup-guides/BigQuery-setup.md index f62f07e986..1ffc64f459 100644 --- a/src/unify/data-graph/setup-guides/BigQuery-setup.md +++ b/src/unify/data-graph/setup-guides/BigQuery-setup.md @@ -78,7 +78,7 @@ GRANT `roles/bigquery.dataViewer` ON SCHEMA `YOUR_DATASET_NAME` TO "serviceAccou ## Update user access for Segment Reverse ETL dataset If you ran Segment Reverse ETL in the project you are configuring as the Segment connection project, a Segment-managed dataset is already created and you need to provide the new Segment user access to the existing dataset. -Grant the [BigQuery Data Editor](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataEditor){:target="_blank"} role on the `__segment_reverse_etl` dataset to the service account if you run into an error on the Segment app indicating that the user doesn’t have sufficient privileges on an existing `__segment_reverse_etl` dataset. Note that the `__segment_reverse_etl` dataset is hidden in the console. Run the following SQL command: +If you run into an error on the Segment app indicating that the user doesn’t have sufficient privileges on an existing `__segment_reverse_etl` dataset, grant the [BigQuery Data Editor](https://cloud.google.com/bigquery/docs/access-control#bigquery.dataEditor){:target="_blank"} role on the `__segment_reverse_etl` dataset to the service account . Note that the `__segment_reverse_etl` dataset is hidden in the console. Run the following SQL command: ``` GRANT `roles/bigquery.dataEditor` ON SCHEMA `__segment_reverse_etl` TO "serviceAccount:";