Skip to content

Latest commit

 

History

History

workflows-bigquery-load

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Load data from Cloud Storage to BigQuery using Workflows

This guide explains how to orchestrate a serverless scheduled data load from Cloud Storage and transform the data in BigQuery using Workflows, Cloud Functions, and Firestore.

For detailed steps, refer to the tutorial: Load data from Cloud Storage to BigQuery using Workflows.

Contents of this repository:

  • main.tf: Terraform template to set up the demo.
  • file_change_handler: Cloud Function trigger (Python 3.7) to handle object finalized events from Cloud Storage.
  • workflow_handlers: Cloud Functions to handle BigQuery jobs and the workflow YAML.
  • generator: Script (Python 3.7) to generate AVRO files and upload to a Cloud Storage bucket.