This guide explains how to orchestrate a serverless scheduled data load from Cloud Storage and transform the data in BigQuery using Workflows, Cloud Functions, and Firestore.
For detailed steps, refer to the tutorial: Load data from Cloud Storage to BigQuery using Workflows.
Contents of this repository:
main.tf: Terraform template to set up the demo.file_change_handler: Cloud Function trigger (Python 3.7) to handleobject finalizedevents from Cloud Storage.workflow_handlers: Cloud Functions to handle BigQuery jobs and the workflow YAML.generator: Script (Python 3.7) to generate AVRO files and upload to a Cloud Storage bucket.