Although Workflows provides a Google Cloud Storage connector, it is not yet possible to read and write files with its API. However, it's possible to use GCS's REST JSON API directly for that purpose, from your workflow definition.
Note: Your workflow should be deployed with a service account that has Google Cloud Storage Object Viewer permission.
As shown in gcs-write-workflow.yaml,
you can write into the data.json
file as follows:
- write_to_gcs:
call: http.post
args:
url: ${"https://storage.googleapis.com/upload/storage/v1/b/" + bucket + "/o"}
auth:
type: OAuth2
query:
name: data.json
body:
greeting: "Hello World"
year: 2022
The body contains your structured data in YAML, but it is converted into JSON as we write into a JSON file.
In order to read from a file in a GCS bucket, we take the same approach, as shown in gcs-read-workflow.yaml:
- read_from_gcs:
call: http.get
args:
url: ${"https://storage.googleapis.com/download/storage/v1/b/" + bucket + "/o/" + name}
auth:
type: OAuth2
query:
alt: media
result: data_json_content
- return_content:
return: ${data_json_content.body}
The data_json_content.body
value is an array or dictionary (your JSON data)
that you can use directly elsewhere in your workflow definition.
Currently, Workflows doesn't (yet!) support custom environment variables. To avoid hard-coding environment-specific information like a service URL, you can store and load configuration files as JSON documents in cloud storage, and access the keys and values in the JSON file from within your workflow, as shown in this example.
You can define a reusable subworkflow tha reads and loads the JSON file from GCS:
read_env_from_gcs:
params: [bucket, object]
steps:
- read_from_gcs:
call: http.get
args:
url: ${"https://storage.googleapis.com/download/storage/v1/b/" + bucket + "/o/" + object}
auth:
type: OAuth2
query:
alt: media
result: env_file_json_content
- return_content:
return: ${env_file_json_content.body}
Calling this subworkflow will return the JSON content, usable from your workflow.
First, you'll need to call this subworkflow,
and store the result in env-details
variable:
- load_env_details:
call: read_env_from_gcs
args:
bucket: workflow_environment_info
object: env-info.json
result: env_details
And then, if the JSON file contains a SERVICE_URL
key/value pair,
you can access as follows thanks to dollar expressions:
- call_service:
call: http.get
args:
url: ${env_details.SERVICE_URL}
result: service_result