When you receive a heavy burst of traffic, it's tempting to create a high number of concurrent executions and let Workflows deal with them. This works until you hit the quotas Workflows enforce. For example, Workflows enforces 2000 maximum number of concurrent workflow executions per region, per project. Once the limit is reached, new executions fail with an HTTP 429.
Thankfully, a Cloud Tasks queue can help. Instead of creating Workflows executions directly, you can add tasks to a Cloud Tasks queue and let Cloud Tasks drain that queue at a rate you define. This way, you can better utilize your workflow quota.
In this example, we show how a parent workflow can create tasks to execute a child workflow at a specified rate using a Cloud Tasks queue in the middle.
First, create a Cloud Tasks queue with no rate limits:
QUEUE=queue-workflow-child
LOCATION=us-central1
gcloud tasks queues create queue-workflow-child --location=$LOCATION
Next, let's create a child workflow in workflow-child.yaml.
The child workflow receives an iteration
argument from the parent workflow. It
waits 10 seconds to simulate doing work and returns a string as result:
main:
params: [args]
steps:
- init:
assign:
- iteration : ${args.iteration}
- wait:
call: sys.sleep
args:
seconds: 10
- return_message:
return: ${"Hello world"+iteration}
Deploy the child workflow:
gcloud workflows deploy workflow-child --source=workflow-child.yaml --location=$LOCATION
Next, create a parent workflow in workflow-parent.yaml.
The workflow assigns some constants first. Note that it's referring to the child workflow and the queue name between the parent and child workflows:
main:
steps:
- init:
assign:
- project_id: ${sys.get_env("GOOGLE_CLOUD_PROJECT_ID")}
- project_number: ${sys.get_env("GOOGLE_CLOUD_PROJECT_NUMBER")}
- location: ${sys.get_env("GOOGLE_CLOUD_LOCATION")}
- workflow_child_name: "workflow-child"
- queue_name: "queue-workflow-child"
In the next step, it adds a high number of tasks to the Cloud Tasks queue to execute the child workflow:
- enqueue_tasks_to_execute_child_workflow:
for:
value: iteration
range: [1, 100]
steps:
- iterate:
assign:
- data:
iteration: ${iteration}
- exec:
# Need to wrap into argument for Workflows args.
argument: ${json.encode_to_string(data)}
- create_task_to_execute_child_workflow:
call: googleapis.cloudtasks.v2.projects.locations.queues.tasks.create
args:
parent: ${"projects/" + project_id + "/locations/" + location + "/queues/" + queue_name}
body:
task:
httpRequest:
body: ${base64.encode(json.encode(exec))}
url: ${"https://workflowexecutions.googleapis.com/v1/projects/" + project_id + "/locations/" + location + "/workflows/" + workflow_child_name + "/executions"}
oauthToken:
serviceAccountEmail: ${project_number + "[email protected]"}
Deploy the parent workflow:
gcloud workflows deploy workflow-parent --source=workflow-parent.yaml --location=$LOCATION
Time to execute the parent workflow:
gcloud workflows run workflow-parent --location=$LOCATION
As the parent workflow is running, you can see parallel executions of the child workflow, all executed roughly around the same:
In this case, 100 executions is a piece of cake for Workflows, but you can run into quota issues, if you submit 1000s of executions all at once. This in when Cloud Tasks queue and its rate limits become useful.
Let's now apply a rate limit to the Cloud Tasks queue. In this case, 1 dispatch per second:
gcloud tasks queues update $QUEUE --max-dispatches-per-second=1 --location=$LOCATION
Execute the parent workflow again:
gcloud workflows run workflow-parent --location=$LOCATION
This time, you see a more smooth execution rate (1 execution request per second):
Playing with different dispatch rates and concurrency settings of Cloud Tasks queues, you can better utilize your quota and stay below the limits without triggering unnecessary quota related failures.