Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BigQuery Worker Wait Adjustments #515

Merged

Conversation

blackwire
Copy link
Collaborator

Removes unnecessary exponential backoff as it's max wait was 5 minutes as written until it kicked off a waiter that polled every minute. The logic has been simplified to run, if it's not complete within 5 seconds then queue it for the waiter job to pick it up again in 30 seconds, and if the waiter job finds it's still incomplete it will queue it for another minute. It checks for errors before queueing it in any situation.

Also, this adds location (where the job was originally run) to the queue message so that when it's picked up it will look for the job in that location and none others. There was an issue where in some instances this was choosing the wrong region or polling both regions for the job_id which returned an error for one of them and threw off the logic.

Robert McMahan added 3 commits July 31, 2024 21:41
…o error then enqueue to check again in 30 seconds. Any kind of exponential backoff or wait period here is unecessary and duplicative. Just let the queue handle the wait if the response is not more or less immediate (within 5 seconds).
…e looking at multiple regions at times and this results in an error for the region where the job does not exist thus confusing the current logic.
Copy link
Collaborator

@jmistral-google jmistral-google left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@blackwire blackwire merged commit 26d1bd6 into google-marketing-solutions:master Aug 20, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants