-
Notifications
You must be signed in to change notification settings - Fork 351
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
JENKINS-64418: add simple exponential backoff to the sleep for bitbucket api rate limits #414
base: master
Are you sure you want to change the base?
Conversation
7ffd348
to
35adbab
Compare
Looking at Atlassians documentation for "Adjusting your code for rate limiting" I would argue that, of the three alternative ways proposed there, the 3rd option makes the most sense here. The 3rd option states that one can adjust the requests sent based on the request limit http headers.
I argue that this would be the way to go, due to the two reasons mentioned in the documentation about exponential backoff
|
@artheus Also, take a look at #187 (comment). There might be a way to mitigate the behavior in some cases by using caching. |
@bitwiseman I will start working on a PR for this.
I think that the scope for this issue should be just to add a way to properly comply with the Bitbucket rate limits. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need a test that exercises this.
@@ -866,7 +869,8 @@ protected CloseableHttpResponse executeMethod(HttpHost host, HttpRequestBase htt | |||
httpMethod.setConfig(requestConfig.build()); | |||
|
|||
CloseableHttpResponse response = client.execute(host, httpMethod, requestContext); | |||
while (response.getStatusLine().getStatusCode() == API_RATE_LIMIT_CODE) { | |||
Random random = new Random(); | |||
for (int attempts = 0; response.getStatusLine().getStatusCode() == API_RATE_LIMIT_CODE; attempts++) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The minimum sleep should remain 5 seconds. Once the limit is hit, the client needs to a wait at least on the order of seconds before trying again.
The maximum around 2 or 3 minutes sounds good.
We don't currently limit the number of attempts. That shouldn't change. We should hit the max and stay there.
I see your point about a caching reverse-proxy. In the best of worlds, people would implement an additional system that gave them more control. However, most Jenkins systems do not exist in the best of worlds. Most don't have what you are describing. As such, we have to look to see if there are any simple changes that can be made in this plugin to mitigate this issue, such as turning on local caching provided by the client http library. OkHttp makes adding local, on-disk, size constrained caching is relatively straightforward. It looks like Apache Take a look at the github-branch-source implementation. There is a cache for each credential (since different users are likely to see different results) and each cache is size-limited on-disk. The model here is that queries are always sent to the server (with an ETag from any previous result), the server then either returns new data or informs the client that their cache is still valid. Cache invalidation is done automatically and there is no timeout: if there's a result that hasn't changed in a day or a week, as long as it is still valid it can be used.
As synchronization job? I'm not sure what you mean, but the most common scenario overall is that some mixture query results are valid and caches are used and some are not and new data is retrieved. In anycase... In the meanwhile, I've requested a few changes to this PR and will happily merge this a first step once they are made. Thanks for contributing! |
@artheus, that article you linked to from Atlassian is for JIRA, not Bitbucket. I logged all the headers that come back from the request that has the 429 response and there's nothing that says
I thought about |
When we have hit the rate limits for Bitbucket API it slows everything down to a crawl. Also other API requests (such as getting the Jenkinsfile for multibranch pipelines) are rate limited and slower.
Adding exponential backoff with a minimum of 100 ms and a max of 1 minute, should alleviate the constant retry loop hitting the limit every 5s.
See JENKINS-64418.
Your checklist for this pull request