Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rate limiting issues with users_list call #234

Open
Limess opened this issue Nov 19, 2020 · 1 comment
Open

Rate limiting issues with users_list call #234

Limess opened this issue Nov 19, 2020 · 1 comment

Comments

@Limess
Copy link

Limess commented Nov 19, 2020

We're seeing issues starting up services due to rate limiting by slack on the users listing calls.

The current retry behaviour is fairly aggressive and retries 10 times with an initial delay of 0.2 seconds.

Slack send a Retry-After header when rate-limiting and suggest their rate limiting is minutely, but that header is not accessible using the api_call method on the slackclient library (at least in v1, I haven't checked v2).

Ideally this would respect that header and retry after a minute or so.

Logs

On startup:

2020-11-19T13:38:46 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 0.20s (attempt 1 of 10)
2020-11-19T13:38:46 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 0.20s (attempt 1 of 10)
2020-11-19T13:38:46 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 0.40s (attempt 2 of 10)
2020-11-19T13:38:46 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 0.40s (attempt 2 of 10)
2020-11-19T13:38:47 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 0.60s (attempt 3 of 10)
2020-11-19T13:38:47 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 0.60s (attempt 3 of 10)
2020-11-19T13:38:48 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 0.80s (attempt 4 of 10)
2020-11-19T13:38:48 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 0.80s (attempt 4 of 10)
2020-11-19T13:38:49 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 1.00s (attempt 5 of 10)
2020-11-19T13:38:50 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 1.00s (attempt 5 of 10)
2020-11-19T13:38:50 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 1.20s (attempt 6 of 10)
2020-11-19T13:38:51 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 1.20s (attempt 6 of 10)
2020-11-19T13:38:51 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 1.40s (attempt 7 of 10)
2020-11-19T13:38:52 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 1.40s (attempt 7 of 10)
2020-11-19T13:38:53 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 1.60s (attempt 8 of 10)
2020-11-19T13:38:55 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 1.80s (attempt 9 of 10)
2020-11-19T13:38:57 - default/backend/050d686b89ce41e98c5aeac3e42fea29 - WARNING:root:Retrying request to users.list after error ratelimited. Backing off 2.00s (attempt 10 of 10)

During this time it seems the server is not available, which with large backoffs also result in our healthcheck killing the starting service after some timeout.

@Limess
Copy link
Author

Limess commented Nov 19, 2020

On further investigation, it seems this is only on startup, prior to the server starting, due to INCIDENT_BOT_ID not being set. We've now set this and it has helped alleviate this issue as it avoids that initial lookup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant