Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support rate limits for git clone #143

Open
andboss opened this issue Mar 3, 2024 · 0 comments
Open

support rate limits for git clone #143

andboss opened this issue Mar 3, 2024 · 0 comments
Milestone

Comments

@andboss
Copy link

andboss commented Mar 3, 2024

We have many repositories that are cloned in parallel and sometimes receiving HTTP code 429 from GitLab server:

File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mkdocs_multirepo_plugin/structure.py", line 424, in batch_import
    await batch_execute(
  File "/usr/local/lib/python3.11/site-packages/mkdocs_multirepo_plugin/structure.py", line 416, in batch_execute
    repo = await future
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/tasks.py", line 615, in _wait_for_one
    return f.result()  # May raise f.exception().
           ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/mkdocs_multirepo_plugin/structure.py", line 388, in import_docs
    await self.sparse_clone([self.docs_dir, self.config] + self.extra_imports)
  File "/usr/local/lib/python3.11/site-packages/mkdocs_multirepo_plugin/structure.py", line 193, in sparse_clone
    await execute_bash_script("sparse_clone.sh", args, self.temp_dir)
  File "/usr/local/lib/python3.11/site-packages/mkdocs_multirepo_plugin/util.py", line 106, in execute_bash_script
    raise BashException(f"\n{stderr}\n")
mkdocs_multirepo_plugin.util.BashException: 
Cloning into 'apps/my-app'...
warning: redirecting to https://my-gitlab/my-app/docs.git/
remote: 429 - Too many requests, see: my-gitlab/settings/#rate-limits
fatal: unable to access 'my-gitlab/my-app/docs/': The requested URL returned error: 429
fatal: could not fetch 44f30c194a684634f2[249](https://my-gitlab/documentation/user-doc/-/jobs/167841352#L249)6040e7d8b8dd9ba1625 from promisor remote
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry with 'git restore --source=HEAD :/'

Is it possible to react on rate limits? Or add a config option to wait (sleep) in between clone executions?

@jdoiro3 jdoiro3 added this to the v0.8 milestone Jun 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants