-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Potential errors when scraping new organization. Skipped repos. #477
Comments
The update can take a long time. Our current daily update typically runs for about an hour. The warning messages with the The generic |
I am running MASTER.sh to download all data from the NREL github organization (which has 350 repos), but it's taking a very long time and I'm not sure if this is normal. For most repositories in the org, the query returns in under a second. It does appear that the script is scraping over 4,000 repositories (possibly dependencies?)
For some repositories, it seems to take much longer and the script prints out warning-like messages such as:
Also, for a very small minority of repos, I get the following error-like message:
These two errors do not seem to occur simultaneously.
The script is still humming along, and I will let it finish, but am wondering if these errors can simply be ignored.
Update: The script has finished and I am able to view the data using the Jekyll dev server. However, it appears that at least 3 repositories (out of 350) were skipped.
Steps to reproduce:
The text was updated successfully, but these errors were encountered: