Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Think how to speed up syncing #134

Open
chshersh opened this issue Aug 14, 2019 · 3 comments
Open

Think how to speed up syncing #134

chshersh opened this issue Aug 14, 2019 · 3 comments
Labels
github Synchronization with GitHub, parsing content from GitHub
Milestone

Comments

@chshersh
Copy link
Contributor

chshersh commented Aug 14, 2019

By @rashadg1030 calculations it takes ~6 hours to sync only all repos. And I can't even imagine how much time it will take to sync all issues as well... For testing purposes, we can have a configurable function that allows us to specify caching in a more flexible way (see issue #126 for more details). But we need some long-term strategy as well...

@chshersh chshersh added the github Synchronization with GitHub, parsing content from GitHub label Aug 14, 2019
@chshersh chshersh added this to the Sync milestone Aug 14, 2019
@rashadg1030
Copy link
Collaborator

@chshersh Maybe we should only go back to like 2015 or something. That will save time. I think right now we sync repos going all the way back to 2011 or so. I doubt users will want to go that far back, but on the other hand, there are old repos that might be popular.

@chshersh
Copy link
Contributor Author

chshersh commented Aug 18, 2019

A repository can be created a long time ago, but it's still can be maintained. So we need all repositories eventually. I propose that we will have 3-stage syncing for the proper implementation eventually:

  1. First initial fat sync that fetches all information. It's supposed to be launched only once when the application is started. But it would be good to implement it as a separate task if we want to perform total sync once per year.
  2. A daily job that will sync for the last 1-2-3 (can be configurable) years of data.
  3. When Admin is implemented, we should be able to force sync specific repo.

@rashadg1030
Copy link
Collaborator

rashadg1030 commented Aug 20, 2019

@chshersh Oooh that's interesting. I'd like to know how to implement an Admin dashboard-type thing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
github Synchronization with GitHub, parsing content from GitHub
Projects
None yet
Development

No branches or pull requests

2 participants