Add async worker for populating DB

[rashadg1030]

I was wondering if it is a good time to work on the async worker that will populate our DB? Once we get a better idea of the data we need I guess. I've implemented an endpoint for our server that touches the DB and everything, so I think this would be a good next step. And one thing about file structure. We have a file that holds the GitHub query functions on the path src/IW/Server/Search.hs. Is it fine if it goes into the Server folder like it is now, or should it go to another folder called Async or Worker. For example, it could be src/IW/Worker/Search.hs. I'm not sure because technically Search.hs does function on the server, but I think files in src/IW/Server should be related to the issue-wanted API.

[chshersh]

@rashadg1030 Let's have IW/Sync directory and corresponding IW.Sync.* modules for the forwker. Async and Worker seem to generic. Sync is for synchronisation which is that it should do.

Regarding implementing the worker, I have some suggestions:

Implement a separate function that fetches data for a single repo or for a single issue. In this case we will be able to test such functions from GHCi and check the data in SQL tables. And it's easy to call such functions for all fetched repos.

[chshersh]

@rashadg1030 I'm going to describe a high-level overview of the sync algorithm.

Main function should look like this:

syncCache = forever $ syncWithGitHub `catch` \...

So we are trying to sync cache in the infinite loop, handling all errors.

syncWithGitHub function looks like this:

syncWithGitHub = do
    repos <- fetchAllRepos
    upsertRepos
    populateCategoriesAsync <- async $ populateCategories repos
    issues <- fetchAllIssues
    upsertIssues issues
    wait populateCategoriesAsync

The idea is the following:

1. First, we fetch all the repositories.
2. Second, we populate categories for repositories in a separate thread.
3. While we are populating categories, we can update all issues.
4. Wait while we update all categories.

Does this make sense for now?

[rashadg1030]

@chshersh This is perfect. Thank you!

[rashadg1030]

@chshersh Now that everything is in place, I'm gonna implement the sync algorithm for issues and then clear this issue. There will be a single function that syncs both repos and issues.

[rashadg1030]

@chshersh
I'm gonna use the algorithm you mentioned above now:

syncWithGitHub = do
    repos <- fetchAllRepos
    upsertRepos
    populateCategoriesAsync <- async $ populateCategories repos
    issues <- fetchAllIssues
    upsertIssues issues
    wait populateCategoriesAsync