Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[#45] Add async worker for synchronizing database #132

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

[#45] Add async worker for synchronizing database #132

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d21cf5d
[#45] Add sync function for synchronizing all repositories and issues
rashadg1030 Aug 8, 2019
09ae6a5
[#45] Rebase
rashadg1030 Aug 24, 2019
4c88408
[#45] Add syncErrHandler function to where clause
rashadg1030 Aug 24, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 35 additions & 11 deletions src/IW/Sync/Update.hs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,23 +4,38 @@ data and insert it into the database.
-}

module IW.Sync.Update
( syncRepos
( syncCache
) where

import Control.Monad.IO.Unlift (MonadUnliftIO)
import UnliftIO.Async (mapConcurrently_)
import UnliftIO.Async (async, mapConcurrently_, wait)

import IW.App (WithError)
import IW.App (AppErrorType, WithError, catchError)
import IW.Core.Repo (Repo (..))
import IW.Db (WithDb, upsertRepos, updateRepoCategories)
import IW.Db (WithDb, upsertRepos, updateRepoCategories, upsertIssues)
import IW.Effects.Cabal (MonadCabal (..), getCabalCategories)
import IW.Sync.Search (searchAllHaskellRepos)
import IW.Sync.Search (searchAllHaskellRepos, searchAllHaskellIssues)
import IW.Time (getToday)


-- | This function fetches all repos from the GitHub API, downloads their @.cabal@ files,
-- and upserts them into the database.
syncRepos
-- | Loop for keeping the database up to date with the latest GitHub data.
syncCache
:: forall env m.
( MonadCabal m
, MonadUnliftIO m
, WithDb env m
, WithLog env m
, WithError m
)
=> Integer
-> m ()
syncCache interval = forever $ syncWithGithub interval `catchError` syncErrHandler
where
syncErrHandler :: AppErrorType -> m a
syncErrHandler = undefined
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chshersh @vrom911 Okay so once I implement this handler I think this PR is done. How should we handle the errors? We don't want it to stop syncing if an error is encountered, so do I have to think of some way to resume the sync process from where the error was thrown? Or is it simpler than that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will think about the problem 🤔


-- | This function synchronizes the database with the latest GitHub data.
syncWithGithub
:: forall env m.
( MonadCabal m
, MonadUnliftIO m
Expand All @@ -30,13 +45,22 @@ syncRepos
)
=> Integer -- ^ The starting date interval used in the search function
-> m ()
syncRepos interval = do
syncWithGithub interval = do
log I "Starting synchronization of all repositories and issues..."
today <- liftIO getToday
log I "Searching for all Haskell repositories..."
repos <- searchAllHaskellRepos today interval
log I "Upserting repositories into the database..."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Logging is very useful in this function! And we definitely should have it. It's just unfortunate that because of calls to logging functions the code becomes less readable and all steps are not apparent immediately... It's an open question, how to improve the situation though. But at least, these logging messages serve as a good documentation

Copy link
Collaborator Author

@rashadg1030 rashadg1030 Aug 23, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chshersh Yes, I agree. I thought the same, but I think it’s definitely worth it. This was my first time using any logging library in Haskell, but I really enjoyed how easy and straightforward it is to use. Very user friendly.

upsertRepos repos
mapConcurrently_ syncCategories repos
populateCategoriesAsync <- async $ mapConcurrently_ syncCategories repos
log I "Searching for all Haskell issues..."
issues <- searchAllHaskellIssues today interval
log I "Upserting issues into the database..."
upsertIssues issues
wait populateCategoriesAsync
log I "All repositories and issues successfully synchronized"

-- | This function takes a @Repo@ and attempts to download its @.cabal@ file.
-- | This function takes a @Repo@ and updates its category field in the database.
syncCategories
:: forall env m.
( MonadCabal m
Expand Down