Implement garbage collecting of old issues

[rashadg1030]

This will require to fetch all closed issues and delete all closed issues from our database.

[rashadg1030]

I'm gonna use the updated_at field to determine if the issue should be deleted. What should time limit between updates be? If the issue isn't updated in 14 days or so or less?

[chshersh]

@rashadg1030 This issue is a bit more complicated. Open issues should never be deleted, no matter how old they are. Old here means is no longer open. So you need to fetch all closed issues and delete all of them from our DB.

[rashadg1030]

@chshersh got it, make sense. Now how do I determine if an issue in the DB has closed or not? Do I just fetch all closed Haskell issues and delete the ones that match in the DB?

[chshersh]

Do I just fetch all closed Haskell issues and delete the ones that match in the DB?

Yep, exactly. This should work for now. But architecture behind this implementation is not that straightforward. If you will repeatably call deleteClosedIssues and upsertIssues in forever loop, DB won't like it...

[rashadg1030]

@chshersh Can you explain why forever is bad in this situation? I'm curious.

[chshersh]

Well, it's just bad in general for performance to do concurrently batch updates of different types and perform SELECT queries on a single time very fast. I'm not sure, but there's a chance that our sync can be very fast (like 5-20 minutes or even faster). Updating table with such frequency might be a high load for DB. But now I think that we don't need to do concurrent garbage collecting (this was my implicit assumption at first). The algorithm can be:

1. Upsert repos
2. Delete closed issues
3. Upsert open issues

[rashadg1030]

@chshersh Sounds good