Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Schedule Regular Scraping of Bitcoin Improvement Proposals (BIPs, BLIPs, BOLTs) #79

Open
kouloumos opened this issue Sep 9, 2024 · 1 comment

Comments

@kouloumos
Copy link
Contributor

Currently, we have scraping code for Bitcoin Improvement Proposals (BIPs, BLIPs, and BOLTs) from GitHub repositories, but it isn't set to run on a schedule. As a result, these sources are not being updated regularly and may contain outdated information.

We need to implement a cron job to ensure that these improvement proposals are scraped on a more frequent basis, so that our data stays current.

Tasks:

  1. Create a cron job for regular scraping of BIPs, BLIPs, and BOLTs.
  2. Define an appropriate frequency for scraping (e.g., daily or weekly).
  3. Ensure the scraped data is stored properly in the Elasticsearch index.
@urvishp80 urvishp80 self-assigned this Nov 14, 2024
@kouloumos
Copy link
Contributor Author

Those repositories are covered in #81, activation of scraperv2 is pending

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants