feat(scrapers): Add GitHub metadata scraper for issues and pull requests #93
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Move Elasticsearch operations into a separate 'elastic' command group that handles index initialization with custom mappings, cleanup operations, and mapping inspection. The new structure supports initializing indices with custom mapping files.
Implement scraper for processing GitHub issue and PR metadata from JSON backups. Add Pydantic models for structured metadata including reviews, comments and review threads. Include Elasticsearch mapping for the metadata schema and register new data sources in sources.yaml.
Use the `number` field instead of `issue` to refer to the number of the related pull request. This follows the same schema used for github metadata
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for scraping all the existing github metadata backups from bitcoin-data which updates every hour for the following repositories:
bitcoin/bips,bitcoin/bitcoin,bitcoin-core/secp256k1,bitcoin-core/gui, thus fixes #92Additionally, this PR introduces a separate 'elastic' command group that handles index initialization with custom mappings, cleanup operations, and mapping inspection. We need index initialization with custom mapping in order to model the relationships between documents in the current github metadata schema using nesting.
An experimental addition to the Bitcoin Search API is currently in progress in order to better showcase and validate this scraper.
The Schema
The schema is designed to store three main types of content:
and can be found at
scraper/models/github_metadata.pyThis schema allows for efficient querying of:
The nested structure maintains the relationships between reviews, threads, and comments while allowing for efficient querying and aggregation at each level.
Example document
{ "id": "github-metadata-bitcoin-core-12345", "title": "Add new validation rules for taproot transactions", "body": "This PR implements additional validation rules for taproot transactions as specified in BIP 341. It adds checks for:\n\n- Script path spending conditions\n- Leaf version requirements\n- Signature verification rules\n\nThe changes have been tested extensively with the test vectors from BIP 341.", "body_type": "markdown", "domain": "https://github.com/bitcoin/bitcoin", "indexed_at": "2024-12-10T12:00:00.000Z", "created_at": "2024-03-15T14:30:00Z", "url": "https://github.com/bitcoin/bitcoin/pull/12345", "type": "pull", "authors": ["alice-developer"], "number": "12345", "updated_at": "2024-03-20T09:15:00Z", "closed_at": "2024-03-25T16:45:00Z", "merged_at": "2024-03-25T16:45:00Z", "state": "merged", "labels": ["consensus", "needs testing", "taproot"], "head_sha": "abc123def456789ghijklmnop0123456789qrstuv", "draft": false, "reviews": [ { "id": 987654321, "author": "bob-reviewer", "commit_id": "abc123def456789ghijklmnop0123456789qrstuv", "submitted_at": "2024-03-18T11:20:00Z", "body": "The implementation looks correct and follows the BIP specification. Tested with the provided test vectors." } ], "review_threads": [ { "pull_request_review_id": 987654321, "path": "src/consensus/validation.cpp", "diff_hunk": "@@ -150,6 +150,15 @@ bool CheckTaprootSignature(...)", "commit_id": "abc123def456789ghijklmnop0123456789qrstuv", "original_commit_id": "abc123def456789ghijklmnop0123456789qrstuv", "position": 8, "original_position": 8, "line": 158, "original_line": 158, "comments": [ { "id": 11111111, "author": "bob-reviewer", "created_at": "2024-03-18T11:25:00Z", "updated_at": "2024-03-18T11:25:00Z", "body": "Consider adding a comment explaining the merkle path verification logic here.", "pull_request_review_id": 987654321 } ] } ], "comments": [ { "id": 22222222, "author": "charlie-commenter", "created_at": "2024-03-16T10:00:00Z", "updated_at": "2024-03-16T10:00:00Z", "body": "Have you considered adding test cases for edge cases with empty script paths?" } ] }