Transitioning from Personal Access Token to GitHub App for Authentication

[maltif]

We currently face GitHub API rate limit issues while collecting and visualizing GitHub metrics for our company's private repositories using GrimoireLab. We have around 1500 private repositories and are collecting various metrics such as git commits, GitHub pull requests data, and GitHub issues.

To address the API rate limit issue, we plan to transition from using a personal access token (PAT) to GitHub App-based authentication. However, we lack clarity on how to configure this transition within GrimoireLab and what permissions are required for the GitHub App to pull all the necessary metrics from our private repositories.

We've reviewed the GrimoireLab documentation, particularly the tutorial section, but couldn't find sufficient guidance on this specific topic.

Objectives:

1. Understand the process of transitioning from PAT to GitHub App-based authentication within GrimoireLab.
2. Determine the required permissions for the GitHub App to access and retrieve metrics from our private repositories.

Proposed Questions for Discussion:

1. Could someone provide guidance on how to configure GrimoireLab to use GitHub App-based authentication instead of a personal access token?
2. Are the repository permissions listed below sufficient for the GitHub App to effectively collect all the required metrics from our private repositories?
   • Actions: Read Only
   • Commit Statuses: Read Only
   • Contents: Read Only
   • Environments: Read Only
   • Merge Queues: Read Only
   • Metadata: Read Only
   • Pull Requests: Read Only
   • Discussions: Read Only
   • Issues: Read Only
   • Secrets: Read Only
   • Variables: Read Only
3. Are there any specific considerations or best practices we should be aware of during this transition process?

Additional Context:

We believe transitioning to GitHub App-based authentication will alleviate the API rate limit issues we're currently facing and ensure smoother data collection for our metrics analysis. Any insights or assistance from the Grimoire project maintainers or knowledgeable community members would be greatly appreciated.

Thank you in advance for your assistance.
@zhquan @sduenas

[sduenas]

To use GitHub App authentication, you will need the ID of your GitHub app and you will also need the private key assigned to that app. With the private key, create a file on your file systems.
Once you have everything ready, you will need to include in your modred configuration (setup.cfg) the fields github-app-id and github-app-pk-filepath under all your github entries. Take the following as an example:

[github]
raw_index = github_issues_raw
enriched_index = github_issues_enriched
github-app-id = <GitHub_App_id>
github-app-pk-filepath = <path_to_pem_file>
sleep-for-rate = true
no-archive = true
category = issue

One of the problems with these tokens is that they expire fast. That's not a problem because the tool will regenerate them when they expire.

[maltif]

Thank you, @sduenas, for your prompt response. Your guidance is greatly appreciated.

I've taken note of the changes required based on your instructions. It appears that we only need to add the github-app-id and github-app-pk-filepath parameters, while commenting out the api-token field in the setup.cfg file.

[github]
raw_index = github_raw
enriched_index = github_enriched
github-app-id = <GitHub_App_id>
github-app-pk-filepath = <path_to_pem_file>
# api-token = [ghp_xxxxxx, ghp_xxxxx]
category = issue
sleep-for-rate = true
no-archive = true
studies = [enrich_onion:github, enrich_geolocation:user, enrich_geolocation:assignee, enrich_extra_data:github, enrich_backlog_analysis, enrich_demography:github]
#from-date = 2022-01-01

However, I still have a query regarding updating the projects.json file, which currently contains both user_id and api-token (PAC). Below is a snippet of my current projects.json file:

---

{
    "Org": {
        "meta": {
            "title": "MyOrg"
        },
        "git": [
            "https://user_id:[email protected]/myorg/hawk-auth.git",
            "https://user_id:[email protected]/myorg/hawk-bodyconfig.git",
            "https://user_id:[email protected]/myorg/hawk-channels.git",
            "https://user_id:[email protected]/myorg/hawk-channels-v2.git",
            "https://user_id:[email protected]/myorg/hawk-common-ts.git",
            "https://user_id:[email protected]/myorg/hawk-core.git",
            "https://user_id:[email protected]/myorg/hawk-coverage.git",
            "https://user_id:[email protected]/myorg/hawk-db-sync-lambda.git",
            "https://user_id:[email protected]/myorg/hawk-device-info.git"
            ]
					}
}

---

Could you please advise on how to proceed with updating the projects.json file to accommodate the transition to GitHub App-based authentication?

Thank you once again for your assistance.

---

[sduenas]

The rate limit only affects to the github backend, not to git. You should still use that configuration to get data from git.

[maltif]

Thank you once again for your assistance, @sduenas, I sincerely apologize for the repetitive nature of my queries.

Considering the transition to GitHub App authentication, I seek clarification on whether it's necessary to update the git backend configuration under projects.json from personal access token (PAT) to GitHub App authentication. If such an update is required, could you kindly provide guidance on the process? Perhaps a sample configuration for reference would be immensely helpful.

Here's a snapshot of the current configurations for setup.cfg and projects.json:

setup.cfg:

[git]
raw_index = git_raw
enriched_index = git_enriched
latest-items = true
studies = [enrich_demography:git, enrich_git_branches:git, enrich_areas_of_code:git, enrich_onion:git]
#from-date = 2022-01-01

[github]
raw_index = github_raw
enriched_index = github_enriched
api-token = [ghp_xxxxxxx]
category = issue
sleep-for-rate = true
no-archive = true
studies = [enrich_onion:github, enrich_geolocation:user, enrich_geolocation:assignee, enrich_extra_data:github, enrich_backlog_analysis, enrich_demography:github]
#from-date = 2022-01-01

[github:pull]
raw_index = github_pull_raw
enriched_index = github_pull_enriched
api-token = [ghp_xxxxxxx]
category = pull_request
sleep-for-rate = true
no-archive = true
studies = [enrich_geolocation:user, enrich_geolocation:assignee, enrich_extra_data:github, enrich_demography:github]
#from-date = 2022-01-01

[github2:pull]
api-token = [ghp_xxxxxxx]
raw_index = github2-pull_raw
enriched_index = github2-pull_enriched
sleep-for-rate = true
category = pull_request
no-archive = true
studies = [enrich_geolocation:user, enrich_geolocation:assignee, enrich_feelings]
#from-date = 2022-01-01

---

projects.json:

{
    "Org": {
        "meta": {
            "title": "MyOrg"
        },
        "git": [
            "https://user_id:[email protected]/myorg/hawk-auth.git",
            "https://user_id:[email protected]/myorg/hawk-bodyconfig.git",
            "https://user_id:[email protected]/myorg/hawk-channels.git",
            "https://user_id:[email protected]/myorg/hawk-channels-v2.git",
            "https://user_id:[email protected]/myorg/hawk-common-ts.git",
            "https://user_id:[email protected]/myorg/hawk-core.git",
            "https://user_id:[email protected]/myorg/hawk-coverage.git",
            "https://user_id:[email protected]/myorg/hawk-db-sync-lambda.git",
            "https://user_id:[email protected]/myorg/hawk-device-info.git"
            ],
        "github": [
            "https://github.com/myorg/hawk-auth.git",
            "https://github.com/myorg/hawk-bodyconfig.git",
            "https://github.com/myorg/hawk-channels.git",
            "https://github.com/myorg/hawk-channels-v2.git",
            "https://github.com/myorg/hawk-common-ts.git",
            "https://github.com/myorg/hawk-core.git",
            "https://github.com/myorg/hawk-coverage.git",
            "https://github.com/myorg/hawk-db-sync-lambda.git",
            "https://github.com/myorg/hawk-device-info.git"
        ],
        "github:pull": [
            "https://github.com/myorg/hawk-auth.git",
            "https://github.com/myorg/hawk-bodyconfig.git",
            "https://github.com/myorg/hawk-channels.git",
            "https://github.com/myorg/hawk-channels-v2.git",
            "https://github.com/myorg/hawk-common-ts.git",
            "https://github.com/myorg/hawk-core.git",
            "https://github.com/myorg/hawk-coverage.git",
            "https://github.com/myorg/hawk-db-sync-lambda.git",
            "https://github.com/myorg/hawk-device-info.git"
        ],
        "github2:pull": [
            "https://github.com/myorg/hawk-auth.git",
            "https://github.com/myorg/hawk-bodyconfig.git",
            "https://github.com/myorg/hawk-channels.git",
            "https://github.com/myorg/hawk-channels-v2.git",
            "https://github.com/myorg/hawk-common-ts.git",
            "https://github.com/myorg/hawk-core.git",
            "https://github.com/myorg/hawk-coverage.git",
            "https://github.com/myorg/hawk-db-sync-lambda.git",
            "https://github.com/myorg/hawk-device-info.git"       ]
    }
}

---

Your clarification on this matter would be greatly appreciated. Thank you once again for your patience and support.