[Help wanted] Speed up push action when using partial clone and sparse checkout

[Dragon1573]

Select Topic Area

Question

Body

Hi, everyone in the community!

I'm currently working as a contributor of microsoft/winget-pkgs. The repository is HUGE for me so I enabled "treeless clone" and "sparse checkout" for it.

D:\Workloads\winget-pkgs [master ≡]> git config --list | findstr tree:0
remote.origin.partialclonefilter=tree:0
remote.upstream.partialclonefilter=tree:0
D:\Workloads\winget-pkgs [master ≡]> git config --list | findstr sparse
index.sparse=true
pack.usesparse=true
core.sparsecheckout=true

When syncing forks using git.exe, I find it seems to pull Git Objects once per commit from upstream and then push them once per commit to github. If github is behind upstream with 100 commits, git.exe will perform 100 seperate pull actions with around 48K objects in each action.

git fetch --progress --depth=1 --all
git merge --progress upstream/master
git push --progress --ff-only origin master

How can I sync them and pull/fetch all necessary Git Objects just once? It waste SO MANY times for me waiting push actions to finish.

Full configurations here

D:\Workloads\winget-pkgs [master ≡]> git config --global --list
core.ignorecase=false
core.quotepath=false
index.sparse=true
user.name=Dragon1573
[email protected]
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
core.ignorecase=false
core.quotepath=false
core.editor=vim.exe
diff.tool=vimdiff.exe
pack.usesparse=true
http.sslbackend=schannel
D:\Workloads\winget-pkgs [master ≡]> git config --system --list
diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
core.autocrlf=true
core.fscache=true
core.symlinks=true
pull.rebase=false
init.defaultbranch=main
D:\Workloads\winget-pkgs [master ≡]> git config --worktree --list
core.repositoryformatversion=1
core.bare=false
core.sparsecheckout=true
[email protected]:Dragon1573/winget-pkgs.git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
remote.origin.promisor=true
remote.origin.partialclonefilter=tree:0
branch.master.remote=origin
branch.master.merge=refs/heads/master
[email protected]:microsoft/winget-pkgs.git
remote.upstream.fetch=+refs/heads/master:refs/remotes/upstream/master
remote.upstream.promisor=true
remote.upstream.partialclonefilter=tree:0
D:\Workloads\winget-pkgs [master ≡]> git config --local --list
core.repositoryformatversion=1
core.bare=false
core.sparsecheckout=true
[email protected]:Dragon1573/winget-pkgs.git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
remote.origin.promisor=true
remote.origin.partialclonefilter=tree:0
branch.master.remote=origin
branch.master.merge=refs/heads/master
[email protected]:microsoft/winget-pkgs.git
remote.upstream.fetch=+refs/heads/master:refs/remotes/upstream/master
remote.upstream.promisor=true
remote.upstream.partialclonefilter=tree:0

[Dragon1573]

The problem is resolved myself. Here's the solution:

1. Install necessary dependencies rely on your operating system. We need them to provide nc on macOS/Unix/Linux or ncat.exe on Windows.
2. Edit configurations in ~/.ssh/config, to enable "SSH over Proxy". This can speed up git clone git fetch and git push.
3. Remove the entire repository. Re-clone it WITHOUT partial (no matter treeless or blobless) clone.

Now Git will squash all differences into 1 "data pack" and fetch from/push to remote just once.

[grandmas-favorite]

Hi @Dragon1573!

Thanks for coming back to share your solution with the Community!