Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase max checksum size (-B) to 1GB #628

Open
YLSnewb opened this issue Aug 13, 2024 · 0 comments
Open

Increase max checksum size (-B) to 1GB #628

YLSnewb opened this issue Aug 13, 2024 · 0 comments

Comments

@YLSnewb
Copy link

YLSnewb commented Aug 13, 2024

I'm trying to transfer a files between 1TB and 6TB in size across a link with high speed, but also high latency due to the distance between the physical servers. Once or twice a week, there's a network issue, and the connection is lost. Restarting the transfer takes a VERY long time -- one transfer of 3.3TB failed at the 550GB mark, and is taking more than an hour to resume.

My best guess is that with the maximum checksum size of 128k, and a file size of 550GB, the back-and-forth chit-chat of checksums is the cause of the slowness. I'm presuming that dramatically increasing the size of the checksum (1MB-1GB) would improve this dramatically, even if there were a slightly higher cost in terms of re-transmitting some of the data.

BONUS POINTS:

I have two ideas for improving this dramatically:

  1. By default, select a checksum size equal to 1% of the file size (rounded up or down if necessary) so that there would be no more than 100 checksums exchanged, or...

  2. A dynamic checksum size that starts at 1MB, and increases by 2x with each successful checksum (up to, say 1GB), and is reduced by 50% with each failure. This would dramatically reduce the number of checksums exchanged between client and server.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant