Server-side checksumming #299
Comments
|
This would be a useful addition. Supporting this on the rest-server needs to be optional, as it's a heavy operation on some deployments, in which case restic should fall back to the old way. Unlike a client-side check, this requires you to trust the server, so perhaps this needs to be opt-in on the client side. The downside of that is that most users will not discover this extra flag. Perhaps recommend it in the log output when not used? |
Yes, optional flag on the client side seems to be the way to go.
Calculating the checksums doesn't even need to have the lock on the repository. BUT if some checksum comparison fails, then the lock should be requested and calculated again. This would prevent files still being written to have the "wrong" checksum. CPU performance should not be an issue, even my router can calculate hashes faster than sata SSDs can read. As we can see here:
The 40MB/s speed measured on PR/130 is actually the SD-Card speed, and not how much hashing the rpi4b is capable of. Specs: Raspberry-pi 4B (4 cores)
OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)
options: bn(64,64)
compiler: gcc -fPIC -pthread -Wa,--noexecstack -Wall -fzero-call-used-regs=used-gpr -DOPENSSL_TLS_SECURITY_LEVEL=2 -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/openssl-928mA1/openssl-3.0.13=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
CPUINFO: OPENSSL_armcap=0x81
Xiaomi AX6S (Router), CPU is a MediaTek MT7622B (2 cores, 1,35GHz)
OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)
options: bn(64,64)
compiler: aarch64-openwrt-linux-musl-gcc -fPIC -pthread -Wa,--noexecstack -Wall -O3 -pipe -mcpu=cortex-a53+crypto+crc -funsafe-math-optimizations -fno-plt -fhonour-copts -ffunction-sections -fdata-sections -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -O3 -DPIC -fPIC -pipe -mcpu=cortex-a53+crypto+crc -funsafe-math-optimizations -fno-plt -fhonour-copts -ffunction-sections -fdata-sections -Wformat -Werror=format-security -fstack-protector -O3 -fPIC -fuse-ld=bfd -znow -zrelro -DOPENSSL_USE_NODELETE -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DZLIB -DZLIB_SHARED -DNDEBUG -D_FORTIFY_SOURCE=1 -DPIC
CPUINFO: OPENSSL_armcap=0x3d
2x Xeon Gold 6244 (2x 8 cores, 16 threads => 16 cores, 32 threads)
OpenSSL 1.1.1w 11 Sep 2023
options:bn(64,64) rc4(16x,int) des(int) aes(partial) blowfish(ptr)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/reproducible-path/openssl-1.1.1w=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
Intel i5-11400 (6 cores, 12 threads)
OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)
options: bn(64,64)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -fzero-call-used-regs=used-gpr -DOPENSSL_TLS_SECURITY_LEVEL=2 -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/reproducible-path/openssl-3.0.13=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
CPUINFO: OPENSSL_ia32cap=0x7ffaf3bfffebffff:0x405f5ef2bf67eb
Intel i7-4790K (4 cores, 8 threads)
OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)
options: bn(64,64)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -fzero-call-used-regs=used-gpr -DOPENSSL_TLS_SECURITY_LEVEL=2 -Wa,--noexecstack -g -O2 -ffile-prefix-map=/build/reproducible-path/openssl-3.0.13=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_BUILDING_OPENSSL -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
CPUINFO: OPENSSL_ia32cap=0x7ffaf3bfffebffff:0x27ab |
This is true for almost all modern devices that have SHA256 acceleration, but not necessarily for older devices. If I remember correctly, my 2013 ARM7 Synology NAS took 20s just to decrypt the restic private key to run a local restic backup, and could only hash at something like 3 MB/s. There should be at least a way to opt-out in rest-server, and maybe require an opt-in. |
Output of
rest-server --version:rest-server version rest-server 0.13.0 compiled with go1.22.5 on linux/amd64
What should rest-server do differently?
Add a function in rest-server that calculates the sha256sum of a given file in the repository and returns the checksum.
What are you trying to do? What is your use case?
Scenario: a server running rest-server <-- limited network bandwidth --> a client running Restic client.
I'm trying to add more consistency to the
restic checkwithout needing to download the whole repository to the client's machine.As every file inside the folder /repository/data has it's sha256sum as it's own name, and the checksum is known for every file. Restic could use it to detect corrupted files when running
restic check --checksum-only.This way, a basic check could be performed on the whole repository without the need to transfer all the data from the server to the client to check.
This check isn't as complete as the default one because it doesn't decrypt the data, but it's way faster.
And yes, some logic will also be needed to be implemented on the Restic client, but as it's this check is only possible if using the rest-server, I found this github repository to be the most reasonable to discuss this feature.
Did rest-server help you today? Did it make you happy in any way?
The text was updated successfully, but these errors were encountered: