Prioritization of snapshots during processing (e.g. prune, copy)

[simonsan]

Thanks for the question @mprasil.
The answer is: It is not that easy. The reason is, snapshots reference blobs. Those are accessed using the index and finally the pack files they are stored in. So we have a dependency chain:

snapshots -> index -> packs

In order to not save invalid snapshots, rustic must be sure that all dependent pack files (i.e. all packs which contain dependent blobs) and all index files which reference the used blobs are really present in the repository. Therefore we have to wait until they are really saved.

If you just copy one snapshot, rustic will therefore copy all blobs and create index entries, but before saving the snapshot, it will finalize the last already started pack and the last already started index file and save it to the storage before finally saving the snapshot to the storage.

So, in your example copying snapshots one-by-one and always finalizing pack and index files may result in lot of small pack and/or index files compared to the current processing.
(And yes, in theory rustic could still try to prioritize blobs and to detect when all needed blobs and index entries are really written. Then the snapshot could be safely written, too. But this would require a big redesing in how rustic currently works and will take some time to get. It would e.g. also allow to delete pack files earlier during a prune run).

Another issue is the priority of copying. You prefer to have newest snapshots first, but maybe someone else would want to have oldest snapshots first...

However, we could make it optional to process snapshots one-by-one, but then users should be able to also choose their own priority (e.g. old vs new first).

Originally posted by @aawsome in #1090 (comment)