scenario for iOS backups created with idevicebackup2

[benjaminweb]

idevicebackup2 [1] allows to backup iOS devices to any path.
That path can be covered from rustic to be backed up.
idevicebackup2 does incremental backups, too.
Since I couldn’t find any information on the data format of the backup:
I am just wondering how the incrementally updated backup will be handled by rustic.

Yes, this post could be categorised as completely off-topic.
But on the other side, think of other use cases which could end up in a gallery of recipes (cookbook) how to get the most out of rustic.

[1] https://github.com/libimobiledevice/libimobiledevice?tab=readme-ov-file#utilities

[aawsome]

Yeah, kind of off-topic ;-)
rustic just backups paths/files and does create snapshots which can be fully restored. It also uses deduplication and compression, so the repository should be small for many use-cases.
I didn't look into idevicebackup, but as long as it creates paths/files it should work with rustic. If some files stay identical during rustic backup runs, they will be deduplicated. But also if it would re-create files with (partly) identical content, rustic would deduplicate them.

I think what could be interesting for a setup is some kind of pre-/post run hooks into rustic, see #902

[benjaminweb]

Assuming it’s an archive, zip or something else, how does deduplication work? When doesn’t? If it doesn’t, pre- and post-hooks are indeed an option to make the backup accessible to deduplication.

…

On Tue, 6 Feb 2024, at 16:15, aawsome wrote: Yeah, kind of off-topic ;-) rustic just backups paths/files and does create snapshots which can be fully restored. It also uses deduplication and compression, to the repository should be small for many use-cases. I didn't look into idevicebackup, but as long as it creates paths/files it should work with rustic. If some files stay identical during rustic backup runs, they will be deduplicated. But also if it would re-create files with (partly) identical content, rustic would deduplicate them. I think what could be interesting for a setup is some kind of pre-/post run hooks into rustic, see #902 <#902> — Reply to this email directly, view it on GitHub <#1054 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ3EVPQW3OAZC3WWZHNIETYSJCJXAVCNFSM6AAAAABC4FR6CCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DGOBTHE2TM>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[kapitainsky]

Since I couldn’t find any information on the data format of the backup:
I am just wondering how the incrementally updated backup will be handled by rustic.

You are affectively saying that you have some software generating data you have no clue what format/structure etc. it has and how rustic incremental backup will work.

But despite this ambiguity I think that the answer can be very clear. rusitc can handle it very well. As long as you make sure that your data is not changing during backup process (filesystem snapshots or just manual timing) your data will be snapshoted in backups and preserved for future retrieval.

Now how space efficient it is going to be? We can speculate...

From my iOS experience I can only guess that in your case "idevicebackup2 does incremental backups" means that this software only needs to add to backup new and changed items. It does no create any snapshots. You always have only the last phone data backup. Also most likely it means that such backup contains individual (optionally compressed/encrypted) files - a lot of small files - 10s of thousands. This is how Apple libraries handle backups.

Good news is that such format is perfect to get maximum benefits from rustic deduplication algorithm.

Proof is in the pudding - so the best is to try. Then you can decide how you want to use it.

[benjaminweb]

Proof is in the pudding - so the best is to try. Then you can decide how you want to use it.

Does rustic support a dry-run option that just tells me the space requirement for the initial backup?

[kapitainsky]

yes. dry-run is supported.

Also you can experiment with local backup destination in another directory on the same machine. This will be fast.

[benjaminweb]

Correct.

Thanks for taking this case inside out.

Also most likely it means that such backup contains individual (optionally compressed/encrypted) files - a lot of small files - 10s of thousands. This is how Apple libraries handle backups.
Good news is that such format is perfect to get maximum benefits from rustic deduplication algorithm.

How could this be generalised?
I mean into a guide one could consult to figure when rustic’s deduplication works and how space efficiency turns out.

Envision NAS systems integrate rustic in their appliances. Their appliances backs up the since “unbackupable”, iOS devices.
Automatically deduplicated and synced to your cost-efficient remote storage.

That would give rustic a much bigger presence.
To get there, I’d find those end-to-end scenarios powered by rustic appealing — since they will pay back to rustic.

[kapitainsky]

How could this be generalised?

I only speculate as you did not provide any details. My thoughts are based on Apple iOS devices backup I use.

“unbackupable”, iOS devices.

??? They are perfectly capable to be backed up. To Apple iCloud and/or to local storage where then you can do with it what you want.

That would give rustic a much bigger presence.

I honestly doubt. As mentioned earlier iOS devices can be backed up very easy today. What extra value rustic can bring here? In my opinion all effort put into rustic should focus on what it is for - backup. Not into looking for some exotic integration options. But if somebody will use it for something bigger then only great! as building block. The same like today rustic is using rclone. Rclone focus is cloud based files' transfers - and praise to its authors for focusing on just that:) In spirit of Unix philosophy: "Write programs that do one thing and do it well. Write programs to work together"

[benjaminweb]

On Tue, 6 Feb 2024, at 19:07, kapitainsky wrote: > How could this be generalised? > I only speculate as you did not provide any details. My thoughts are based in Apple iOS devices backup I use.

Fine. But your reasoning is based on the technical foundation. And I mean those are relevant to all kinds of different use case scenarios: How does deduplication go about ZIP archives? How about files in two iOS backups that are identical? How about a large and a small image? …

> “unbackupable”, iOS devices. > ??? They are perfectly capable to be backed up. To Apple iCloud and/or to local storage where then you can do with it what you want.

It becomes less simple if you exclude: - local storage (because sum of iOS devices exceeds local storage of device storing backups) - iCloud: because the user does not want to use it for whatever reason

> That would give rustic a much bigger presence. > I honestly doubt. As mentioned earlier iOS devices can be backed up very easy today. What extra value `rustic` can bring here?

As one factor for seemless integration. By providing the end-to-end scenario TO finally get integrated in solutions. For non-cli users. Well, as a matter of fact, it’s not fully integrated as in “select iOS device in NAS and get it backed up to your NAS and a remote destination”.

[kapitainsky]

if you choose rustic for your project then really great. In the meantime I think as for rustic the best is to make sure that it is even better in what it does the best:)

It will be genuine flattery when people start using it in higher level solutions. Watching this project for some time I think it can really happen as there is a lot of thought (and work) put into decoupling pure logic from UI etc. opening endless options later.

From this project motto: "rustic - fast, encrypted, and deduplicated backups". I would prefer myself it stays like that:)

[benjaminweb]

Yup, that’s my idea. Giving people higher level recipes of how to use it. Because they will not search rather for the specific use case instead of a specific definition of backup application. So it’d have a wider stage to shine.

…

On Tue, 6 Feb 2024, at 19:31, kapitainsky wrote: if you choose `rustic` for your project then really great. In the meantime I think as for `rustic` the best is to make sure that it is even better in what it does the:) It will be genuine flattery when people start using it in higher level solutions. Watching this project for some time I think it can really happen as there is a lot of thought (and work) put into decoupling pure logic from UI etc. — Reply to this email directly, view it on GitHub <#1054 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ3EVK4HLD2PFSFEFVJDRDYSJZHTAVCNFSM6AAAAABC4FR6CCVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DGOBWGUYTG>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[aawsome]

Just note that there are the libraries rustic_core and rustic_backend which empower rustic.
And of course it's all open source and you are free to use parts of it. So, one can easily roll a self-mad backup solution on top of it.

Disclaimer: It's under development meaning the APIs are not yet stable - be prepared for higher work adjusting API changes in future.