PVC mover

[leoluk]

We want a simple way of moving PVCs elsewhere when draining a node.

[lorenz]

The moving part itself is not too hard, we can basically push a tar or similar over an mTLS connection. Serialization/deserialization of most things is relatively easy, special attention needs to be paid to xattrs, sparse files and hard links. Reflinks are a pain though, the best approach to reliably serialize them is hope that the underlying filesystem supports FIEMAP and keeping track of all the mappings which can consume non-trivial amounts of memory.

Also block positions are accessible to userspace and will expose that there was a filesystem move. On the other hand this mechanism supports migrating between very different PV implementations as it sits on the VFS layer.

The most complicated part is figuring out when to move and what decides that. K8s can do it, but will do it gratuitously as PV affinity can only be hard (required), a preference for the same node cannot be expressed. Advantage of having K8s do it is we do not need to become involved in scheduling decisions. If we do it, we need to decide where to move the PV, hope that the pod can schedule onto that node and then update the affinity, which is a bit painful.