make-disk-image: improve xargs invocation

[danjujan]

When the disk image is too small, cp fails because it is out of space. However, xargs does not abort immediately but still invokes all remaining cp commands, which will of course also fail. That can lead to long waiting times until the make-disk-image script exits with a failure. This PR decreases the time to failure by letting xargs abort directly by causing cp to fail with exti code 255.

https://www.man7.org/linux/man-pages/man1/xargs.1.html#DESCRIPTION

       If any invocation of the command exits with a status of 255,
       xargs will stop immediately without reading any further input.
       An error message is issued on stderr when this happens.

Additionally allow xargs to invoke cp nproc times in parallel with -P 0 as argument. Since by default the vm uses only one core, this does not change anything. However, for custom imageBuilder.qemu options like -smp 4 this can cause a major speedup. For example, I observed a time reduction from 22 to 12 minutes when building an image with aarch64 binfmt emulation.

[Enzime]

LGTM

the flags -P and -I both exist on macOS as well 👍

[Enzime]

@mergify queue

[mergify]

queue

🛑 The pull request has been removed from the queue default

Pull request #910 has been dequeued by a dequeue command.

You can take a look at Queue: Embarked in merge queue check runs for more details.

In case of a failure due to a flaky test, you should first retrigger the CI.
Then, re-embark the pull request into the merge queue by posting the comment
@mergifyio refresh on the pull request.

[Mic92]

Does this not run than one cp for every store path?

[danjujan]

Currently yes. We could wrap the cp invocation in a short script with $@ to avoid that.

[Mic92]

I would prefer that to reduce the overhead.

[danjujan]

Addressed by e6c2ca8

[Mic92]

@mergify dequeue

[mergify]

dequeue

✅ The pull request has been removed from the queue default

[Mic92]

Btw we also have parallel copy implemented here: #788

[danjujan]

Btw we also have parallel copy implemented here: #788

I have seen that. However, this does not do anything about aborting early in out of space situations which was my main motivation for this PR.

[Mic92]

I also tried xargs -P 0 but so far I wasn't able to get any speed up for it. It only got slower. Do you have an example?

[Mic92]

Also if the disk is full, how long does it actually take for cp to finish? Should this not rather quickly?

[danjujan]

I also tried xargs -P 0 but so far I wasn't able to get any speed up for it. It only got slower. Do you have an example?

No public example code but I build a 2GB sized aarch64-linux image on my x86_64-linux host. Just tested it again.
12 mins with xargs -P 0 and 8 cores.
18 mins with just xargs and 8 cores.
22 mins with just xargs and 1 core.

Also if the disk is full, how long does it actually take for cp to finish? Should this not rather quickly?

cp exits instantly. The problem is that xargs keeps invoking cp with the remaining arguments. I've observed 1-5 mins delay because of that depending at which point the disk gets full.

[danjujan]

Made the xargs flags configurable. Since I had to use -L 100 after dropping -I to get to the 12 minutes. Just -P 0 without a limit took 16 mins for me.