How does artifact streaming work? (and lead to slow task initialization)

[chbiel]

I tried to find any documentation on that topic but it seems that non exists.

So what I experience is:
I have one job with 5 tasks. All tasks have input and outputs, so that the following task depends on the outputs of the task before.
I have the placement strategy "volume-locality" enabled.
I have 5 workers.

But anyway, every task mostly get executed on a different worker.
That leads to sometime 2-3 minutes start up time, as 1.5GB artifact needs to be copied to the next worker.

So why does concourse place the tasks like this?
Why does concourse not detect, that task-1 has output "xyz" and task-2 and input "xyz" and then run them on the same worker?
Sometimes this happens (placed on the same worker) but it feels more or less random.

For me it feels like a bug but as there is no documentation, I cannot really confirm.

Does someone know if there is some documentation on artifact streaming?
How it works?
When it happens?
End (to all workers) or beginning (only the selected worker) of a task?
Can 1.5 minutes at the end of a task "doing nothing" be related to streaming?
What happens between "selected worker: ..." and the first log line coming from my build scripts?

Or someone as any clue what the source of the problem could be?

[taylorsilva]

This does seem like a bug with the volume-locality container placement strategy. Your expectations are correct, I would expect Concourse to schedule all 5 tasks on the same worker, not on different workers. Are you using any other container placement strategies in a chain?

Generally, artifact streaming happens at the step-level when a build is running. Concourse will look at what artifacts are required to run a step and stream them to the worker the step landed on, if the artifact is not already on the worker.

[chbiel]

thanks for getting back :)

we are currently using limit-active-containers,volume-locality,fewest-build-containers.
I already confirmed, that "active containers" limit is not reached on any of the workers, so this should not be the reason for that kind of placement.

I am currently trying to create a reproducable example with this. but I had no success yet. I will create a bug if I have one.

I would also expect, that a step with three input would choose the worker with the most inputs available. Am I right?

But, do you know, if streaming happens before or after "selected worker: ..." is printed to the build log?

I feel that there should be some kind of debug logging, that could be enabled, to write concourse internals into the build log so one can understand what is happening in detail. Is there such option?
Only simple stuff like "Streaming artifact X from worker Y ..." or "Caching directory Z ..."

[taylorsilva]

I would also expect, that a step with three input would choose the worker with the most inputs available. Am I right?

Correct!

But, do you know, if streaming happens before or after "selected worker: ..." is printed to the build log?

Happens after selected worker is printed.

I feel that there should be some kind of debug logging

Great idea! Sadly not a thing that's currently in there :(

[xdu-opendoor]

This does seem like a bug with the volume-locality container placement strategy.

would setting logLevel to debug help reveal the logic in scheduling to help us understand what strategy was employed?