Multi-base-build artifacts

[dominics]

The inject-artifact step can become quite unwieldy with large, or large numbers of, artifacts. It's necessary because of a few reasons:

1. monofo has a single base build that it pulls artifacts from
2. monofo doesn't attempt to look at what artifacts exist on a build at all
   • it assumes the base build contains all the artifacts
3. monofo doesn't interfere with how artifacts are downloaded in build steps
   • what works outside of monofo works exactly the same way inside monofo

So, for these reasons, monofo tries to ensure that every build contains every artifact (that's used by another pipeline), so that if a pipeline is skipped, that artifact can be injected and the cached version used.

But inject-artifacts is not a good mechanism. Ideally, it should be unnecessary, if we can work around the above reasons. This requires a few things:

First, break 3

The hardest bit:

• Make monofo aware of how steps download artifacts (i.e. the artifacts plugin)
  • Make it inject into those steps environment variables like BUILDKITE_PLUGIN_ARTIFACTS_BUILD that tell it to use a specific build ID
  • Monofo can now make a step use an artifact from another build
• Complication: some plugins we probably won't be able to identify/adjust (e.g. the JUnit plugin does an internal download, or e.g. a custom pre-command hook that does downloading of artifacts in parallel)
  • Solution: provide enough context in env vars for a custom pre-command hook to work out where to get things from (MONOFO_<COMPONENT>_BASE_BUILD_ID?)
  • Solution: also allow opt-out for specified artifacts (e.g. junit.xml if needed), which goes back to using inject-artifacts for them

Then, break 2

• Make monofo put "skip flags" into Build Metadata when a sub-pipeline that produces an artifact is skipped
  • This metadata is keyed by the artifact name, and the value is the current base build
  • This is a build saying "I didn't build this, but I got it from buildId=X which did"
• Now monofo can look at the single base build, and identify that it should instead pull artifacts from some other base build for some particular artifact

Then, break 1

• Make monofo do the lookup of metadata on the base build, then inject the required build IDs into steps that download or depend on that artifact

🎉 That should be it: no more inject-artifacts step except for things that really need to be in the Buildkite artifacts system for the current build