Support for versioned pipelines

[robwhitby]

Hi,

I've been reading up on how all the new roadmap features fit together. Lots of exciting developments with projects, prototypes and instanced pipelines! The one piece of the jigsaw I don't see is versioned pipelines. For example..

• after a pipeline has changed, how can you see its history, "as it was" for older versions? e.g. view or re-run a job that no longer exists in the current pipeline definition.

• the pipeline definition can be versioned in a git resource along with the code, but the pipeline itself - or the jobs it comprises of - are not version aware. e.g. adding a new job causes it to trigger immediately with the previous version of the git resource.

It seems to be a fundamental part of GitOps "config as code" and automating pipeline management in general, or maybe I'm not thinking about it in the right way for Concourse?

Thanks

[chenbh]

after a pipeline has changed, how can you see its history, "as it was" for older versions? e.g. view or re-run a job that no longer exists in the current pipeline definition.

Interesting enough, builds for jobs that no longer exist are still kept around, maybe there's someway for us to surface that information to the UI. As for re-running, there's no good way other than setting the old pipeline again. Btw, because jobs are identified by name, setting a job with the same name or setting the old pipeline again will restore the history of any old jobs.

the pipeline definition can be versioned in a git resource along with the code, but the pipeline itself - or the jobs it comprises of - are not version aware. e.g. adding a new job causes it to trigger immediately with the previous version of the git resource.

This is doable via resource pinning and set_pipeline step, we have a separate pipeline (config found here) responsible for configuring all other pipelines including itself. We can pin the git repo to a particular commit and run the reconfigure pipeline to get an older version of our other pipelines.

[DanielJonesEB]

For me it's a deal breaker, but as I don't know Concourse a lot I might either not understand the scope of the tool or be misusing it. So if anyone knows, I'm all ears !

@robwhitby is correct in that Concourse has no notion of a 'pipeline run'.

@Erwyn, if you need to know that this specific build of a job ran using the pipeline configuration from a specific Git commit of a repo containing the pipeline definition, you can achieve this by:

• passing the resource that contained the pipeline definition to all jobs
• using the pool resource to claim a lock on the pipeline, when the set-pipeline job triggers. The lock remain claims until the pipeline passes (at which point you release it), or when the build fails.

The latter is an approach we use to avoid 'spooky action at distance' (credit: A. Einstein via @jpluscplusm) in pipelines. You don't want a pipeline to be halfway through executing on one version of the pipeline definition, when a new pipeline definition comes in and re-sets the pipeline, potentially changing the end of the pipeline before the previous run got there. That would break causality. The way Concourse enforces causality is via the passing of versioned resources, thus you must pass pipeline-definition-repo resource all the way down the pipeline.

Does that make sense? It's hard to explain this stuff in prose.

[DanielJonesEB]

TL;DR: it's totally possible to enforce the rigour that I think you seek.

[Erwyn]

@DanielJonesEB thanks for your answer. The question was more about the versioning of the pipeline definition alongside the code that is run through this pipeline. It is a common practice among other tools (Jenkins through the Jenkinsfile, Drone through drone.yml) to version your pipeline configuration with your code, hence both are in the same repository at the same version. This way, when the tool wants to build a repository, it takes the pipeline definition from the commit that is being built and configure itself to run the code through this definition.

The question is then, how do we achieve this, is it possible with concourse to have in the same repository the code and the pipeline definition and that when I try to build a commit I build it against its own pipeline definition.

Maybe I didn't get what you were trying to explain, and if that's the case I'm sorry.

[DanielJonesEB]

@Erwyn No worries, this stuff is sometimes hard to explain.

Yes, this is totally possible. Most of our code repos have ci/pipeline.yml in them. The first job is set-pipeline which triggers when a new version of the repo-under-test is available. The first job sets the pipeline using the definition from the new commit. This resource is fed into the next job (often unit tests) via the passed constraint on the unit test job.

See the pipeline and repo for ironbird as an example.

[Erwyn]

Oh I think I get it. And thanks to the resource locking you were talking about before I can ensure that this is the correct pipeline version that is kept during the whole run. Okay nice, thanks for the links, I'm going to test that see if it fits what I'm trying to do, but it does look like it.

Thanks !

[Erwyn]

Could we maybe switch categories of this discussion from General to Ideas ?

[taylorsilva]

Done!

[Erwyn]

thanks !