Easier integration of external CICD metadata into GoCD

[vernonr3]

Issue Type

Feature Proposal

Summary

First things first. I like GoCD. The concepts, architecture and facilities seem to fit well generally with the "CICD challenge". It operates with smaller hardware resources than Jenkins ( important for lower cost cloud servers)- and the split between server and remote agents is neat.

Justification

Getting files (material) from source control systems to include in build, test pipelines etc. is fairly simple. However getting general CICD "metadata" (e.g. data required to name a build rather than included in the build itself) from elsewhere is harder.
This metadata might include:

• tags from Source Control systems
• data derived from project management/other systems.

Why is this an issue?

My use case is building external CICD metadata into the naming of components produced by GoCD pipelines.

I don't believe users should:

• have to treat GoCD as the "master data source".
• have to copy/create all the master metadata inside GoCD as environment variables.

I'm sure others have found workarounds - but the core system doesn't make this as easy as I believe it should be. GoCD would be better if it didn't require hard work to fork GoCD or building logic to "make integration work".

Proposed Action

I'm well down the track of preparing a Pull Request to allow a task plugin to override "placeholder parameters". Using placeholders will allow existing parameter verification routines to operate unchanged. I'll supply a sample plugin.

The solution I see would:

• allow external metadata to be fed in and become parameters to custom tasks (e.g. docker builds/tagging)
• allow environment variables from "any hierarchical level" within GoCD to be recast as parameters. This might help users avoid a potentially "awkward dance with shell scripts" ( I believe people should be able to use GoCD without being expert in bash etc.)
• allow non trivial text processing of metadata outside the core of GoCD.

What do people think? How could this be improved?

[chadlwilson]

I think it's generally tough to make this generic especially across source control systems (with respect to tags etc) and certainly with external systems like issue trackers which GoCD isn't directly integrated with.

Have you looked at pipeline labeling https://docs.gocd.org/current/configuration/pipeline_labeling.html and the existing env vars at https://docs.gocd.org/current/faq/dev_use_current_revision_in_build.html ?

I believe env vars were generally chosen to supply these pieces (including the label) as these can also be worked with in a build tool or scripting language of choice, without shell scripting.

I'm not really sure what your proposed solution would look like, or what additional information a task plugin could be supplied that isn't already available as an env var.

If you want to use a task plugin to talk to some external system what additional metadata would be needed which GoCD has but is not currently supplied? Maybe your sample plugin and/or a definition of 'placeholder parameters' might clarify.

What GoCD does (when using GoCD to build GoCD itself) is to generate that metadata inside build tooling (Gradle, Rube/rake) as an artifact (usually some JSON) and carry it along the value stream through multiple pipelines alongside the artifact. Subsequent pipelines can use that to make additional decisions, or say, know how to publish a container image to docker hub, or upload binaries to www.gocd.org.

[vernonr3]

Thank you Chad for the prompt, encouraging response. I'll bear these comments in mind...

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 90 days.
If you can still reproduce this error on the master branch using local development environment or on the latest GoCD Release, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.