How to detect a branch is created? The pipeline is only active when the branch is created/exists?

[fong1999]

Hi! The use case is I have a concourse pipeline which runs against a certain branch called release-candidate branch. Below is the fly command.

fly sp -t team-name -c ./pipeline.yml -p app-repo-release-candidate -l settings-default.yml -v BRANCH=release-candidate

Below is how I set up gitrhub repo as a resource.

  - name: source-code
    type: git
    source:
      branch: ((BRANCH))
      uri: ((resources.git.code_repo_url))
      private_key: ((pipeline-creds-ssh.privatekey))
    webhook_token: ((pipeline-webhook-token))
    check_every: 1m

But the release-branch only exists when we have a release, it is deleted after the release. Once it is being deleted, I see concourse pipeline shows red mark around the resource source-code. But if I can have a way to let concourse yaml only check when the branch exists/created. Or the pipeline is only active when the branch exists or is created? Any suggestions?

Thanks much!

[suhlig]

I don't think that a resource exists that discovers git branches.

You could either build such a resource, e.g. using the GitHub API, or you could write a job that does the discovery and the modifies the pipeline using set_pipeline.

[fong1999]

@suhlig Thank you for your message. Could you please be a little specific? how to use set_pipeline to discover it? do you mean in the same concourse pipeline yaml file? Any examples? Thanks!

[suhlig]

Here is an example: suhlig/concourse-branch-detection.

• It detects all branches of suhlig/concourse-rss-resource and creates a pipeline instance for each.
• If a branch is removed, the pipeline instance disappears on the next pipeline run.

Beware: this is just a spike. If you want to use this for real, turn the branch detection into a proper resource so that pipelines can trigger on branches added or removed (instead of polling via job trigger).

[fong1999]

@suhlig Thank you!!

Do you mean here to change to use a real branch name instead? I see you have resource defined .... Thanks!

https://github.com/suhlig/concourse-branch-detection/blob/main/pipelines/branch-detector.yml#L2

[fong1999]

@suhlig Hi! I have another question:

What does this line do? Could you please help me understand? Do I need to create a list.yaml file?

https://github.com/suhlig/concourse-branch-detection/blob/main/pipelines/branch-detector.yml#L25

Thanks much!

[suhlig]

Do you mean here to change to use a real branch name instead? I see you have resource defined ....

No, the idea of this task is to list all branches that exist in the remote git repo (defined by the source resource). This YAML file is created in line 25, read in load_var step, and iterated over in the across step (running spike/pipelines/show-branch.yml on each branch).

What does this line do? Could you please help me understand? Do I need to create a list.yaml file?

All of the previous lines create a list of branches on stdout. The tee command consumes them on stdin (via pipe |), prints them on stdout again and also writes them to the given file (../branches/list.yml). So you don't have to create that file, the task does it for you.

[fong1999]

@suhlig Thanks for your reply!

When you fly the pipeline, have you run into this error?

invalid jobs:
jobs.detect-branch.plan.do[3].across: the across step must be explicitly opted-in to using the --enable-across-step flag

I was reading the document here - https://concourse-ci.org/across-step.html

Not sure how to enable it. Any suggestions?

Thanks!

[suhlig]

you must set the feature flag CONCOURSE_ENABLE_ACROSS_STEP

Set this environment variable when starting the Concourse server.

[fong1999]

I see. The concourse is managed by another team in my company. Let us check to see if our DevOps team can do that. Thanks!

[marco-m]

We had the same problem. Currently we poll the GitHub API, but we realized that a better approach is to react on webhooks, since GitHub can be configured to send a webhook (also) for branch creation.
Concourse supports webhooks, see https://concourse-ci.org/resources.html#schema.resource.webhook_token.
Using only Concourse is a bit painful, because you must associate one webhook per repo per branch.
There are tools such as https://github.com/sapcc/webhook-broadcaster that simplify this.
We are currently working on our own version of webhook demultiplexer, which we plan to open source when ready.