Feedback thread for "Getting Started" docs

[taylorsilva]

• Did you have any questions after following the tutorial?
• Any sections that were unclear or confusing to follow?
• Other topics or areas of Concourse you think the tutorial should cover?

[wsierakowski]

I wonder if the screenshot on this page tutorial-hello-world is relevant to the topic of this tutorial piece:

I would say that instead of 404, we should see a screenshot like below:

[wsierakowski]

Obviously, I didn't realise that this wasn't just a screenshot but a frame with a view pointing at the reader's own local instance.

[taylorsilva]

Added this line to avoid future confusion:

[roerjo]

I believe the git commit task needs to be updated to configure git with the email and name. I'm getting an error because of that running it off of the current documentation. I also needed to specify the branch name in the resource configuration for the put step to work. The git resource docs mention that branch is required for the put step. https://github.com/concourse/git-resource

[taylorsilva]

Thank you! Fixed it concourse/docs@cc0ccd8

[roerjo]

@taylorsilva looks like there are still a couple pieces missing still. Need to call the git config commands still:

          params: # Change these to match your personal info
            EMAIL: [email protected]
            GIT_AUTHOR_NAME: Tester
          run:
            path: sh
            args:
              - -cx
              # this is just a bash script
              - |
                cd repo
                date +%Y-%m-%d > todays-date
                git add ./todays-date
                git config --global user.email $EMAIL
                git config --global user.name $GIT_AUTHOR_NAME
                git commit -m "Update todays date"

[taylorsilva]

I don't think so. I tested with just the environment variables and the task step completed without error. From https://git-scm.com/book/en/v2/Git-Internals-Environment-Variables see the Committing section.

[roerjo]

Interesting, I kept getting the error until I added the git config commands. I guess either way it will be documented here now. Thanks!

[taylorsilva]

Decided to add it just to keep the chance of people running into this error as close to zero as possible :) Thanks for your feedback and going through the tutorial!

[stranger-danger-zamu]

This ended up a wall of text going through my first experience using Concourse. There are relevant comments to the tutorial, but also just pain points in general with Concourse. I tried quoting the general pain points and annoyances so the tutorial relevant sections are easier to find.

---

Overall the tutorial was great. It got me up and running fairly quickly and I was able to eventually grok how to put a reasonable pipeline together. I was able to generate artifacts and push them elsewhere by just reading the tutorial.

It didn't cover how to generate container images so I had to go down a rabbit hole of Github issue spelunking, Github ReadMe's, and searching online for tidbits to figure out how to generate all the artifacts I needed.

Specifically, I was able to generate python .whl and .pyz files and name them based on the pyproject.toml file's version number and push them to an s3 bucket with just the tutorial. When I tried to figure out how to build container images from the artifacts, I spent more than five hours try to find out how to do it idiomatically.

Eventually, I figured out I had to use concourse/oci-build-task to generate the images and did so, but I couldn't figure out how to tag an image when puting it via the concourse/registry-image resource.

I know I should use the semver resource, but that required a backend and I didn't need any of the other functionality it provided and wasn't sure if it would do what I needed it to do without messing with other things. Even then, setting the semvar via put seemed very complicated for just getting the git repo's matched tag.

I realized what annoyed me was that some values weren't just added to the variable space even though they were relevant to that particular task (eg. if the git resource is matching against a tag filter, wouldn't it make sense to expose the tag it matched on so it could be used as a variable in that job).

I know that resources have a defined interface (check, in, and out) and that in V10 this might be a non-issue, but it's still a pain point now.

At this point I gave up on the idiomatic way and wrangled it together with a custom container that utilized buildah to build the images. This was so much easier since I could run commands so I had a lot more control than oci-build-task. I even decided to then save the image and put the image tagged with latest via registry-image.

Another reason I went with custom containers was the fact that I couldn't figure out how to pass the self-signed certs to everything. I couldn't find my cert in the task so I could to point to them when running some commands (though resources didn't have cert issues, so it was sorta working? IDK).

I know it was supposedly as simple as mounting the certs into /etc/ssl/certs/<cert>.crt and it should propagate them to all the workers and containers, but that didn't always work since it didn't propagate them into everything (or at least it didn't copy them into the spawned container for a task, so when I ran buildah it didn't automatically find the cert in /etc/ssl/certs). So instead I made more pipelines to construct the custom build containers which embedded the certificate within and used the generated image for other pipelines.

Then I left it running for a bit and eventually it started yelling at me that I hit the max number of workers. But when I look at the resource usage of the host system and I'm idling at most 5% CPU and less than 8% of the 32GiB of memory available. Looking at a process manager, I can see the count of containerd-shim-runc-v2 rise steadily (with memory increasing, but no CPU usage), so I'm not sure if this is just PEBCAK or an actual issue.

Overall, Concourse is so much better than other CI's I've tried. After figuring out what I needed to search for it was fairly trivial to get something working, but:

• the fact that a lot of documentation sits on Github ReadMe's is painful
  • searching for answers either points to old Concourse issues (circa v3.x) or points to RFCs or closed issues pointing to an RFC that's still not implemented
• I'm still not sure if docker-compose is a valid installation method or if it's just a kludged together quickstart-only method
• I didn't leave the tutorial understanding why I should use multiple jobs or how to create dependencies between jobs
  • I get that I could use them running test, then building and pushing artifacts, and then smoke testing, and then end to end, but I can also just have them as tasks in a job?
• I couldn't find a way to kill a stuck task
  • This might just be a docker-compose quickstart quirk, a concourse quirk, or a buildah quirk , not sure.
  • If buildah is building based on a Dockerfile and it's ambiguous which registry to use (eg. docker.io, quay.io, some internal registry), it'll prompt for user input. This literally causes an impossible to stop task. You can't abort it, cancel it, or anything. I had to blow away the pipeline, land the worker, and restart concourse.

[chandradeepak]

→ fly -t tutorial workers
error: unknown target: tutorial

i get this error when i follow the hello-world tutorial

[taylorsilva]

Did you fly login first? The login command creates the target in your ~/.flyrc

[dannystaple]

I am getting an interesting error in running the hello world tutorial:

$ fly -t tutorial trigger-job --job hello-world/hello-world-job --watch
started hello-world/hello-world-job #1

initializing
selected worker: ef0e3766321e
find or create container on worker ef0e3766321e: new container: garden spec to oci spec: rootfs must not be empty
errored

I've torn down the composed concourse, pruned the volumes and recreated it, and still see this error. Any ideas why?

My (hopefully unsurprising) hello world pipeline file is:

jobs:
- name: hello-world-job
  plan:
  - task: hello-world-task
    config:
      platform: linux
      image-resource:
        type: registry-image
        source:
          repository: busybox
      run:
        path: echo
        args: ["Hello concourse!"]

I have run the fly commands to login, set-pipeline , unpause and trigger. I have half a TB of disk space remaining - so it's not that. Running on MacOS 11.6.2 with Docker Desktop.

[dannystaple]

Doh - found it. My image-resource was Kebab-case. It should be snake_case.

[tylershelton]

Doh - found it. My image-resource was Kebab-case. It should be snake_case.

One and a half years later, thank you so much for a) teaching me the term "kebab case" and b) pointing out to my blearly, late-night, been-at-this-for-hours eyes that I, too, kebab'd my snake.

[dannystaple]

I, too, kebab'd my snake

That phrase is a keeper...

[richard-gebbia]

The base docker-compose file didn't work on my m1 Mac. I had to change the concourse service's image from concourse/concourse to rdclda/concourse:7.5.0 (which is built for arm architectures).

[taylorsilva]

Correct, we don't build an official ARM image of Concourse. I recently got an M1 so I'll be much more motivated to fix this pain sooner rather than later 😁

[nickmcostello]

@taylorsilva has this been updated? Is there a native ARM image of Concourse? Could I help get one set up?

[taylorsilva]

@nickmcostello sadly it has not been updated. Haven't been spending my spare time on Concourse stuff like I thought I would. Hoping to carve out some time during work hours. Will probably be a few more months until I can make time for that though

[gsaslis]

I wish I'd found this thread sooner...

I've left some relevant comments on #8719 (comment)

Even if concourse doesn't ship an arm64 image, I think a comment in the docker-compose.yml that mentions that an arm64 image is required (it currently directs folks to switch to the houdini runtime) would be very helpful!

(I'd open the PR if I knew where to do that - the link in the quickstart doesn't point to a file in some git repo)

[Kevvix]

Hi,

I'm trying to do the tutorial but I'm blocked at the part "1.1.2 Hello World Pipeline".
I have been able to do the first part before without problems

I don't understand the error.

Thanks in advance ^^

[taylorsilva]

Third line from the bottom looks to be the main error message you want to debug. no such host. Usually means it was a DNS resolution error. Before that part I can see it's hitting 8.8.8.8 for the DNS lookup, so you already have custom DNS configured. Not sure why you're getting this 🙁

Is there something up with your Docker networking? Rebooting the docker vm or your computer may resolve this.

[muzhaqi16]

Unable to run concourse with apple m1, receiving an error regarding iptables.

error: Exit trace for group:
worker exited with error: Exit trace for group:
garden exited with error: Exit trace for group:
containerd-garden-backend exited with error: setup host network failed: create chain or flush if exists failed: running [/sbin/iptables -t filter -N CONCOURSE-OPERATOR --wait]: exit status 3: iptables v1.6.1: can't initialize iptables table `filter': iptables who? (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

[paulmiller3000]

It may be in the docs and I'm just an idiot, but after hours of searching I finally learned that my pipeline cannot pull in a task from another file in memory. For example, I can't get ./scripts/test.sh and run it as part of my pipeline UNLESS it exists somewhere on the web. For me, this was counterintuitive and represented hours of banging my head against a wall.

Fortunately, I found the statement below at this site. Making this a bit more clear -- or adding a flashing light for dummies like me -- would be a huge help. And I'll buy you a drink!

When creating separate task files, it’s important to understand, that only the main pipeline definition will be stored in system memory. Everything else, including separate task definition files, will be fetched from some external resource (e.g. git in our case).

[Radmanded]

This isn't a huge problem, but the instructions for installing fly are confusing. In the code block, I copied the commands and the response. Creating a space would help with that. It took me about 10 mins to figure it out, but I'm sure it would stump others.

$ fly -t tutorial login -c http://localhost:8080 -u test -p test
logging in to team 'main'

Thank you!

[Radmanded]

Also, the instructions say to download fly from the web UI. Is it still necessary to use cURL and/or download?

Thanks

[Gr1zzz]

Hello, i have problem, when i want to connect github. when i started pipeline i have this mistake:
selected worker: 8223477c5283
Cloning into '/tmp/build/get'...
fatal: detected dubious ownership in repository at '/tmp/build/get'
To add an exception for this directory, call:

git config --global --add safe.directory /tmp/build/get

failed
I alrady update git, and write this command, and write the command for all package save