Artifact creation issue with API Call

[dauphinpasdroit]

Hey,

I encounter some issues with the artifact creation in a job using an API call.

Here is the sample of API Query:

                    "stages": [
                      {
                        "name": "preconfig",
                        "fetch_materials": true,
                        "clean_working_directory": false,
                        "never_cleanup_artifacts": false,
                        "approval": {
                          "type": "success",
                          "allow_only_on_success" : false,
                          "authorization": {
                            "roles": [],
                            "users": []
                          }
                        },
                        "environment_variables": [],
                        "jobs": [
                          {
                            "name": "preConfig",
                            "run_instance_count": null,
                            "timeout": null,
                            "environment_variables": [],
                            "resources": [],
                            "tasks": [
                              {
                                "type": "exec",
                                "attributes": {
                                  "run_if": [
                                    "passed"
                                  ],
                                  "command":  "/bin/bash",
                                  "arguments" : [ "./pipelines/deployVm/windowsAzure/jobs/preConfig/preConfig.sh" ],
                                  "working_directory": null
                                }
                              }
                            ]
                          }
                        ]
                      },
                      {
                        "name": "infrastructureDeployment",
                        "fetch_materials": true,
                        "clean_working_directory": false,
                        "never_cleanup_artifacts": false,
                        "approval": {
                          "type": "success",
                          "allow_only_on_success" : false,
                          "authorization": {
                            "roles": [],
                            "users": []
                          }
                        },
                        "environment_variables": [],
                        "jobs": [
                          {
                            "name": "infrastructureDeployment",
                            "run_instance_count": null,
                            "timeout": null,
                            "environment_variables": [],
                            "resources": [],
                            "tasks": [
                              {
                                "type": "exec",
                                "attributes": {
                                  "run_if": [
                                    "passed"
                                  ],
                                  "command":  "/bin/bash",
                                  "arguments" : [ "./pipelines/deployVm/windowsAzure/jobs/infrastructureDeployment/terraform.sh" ],
                                  "working_directory": null
                                }
                              }
                            ],
                            "artifacts": [
                                {
                                  "type": "build",
                                  "source": "snake-3740-09042023354",
                                  "destination": "snake-3740-09042023354"
                                }   
                                ]
                            }
                            ]
                        },

Here is the error message:

[go] The rule [snake-3740-09042023354] cannot match any resource under [pipelines/snake-3740-09042023354]
[go] [go] Uploading finished. Failed to upload [snake-3740-09042023354].

Here is the server configuration for the artifact settings:

I've mostly use the same settings in a configuration file and that's one works:

    stages:  
    - preconfig:
        fetch_materials: true
        keep_artifacts: false
        clean_workspace: false
        approval:
          type: success
          allow_only_on_success: false
        jobs:
          preConfig:
            # elastic_profile_id: eaProd
            tasks:
            - exec:
                command: /bin/bash          
                arguments: 
                - ./pipelines/deployVm/windowsAzure/jobs/preConfig/preConfig.sh
                run_if: passed
    - infrastructureDeployment:
        fetch_materials: true
        keep_artifacts: false
        clean_workspace: false
        approval:
          type: success
          allow_only_on_success: false
        jobs:
          infrastructureDeployment:
            artifacts:
            - build:
                source: vm_info
                destination: vm_info  
            # elastic_profile_id: eaProd
            tasks:
            - exec:
                command: /bin/bash          
                arguments: 
                - ./pipelines/deployVm/windowsAzure/jobs/infrastructureDeployment/terraform.sh
                run_if: passed

The only thing that change is the name of the artifact. Does the new name is too long ? Otherwise, I can't understand what am I doing wrong here.

Thank you for the help

[chadlwilson]

There isn’t really a name of the artifact per se. The source is different which means GoCd is looking to archive everything at /snake-3740-09042023354. Are you sure that’s the right location relative to the pipeline workspace folder? You normally wouldn’t want to refer to the pipeline name inside the “source”. This is a relative location inside the pipeline’s working area.

Presumably your script terraform.sh creates something at the folder vm_info the same as your “working” pipeline and the source dir needs to have the same name not something specific to the pipeline?

[dauphinpasdroit]

I'm an idiot, I forgot to update the name of the ouput I make in the terraform.sh. Sorry for that and thank you for the help again. Appreciate your reactivity !

[chadlwilson]

No worries 👍

[dauphinpasdroit]

Hey @chadlwilson,

Sorry to bother again about this use case. There is definitely something I dont understand here.

Basically, here is what's been exported from the terraform.sh script:

touch /godata/pipelines/windowsAzure/$pipelineName

cat > /godata/pipelines/windowsAzure/$pipelineName   <<EOF
$TF_OUT_BACKUP_POLICY
$TF_OUT_RESOURCE_GROUP
$TF_OUT_SUBNET_NAME
$TF_OUT_VM_FQDN
$TF_OUT_VM_IP
$TF_OUT_M_NAME
EOF

cat /godata/pipelines/windowsAzure/$pipelineName

Here, the variable pipelineName refers to snake-3740-090520231116

So, terraform.sh output in a file the output from a terraform deployment I've made earlier.

The cat of the file is good and the export works.

Unfortunately, I still encounter the error:

[go] Task: /bin/bash ./pipelines/deployVm/windowsAzure/jobs/infrastructureDeployment/terraform.shtook: 29.430s
[go] Current job status: passed
[go] Start to upload snake-3740-090520231116/2/infrastructureDeployment/1/infrastructureDeployment on 54d2d21c68fe [/go]
[go] The rule [snake-3740-090520231116] cannot match any resource under [pipelines/snake-3740-090520231116]
[go] [go] Uploading finished. Failed to upload [snake-3740-090520231116].

What does " The rule [snake-3740-090520231116] " means exactly ? Is there something I miss from any other configuration ?

[chadlwilson]

Your script is outputting your file to an absolute path at /godata/pipelines/windowsAzure/$pipelineName

However the artifact is trying to read a relative path, which ends up as <agent install loc>/<pipelineWorkspace>/<artifactSrc> i.e <agent install loc>/pipelines/snake-3740-090520231116/snake-3740-090520231116.

These aren't the same, so the agent probably can't find it.

The rule [artifactSrc] cannot match any resource under [<pipelineWorkspace>]
The rule [snake-3740-090520231116] cannot match any resource under [pipelines/snake-3740-090520231116]

I don't really understand what you are doing here or your pipeline design, but generally speaking you should avoid using absolute paths in your scripts/jobs/tasks. They should operate independently of where they are run, and generally unaware of the pipeline/stage/job that they belong to. Generally you want them to output files in the current workspace area using paths relative to where the job is run, and they should also read/load/artifact things using similar relative paths.

I would consider altering your script to be something like mkdir -p output && cat > output/tf-metadata. And then change your artifact source in the pipeline definition to be output/tf-metadata (and set the destination to whatever you like this to be within the GoCD server artifact storage). If you really want to include the pipeline name, the best way is to interpret it at runtime using standard GoCD env vars such as ${GO_PIPELINE_NAME} that are available to all jobs/tasks rather than "baking" it into the pipeline definition/configuration.

Even ./pipelines/deployVm/windowsAzure/jobs/infrastructureDeployment/terraform.sh looks wrong/strange to me. It's a bit of a pipeline/job design anti-pattern to be including pipeline and job specific names coupled to the GoCD config inside the jobs itself. Normally you would clone a repository or download an artifact that contains this script and then refer to it with a relative path such as <checkoutLoc>/terraform.sh or <artifactFetchLoc>/terraform.sh that GoCD will interpret relative to <pipelineWorkspace>