Strengthen build-arch rebuild capabilities
#566
Labels
Comments
|
One simple solution @dustymabe and I we came up with is to check upfront if the artifacts already exist, and if so just skip straight to the |
34 tasks
|
A temporary hack on this is to have a |
|
This might be a bit complicated, for little gain; revisit with @jlebon; This #566 (comment) seems the most relevant. |
|
One option is to have the steps that notifies the ostree repo to be a warning instead of a failure, saving us from having to run cloud and update tests manually for the build. |
jlebon
added a commit
to jlebon/fedora-coreos-pipeline
that referenced
this issue
Jun 17, 2024
Very often, our pipeline flakes after the final archiving stage, e.g. if the fedora-messaging queue is down, or Robosignatory is backed up. Ideally, as discussed in coreos#566, we could just rerun the job and have it know to salvage the builds, but there's a lot of details there that we need to get right. (And... it's probably not worth tackling if we're eventually moving away from Jenkins anyway.) For now at least, it's not hard to just have a separate job that reruns those last final steps from the `build` and `build-arch` jobs so we can salvage those builds. You need to know what you're doing when using this job. It should only be used when strictly necessary. It's often better to just do a new build instead. Fixes: coreos#566
jlebon
added a commit
to jlebon/fedora-coreos-pipeline
that referenced
this issue
Jun 17, 2024
Very often, our pipeline flakes after the final archiving stage, e.g. if the fedora-messaging queue is down, or Robosignatory is backed up. Ideally, as discussed in coreos#566, we could just rerun the job and have it know to salvage the builds, but there's a lot of details there that we need to get right. (And... it's probably not worth tackling if we're eventually moving away from Jenkins anyway.) For now at least, it's not hard to just have a separate job that reruns those last final steps from the `build` and `build-arch` jobs so we can salvage those builds. You need to know what you're doing when using this job. It should only be used when strictly necessary. It's often better to just do a new build instead. Fixes: coreos#566
jlebon
added a commit
that referenced
this issue
Jun 17, 2024
Very often, our pipeline flakes after the final archiving stage, e.g. if the fedora-messaging queue is down, or Robosignatory is backed up. Ideally, as discussed in #566, we could just rerun the job and have it know to salvage the builds, but there's a lot of details there that we need to get right. (And... it's probably not worth tackling if we're eventually moving away from Jenkins anyway.) For now at least, it's not hard to just have a separate job that reruns those last final steps from the `build` and `build-arch` jobs so we can salvage those builds. You need to know what you're doing when using this job. It should only be used when strictly necessary. It's often better to just do a new build instead. Fixes: #566
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Right now, if
build-archfails, one can mostly rerun it with the same parameters. There is one gotcha: if the failure happens at any point after theArchivestep, the artifacts for the given build ID and arch will already exist.We should strengthen the job so that it handles this window better.
The text was updated successfully, but these errors were encountered: