You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We recently encountered an issue where the version of a resource type used by concourse was not the most recent one despite the fact that check correctly reported the most recent version available.
tag latest has the highest check_order and so is considered the most recent version
the digest of tag latest is the same as the one of tag 0.6.1
tag 0.6.2 exists and is indeed more recent then 0.6.1
Forcing a check prints out 0.6.2 as expected.
After looking at the code, the problem can be explained as a race condition between tagging the repository images and checking them. Here is the scenario:
Start with a docker repository containing a single image with digest hash1 and tags latest and 0.6.1.
Configure a resource type for it and run the check
all the tags are retrieved from the docker registry
tag latest is processed and variable latestTag set and its digest stored in tagDigests["latest"]
then tag 0.6.1 is processed and digestVersions["hash1"] is set to "0.6.1"
the digestVersions map is turned into the list tagVersions which is then sorted
the check response is built from tagVersions
as latestTag is set, its digest (hash1) is looked up in digestVersions and found and so existsAsSemver is true and so response is left unchanged
the response, which is [{"tag": "0.6.1", "digest": "hash1"}] is returned
Push a new docker image to the registry with tag 0.6.2 only. Now it contains two images with hashes hash1 and hash2. hash2 has tag 0.6.2 and hash1 has the tags 0.6.1 and latest.
Run the check
all the tags are retrieved from the docker registry
tags is shuffled to put 0.6.1 first because from is set to {"tag": "0.6.1", "digest": "hash1"} by concourse following the previous check
tag 0.6.1 is processed, digestVersions["hash1"] is set to "0.6.1" and cursorVer is set to "0.6.1"
tag latest is processed and variable latestTag set and its digest stored in tagDigests["latest"]
tag 0.6.2 is processed and digestVersions["hash2"] is set to "0.6.2" because it is greater than or equal to cursorVer
the digestVersions map is turned into the list tagVersions which is then sorted
the check response is built from tagVersions
as latestTag is set, its digest (hash1) is looked up in digestVersions and found and so existsAsSemver is true and so response is left unchanged
the response, which is [{"tag": "0.6.1", "digest": "hash1"}, {"tag": "0.6.2", "digest": "hash2"}] is returned
So far, so good. Now, run the check again
all the tags are retrieved from the docker registry
tags is shuffled to put 0.6.2 first because from is set to {"tag": "0.6.2", "digest": "hash2"} by concourse following the previous check
tag 0.6.2 is processed, digestVersions["hash2"] is set to "0.6.2" and cursorVer is set to "0.6.2"
tag latest is processed and variable latestTag set and its digest stored in tagDigests["latest"]
tag 0.6.1 is processed and ignored because it is less than cursorVer
the digestVersions map is turned into the list tagVersions which is then sorted
the check response is built from tagVersions
as latestTag is set, its digest (hash1) is looked up in digestVersions and not found and so existsAsSemver is false and so response is extended with {"tag": "latest", "digest": "hash1"}
the response, which is now [{"tag": "0.6.2", "digest": "hash2"}, {"tag": "latest", "digest": "hash1"}] is returned and so concourse now believes that {"tag": "latest", "digest": "hash1"} is the most recent version, which is wrong
Eventually the tag latest is moved to the image with tag 0.6.2 and digest hash2
The check runs again
all the tags are retrieved from the docker registry
tags is shuffled to put latest first because from is set to {"tag": "latest", "digest": "hash1"} by concourse following the previous check, note that the digest in from is not actually the digest linked to latest in the registry now
tag latest is processed and variable latestTag set and its digest stored in tagDigests["latest"]
tag 0.6.2 is processed, digestVersions["hash2"] is set to "0.6.2"
tag 0.6.1 is processed, digestVersions["hash1"] is set to "0.6.1"
Notice that this time digestVersions has entries for both 0.6.1 and 0.6.2 because cursorVer is never set due to the fact that from != nil && identifier == from.Tag && digest.String() == from.Digest is never true, even when identifier is latest due to the fact that it's digest is hash2 now but from.Digest is hash1
the digestVersions map is turned into the list tagVersions which is then sorted
the check response is built from tagVersions
as latestTag is set, its digest (hash2) is looked up in digestVersions and found and so existsAsSemver is true and so response left unchanged
the response is now [{"tag": "0.6.1", "digest": "hash1"}, {"tag": "0.6.2", "digest": "hash2"}] and so 0.6.2 appears as the most recent version in the check output
Concourse does not update the version history because these two versions already exist in it and so {"tag": "latest", "digest": "hash1"} remains the most recent version
The text was updated successfully, but these errors were encountered:
I'm running into something similar just using a simple resource not a resource-type. I'm not specifying a source.tag in my resource and was surprised when my job ran with tag latest. Since the documentation mentions:
With tag omitted, check will instead detect tags based on semver versions (e.g. 1.2.3) and return them in semver order
Note that this was a brand new pipeline so all the current images tags were all existing.
We recently encountered an issue where the version of a resource type used by concourse was not the most recent one despite the fact that check correctly reported the most recent version available.
Here is the definition of the resource:
As you can see,
source.tag
is not set and we expect semver versioning of the resource type.Looking at the resource_config_versions table in the DB, here is what I saw:
As you can see:
latest
has the highestcheck_order
and so is considered the most recent versiondigest
of taglatest
is the same as the one of tag0.6.1
0.6.2
exists and is indeed more recent then0.6.1
Forcing a check prints out
0.6.2
as expected.After looking at the code, the problem can be explained as a race condition between tagging the repository images and checking them. Here is the scenario:
hash1
and tagslatest
and0.6.1
.latest
is processed and variablelatestTag
set and its digest stored intagDigests["latest"]
0.6.1
is processed anddigestVersions["hash1"]
is set to"0.6.1"
digestVersions
map is turned into the listtagVersions
which is then sortedresponse
is built fromtagVersions
latestTag
is set, its digest (hash1
) is looked up indigestVersions
and found and soexistsAsSemver
is true and soresponse
is left unchangedresponse
, which is[{"tag": "0.6.1", "digest": "hash1"}]
is returned0.6.2
only. Now it contains two images with hasheshash1
andhash2
.hash2
has tag0.6.2
andhash1
has the tags0.6.1
andlatest
.tags
is shuffled to put0.6.1
first becausefrom
is set to{"tag": "0.6.1", "digest": "hash1"}
by concourse following the previous check0.6.1
is processed,digestVersions["hash1"]
is set to"0.6.1"
andcursorVer
is set to"0.6.1"
latest
is processed and variablelatestTag
set and its digest stored intagDigests["latest"]
0.6.2
is processed anddigestVersions["hash2"]
is set to"0.6.2"
because it is greater than or equal tocursorVer
digestVersions
map is turned into the listtagVersions
which is then sortedresponse
is built fromtagVersions
latestTag
is set, its digest (hash1
) is looked up indigestVersions
and found and soexistsAsSemver
is true and soresponse
is left unchangedresponse
, which is[{"tag": "0.6.1", "digest": "hash1"}, {"tag": "0.6.2", "digest": "hash2"}]
is returnedtags
is shuffled to put0.6.2
first becausefrom
is set to{"tag": "0.6.2", "digest": "hash2"}
by concourse following the previous check0.6.2
is processed,digestVersions["hash2"]
is set to"0.6.2"
andcursorVer
is set to"0.6.2"
latest
is processed and variablelatestTag
set and its digest stored intagDigests["latest"]
0.6.1
is processed and ignored because it is less thancursorVer
digestVersions
map is turned into the listtagVersions
which is then sortedresponse
is built fromtagVersions
latestTag
is set, its digest (hash1
) is looked up indigestVersions
and not found and soexistsAsSemver
is false and soresponse
is extended with{"tag": "latest", "digest": "hash1"}
response
, which is now[{"tag": "0.6.2", "digest": "hash2"}, {"tag": "latest", "digest": "hash1"}]
is returned and so concourse now believes that{"tag": "latest", "digest": "hash1"}
is the most recent version, which is wronglatest
is moved to the image with tag0.6.2
and digesthash2
tags
is shuffled to putlatest
first becausefrom
is set to{"tag": "latest", "digest": "hash1"}
by concourse following the previous check, note that the digest infrom
is not actually thedigest
linked tolatest
in the registry nowlatest
is processed and variablelatestTag
set and its digest stored intagDigests["latest"]
0.6.2
is processed,digestVersions["hash2"]
is set to"0.6.2"
0.6.1
is processed,digestVersions["hash1"]
is set to"0.6.1"
digestVersions
has entries for both0.6.1
and0.6.2
becausecursorVer
is never set due to the fact thatfrom != nil && identifier == from.Tag && digest.String() == from.Digest
is never true, even whenidentifier
islatest
due to the fact that it'sdigest
ishash2
now butfrom.Digest
ishash1
digestVersions
map is turned into the listtagVersions
which is then sortedresponse
is built fromtagVersions
latestTag
is set, its digest (hash2
) is looked up indigestVersions
and found and soexistsAsSemver
is true and soresponse
left unchangedresponse
is now[{"tag": "0.6.1", "digest": "hash1"}, {"tag": "0.6.2", "digest": "hash2"}]
and so0.6.2
appears as the most recent version in the check output{"tag": "latest", "digest": "hash1"}
remains the most recent versionThe text was updated successfully, but these errors were encountered: