You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Even after #924, publishing to an S3 bucket with KMS encryption enabled results in 'file already exists and is different' errors like the following.
ERROR: unable to publish: unable to process packages: error putting file to aptly/focal/pool/main/a/a7xpg/a7xpg_0.11.dfsg1-10build3.1_amd64.deb: file already exists and is different: S3: us-east-1:orca-sre-staging-na1-repo-36812/aptly
The objects throwing the errors were PUT by Aptly 1.5.0 to S3 with KMS encryption and contain the user-defined x-amz-meta-md5 metadata with the correct MD5 checksum. The ETAG of the objects have a length of 32, which is the condition Aptly currently uses to assume the ETAG is an MD5 checksum.
AWS documentation states KMS encrypted objects do not use MD5 checksums of the content for their ETAG. It also makes no guarantees about the length of such ETAGs.
Instead of assuming an ETAG of length 32 is an MD5 checksum of the content, Aptly could do one of the following:
Use object metadata or upload details (Is this multi-part? Is this KMS encrypted?) to determine if the ETAG can be used for md5 checksum.
In the S3PublishEndpoints portion of the Aptly config, allow users to specify whether or not to use object metadata rather than ETAGs for checksum values.
Always use object metadata for checksum values on S3 (this would have a performance impact).
Option 2 is probably the simplest solution.
Your Environment
Aptly version 1.5.0
Running aptly on Ubuntu Focal LTS
Publishing Ubuntu Focal LTS mirrors to S3 buckets with KMS encryption
The text was updated successfully, but these errors were encountered:
Detailed Description
Even after #924, publishing to an S3 bucket with KMS encryption enabled results in 'file already exists and is different' errors like the following.
The objects throwing the errors were PUT by Aptly 1.5.0 to S3 with KMS encryption and contain the user-defined
x-amz-meta-md5
metadata with the correct MD5 checksum. The ETAG of the objects have a length of 32, which is the condition Aptly currently uses to assume the ETAG is an MD5 checksum.AWS documentation states KMS encrypted objects do not use MD5 checksums of the content for their ETAG. It also makes no guarantees about the length of such ETAGs.
aptly/s3/public.go
Line 326 in e90ac67
Context
Possible Implementation
Instead of assuming an ETAG of length 32 is an MD5 checksum of the content, Aptly could do one of the following:
Option 2 is probably the simplest solution.
Your Environment
Aptly version 1.5.0
Running aptly on Ubuntu Focal LTS
Publishing Ubuntu Focal LTS mirrors to S3 buckets with KMS encryption
The text was updated successfully, but these errors were encountered: