Detailed Description

Even after #924, publishing to an S3 bucket with KMS encryption enabled results in 'file already exists and is different' errors like the following.

ERROR: unable to publish: unable to process packages: error putting file to aptly/focal/pool/main/a/a7xpg/a7xpg_0.11.dfsg1-10build3.1_amd64.deb: file already exists and is different: S3: us-east-1:orca-sre-staging-na1-repo-36812/aptly

The objects throwing the errors were PUT by Aptly 1.5.0 to S3 with KMS encryption and contain the user-defined x-amz-meta-md5 metadata with the correct MD5 checksum. The ETAG of the objects have a length of 32, which is the condition Aptly currently uses to assume the ETAG is an MD5 checksum.

AWS documentation states KMS encrypted objects do not use MD5 checksums of the content for their ETAG. It also makes no guarantees about the length of such ETAGs.

Context

Possible Implementation

Instead of assuming an ETAG of length 32 is an MD5 checksum of the content, Aptly could do one of the following:

1. Use object metadata or upload details (Is this multi-part? Is this KMS encrypted?) to determine if the ETAG can be used for md5 checksum.
2. In the S3PublishEndpoints portion of the Aptly config, allow users to specify whether or not to use object metadata rather than ETAGs for checksum values.
3. Always use object metadata for checksum values on S3 (this would have a performance impact).

Option 2 is probably the simplest solution.

Your Environment

Aptly version 1.5.0
Running aptly on Ubuntu Focal LTS
Publishing Ubuntu Focal LTS mirrors to S3 buckets with KMS encryption