aws-sigv4 failing to calculate the right signature when using "content-type: multipart/form-data"

[Viphor]

I did this

NOTE: Changing the actual paths and hashes to some redacted values.

curl -vvv -X POST \
    "https://my-api-gateway-endpoint.com/my-resource?my_query=param" \
    --user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
    --aws-sigv4 "aws:amz:${AWS_REGION}:execute-api" \
    --header "x-amz-security-token: $AWS_SESSION_TOKEN" \
    -H "accept: application/json" \
    -H "Content-Type: multipart/form-data" \
    -F "content=@${FILE};type=application/x-zip-compressed" \
    -F "extra_metadata=@${JSON_META};type=application/json"

Which gave me the following output:

Note: Unnecessary use of -X or --request, POST is already inferred.
* Host my-api-gateway-endpoint.com:443 was resolved.
* IPv6: (none)
* IPv4: x.x.x.x, x.x.x.x
*   Trying x.x.x.x:443...
* Connected to my-api-gateway-endpoint.com (x.x.x.x) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=my-api-gateway-endpoint.com
*  start date: Apr  4 00:00:00 2024 GMT
*  expire date: May  3 23:59:59 2025 GMT
*  subjectAltName: host "my-api-gateway-endpoint.com" matched cert's "my-api-gateway-endpoint.com"
*  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M03
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* Server auth using AWS_SIGV4 with user 'ASIA*******'
* [HTTP/2] [1] OPENED stream for https://my-api-gateway-endpoint.com/my-resource?my_query=param
* [HTTP/2] [1] [:method: POST]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: my-api-gateway-endpoint.com]
* [HTTP/2] [1] [:path: /my-resource?my_query=param]
* [HTTP/2] [1] [authorization: AWS4-HMAC-SHA256 Credential=ASIA*******/20240411/eu-central-1/execute-api/aws4_request, SignedHeaders=accept;host;x-amz-date;x-amz-security-token, Signature=2fee53*****]
* [HTTP/2] [1] [x-amz-date: 20240411T130311Z]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [x-amz-security-token: IQoJb3JpZ********+la3yaTN6lr9AH+3JJCPquZB2VoKT]
* [HTTP/2] [1] [accept: application/json]
* [HTTP/2] [1] [content-length: 655]
* [HTTP/2] [1] [content-type: multipart/form-data; boundary=------------------------GlkWoiCdSW4YuTzdI6B8TJ]
> POST /my-resource?my_query=param HTTP/2
> Host: my-api-gateway-endpoint.com
> Authorization: AWS4-HMAC-SHA256 Credential=ASIA*******/20240411/eu-central-1/execute-api/aws4_request, SignedHeaders=accept;host;x-amz-date;x-amz-security-token, Signature=2fee53******
> X-Amz-Date: 20240411T130311Z
> User-Agent: curl/8.7.1
> x-amz-security-token: IQoJb3JpZ************+oSscHqSd+la3yaTN6lr9AH+3JJCPquZB2VoKT
> accept: application/json
> Content-Length: 655
> Content-Type: multipart/form-data; boundary=------------------------GlkWoiCdSW4YuTzdI6B8TJ
> 
* upload completely sent off: 655 bytes
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
< HTTP/2 403 
< date: Thu, 11 Apr 2024 13:03:11 GMT
< content-type: application/json
< content-length: 1701
< x-amzn-requestid: 050dbcba-4d82-46df-9c16-5b96fbeeb2b4
< x-amzn-errortype: InvalidSignatureException
< x-amz-apigw-id: WD_****Eaqg=
< 
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'POST\n/my-resource\nmy_query=param\naccept:application/json\nhost:my-api-gateway-endpoint.com\nx-amz-date:20240411T130311Z\nx-amz-security-token:IQoJb3JpZ*******+oSscHqSd+la3yaTN6lr9AH+3JJCPquZB2VoKT\n\naccept;host;x-amz-date;x-amz-security-token\n2fa2cc887bff*******c9d22cce'\n\nThe String-to-Sign should have been\n'AWS4-HMAC-SHA256\n20240411T130311Z\n20240411/eu-central-1/execute-api/aws4_request\n6b6f17fc5e653b********c952640aa3d66'\n"}
* Connection #0 to host my-api-gateway-endpoint.com left intact

When using the curl command with only a single file and without Content-Type: multipart/form-data everything works as expected, but fails to sign properly when given Content-Type: multipart/form-data.

I expected the following

I expect for the signing to work and hit the logic behind my endpoint.

curl/libcurl version

curl 8.7.1 (aarch64-apple-darwin23.4.0) libcurl/8.7.1 (SecureTransport) OpenSSL/3.2.1 zlib/1.2.12 brotli/1.1.0 zstd/1.5.6 libidn2/2.3.7 libssh2/1.11.0 nghttp2/1.60.0 librtmp/2.3 OpenLDAP/2.6.7
Release-Date: 2024-03-27
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz MultiSSL NTLM SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

operating system

Darwin M-01619 23.4.0 Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:49 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6020 arm64

[bagder]

fails to sign properly when given Content-Type: multipart/form-data.

So why do you?

When sending a multipart formpost, the Content-Type header also needs a boundary string set (and curl needs to know it) and your provided header does not so curl is not able to produce a proper POST. You broke curl by providing a broken header.

update: curl actually can survive the situation and still do a formpost with a boundary so it does not exactly explain the error. Even though the original question why you would use this command line remains.

[Viphor]

So the same error occurs whether or not I give it the -H "Content-Type: multipart/form-data.
I left it there as this was how we received the bug internally.

[bagder]

This confuses me. Your report says

When using the curl command with only a single file and without Content-Type: multipart/form-data everything works as expected

Isn't that quite the opposite of what you just said here?

[Viphor]

I think I might need to clarify then. :)
Whenever curl tries to send a request where curl uses the Content-Type: multipart/form-data, then it fails to sign correctly.
So in this case whether it automatically adds it because of -F or if I add it manually, then it fails to sign the message correctly.

Sorry for the confusion. :)

[bagder]

Aha, ok. Thanks for clarifying.

[bagder]

I suspect this is because the generated Content-Type: type header used for multipart formposts is not taken into account correctly when the header hash is calculated.

[Viphor]

For context what we have done previously that worked was:

curl --verbose -X POST "${ENDPOINT}" \
-d "$DATA" \
--user "$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY" \
--aws-sigv4 "aws:amz:${AWS_REGION}:${SERVICE}" \
--header "x-amz-security-token: $AWS_SESSION_TOKEN"

This has worked fine until we needed to upload multiple files.
Here we end up with Content-Type: application/x-www-form-urlencoded.