aws-sigv4 request signature does not match expected value

[jaw111]

I did this

Using latest curl version 8.6.0 I am running into problems with the signatures when trying to run SPARQL queries using --url-query as described here.

An example request:

curl https://${NEPTUNE_HOSTNAME}:8182/sparql \
  --aws-sigv4 "aws:amz:eu-west-1:neptune-db" \
  --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
  --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" \
  --header "Accept: application/sparql-results+json" \
  --url-query "query=select * where { ?s ?p ?o } limit 1" \
  --no-progress-meter -i

Response headers

HTTP/1.1 403 Forbidden
Content-Length: 473
Content-Type: application/json
x-amzn-errortype: AccessDeniedException
x-amzn-requestid: 3cc709b0-16e5-78ef-f335-81431d8d2027

Response body:

{
  "code": "AccessDeniedException",
  "requestId": "10c709b0-16e5-56ed-2c66-63071e997e70",
  "detailedMessage": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.",
  "message": "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."
}

Some problems were already resolved by #11794 but it seems some characters used in SPARQL queries are still not handled correctly. I think the spaces are the cause as it works OK with --url-query "query=select*where{?s?p?o}limit1".

I expected the following

The calculated signature should be correct.

curl/libcurl version

curl 8.6.0 (x86_64-w64-mingw32) libcurl/8.6.0 LibreSSL/3.8.2 zlib/1.3.1 brotli/1.1.0 zstd/1.5.5 WinIDN libpsl/0.21.5 libssh2/1.11.0 nghttp2/1.59.0 ngtcp2/1.2.0 nghttp3/1.1.0
Release-Date: 2024-01-31
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Largefile libz NTLM PSL SSL threadsafe UnixSockets zstd

operating system

MINGW32_NT-10.0-22621-WOW64 SMW008711 3.3.6-341.i686 2022-09-05 20:09 UTC i686 Msys

[bagder]

@outscale-mgo any idea?

[outscale-mgo]

I think the problem is that --url-query do the url-encoding the standard way, and AWS isn't that standard.
So spaces are encoded as '+', which make the signature fail, as AWS signature want space to be encoded as %20.
I guess, you could bypass the problem using something like:

--url-query "+query=select%20*%20where%20{%20?s%20?p%20?o%20}%20limit%201"

[bagder]

spaces are encoded as '+', which make the signature fail, as AWS signature want space to be encoded as %20

Ouch, that's... weird. I suppose we can run an extra pass over the query part when AWS-SIGV4 is used and force-replace all pluses with %20 ?

[outscale-mgo]

AWS sigv4 have this function.
I guess we should reuse it here: https://github.com/curl/curl/blob/master/src/tool_getparam.c#L1044, if aws encoding is used. (or a syntax +aws)

[jaw111]

@outscale-mgo the mentioned bypass works, but kind of defeats the purpose of using --url-query in the first place for my use case.

You say --url-query does encoding the standard way, can you provide a reference to that standard?

As I mentioned above, I am running a SPARQL query using the W3C SPARQL 1.1 Protocol. Specifically I am using the query via GET operation which specifies:

When using the GET method, clients must URL percent encode all parameters and include them as query parameter strings...

The referenced RFC 3986 specifies percent-encoding on page 12.

To my understanding, encoding spaces as a + in URL query string is more a "legacy" thing (for backwards compatibility), whilst percent-encoding is the "modern" approach. Would using percent-encoding for --url-query actually "break" anything?

See also: https://stackoverflow.com/questions/1634271/url-encoding-the-space-character-or-20

[outscale-mgo]

Here's the documentation about aws query encoding:
https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
and here is what it says about query encoding:

URI encode every byte. UriEncode() must enforce the following rules:

    URI encode every byte except the unreserved characters: 'A'-'Z', 'a'-'z', '0'-'9', '-', '.', '_', and '~'.

    The space character is a reserved character and must be encoded as "%20" (and not as "+").

    Each URI encoded byte is formed by a '%' and the two-digit hexadecimal value of the byte.

    Letters in the hexadecimal value must be uppercase, for example "%1A".

    Encode the forward slash character, '/', everywhere except in the object key name. For example, if the object key name is photos/Jan/sample.jpg, the forward slash in the key name is not encoded. 

Important

The standard UriEncode functions provided by your development platform may not work because 
of differences in implementation and related ambiguity in the underlying RFCs. 
We recommend that you write your own custom UriEncode function to ensure that your encoding will work.

So saying "AWS s3 isn't that standard" was a little of an overstatement.
The encoding require by s3 is standard, but doesn't support the whole standard.
The way curl encode query isn't wrong in any way, but presently doesn't work with V4 signature.

[outscale-mgo]

I'm thinking about it now @jaw111 .
But as curl v4 signature code already do the encoding (the way it is asked by v4 "standard"),
can you try :

--url-query "+query=select * where { ?s ?p ?o } limit 1" 

Because it might just work.

[jaw111]

The way curl encode query isn't wrong in any way

I'd prefer to say that curl is currently running in "backwards compatible" mode :)

It might not be wrong per-se, but it is also not compliant with what the SPARQL 1.1 Protocol sees as a must.

Whilst I expect that many servers will accept requests with + in the query parameter string without any errors (backwards compatibility), there are others that will not.

So the real question we should ask is if there are any servers out that that will throw errors if the query string parameter is percent encoded i.e. %20 instead of +? My hunch is they would be few and far between.

IMHO curl should be forward looking here and use percent encoding. Maybe for backwards compatibility you could provide a way to encode spaces as +, then using the + prefix before the parameter name makes sense:

--url-query "+param=hello+world"

[jaw111]

@outscale-mgo using non-encoded spaces with the plus prefix makes curl throw an error:

curl: (3) URL using bad/illegal format or missing URL

[outscale-mgo]

I'd prefer to say that curl is currently running in "backwards compatible" mode :)

But it's not only about the '+'.
AWS also require hexa to but uppercase, and encode '/' in some case, and not other as explain by this line:

Encode the forward slash character, '/', everywhere except in the object key name. For example, if the object key name is photos/Jan/sample.jpg, the forward slash in the key name is not encoded.

[jaw111]

@outscale-mgo let's not conflate two different things here.

Although I see the initial issue when signing the AWS requests, further analysis determines that the --url-query is encoding spaces as + rather than %20 in the query string, which is incompatible with the SPARQL 1.1 Protocol (which states clients must use percent-encoding).

So this current behavior may cause problems when sending requests to any SPARQL endpoint using --url-query, independent of the AWS signing or not.

I can agree the AWS Signature Version 4 documentation is a bit vague in the wording, maybe too S3-centric? I found more documentation under IAM and there it specifies the canonicalization algorithm in more detail. From that it seems that you should not encode the "/" in the absolute path, but would do elsewhere:

https://docs.aws.amazon.com/IAM/latest/UserGuide/create-signed-request.html#create-canonical-request

So seems a bit like encodeURI() vs. encodeURIComponent() in JavaScript.