AWS SDK Core 3.192.0 broke older SQS gem ability to receive messages

[dannyfallon]

Describe the bug

We're running aws-sdk-sqs v1.40.0 and aws-sdk-autoscaling v1.59.0 both via a loose ~> 1 version in the Gemfile.

We needed to update the awk-sdk-autoscaling gem to get new functionality, which bumped the version of aws-sdk-core to the current latest version (3.200.0). After this, every request to the SQS ReceiveMessage API returned no messages even though it took a message from the queue.

Expected Behavior

I expected any one of the following to happen, in order of preference:

1. The SQS gem to continue to receive messages as it did before
2. The SDK to raise an exception of some kind instead of swallowing the message silently. Polling for messages can return no results, so I can't check this on the application side.
3. The bundle update / install to error with a version conflict caused by my older gems if this error can't be resolved.

Current Behavior

Aws::SQS::Client.new.receive_message(params)
[Aws::SQS::Client 200 0.381394 0 retries] receive_message(queue_url:"https://sqs.us-east-1.amazonaws.com/__account__/__queue__",wait_time_seconds:20,visibility_timeout:nil,attribute_names:["All"],message_attribute_names:["All"],max_number_of_messages:1)

=> #<struct Aws::SQS::Types::ReceiveMessageResult messages=[]>

It pulls the message, hiding it from the rest of the queue's consumers but does not make the message available for use.

Reproduction Steps

# frozen_string_literal: true

require "bundler/inline"

gemfile(true) do
  source "https://rubygems.org"

  gem "aws-sdk-core", "3.192.0"
  gem "aws-sdk-sqs", "1.40.0"
  gem "nokogiri"
end

require "aws-sdk-sqs"
require "logger"

client = Aws::SQS::Client.new(region: "us-east-1", http_wire_trace: true, logger: Logger.new($stdout))
queue_url = "https://sqs.us-east-1.amazonaws.com/__account__/__queue__"
puts client.receive_message({ queue_url: queue_url })

Possible Solution

No response

Additional Information/Context

Here's the HTTP trace from the same request as the repro above, clearly showing there was a message in the response:

opening connection to sqs.us-east-1.amazonaws.com:443...
opened
starting SSL for sqs.us-east-1.amazonaws.com:443...
SSL established, protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256
<- "POST /__account__/__queue__ HTTP/1.1\r\nAccept-Encoding: \r\nContent-Type: application/x-www-form-urlencoded; charset=utf-8\r\nUser-Agent: aws-sdk-ruby3/3.192.0 ua/2.0 api/sqs#1.40.0 os/macos#23 md/arm64 lang/ruby#3.2.2 md/3.2.2 cfg/retry-mode#legacy\r\nHost: sqs.us-east-1.amazonaws.com\r\nX-Amz-Date: 20240702T144627Z\r\nX-Amz-Security-Token: <snip>\r\nX-Amz-Content-Sha256: 50ab40d11201b3e21753039def6064663b8545b11ffa85cd2932b833ef94d36a\r\nAuthorization: AWS4-HMAC-SHA256 Credential=<snip>, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=d214b3ef90a7ae40b603562e6b62df6b3dcacc49c5fefa61ab8e22263971c262\r\nContent-Length: 238\r\nAccept: */*\r\n\r\n"
<- "Action=ReceiveMessage&AttributeNames.1=All&MaxNumberOfMessages=1&MessageAttributeNames.1=All&QueueUrl=https%3A%2F%2Fsqs.us-east-1.amazonaws.com%2F__account__%2F__queue__&Version=2012-11-05&WaitTimeSeconds=20"
-> "HTTP/1.1 200 OK\r\n"
-> "x-amzn-RequestId: 3ed3869c-d846-537d-93e9-d862cd42f67f\r\n"
-> "Date: Tue, 02 Jul 2024 14:46:28 GMT\r\n"
-> "Content-Type: text/xml\r\n"
-> "Content-Length: 1244\r\n"
-> "connection: keep-alive\r\n"
-> "\r\n"
reading 1244 bytes...
-> "<?xml version=\"1.0\"?><ReceiveMessageResponse xmlns=\"http://queue.amazonaws.com/doc/2012-11-05/\"><ReceiveMessageResult><Message><MessageId>336bc484-0746-48a5-b62b-e1dca16a20af</MessageId><ReceiptHandle>AQEBiWfo5hLCgDXepDR67duzntUGFFnYP40/bZmy69wApwNriyMlFOtUBAcrzeGZRpwPz+D7gvn5K73vrERmrfhS4n4eutKcj0xNJdrbzrlEvtxV5Aj2sg/AXRJsUR5BUZfc3kyLlNnGl+FhxfjObDKLLXWJFxVF6TJMvs29bNxfV6YIMdzFcwhDtleo0QeP49kNIPigNWLNbKXcr/I1IVnC3ZH2H6+YxUUXN1P4vqGNa6H0ZIWgx67TgUyWs0R+zYJGr4MPtDNIwBbnxCDeyCQ/WjLrWqCQ6Nstnl2TWBO+zcKcfq02avOzvM6PXSZxeo+fNq10yk8y/0jZJSnLuNchBI2W9PwrtGdNFNd1w/Zs1mF+ow2iC9awUr6eXEMy3hHJlU7oaj4T4zrByA3pTTMILT+wM4cNkD/8uaMGO7kWTFc=</ReceiptHandle><MD5OfBody>00dbe7ca5d2da5343982d2af774299fd</MD5OfBody><Body>{ &quot;danny&quot;: &quot;test&quot; }</Body><Attribute><Name>SenderId</Name><Value>AROAWASXDESQVM6QBZVPH:danny</Value></Attribute><Attribute><Name>ApproximateFirstReceiveTimestamp</Name><Value>1719930948583</Value></Attribute><Attribute><Name>ApproximateReceiveCount</Name><Value>2</Value></Attribute><Attribute><Name>SentTimestamp</Name><Value>1719930368330</Value></Attribute></Message></ReceiveMessageResult><ResponseMetadata><RequestId>3ed3869c-d846-537d-93e9-d862cd42f67f</RequestId></ResponseMetadata></ReceiveMessageResponse>"
read 1244 bytes
Conn keep-alive

Gem name ('aws-sdk', 'aws-sdk-resources' or service gems like 'aws-sdk-s3') and its version

aws-sdk-core 3.192.0+, aws-sdk-sqs 1.40.0

Environment details (Version of Ruby, OS environment)

Ruby 3.2.2

[mullermp]

We were aware of this. Please roll forward to at least sqs 1.67.0 where json protocol is used.

[dannyfallon]

Are there any plans to address it in the core at some stage? This was deeply surprising behaviour that flew completely under the radar.

[jterapin]

I'm very sorry about this. There was an organizational effort to align all protocol behaviors across AWS SDKs and these changes were not intended to break SQS.

SQS has switched from using aws query protocol to using the json 1.0 protocol in version 1.167.0. From my investigation, SQS (query) was an exception to the query alignment addressed in this protocol test case.

My recommendation will be upgrading the sqs gem, since new features of awk-sdk-autoscaling will require the most recent aws-sdk-core.

[mullermp]

This would not be feasible to fix because it would require a service specific customization in core. Typically we would implement service customizations in the service gems themselves but in this case, the latest sqs is no longer query. I will bubble this issue up to the service frameworks team who provided these tests that SDKs follow, however, I think unfortunately this is just a miss from our side and the best mitigation is just to upgrade, as it should also be backwards compatible and easy to upgrade the sqs gem.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.