Bump botocore dependency specification

[jakob-keller]

Description of Change

This PR intends to improve general compatibility of aiobotocore within the Python ecosystem by bumping the dependency specification of botocore, as well as boto3 and awscli.

Assumptions

Upstream diff contains changes that require adjustments to the aiobotocore codebase.

Checklist for All Submissions

• I have added change info to CHANGES.rst
• If this is resolving an issue (needed so future developers can determine if change is still necessary and under what conditions) (can be provided via link to issue with these details): closes Bump botocore dependency #1229
  • Detailed description of issue
  • Alternative methods considered (if any)
  • How issue is being resolved
  • How issue can be reproduced
• If this is providing a new feature (can be provided via link to issue with these details):
  • Detailed description of new feature
  • Why needed
  • Alternatives methods considered (if any)

Checklist when updating botocore and/or aiohttp versions

• I have read and followed CONTRIBUTING.rst
• I have updated test_patches.py where/if appropriate (also check if no changes necessary)
• I have ensured that the awscli/boto3 versions match the updated botocore version

[codecov]

Codecov Report

Attention: Patch coverage is 98.95833% with 1 line in your changes missing coverage. Please review.

Project coverage is 88.05%. Comparing base (f7e5acb) to head (a73f8c2).
Report is 5 commits behind head on master.

Files with missing lines	Patch %	Lines
tests/boto_tests/__init__.py	94.44%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1221      +/-   ##
==========================================
+ Coverage   87.73%   88.05%   +0.32%     
==========================================
  Files          65       67       +2     
  Lines        5823     5904      +81     
==========================================
+ Hits         5109     5199      +90     
+ Misses        714      705       -9     

Flag	Coverage Δ
unittests	88.05% <98.95%> (+0.32%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jakob-keller]

Notes to reviewers:

• I believe that upstream does not affect any codepaths relevant for us, beyond what is included in this PR. However, I am not 100% sure about _handle_200_error() in handlers.py. Please take a closer look at that aspect, if you can.
• FWIW: test coverage is not actually any worse than it was before.

[thehesiod]

Notes to reviewers:

• I believe that upstream does not affect any codepaths relevant for us, beyond what is included in this PR. However, I am not 100% sure about _handle_200_error() in handlers.py. Please take a closer look at that aspect, if you can.
• FWIW: test coverage is not actually any worse than it was before.

ya I'm not sure if response_dict['body'] is a coroutine or the actual bytes. I'd put a breakpoint and see what it looks like

[thehesiod]

I think it's populated here:

aiobotocore/aiobotocore/endpoint.py

Lines 57 to 65 in 8a50b61

	if response_dict['status_code'] >= 300:
	response_dict['body'] = await http_response.content
	elif operation_model.has_event_stream_output:
	response_dict['body'] = http_response.raw
	elif operation_model.has_streaming_output:
	length = response_dict['headers'].get('content-length')
	response_dict['body'] = StreamingBody(http_response.raw, length)
	else:
	response_dict['body'] = await http_response.content

so I think it depends on the response type

[thehesiod]

I think here's an example: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/eventstream.html

[thehesiod]

isn't line 63 also a coroutine? That's like a s3 get-object body i think

[thehesiod]

ideally we could add tests for this with moto

[jakob-keller]

I've included all pending botocore updates into this PR, including support for DSQL auth and all the other new and updated AWS services.

We still need include unit tests, in particular with regards to the issue discussed above. Hopefully this weekend...

[jakob-keller]

I think here's an example: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/eventstream.html

Well, that example doesn't work for me: AWS no longer accepts new users for S3 Select. I will need to find another service that uses event streams.

[thehesiod]

looks like bedrock supports eventstreams as well: https://docs.aws.amazon.com/bedrock/latest/userguide/agents-test.html

[jakob-keller]

seems like it's fairly easy to pull their tests in now, perhaps we should try to pull the whole thing over

I've included relevant unit tests from upstream, resulting in full test coverage.

[thehesiod]

sorry I thought I approved this yesterday

[thehesiod]

oh, shoot forgot to see if we got coverage now for that new method

[jakob-keller]

oh, shoot forgot to see if we got coverage now for that new method

Patch coverage is not relevant here, since the method remains in botocore and is not replaced by us.

I was not able to come up with an example that triggers that method locally. I tried the S3 Select API with and without moto and for some reason the breakpoints never hit. Probably my bad.

Having analyzed the code path, I would be surprised if we have an issue, but still can't fully rule it out. Looking at possible values for response_dict['body'] there are four cases, as you rightly pointed out:

I think it's populated here:

aiobotocore/aiobotocore/endpoint.py

Lines 57 to 65 in 8a50b61

	if response_dict['status_code'] >= 300:
	response_dict['body'] = await http_response.content
	elif operation_model.has_event_stream_output:
	response_dict['body'] = http_response.raw
	elif operation_model.has_streaming_output:
	length = response_dict['headers'].get('content-length')
	response_dict['body'] = StreamingBody(http_response.raw, length)
	else:
	response_dict['body'] = await http_response.content

so I think it depends on the response type

• case 1 results in bytes
• case 2 cannot be reached in this case
• case 3 might be reached, but I don't know how that could be handled. We are checking for an error message encoded in the response body. We would need to consume the streaming response to perform that check. I assume it would then be unavailable for consumption by the user?
• case 4 results in bytes

[thehesiod]

welp lets get it out, if something breaks we can figure out what tests we need to port over or new ones to write

[thehesiod]

this doesn't appear to have been needed