S3 ListObjectsV2OutputTypeDef Contents key should be NotRequired

[raymondbutcher]

Describe the bug

ListObjectsV2OutputTypeDef is a TypedDict which specifies the Contents key as List[ObjectTypeDef] but the API can return a response without this key. Can we please change this to NotRequired[List[ObjectTypeDef]]?

To Reproduce
Steps to reproduce the behavior:

1. Install boto3 and boto3-stubs[s3] and pyright
2. Run pyright on the following code sample:

import boto3

from typing_extensions import reveal_type

BUCKET_NAME = "your-bucket-goes-here"
PREFIX = "there-are-no-objects-with-this-prefix/"

s3_client = boto3.client("s3")
s3_paginator = s3_client.get_paginator("list_objects_v2")
for page in s3_paginator.paginate(Bucket=BUCKET_NAME, Prefix=PREFIX):
    reveal_type(page)
    for obj in page.get("Contents", []):
        print(obj)
    for obj in page["Contents"]:
        print(obj)

Actual output

bug.py
  bug.py:11:17 - information: Type of "page" is "ListObjectsV2OutputTypeDef"
0 errors, 0 warnings, 1 information

Expected output

If I manually change type_defs.pyi to use NotRequired[List[ObjectTypeDef]] then it does what I was expecting:

bug.py
  bug.py:11:17 - information: Type of "page" is "ListObjectsV2OutputTypeDef"
  bug.py:14:16 - error: Could not access item in TypedDict
    "Contents" is not a required key in "ListObjectsV2OutputTypeDef", so access may result in runtime exception (reportTypedDictNotRequiredAccess)
1 error, 0 warnings, 1 information

Additional context

Using macOS 14.2.1, Python 3.11.2, Poetry (version 1.4.2) for managing dependencies, boto3 1.34.23, boto3-stubs 1.34.23.

I printed the page dictionary and got the following. You can see that there is no Contents key.

{
    "ResponseMetadata": {
        "RequestId": "REDACTED",
        "HostId": "REDACTED",
        "HTTPStatusCode": 200,
        "HTTPHeaders": {
            "x-amz-id-2": "REDACTED",
            "x-amz-request-id": "REDACTED",
            "date": "Mon, 22 Jan 2024 13:26:25 GMT",
            "x-amz-bucket-region": "us-east-1",
            "content-type": "application/xml",
            "transfer-encoding": "chunked",
            "server": "AmazonS3",
        },
        "RetryAttempts": 0,
    },
    "IsTruncated": False,
    "Name": "REDACTED",
    "Prefix": "there-are-no-objects-with-this-prefix/",
    "MaxKeys": 1000,
    "EncodingType": "url",
    "KeyCount": 0,
}

[vemel]

Unfortunately, botocore shapes mark all output shape keys as required. I will check if there is a way to distinguish between keys that can be missing.

[guilhem-dvr]

A similar issue happens with the key CommonPrefixes in ListObjectVersionsOutputTypeDef in the response of ListObjectVersion. It is typed as a List[CommonPrefixTypeDef] in ListObjectVersionsOutputTypeDef but can be missing from the response - likely if the list of common prefixes is empty.

I can open an issue if you want to keep track of both bugs separately --> done here #248

[vemel]

The issue has finally been fixed in mypy_boto3_builder 7.24.0 release. I have also released mypy-boto3-s3 1.34.91 with the fix included. Please update and let me know if it works as expected.