Warnings when using Python 3.12

[Nuru]

Describe the bug

When compiling CLI v1 under Python 3.12, Python emits a long string of warnings related to invalid escape sequences in regular expressions:

$ aws --version
/usr/local/lib/python3.12/site-packages/awscli/compat.py:462: SyntaxWarning: invalid escape sequence '\s'
  _distributor_id_file_re = re.compile("(?:DISTRIB_ID\s*=)\s*(.*)", re.I)
...

Full version string:

aws-cli/1.32.48 Python/3.12.2 Linux/6.6.12-linuxkit botocore/1.34.48

Expected Behavior

No warnings emitted from a supported version of Python.

Current Behavior

9 separate warnings emitted about "invalid escape sequence".

Note this only happens when the code is compiled, which can be hidden by install process. However, when packaged in a Docker container, the compiled code is often removed, and the error shows up on first use when the code is recompiled.

Reproduction Steps

1. pip install --prefix=/usr/local --no-build-isolation awscli==1.32.48
2. find / -xdev -name __pycache__ -exec rm -rf {} \; -prune
3. aws --version

Possible Solution

Use raw strings instead of string literals. See example here

Additional Information/Context

From Stack Overflow:

Using invalid escape sequences in string literals has been deprecated since Python 3.6. Since then, attempting to use an invalid escape sequence has emitted a DeprecationWarning. This often goes unnoticed if you never run Python with warnings enabled. DeprecationWarnings are silenced by default.

Python 3.12 upgraded the DeprecationWarning to a SyntaxWarning. SyntaxWarnings are emitted by the compiler when the code is parsed, not when it's being run, so they cannot be ignored using a runtime warning filter. Unlike DeprecationWarnings, SyntaxWarnings are displayed by default, which is why you're seeing it now. This increase in visibility was intentional. In a future version of Python, using invalid escape sequence in string literals is planned to eventually become a hard SyntaxError.

CLI version used

1.32.48

Environment details (OS name and version, etc.)

aws-cli/1.32.48 Python/3.12.2 Linux/6.6.12-linuxkit botocore/1.34.48 Debian 12.4

[tim-finnigan]

Thanks for reaching out and for your patience here I could not reproduce this issue with that AWS CLI version using Python 3.12. Is this still something that you are seeing? And have you tried updating your version of the CLI? The latest version of v1 is 1.32.100 per the CHANGELOG.

[Nuru]

Yes, still seeing it in v1.32.100. Click below to see full details of reproduction.

Note that when aws is first installed by pip, it is compiled by Python with warnings suppressed. You have to delete the cached compiled Python files using

find / -xdev -name __pycache__ -exec rm -rf {} \; -prune

to see the error when the files are recompiled on demand. Removing the cached files is common when creating a Docker image, in order to reduce the size of the image.

My understanding is that in Python 3.13, these warnings will no longer be able to be suppressed, and will instead be full errors, so time spent fixing this now will be time saved later.

Click to reveal terminal capture of reproduction

HostPrompt$ docker run --rm -it  --entrypoint /bin/bash python:3.12.2-slim-bookworm
Unable to find image 'python:3.12.2-slim-bookworm' locally
3.12.2-slim-bookworm: Pulling from library/python
59f5764b1f6d: Already exists 
55af26b7addf: Already exists 
d3caa3c29ca1: Already exists 
e1240f83dd7d: Already exists 
6318e81a3dcc: Already exists 
Digest: sha256:5dc6f84b5e97bfb0c90abfb7c55f3cacc2cb6687c8f920b64a833a2219875997
Status: Downloaded newer image for python:3.12.2-slim-bookworm
root@b6dc205cc7b1:/# pip install --prefix=/usr/local --no-build-isolation awscli==1.32.100
Collecting awscli==1.32.100
  Downloading awscli-1.32.100-py3-none-any.whl.metadata (11 kB)
Collecting botocore==1.34.100 (from awscli==1.32.100)
  Downloading botocore-1.34.100-py3-none-any.whl.metadata (5.7 kB)
Collecting docutils<0.17,>=0.10 (from awscli==1.32.100)
  Downloading docutils-0.16-py2.py3-none-any.whl.metadata (2.7 kB)
Collecting s3transfer<0.11.0,>=0.10.0 (from awscli==1.32.100)
  Downloading s3transfer-0.10.1-py3-none-any.whl.metadata (1.7 kB)
Collecting PyYAML<6.1,>=3.10 (from awscli==1.32.100)
  Downloading PyYAML-6.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.metadata (2.1 kB)
Collecting colorama<0.4.7,>=0.2.5 (from awscli==1.32.100)
  Downloading colorama-0.4.6-py2.py3-none-any.whl.metadata (17 kB)
Collecting rsa<4.8,>=3.1.2 (from awscli==1.32.100)
  Downloading rsa-4.7.2-py3-none-any.whl.metadata (3.6 kB)
Collecting jmespath<2.0.0,>=0.7.1 (from botocore==1.34.100->awscli==1.32.100)
  Downloading jmespath-1.0.1-py3-none-any.whl.metadata (7.6 kB)
Collecting python-dateutil<3.0.0,>=2.1 (from botocore==1.34.100->awscli==1.32.100)
  Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting urllib3!=2.2.0,<3,>=1.25.4 (from botocore==1.34.100->awscli==1.32.100)
  Downloading urllib3-2.2.1-py3-none-any.whl.metadata (6.4 kB)
Collecting pyasn1>=0.1.3 (from rsa<4.8,>=3.1.2->awscli==1.32.100)
  Downloading pyasn1-0.6.0-py2.py3-none-any.whl.metadata (8.3 kB)
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1->botocore==1.34.100->awscli==1.32.100)
  Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Downloading awscli-1.32.100-py3-none-any.whl (4.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.5/4.5 MB 27.6 MB/s eta 0:00:00
Downloading botocore-1.34.100-py3-none-any.whl (12.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12.2/12.2 MB 43.0 MB/s eta 0:00:00
Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Downloading docutils-0.16-py2.py3-none-any.whl (548 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 548.2/548.2 kB 41.4 MB/s eta 0:00:00
Downloading PyYAML-6.0.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (696 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 696.9/696.9 kB 42.1 MB/s eta 0:00:00
Downloading rsa-4.7.2-py3-none-any.whl (34 kB)
Downloading s3transfer-0.10.1-py3-none-any.whl (82 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.2/82.2 kB 33.3 MB/s eta 0:00:00
Downloading jmespath-1.0.1-py3-none-any.whl (20 kB)
Downloading pyasn1-0.6.0-py2.py3-none-any.whl (85 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.3/85.3 kB 29.4 MB/s eta 0:00:00
Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.9/229.9 kB 53.1 MB/s eta 0:00:00
Downloading urllib3-2.2.1-py3-none-any.whl (121 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.1/121.1 kB 45.9 MB/s eta 0:00:00
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: urllib3, six, PyYAML, pyasn1, jmespath, docutils, colorama, rsa, python-dateutil, botocore, s3transfer, awscli
Successfully installed PyYAML-6.0.1 awscli-1.32.100 botocore-1.34.100 colorama-0.4.6 docutils-0.16 jmespath-1.0.1 pyasn1-0.6.0 python-dateutil-2.9.0.post0 rsa-4.7.2 s3transfer-0.10.1 six-1.16.0 urllib3-2.2.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@b6dc205cc7b1:/# 
root@b6dc205cc7b1:/# # NOTE: installing as normal user does not fix the problem
root@b6dc205cc7b1:/# # but does require a different starting directory for `find`
root@b6dc205cc7b1:/# 
root@b6dc205cc7b1:/# find / -xdev -name __pycache__ -exec rm -rf {} \; -prune
root@b6dc205cc7b1:/# aws --version
/usr/local/lib/python3.12/site-packages/awscli/compat.py:462: SyntaxWarning: invalid escape sequence '\s'
  _distributor_id_file_re = re.compile("(?:DISTRIB_ID\s*=)\s*(.*)", re.I)
/usr/local/lib/python3.12/site-packages/awscli/compat.py:463: SyntaxWarning: invalid escape sequence '\s'
  _release_file_re = re.compile("(?:DISTRIB_RELEASE\s*=)\s*(.*)", re.I)
/usr/local/lib/python3.12/site-packages/awscli/compat.py:464: SyntaxWarning: invalid escape sequence '\s'
  _codename_file_re = re.compile("(?:DISTRIB_CODENAME\s*=)\s*(.*)", re.I)
/usr/local/lib/python3.12/site-packages/awscli/shorthand.py:132: SyntaxWarning: invalid escape sequence '\!'
  _START_WORD = u'\!\#-&\(-\+\--\<\>-Z\\\\-z\u007c-\uffff'
/usr/local/lib/python3.12/site-packages/awscli/shorthand.py:133: SyntaxWarning: invalid escape sequence '\s'
  _FIRST_FOLLOW_CHARS = u'\s\!\#-&\(-\+\--\\\\\^-\|~-\uffff'
/usr/local/lib/python3.12/site-packages/awscli/shorthand.py:134: SyntaxWarning: invalid escape sequence '\s'
  _SECOND_FOLLOW_CHARS = u'\s\!\#-&\(-\+\--\<\>-\uffff'
/usr/local/lib/python3.12/site-packages/awscli/customizations/cloudtrail/validation.py:76: SyntaxWarning: invalid escape sequence '\d'
  pattern = re.compile('arn:.+:cloudtrail:.+:\d{12}:trail/.+')
/usr/local/lib/python3.12/site-packages/awscli/customizations/codedeploy/push.py:65: SyntaxWarning: invalid escape sequence '\<'
  'uploading. Use the format s3://\<bucket\>/\<key\>'
/usr/local/lib/python3.12/site-packages/awscli/customizations/emr/createcluster.py:204: SyntaxWarning: invalid escape sequence '\d'
  is_valid_ami_version = re.match('\d?\..*', parsed_args.ami_version)
aws-cli/1.32.100 Python/3.12.2 Linux/6.6.22-linuxkit botocore/1.34.100
root@b6dc205cc7b1:/# 

[tim-finnigan]

Thanks for clarifying — there have been other DeprectionWarnings issues reported across our repositories for Python 3.12 but those seem to relate to datetime issues. Will mark this issue for further review by the team.