aws ecs execute-command TargetNotConnectedException

[grepsedawk]

Confirm by changing [ ] to [x] below to ensure that it's a bug:

• [ x] I've gone though the User Guide and the API reference
• [ x] I've searched for previous similar issues and didn't find any solution

Describe the bug
When I run aws ecs execute-command it results in:

An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

However, the issue seems different than previous issues. When I check for exec status via https://github.com/aws-containers/amazon-ecs-exec-checker/blob/main/check-ecs-exec.sh it shows it should be working, but I get still get TargetNotConnectedException.

CLI version: aws-cli/2.2.43 Python/3.8.8 Linux/5.11.0-7620-generic exe/x86_64.pop.20

To Reproduce (observed behavior)
Steps to reproduce the behavior

• Create task running on ec2
• Try to connect to task via aws cli exec

Expected behavior
Connect to ecs container using exec

Logs/output

2021-10-07 00:27:02,310 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.2.43 Python/3.8.8 Linux/5.11.0-7620-generic exe/x86_64.pop.20
2021-10-07 00:27:02,310 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['ecs', 'execute-command', '--task', '43048ec3a1d149a2a0eceee69fe8d02d', '--command', '/bin/sh', '--interactive', '--cluster', 'group-duration-month-option', '--debug']
2021-10-07 00:27:02,316 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7f48e380cb80>
2021-10-07 00:27:02,316 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7f48e39655e0>
2021-10-07 00:27:02,316 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2021-10-07 00:27:02,316 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7f48e3a08f70>
2021-10-07 00:27:02,316 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7f48e3a10dc0>
2021-10-07 00:27:02,316 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7f48e381b5e0>
2021-10-07 00:27:02,316 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7f48e392f430>
2021-10-07 00:27:02,316 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2021-10-07 00:27:02,316 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7f48e3812820>
2021-10-07 00:27:02,316 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.43/dist/awscli/data/cli.json
2021-10-07 00:27:02,318 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7f48e3861670>
2021-10-07 00:27:02,318 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7f48e38621f0>
2021-10-07 00:27:02,318 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7f48e3862160>
2021-10-07 00:27:02,318 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7f48e3862310>
2021-10-07 00:27:02,318 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7f48e3862280>
2021-10-07 00:27:02,318 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7f48e37361c0>
2021-10-07 00:27:02,319 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.2.43 Python/3.8.8 Linux/5.11.0-7620-generic exe/x86_64.pop.20 prompt/off
2021-10-07 00:27:02,319 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['ecs', 'execute-command', '--task', '43048ec3a1d149a2a0eceee69fe8d02d', '--command', '/bin/sh', '--interactive', '--cluster', 'group-duration-month-option', '--debug']
2021-10-07 00:27:02,319 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7f48e380f1f0>
2021-10-07 00:27:02,319 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7f48e425c310>
2021-10-07 00:27:02,319 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7f48e3779040>
2021-10-07 00:27:02,319 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7f48e4253670>
2021-10-07 00:27:02,319 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7f48e41adee0>
2021-10-07 00:27:02,320 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2021-10-07 00:27:02,321 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7f48e392f310>
2021-10-07 00:27:02,321 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7f48e3963430>
2021-10-07 00:27:02,326 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.43/dist/botocore/data/ecs/2014-11-13/service-2.json
2021-10-07 00:27:02,333 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ecs: calling handler <function inject_commands at 0x7f48e38d1e50>
2021-10-07 00:27:02,333 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ecs: calling handler <function add_waiters at 0x7f48e3812820>
2021-10-07 00:27:02,338 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.43/dist/botocore/data/ecs/2014-11-13/waiters-2.json
2021-10-07 00:27:02,339 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('cluster', <awscli.arguments.CLIArgument object at 0x7f48e2d3e4f0>), ('container', <awscli.arguments.CLIArgument object at 0x7f48e2d3e430>), ('command', <awscli.arguments.CLIArgument object at 0x7f48e2d3e400>), ('interactive', <awscli.arguments.BooleanArgument object at 0x7f48e2d3e3a0>), ('no-interactive', <awscli.arguments.BooleanArgument object at 0x7f48e2d3e370>), ('task', <awscli.arguments.CLIArgument object at 0x7f48e2d3e340>)])
2021-10-07 00:27:02,339 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function add_streaming_output_arg at 0x7f48e380f790>
2021-10-07 00:27:02,339 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function rename_arg.<locals>._rename_arg at 0x7f48e379eee0>
2021-10-07 00:27:02,339 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function rename_arg.<locals>._rename_arg at 0x7f48e379ef70>
2021-10-07 00:27:02,339 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function add_cli_input_json at 0x7f48e41b9700>
2021-10-07 00:27:02,339 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function add_cli_input_yaml at 0x7f48e41b99d0>
2021-10-07 00:27:02,339 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function unify_paging_params at 0x7f48e3965c10>
2021-10-07 00:27:02,345 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.43/dist/botocore/data/ecs/2014-11-13/paginators-1.json
2021-10-07 00:27:02,345 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ecs.execute-command: calling handler <function add_generate_skeleton at 0x7f48e38d6c10>
2021-10-07 00:27:02,345 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ecs.execute-command: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7f48e2d3e670>>
2021-10-07 00:27:02,345 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ecs.execute-command: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7f48e2d3e6a0>>
2021-10-07 00:27:02,345 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ecs.execute-command: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7f48e2d3e7f0>>
2021-10-07 00:27:02,346 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.cluster: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f48e2efbfa0>
2021-10-07 00:27:02,346 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.ecs.execute-command: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f48e75c6460>
2021-10-07 00:27:02,346 - MainThread - awscli.arguments - DEBUG - Unpacked value of 'group-duration-month-option' for parameter "cluster": 'group-duration-month-option'
2021-10-07 00:27:02,346 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.container: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f48e2efbfa0>
2021-10-07 00:27:02,346 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.command: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f48e2efbfa0>
2021-10-07 00:27:02,346 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.ecs.execute-command: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f48e75c6460>
2021-10-07 00:27:02,346 - MainThread - awscli.arguments - DEBUG - Unpacked value of '/bin/sh' for parameter "command": '/bin/sh'
2021-10-07 00:27:02,346 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.interactive: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f48e2efbfa0>
2021-10-07 00:27:02,346 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.task: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f48e2efbfa0>
2021-10-07 00:27:02,346 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.ecs.execute-command: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7f48e75c6460>
2021-10-07 00:27:02,346 - MainThread - awscli.arguments - DEBUG - Unpacked value of '43048ec3a1d149a2a0eceee69fe8d02d' for parameter "task": '43048ec3a1d149a2a0eceee69fe8d02d'
2021-10-07 00:27:02,346 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f48e2efbfa0>
2021-10-07 00:27:02,346 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f48e2efbfa0>
2021-10-07 00:27:02,346 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ecs.execute-command.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7f48e2efbfa0>
2021-10-07 00:27:02,346 - MainThread - botocore.hooks - DEBUG - Event calling-command.ecs.execute-command: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7f48e2d3e670>>
2021-10-07 00:27:02,346 - MainThread - botocore.hooks - DEBUG - Event calling-command.ecs.execute-command: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7f48e2d3e6a0>>
2021-10-07 00:27:02,346 - MainThread - botocore.hooks - DEBUG - Event calling-command.ecs.execute-command: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7f48e2d3e7f0>>

The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.

2021-10-07 00:27:02,358 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2021-10-07 00:27:02,358 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2021-10-07 00:27:02,358 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/v2/2.2.43/dist/botocore/data/endpoints.json
2021-10-07 00:27:02,362 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f48e55e93a0>
2021-10-07 00:27:02,363 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.ecs: calling handler <function add_generate_presigned_url at 0x7f48e5618940>
2021-10-07 00:27:02,365 - MainThread - botocore.endpoint - DEBUG - Setting ecs timeout as (60, 60)
2021-10-07 00:27:02,366 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.ecs.ExecuteCommand: calling handler <function base64_decode_input_blobs at 0x7f48e3779790>
2021-10-07 00:27:02,366 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.ecs.ExecuteCommand: calling handler <function generate_idempotent_uuid at 0x7f48e56094c0>
2021-10-07 00:27:02,366 - MainThread - botocore.hooks - DEBUG - Event before-call.ecs.ExecuteCommand: calling handler <function inject_api_version_header_if_needed at 0x7f48e558cd30>
2021-10-07 00:27:02,366 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=ExecuteCommand) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AmazonEC2ContainerServiceV20141113.ExecuteCommand', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'aws-cli/2.2.43 Python/3.8.8 Linux/5.11.0-7620-generic exe/x86_64.pop.20 prompt/off command/ecs.execute-command'}, 'body': b'{"cluster": "group-duration-month-option", "command": "/bin/sh", "interactive": true, "task": "43048ec3a1d149a2a0eceee69fe8d02d"}', 'url': 'https://ecs.us-east-1.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f48e2a6ba60>, 'has_streaming_input': False, 'auth_type': None}}
2021-10-07 00:27:02,366 - MainThread - botocore.hooks - DEBUG - Event request-created.ecs.ExecuteCommand: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f48e2a6bb20>>
2021-10-07 00:27:02,366 - MainThread - botocore.hooks - DEBUG - Event choose-signer.ecs.ExecuteCommand: calling handler <function set_operation_specific_signer at 0x7f48e56093a0>
2021-10-07 00:27:02,367 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2021-10-07 00:27:02,367 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:ecs.us-east-1.amazonaws.com
x-amz-date:20211007T062702Z
x-amz-target:AmazonEC2ContainerServiceV20141113.ExecuteCommand

content-type;host;x-amz-date;x-amz-target
5cbc4f06963921e8a6f28fe0074abb6a1851b4dba48d2502440ab681942b8d62
2021-10-07 00:27:02,367 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20211007T062702Z
20211007/us-east-1/ecs/aws4_request
0dba831b1c835517e447946229c99e9f7ce16a78a1b40474b8d83c32928843a2
2021-10-07 00:27:02,367 - MainThread - botocore.auth - DEBUG - Signature:
9f9f0a6e31205a280bd3994e4e8ee932d7fc316305c3a47a9faf0335562471e0
2021-10-07 00:27:02,367 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ecs.us-east-1.amazonaws.com/, headers={'X-Amz-Target': b'AmazonEC2ContainerServiceV20141113.ExecuteCommand', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'aws-cli/2.2.43 Python/3.8.8 Linux/5.11.0-7620-generic exe/x86_64.pop.20 prompt/off command/ecs.execute-command', 'X-Amz-Date': b'20211007T062702Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=AKIAWWFESX62U4LDHBP4/20211007/us-east-1/ecs/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-target, Signature=9f9f0a6e31205a280bd3994e4e8ee932d7fc316305c3a47a9faf0335562471e0', 'Content-Length': '129'}>
2021-10-07 00:27:02,367 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/v2/2.2.43/dist/botocore/cacert.pem
2021-10-07 00:27:02,368 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): ecs.us-east-1.amazonaws.com:443
2021-10-07 00:27:03,020 - MainThread - urllib3.connectionpool - DEBUG - https://ecs.us-east-1.amazonaws.com:443 "POST / HTTP/1.1" 400 122
2021-10-07 00:27:03,020 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': 'a663ad46-bdd2-4cf1-8b81-69479cddc52b', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '122', 'Date': 'Thu, 07 Oct 2021 06:27:02 GMT', 'Connection': 'close'}
2021-10-07 00:27:03,021 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"__type":"TargetNotConnectedException","message":"The execute command failed due to an internal error. Try again later."}'
2021-10-07 00:27:03,023 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': 'a663ad46-bdd2-4cf1-8b81-69479cddc52b', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '122', 'Date': 'Thu, 07 Oct 2021 06:27:02 GMT', 'Connection': 'close'}
2021-10-07 00:27:03,023 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"__type":"TargetNotConnectedException","message":"The execute command failed due to an internal error. Try again later."}'
2021-10-07 00:27:03,023 - MainThread - botocore.hooks - DEBUG - Event needs-retry.ecs.ExecuteCommand: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7f48e2a2f5b0>>
2021-10-07 00:27:03,023 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2021-10-07 00:27:03,023 - MainThread - botocore.hooks - DEBUG - Event after-call.ecs.ExecuteCommand: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7f48e2a2f130>>
2021-10-07 00:27:03,024 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli/clidriver.py", line 459, in main
  File "awscli/clidriver.py", line 594, in __call__
  File "awscli/clidriver.py", line 770, in __call__
  File "awscli/customizations/ecs/executecommand.py", line 91, in invoke
  File "botocore/client.py", line 278, in _api_call
  File "botocore/client.py", line 597, in _make_api_call
botocore.errorfactory.TargetNotConnectedException: An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

[kdaily]

Hi @grepsedawk,

Sorry to hear you're having trouble. To clarify, your issue is not related to this one:

#6070

This ended up being related to needing a writable filesystem:

aws-containers/amazon-ecs-exec-checker#21 (comment)

I might need to ping the ECS team to look into this more if not.

[grepsedawk]

Nope! It's a read-write file system!
Additionally the ecs-exec-checker you linked is what I used, and that passes with flying colors

[mateuszwieloch]

I'm experiencing exactly the same issue. Just like OP, I have granted all required permissions to ECS task and IAM role. Read-write file system. ExecuteCommandAgent running.

[grepsedawk]

@mateuszwieloch does https://github.com/aws-containers/amazon-ecs-exec-checker/blob/main/check-ecs-exec.sh say everything is good to go for you as well?

[mateuszwieloch]

@grepsedawk It's a little complicated answer. The tool is showing ecs:implicitDeny and TaskRolePermissions as failing, due to a known bug/limitation in the ecs-exec-checker tool itself. See aws-containers/amazon-ecs-exec-checker#44

To be precise, my organization has AWS Organization SCP rules that limit users to certain regions and the ecs-exec-checker tool currently doesn't pass in RequestedRegion in the call to IAM permission checker. Once I manually added the region, the rules passed.

[kdaily]

Just checking in here, I've reached out to the ECS team, waiting to hear back.

[grepsedawk]

Cool beans!
Let me know if there's anything I can do to help here.
We're willing to hop on a call if a tech wants to explore what we have configured

[grepsedawk]

I never set up routes for my containers to talk out. (In this case, NAT rules).
Now that my containers can talk out, ecs exec works.
This was not a check in the ecs checker tool

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.