aws_completer doesn't complete filenames

[Malvineous]

If you use a command like this:

aws s3 cp a<tab>

Then normally the <tab> will result in autocompleting a file in the current directory starting with a. However this does not happen, and pressing tab has no effect.

Can the autocompleter be updated to allow completing of filenames at the appropriate times?

[kyleknap]

Marking as a feature request. I think the difficult part about this is determining if a user wants to autocomplete a parameter/command or autocomplete a local file.

[ahuwaa]

@Malvineous what OS and terminal are you using? I'm able to autocomplete local filenames using tab on Mac OSX in iTerm2 and regular Terminal

[Malvineous]

@ahuwaa I'm using bash under Linux with aws-cli/bin/aws_bash_completer. Have you enabled an autocomplete script for your shell like this one? You might be using your shell's default autocomplete process which will only complete filenames but not aws-cli command parameters.

[ahuwaa]

Sorry, I misunderstood. I did not know there was an autocomplete feature but have now reproduced the issue.

I'd like to take a shot at this, if someone hasn't already started. Any advice/tips appreciated.

[ahuwaa]

Going to need help with this due to time constraints. So far I thought it was simplest to just default to the shell's autocomplete e.g. this works in bash

import subprocess
import pipes
# ...
def complete(cmdline, point):
    choices = Completer().complete(cmdline, point)
    if len(choices) > 0:
        print(' \n'.join(choices))
    else:
        cmd = cmdline.split(" ").pop()
        try:
            out = subprocess.check_output("compgen -o default " + pipes.quote(cmd), shell=True)
        except subprocess.CalledProcessError:
            out = ""
        print out

but of course this relies on something like compgen being available. If anyone has a better suggestion or would like to take over, feel free

[nhed]

@kyleknap remote filename completion is the primary thing I hoped for when I learned that there is a complete module, calling it a feature-request is somewhat of an understatement. Even scp in bash completion goes to the remote server over ssh to obtain list of files/paths. Having the path start with s3:// should give you the hint (as its the hint that aws cp uses to figure out which side of the copy is remote which is local)

(also should note that this should also complete aws s3 ls a<TAB> with the caveat that aws s3 ls is forgiving about the s3:// prefix and one could say that this is required for completion to work)

[CBribiescas]

Is there any ETA on this feature? Now with the new AWS S3 interface its a pain to have to type out paths. I have no idea why they took out the full path from the to of the page.

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

This entry can specifically be found on UserVoice at: https://aws.uservoice.com/forums/598381-aws-command-line-interface/suggestions/33168376-aws-completer-doesn-t-complete-filenames

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.

[excenter]

Recently installed
aws --version > aws-cli/1.15.60 Python/2.7.15 Darwin/17.7.0 botocore/1.10.59
still no tab completion on local files, which makes aws s3 cp meaningfully harder to use than it has to be.
If this isn't going to be fixed I'd rather uninstall tab completion for the cli and have file completion back, than have cli completion and not local file names.

[nhed]

[brunogasperini]

Any updates on this feature request? It would be extremely useful to have tabcompletion for s3 buckets and files.

[polarizeme]

Just chiming in with another voice on this request. I honestly thought my AWS CLI was just broken this entire time. 😬

[alisade]

really need this to work pretty please!

[erankor]

Hi all,

I patched the AWS completer to complete s3 paths, for example - typing:
aws s3 cp s3://mybucket/a(TAB)
will complete all prefixes starting with /a under mybucket.

I'm not submitting it as pull request since it's a bit hacky, but if anyone's interested - here is the patch -
completer-s3-paths.zip

Eran

[ragulpr]

@erankor, thanks for sharing! Haven't tried it yet but I inlined erankors patch below for those wary about downloading zips 🙂

completer-s3-paths.zip
        completer-s3-paths.patch
--- completer.py	2019-08-12 11:01:24.536333508 +0000
+++ /usr/lib/python3/dist-packages/awscli/completer.py	2019-08-12 11:00:00.152687298 +0000
@@ -14,6 +14,13 @@
 import logging
 import copy
 
+try:
+	import botocore
+	session = botocore.session.Session()
+	s3 = session.create_client('s3')
+except ImportError:
+	s3 = None
+
 LOG = logging.getLogger(__name__)
 
 
@@ -61,9 +68,38 @@
                 command_help.command_table, current_arg)
         return []
 
+    def _complete_s3_path(self, current_arg, depth = 10):
+        if depth <= 0:
+            return [current_arg]
+        splitted_arg = current_arg.split('/', 1)
+        if len(splitted_arg) < 2:
+            return []
+        bucketName, prefix = splitted_arg
+        paginator = s3.get_paginator('list_objects')
+        page_iterator = paginator.paginate(Bucket=bucketName,Prefix=prefix,Delimiter='/')
+
+        result = []
+        for page in page_iterator:
+            if 'CommonPrefixes' in page:
+                for item in page['CommonPrefixes']:
+                    result.append('%s/%s' % (bucketName, item['Prefix']))
+            if 'Contents' in page:
+                for item in page['Contents']:
+                    result.append('%s/%s' % (bucketName, item['Key']))
+
+        if len(result) == 1:
+            return self._complete_s3_path(result[0], depth - 1)
+        return result
+
     def _complete_subcommand(self, subcmd_name, subcmd_help, current_arg, opts):
         if current_arg != subcmd_name and current_arg.startswith('-'):
             return self._find_possible_options(current_arg, opts, subcmd_help)
+        elif current_arg.startswith('s3://') and s3 is not None:
+            current_arg = current_arg[len('s3://'):]
+            possibilities = self._complete_s3_path(current_arg)
+            if possibilities == [current_arg]:  # avoid duplication when clicking tab after reaching a leaf node
+                return []
+            return ['//' + n for n in possibilities]
         return []
 
     def _complete_option(self, option_name):

[diesal11]

Do any solutions for local files exist yet?

[max0r]

+1 for local file completion.

[erhhung]

@erankor I haven't tried your patch, but do you know if the list_objects call uses the preceding --profile myprofile option to obtain the proper credentials? I have lots of profiles for different accounts, so this would be a must.

[erankor]

Probably won't work, since I'm creating the s3 client directly in my patch, not using something provided by the aws cli.

[cervantek]

This feature request effects v2 as well.

The rules for when to drop through to client side file completion seem straightforward. This is not a complete list, but here are some examples that would be really useful.

For any command, look for file:// to do client side as in...
aws cloudformation create-stack --cli-input-json file://<tab for client-side completion of folders/files>

A few special cases that don't follow that rule of needing/expecting file://.
aws s3 cp
aws s3 cp s3://<tab for server-side completion, i.e. buckets>
aws s3 cp <tab for client-side completion folders/files>, do client-side

aws cloudformation package --template-file <tab for client-side completion of folders/files>

[gareth-li]

I also assumed this would be already built into aws_completer.

Making this a standard feature would be immensely useful.

[jaklan]

The issue was created more than 3 years ago, and such a completely fundamental functionality is still missing? No words...

[shugybugy-assaf]

It will be super useful - please implement
This one
aws s3 cp s3://<tab for server-side completion, i.e. buckets>
is the critical feature

[erankor]

After more than a year, I circled back to this open feature request...
Opened a PR this time #5849, it passed all checks now, hope it will be merged soon

[shawnzam]

Until this is fixed, you can use https://rclone.org/

[kdaily]

Hi @erankor,

I'm going to take a look at your PR. This is a pretty completx feature, especially for the S3 file name completion. I'll comment some more over there. Thanks!

[erankor]

@kdaily, did you check it out?

[erankor]

So, two months passed since I opened the PR... is this going to happen at some point?

[nikhilweee]

One word: Please. 🥺

[JonathanWeinrib]

Adding a +1 for this!

[spolloni]

hi

[dennis-park-TRI]

please.

[tnightengale]

OMFG just merge #5849 already, lfg are you serious AWS? Don't make me email Bezo-daddy directly and have an email like:

Sender: [email protected]
Subject: ?

show up in the inbox of the PM for this effing tool. Holy honkers, why have tests if you won't merge PRs that pass them!!!! 😠

[nhed]

Marking as a feature request. I think the difficult part about this is determining if a user wants to autocomplete a parameter/command or autocomplete a local file.

weak, how does scp completion work ?

[jwisser-heap]

Would love to see support for this added.

[timotheecour4]

+1; saws has had this feature for a long time (https://github.com/donnemartin/saws) ; unfortunately, saws doesn't support aws-cli v2 (donnemartin/saws#122)

[mc-borscht]

Please let this happen already

[dossy]

For anyone who lands here, frustrated that this issue is still unfixed, going on 5.5 years, and wants to temporarily disable the broken functionality and uses the bash shell, here you go:

# remove broken tab completion from shell
$ complete -r aws

# reinstate the broken tab completion
$ complete -C aws_completer aws

And, if you want to disable the functionality completely, comment out the complete -C aws_completer aws line from /usr/local/etc/bash_completion.d/aws_bash_completer.

[veriff-javiervgd]

Still no completions on s3:// paths?

[wwalker]

To get local path completion, the ugly way:

• Jump to start of the line, add an a (so now the command is aaws)
• go to the end of the line
  • do your tab completion for local file
• Then when you're ready to run
• jump to the start of the line
• remove the extra a
• hit enter.

[wwalker]

AWS, COME ON! It has been 5 years and you have a PR!!!! What the hell?

[zzhuolun]

Local path completion works now, s3:// path completion still not there.

My aws version:
aws-cli/2.11.23 Python/3.11.3 Linux/5.19.0-46-generic exe/x86_64.ubuntu.22 prompt/off

[mtravis]

@zzhuolun is that on linux? I still can't get the local path completion working when using v2.

[zzhuolun]

@zzhuolun is that on linux? I still can't get the local path completion working when using v2.

Yes on Linux.

Btw, with the bash shell, only local path completion works. And with fish shell, both local path and s3:// remote path completion work!

[mtravis]

@zzhuolun Thanks. I will check out the fish shell. Did you completely remove V1 of the aws-cli to get this working?

[zzhuolun]

@zzhuolun Thanks. I will check out the fish shell. Did you completely remove V1 of the aws-cli to get this working?

I started with aws-cli v2 without installing v1 in the first place.

[scottedwards2000]

@zzhuolun Thanks. I will check out the fish shell. Did you completely remove V1 of the aws-cli to get this working?

I started with aws-cli v2 without installing v1 in the first place.

wow are you saying that if we use fish shell we get the remote completion for s3?! whoa - wonder why that would just suddenly work without any code changes to the CLI? do you think this would work if I installed it on mac?

[zzhuolun]

@zzhuolun Thanks. I will check out the fish shell. Did you completely remove V1 of the aws-cli to get this working?

I started with aws-cli v2 without installing v1 in the first place.

wow are you saying that if we use fish shell we get the remote completion for s3?! whoa - wonder why that would just suddenly work without any code changes to the CLI? do you think this would work if I installed it on mac?

Yes, remote auto-completion of s3 works with fish shell. I'm not sure about mac OS.

My versions:
OS:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy

aws:
aws-cli/2.11.23 Python/3.11.3 Linux/6.5.0-25-generic exe/x86_64.ubuntu.22
fish:
fish, version 3.7.0

[mtravis]

@scottedwards2000

My colleague uses a Mac and he told me zsh works the same way as fish does here.

[scottedwards2000]

@scottedwards2000

My colleague uses a Mac and he told me zsh works the same way as fish does here.

Thanks @mtravis (and @zzhuolun )! I have zsh and i get aws autocomplete for everything else it seems (like even dynamo table names) but nothing for s3 (e.g. "aws s3api list-object-versions --bucket" when I tab after that, crickets. same with prefixing with "s3://"). Alot of my s3 bucket paths are pretty long, so this would be a HUGE help if I could get it to work. Do you think your colleague would be willing to give me some more details? I've checked my setup against all the things aws says to do and it's all correct. My contact details are in my profile here, or he could join our convo here. Thanks!