Storage mixin does not work with boto3

[merwok]

Hello! To use this filebrowser with django-tinymce and S3 storage for uploaded files, I combined S3Boto3Storage backend with your S3BotoStorageMixin.

I got this exception:

AttributeError
filebrowser.decorators.path_exists..decorator
's3.Bucket' object has no attribute 'list'

[smacker]

Hi @merwok ! Thanks for the bug report!

Sadly, I don't use S3 storage but I believe you can make it work by using an older version of S3Boto3Storage which is compatible with this mixin.

Anyway I would recommend you create an issue or pull request to upstream: https://github.com/sehmaschine/django-filebrowser

[merwok]

I wrote my own mixin and will put it in a gist!

[smacker]

Hi @merwok !

Please feel free to open PR with you mixin.

I also see @FutureMind made this fix: FutureMind@0b9cda4

Does it work for you? I'll be happy to fix the issue if I somebody provide PR with a tested mixin.

[merwok]

This is a bit rough but functional:

https://gist.github.com/merwok/3365ed649500baf0aae3a5f3263fa7b5

The issues are:

1. needs custom code to support «directories» on S3, e.g. by saving empty .keep files
2. for my project, a tree of directories doesn’t really help, and most uploaded files are unique images (so I just need easy upload for my rich text fields)
3. with django 2.1 and https://github.com/fabiocaccamo/django-admin-interface , the filebrowser templates and styles do not fit at all

I will probably remove/rewrite the whole filebrowser, but feel free to start from my code and improve it!

[kudlatyamroth]

i used django-s3-storage
and with this code:

class S3Boto3Storage(S3Storage, StorageMixin):
    def path(self, name):
        return self._get_key_name(name)

    @_wrap_errors
    def meta(self, name):
        object_params = self._object_params(name)
        try:
            return self.s3_connection.head_object(**object_params)
        except (S3Error, ClientError):
            object_params['Key'] = object_params['Key'] + '/'
            return self.s3_connection.head_object(**object_params)

    def isfile(self, name):
        return self.exists(name)

    def isdir(self, name):
        dir_list = self.listdir(name)
        return any(dir_list)

    def move(self, old_file_name, new_file_name, allow_overwrite=False):
        if self.exists(new_file_name):
            if allow_overwrite:
                self.rmtree(new_file_name)
            else:
                raise S3Error(f"The destination file '{new_file_name}' exists and allow_overwrite is False")

        if self.isdir(old_file_name):
            self.makedirs(new_file_name)
            self._move_dir(old_file_name, new_file_name)
        else:
            self._move_file(old_file_name, new_file_name)

    def _move_dir(self, source, destination):
        dirs, files = self.listdir(source)
        for directory in dirs:
            source_dir = f"{source.rstrip('/')}/{directory}/"
            destination_dir = f"{destination.rstrip('/')}/{directory}/"
            self.move(source_dir, destination_dir)

        for file in files:
            if file == '.':
                continue
            source_path = '/'.join([source, file])
            destination_path = '/'.join([destination.rstrip('/'), file])
            self._move_file(source_path, destination_path)
        self.rmtree(source)

    def _move_file(self, source, destination):
        source_params = self._object_params(source)
        new_key_name = self._get_key_name(destination)
        extra_args = {
            'ACL': 'public-read'
        }

        try:
            self.s3_connection.copy(source_params, source_params['Bucket'], new_key_name, extra_args)
        except ClientError:
            raise S3Error(f"Couldn't copy '{source}' to '{destination}'")
        self.delete(source)

    def makedirs(self, name):
        put_params = self._object_put_params(name)
        if not put_params['Key'].endswith('/'):
            put_params['Key'] = f"{put_params['Key']}/"
        self.s3_connection.put_object(**put_params)

    def rmtree(self, name):
        dirs, files = self.listdir(name)
        for item in dirs:
            dir_path = '/'.join([name, item])
            self.rmtree(dir_path)
        for item in files:
            if item == '.':
                continue
            self.delete('/'.join([name, item]))
        self.delete(name)

    @_wrap_errors
    def delete(self, name):
        object_params = self._object_params(name)
        try:
            head = self.s3_connection.head_object(**object_params)
            self.s3_connection.delete_object(**object_params)
        except (S3Error, ClientError):
            object_params['Key'] = object_params['Key'] + '/'
            self.s3_connection.delete_object(**object_params)

    def setpermission(self, name):
        pass

it works for me.
but when we change dir name, it can take long time, so remember about setting long timeout in server...

---

updated rmtree function