What is this chunked upload ?

[AyushSehrawat]

I was going through http docs of Deta and found https://docs.deta.sh/docs/drive/http#initialize-chunked-upload... Can anyone tell what this is about ? Something like uploading a file larger than x mb chunks by chunks via multiple requests ?
If possible provide some code ( just a sample kind of )

[jnsougata]

Yes you got it right!
Here's an example how I implemented it my async deta library! the quality of the code is a bit nasty...though

    async def _push_file(self, drive_name: str, remote_path: str, path: PathLike):

        if isinstance(path, str):
            file = open(path, 'rb')
        elif isinstance(path, bytes):
            file = io.BytesIO(path)
        else:
            raise ValueError('path must be a string or bytes')

        chunks = file.read()

        if not len(chunks) > self.__SINGLE_REQ_UPLOAD_SIZE:
            ep = self.__drive_root + drive_name + '/files?name=' + quote_plus(remote_path)
            resp = await self.__session.post(ep, headers=self.__drive_headers, data=chunks)
            file.close()
            if resp.status == 201:
                return await resp.json()
            elif resp.status == 400:
                raise BadRequest(self.__err(await resp.json()))
            else:
                raise Exception(self.__err(await resp.json()))

        ep = self.__drive_root + drive_name + '/uploads?name=' + quote_plus(remote_path)
        resp = await self.__session.post(ep, headers=self.__drive_headers)
        if resp.status == 202:
            upload_id = (await resp.json())['upload_id']
            chunked = [
                chunks[i:i+self.__SINGLE_REQ_UPLOAD_SIZE]
                for i in range(0, len(chunks), self.__SINGLE_REQ_UPLOAD_SIZE)
            ]
            uploads = []
            for i, chunk in enumerate(chunked[:-1]):
                post_ep = (
                    f"{self.__drive_root}{drive_name}/uploads/{upload_id}/parts?name={remote_path}&part={i+1}"
                )
                uploads.append(
                    asyncio.create_task(self.__session.post(post_ep, headers=self.__drive_headers, data=chunk))
                )
            gathered = await asyncio.gather(*uploads)
            file.close()
            for item in gathered:
                if isinstance(item, Exception):
                    abort_ep = f"{self.__drive_root}{drive_name}/uploads/{upload_id}?name={remote_path}"
                    resp = await self.__session.delete(abort_ep, headers=self.__drive_headers)
                    if resp.status == 200:
                        return await resp.json()
                    if resp.status == 400:
                        raise BadRequest(self.__err(await resp.json()))
                    if resp.status == 404:
                        raise NotFound(self.__err(await resp.json()))

            patch_ep = f"{self.__drive_root}{drive_name}/uploads/{upload_id}?name={remote_path}"
            resp = await self.__session.patch(patch_ep, headers=self.__drive_headers, data=chunked[-1])
            if resp.status == 200:
                return await resp.json()
            if resp.status == 400:
                raise BadRequest(self.__err(await resp.json()))
            if resp.status == 404:
                raise NotFound(self.__err(await resp.json()))

[jnsougata]

Man I already tried to make a cdn with deta but the 5.5mb will bottleneck the whole thing... deta micro is not meant for it

[AyushSehrawat]

Yeah, i was thinking bout it too, maybe something which can limit the upload to 5.5 mb and continute it from there in another request while temporary storing it.
Maybe js can do, dunno :/ . Hope it gets removed, atleast 100 should be fine ig.

[jnsougata]

I used heroku instead! https://github.com/jnsougata/cdn-flash

[AyushSehrawat]

Oh, heroku hmm.... Doesn't go well with me, instead i will use a server ;-;... For now, i will compress and upload or try some other way... Thnx for the chunked upload code though :)

[thomasmuus]

Hi @jnsougata and @AyushSehrawat ,

I have the following code for making upload of file in Deta drive. However as its deployed as Micro it cannot upload file larger than 5.5 MB as far as I know. I could not make sense of the above-mentioned code and also could not make it work.

@app.post("/upload", status_code=status.HTTP_201_CREATED)
async def upload_file(cvr: str = Form(...), file: UploadFile = File(...)):
    if file.content_type == 'text/csv' or file.content_type == 'application/vnd.ms-excel' or file.content_type == 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet':
        try:
            name = cvr + '_'+ datetime.date.today().strftime('%d_%m_%Y') + '_' + file.filename
            f = file.file
            res = email_vask_file_drive.put(name, f)
            return res
        except:
            raise HTTPException(status_code=422, detail={"error_message": f"Uploading file failed"})
    else:
        raise HTTPException(status_code=423, detail={"error_message": f"File extension must be either in .csv, .xls or .xlsx format"})
        

Is it possible to show me a chunked upload code snippet in Deta drive using FastAPI? This would help a lot of aspiring novice users like me :) Thanks!