Atomicity of Deta Drive file

[begoon]

Hey,

What happens when two workers of a micro (according to logs, I see at least two of them) are writing to the same Deta Drive file?

If the file is big enough, both workers may write simultaneously to the same file.

What is it expected behaviour of the file consistency?

[jnsougata]

big files are uploaded in chunked parts so that won't be an issue afaik

[begoon]

May the chunks from different writers be be mixed up eventually in the file? Based on the API description, the chunks are uploaded in separate requests. How will the server know which chunk was saved before the current it is now processing? Anyhow, let’s ask it this way: is “drive.put(filename)” call is atomic?

…

On 5 Dec 2022 at 23:54 +0100, Sougata Jana ***@***.***>, wrote: big files are uploaded in chunked parts so that won't be an issue afaik — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[jnsougata]

May the chunks from different writers be be mixed up eventually in the file? Based on the API description, the chunks are uploaded in separate requests. How will the server know which chunk was saved before the current it is now processing? Anyhow, let’s ask it this way: is “drive.put(filename)” call is atomic?
…
On 5 Dec 2022 at 23:54 +0100, Sougata Jana @.>, wrote: big files are uploaded in chunked parts so that won't be an issue afaik — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.>

To my understanding... let's assume two workers sending same chunk then the chunk from the 1st worker will be overwritten by the second. Technically two workers can't write at the same time with nanosecond order precison so it's safe to assume the mixup won't happen. more over one has to send a patch request to end the chunk upload ig at that time all chunks are joined by their index to form a full file.