[Issue] sending binary data

[armandofln]

I think there's a wrong configuration on the Deta platform which prevents me from sending binary data, like an image.

Following the example from the Deta guide, I used this snippet in my project:

app.get("/download/:name", async (req, res) => {
    const name = req.params.name;
    const img = await drive.get(name);
    const buffer = await img.arrayBuffer();
    res.send(Buffer.from(buffer));
}); 

app.listen(3000);

https://docs.deta.sh/docs/drive/node_tutorial#downloading-images

This code is supposed to extract data from Deta Drive and then serve the data with Deta Micros, like for example serving an image. I did the same in a node.js project, trying to serve a .png image.

This works when I host the express.js web server locally (while using the Deta Drive to retrieve the image). But it stops working when I deploy the server on the Deta platform and follow the appropriate id.deta.dev link.

Basically the data that is retrieved by the drive.get(name) function and the data that is then sent with res.send(buffer) is different. I dug a little bit to understand what's the issue and finally I came up with an answer.

If we first look at the lengths of the two different versions of the same data, the data received by the client is always longer than the data that is sent by the webserver. The data is treated correctly in the snippets of code above, so the following holds:

img.size == buffer.byteLength == Buffer.from(buffer).length == <correct size of the image in number of bytes>

But then the data that is received from the browser is larger.

Then I decided to take a look at the hex data of the two different versions of the file. Here some examples:

89 50 4E 47 0D 0A 1A ... (original/sent) (start of the data, with the magic number of the .png file)
EF BF BD 50 4E 47 0D 0A 1A ... (received)

... 00 D3 10 3F 31 00 ... (original/sent)
... 00 EF BF BD 10 3F 31 00 ... (received)

... 54 78 9C ED DD 77 ... (original/sent)
... 54 78 EF BF BD EF BF BD EF BF BD 77 ... (received)

If we take a closer look, we can see that some bytes are transformed into a group of three bytes: 0xEF 0xBF 0xBD. This is probably because before sending out the data, it is transformed by the Deta platform in this way:

Buffer.toString('utf8')

This command maps invalid sequences of bytes to the above mentioned group of three bytes. This also happens with other services, like Axios (first answer):

https://stackoverflow.com/questions/66807052/utf-8-encoded-string-to-buffer-node-js#answer-66807640

The thing is, with Axios you can specify that you're sending raw bytes with responseType: "arraybuffer", but I don't see any option like this in Deta. If there's a way, it should have been included in the guide for serving images that I linked above.

So, to recap: as for now, it's not possible to serve images with the Deta platform. Or, if it is possible, it's not as simple as following the "Node Tutorial". Probably the Deta platform uses the command Buffer.toString('utf8') to serve projects, but it shouldn’t always do it. It would be better to just delete that command and let people decide the type of encoding they want to use.