Highly recommand to move S3_PUBLIC_DOMAIN sharing function to server side

[Lostsite]

My suggestion: request the file to a temp storage location on server, than ask the llm provider to download it directly from the server:

The problem for "S3_PUBLIC_DOMAIN" is many cheap "S3" storage providers are only "S3 compatible", many of them would NOT support "S3_PUBLIC_DOMAIN" on the basket level. Request to use "S3_PUBLIC_DOMAIN" would cause us can't use them.

(Also it is a pretty bad idea to keep a basket level sharing link alive all the time. If it leaks, it could cause trolls evilly pulling everything down again and again just to bill you, since usually "S3" storage are charged by downloads).

My suggestion is: 1st request the file to a temp storage location on server, than ask the llm provider to download it directly from the server:

Old method:
(llm provider) <----- ( File in S3 basket (permanent storage) )

New method:
(llm provider) <----- (File on server (request from S3, save for 1 -2h)) <-------- ( File in S3 basket (permanent storage) )

This would also benefits us if people wish to use the file multiple times, like download it during middle of one conversation, we can save some S3 download fee.

[arvinxx]

• (File on server (request from S3, save for 1 -2h))

I don't think it's a good idea. It's unavailable on serverless deployment in Vercel. If you want to protect the s3, maybe it's better to have a signed url from S3 to for view.

[arvinxx]

If just build this function only for Docker

I think it's not a good idea. I don't have efforts to maintain this part code . I will consider the coder to organize in serverless runtime, than docker.

as for the 4~8, it will take lots of efforts to make the cache correctly and it's not right way. I think it will mass the core biz logic, which I always want to avoid.

[xynova]

• (File on server (request from S3, save for 1 -2h))

I don't think it's a good idea. It's unavailable on serverless deployment in Vercel. If you want to protect the s3, maybe it's better to have a signed url from S3 to for view.

Hi @arvinxx, I actually wanted do try out creating signed URLs with short expiration policy for all Get file operations. I want to privately host Lobechat and wanted to manage the risk of sensitive files being uploaded to s3 by adding KMS encryption (which requires a signed url that can have decrypt permissions when reading). Has anyone discussed this idea before? I am a total noobs to the project

[arvinxx]

Hi @arvinxx, I actually wanted do try out creating signed URLs with short expiration policy for all Get file operations. I want to privately host Lobechat and wanted to manage the risk of sensitive files being uploaded to s3 by adding KMS encryption (which requires a signed url that can have decrypt permissions when reading). Has anyone discussed this idea before? I am a total noobs to the project

There is a PR for it #4254

[arvinxx]

@Lostsite @xynova the #4254 is merged. Now you can set S3_SET_ACL=0 to use private url

[xynova]

@Lostsite @xynova the #4254 is merged. Now you can set S3_SET_ACL=0 to use private url

Awesome, thank you @arvinxx