Consider open source data sharing endpoint?

[ZhiHanZ]

Currently we have already supported data sharing. https://databend.rs/doc/sql-commands/ddl/share/ , thus far we have designed our data sharing protocol with simple RESTful api and allow user to query accross different tenant under fine-grained permissions.
I think it is the right timing to implement an open source sharing endpoint.

Supposed feature:

1. A simple server for sharing operator testing and debugging
2. A lambda-runtime release for serverless deployment
3. Written in rust, thus we could use it with opendal(https://github.com/datafuselabs/opendal) for much easier maintainence on sharing endpoint

Possible features:

1. also support delta-sharing and other data lake sharing protocols on a single sharing endpoint
2. support more runtimes such as cloudflare worker

[ZhiHanZ]

cc @BohuTANG PTAL

[flaneur2020]

maybe we can consider extending the DDL about sharing like this:

CREATE DATABASE xxx FROM SHARING 'https://mytoken:[email protected]'

so we can allow the sharing server be deployed in an on-premise environment.

[ZhiHanZ]

I think it could provide additional configurations such as

1. overriding defaulting sharing endpoint
2. support customized authentication
3. support private deployed databend-query clusters to share data

maybe we could at first promote some SQL DDL API RFC and make further discussions?

[BohuTANG]

Share endpoint can be a catalog?