Adopting an open table format for Databend?

[shaeqahmed]

Nowadays the industry is moving toward open data formats like Iceberg or Delta Lake for tables on object storage. Besides potential architectural benefits of adopting one of these formats for Databend (not sure how valuable, as Databend already maintains specialized external indices on top of parquet data, but could help with schema evolution), this would be really valuable for adoption. For example, users with an iceberg table in s3 may currently be using Athena or Snowflake as their query engine and it would be really nice if one could just switch over to using Databend instead without having to copy the data. Is their plans to support something like this?

[dantengsky]

... this would be really valuable for adoption

can not agree more :)

... Is their plans to support something like this?

Yes.

• Custom table engines could be registered into databend, for example the github database and tables:

  https://github.com/datafuselabs/databend/tree/main/common/storages/preludes/src/github

• or customization at the catalog level if necessary, for example, the hive catalog (contributed & maintained by @sandflee)

  https://github.com/datafuselabs/databend/tree/main/common/storages/hive/src

but,

• definitely, there are things to be improved/polished, for example, the database engine can not be customized yet
• and at least for the short term, AFAIK, due to lack of resources, they are not on the schedule yet