hey @cocoa-xu , thank you for this PR!

I'm going to try to fix the missing parts if you don't mind. I'm going to open another branch to avoid conflicting with yours, but I'm going to be based on this branch, ok?

hey @cocoa-xu , thank you for this PR!

I'm going to try to fix the missing parts if you don't mind. I'm going to open another branch to avoid conflicting with yours, but I'm going to be based on this branch, ok?

No problem!

@cocoa-xu could we perhaps use a library that converts delta lake to an arrow stream, and then we can use the existing logic in df_from_arrow_stream_pointer to convert it to a dataframe?

Alternatively, we could even have delta lake as a separate library, and all it need to do is to expose a pointer to arrow stream, similar to what we do in Adbc. In Adbc we do:

  Adbc.Connection.query_pointer(conn, query, params, fn pointer, _num_rows ->
    Explorer.PolarsBackend.Native.df_from_arrow_stream_pointer(pointer)
  end)

But we could export df_from_arrow_stream_pointer(pointer) as a public function? This way you could do:

  DeltaLake.transfer_arrow_stream_pointer(lake, fn pointer ->
    Explorer.PolarsBackend.from_arrow_stream_pointer(pointer)
  end)

WDYT?

@cocoa-xu could we perhaps use a library that converts delta lake to an arrow stream, and then we can use the existing logic in df_from_arrow_stream_pointer to convert it to a dataframe?

Alternatively, we could even have delta lake as a separate library, and all it need to do is to expose a pointer to arrow stream, similar to what we do in Adbc.

Yeah, I think perhaps I'll try to have delta lake as a separate library, at least it would be a good start point, and probably easier to debug (and possibly integrate it into explorer if it won't cause too much trouble to do so)

So I read more in the deltalake repo and other relevant repos, and I found that we basically have to wait for the upstream to support ADBC.

According to delta-rs issue #954 and the design doc of Delta Lake ADBC,

We would aim to use this to support integration with DuckDB and Polars.

Polars has a different problem: it uses arrow2, which is a separate Arrow implementation from apache/arrow-rs (used by DataFusion).

This was one of the difficulties I encountered here -- there's no easy way to cast from polars_arrow's array to the one deltalake is using, i.e., apache/arrow-rs' array, or the other way around.

Currently, I think we should wait for

• the Rust version of ADBC driver manager, which will be implemented either in
  • feat(rust/drivermgr): start rust driver_manager apache/arrow-adbc#416
  • feat(rust): add driver manager and ability to expose Rust driver as C driver apache/arrow-adbc#1496
• ADBC connector for deltalake, ADBC Driver delta-io/delta-rs#954

Once they're available, we can easily add support for deltalake to explorer.

Besides that, we can try to replace current ADBC binding for Elixir https://github.com/elixir-explorer/adbc with the Rust version of ADBC if we want to and if it's possible/suitable for our use cases.

ADBC sounds like a solid route forward. Shall we close this one then?

ADBC sounds like a solid route forward. Shall we close this one then?

Yeah I think we can close this one and wait for the ADBC version for it.