Arrow Interop?

[thatcort]

Does Databend use Arrow as the internal data representation?

Would it make sense to be able to access this for zero copy computation with other arrow tools?

[sundy-li]

Yes, its memory model is based on Arrow.

[thatcort]

Does it make sense to try to hand off results to other processes on the same node instead of sending it via HTTP/etc.? This might be complicated in terms of how k8s organizes resources, but it would be great to be able to move results from Databend into a other arrow tools with zero copy.

[sundy-li]

Yes, we have supported FlightSQL interface, you can use bendsql --flight -P8900 client to connect the server.

Also, the RPC service between query nodes is using flight which is zero copy based on arrow.

        --flight-api-address <FLIGHT_API_ADDRESS>
        --flight-sql-handler-host <FLIGHT_SQL_HANDLER_HOST>
        --flight-sql-handler-port <FLIGHT_SQL_HANDLER_PORT>
        --flight-sql-tls-server-cert <FLIGHT_SQL_TLS_SERVER_CERT>
        --flight-sql-tls-server-key <FLIGHT_SQL_TLS_SERVER_KEY>