ETL workloads on binary data

[JanBeelte]

Hello everyone,

I am currently designing a new architecture for ETL workloads of custom binary data and was wondering if I could use ballista for this usecase. One assumption that should hold for a long time is that the binary files are small enough to fit onto the disk of a single executor node, the first datasets might even fit into memory directly. I want to write my own custom parser for the data formats in Rust.

• Are that kind of "binary ETL" workloads possible with ballista today?
• Are there already plugin / extension mechanisms to write my own rust based parsers for the files?
• How would you approach the "binary ETL" use case under the above assumptions?

Best regards,
Jan