Integrate genomics libraries in Spark support

[heuermh]

Hello, congrats on the recent blog post on the AWS HPC Blog!

I intend to look closer into your Spark support to see how it works. While I imagine it straightforward to use Spark-based tools via command line calls, it might be interesting to integrate these libraries and tools into the redun APIs directly.

• https://github.com/bigdatagenomics/adam
• https://github.com/bigdatagenomics/cannoli
• https://github.com/disq-bio/disq
• https://github.com/projectglow/glow
• etc.

[mattrasmus]

Thanks @heuermh for the ideas and links. It's very helpful.

Hopefully, many of these tools can be used within a redun task as-is. One design goal of redun was to make very few assumptions about what goes on inside as task function ("no dirty tricks") in order to be as compatible as possible with other data science libraries.

But I'll keep an eye out for opportunities for a more meaningful integration between redun and other spark libs.