Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for multiple Spark versions #462

Open
gerashegalov opened this issue Feb 15, 2020 · 2 comments
Open

Support for multiple Spark versions #462

gerashegalov opened this issue Feb 15, 2020 · 2 comments

Comments

@gerashegalov
Copy link
Contributor

Problem
We need to be able to support multiple Spark versions to avoid painful upgrades

Solution

  • Create build profiles capturing dependencies of a particular Spark version. Add a Shim library with the stable facade translating API's that change let us say from 2.3.2 to 2.4. x to 3.x

Alternatives
continue being in lock step with Spark and suffer

Additional context
Support for wider variety of users with differing pace of Spark version adoption

@koertkuipers
Copy link

not just multiple spark versions... also multiple scala versions

@SemanticBeeng
Copy link

SemanticBeeng commented Sep 11, 2020

Maybe we can reuse the approach from frameless https://github.com/typelevel/frameless

Frameless intentionally does not have a compile dependency on Spark. This essentially allows you to use any version of Frameless with any version of Spark.

Since the 0.9.x release, Frameless is compiled only against Scala 2.12.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants