Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace typesystem with Spark-/Hive- types #367

Open
5 tasks
okennedy opened this issue Jan 10, 2020 · 0 comments
Open
5 tasks

Replace typesystem with Spark-/Hive- types #367

okennedy opened this issue Jan 10, 2020 · 0 comments

Comments

@okennedy
Copy link
Member

okennedy commented Jan 10, 2020

Possibly required for #361
Would subsume #333

Mimir has a much cruder internal type system than Spark. In addition to lacking collection types, there's a lot of capabilities (e.g., integer precision) that Mimir just does not support at the moment. Spark already has a well-developed type system, and it would be fantastic if we could replace this part of Mimir with the corresponding parts from Spark.

  • Replace Mimir's Type with Spark's DataType
  • Update or replace Typechecker with corresponding parts of Spark (not sure where this lives... perhaps the Catalyst analyzer
  • Tack on support for unit-based types (Maybe use User-Defined Types?)
  • Move to serialization using foo.json and foo.as[DataType] (make sure to keep track of everywhere this happens)
  • Add an upgrade path to migrate old, encoded types to the new form
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant