Skip to content

Releases: G-Research/spark-extension

[2.2.0] - 2022-07-21

21 Jul 11:40
Compare
Choose a tag to compare

Added

  • Add (global) row numbers transformation to Scala, Java and Python API. (#97)

Removed

  • Removed support for Pyton 3.6

[2.1.0] - 2022-04-07

07 Apr 19:40
Compare
Choose a tag to compare

Added

  • Add sorted group methods to Dataset. (#76)

[2.0.0] - 2021-10-29

07 Apr 19:40
Compare
Choose a tag to compare

Added

  • Add support for Spark 3.2 and Scala 2.13.
  • Support to ignore columns in diff API. (#63)

Removed

  • Removed support for Spark 2.4.

[1.3.3] - 2020-12-17

07 Apr 19:39
Compare
Choose a tag to compare

Added

  • Add support for Spark 3.1.

[1.3.2] - 2020-12-17

07 Apr 19:39
Compare
Choose a tag to compare

Changed

  • Refine conditional transformation helper methods.

[1.3.1] - 2020-12-10

07 Apr 19:38
Compare
Choose a tag to compare

Changed

  • Refine conditional transformation helper methods.

[1.3.0] - 2020-12-07

07 Apr 19:38
Compare
Choose a tag to compare

Added

  • Add transformation to compute histogram. (#26)
  • Add conditional transformation helper methods. (#27)
  • Add partitioned writing helpers that simplifies writing optimally ordered partitioned data. (#29)

[1.2.0] - 2020-10-06

07 Apr 19:37
Compare
Choose a tag to compare

Added

  • Add diff modes (#22): column-by-column, side-by-side, left and right side diff modes.
  • Add sparse mode (#23): diff DataFrame contains only changed values.

[1.1.0] - 2020-08-24

07 Apr 19:36
Compare
Choose a tag to compare

Added

  • Add Python API for Diff transformation.
  • Add change column to Diff transformation providing column names of all changed columns in a row.
  • Add fluent methods to change immutable diff options.
  • Add backticks method to handle column names that contain dots (.).

[1.0.0] - 2020-03-12

07 Apr 19:35
Compare
Choose a tag to compare

Added

  • Add Diff transformation for Datasets.