Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] [Audit] potential performance improvement in Scala 2.13: Replace explicit ArrayOps#toSeq with s.c.immutable.ArraySeq.unsafeWrapArray #11839

Open
res-life opened this issue Dec 9, 2024 · 0 comments
Labels
audit_4.0.0 Audit related tasks for 4.0.0 feature request New feature or request performance A performance related task/issue

Comments

@res-life
Copy link
Collaborator

res-life commented Dec 9, 2024

Is your feature request related to a problem? Please describe.
Scala 2.13 changed the behavior for the toSeq API, it does with collection copy.
Scala 2.12 does not copy for toSeq.
So Apache Spark posted a commit (apache/spark@8a4890ddb55) : Replace explicit ArrayOps#toSeq with s.c.immutable.ArraySeq.unsafeWrapArray.
This commit has potential benefits for performance for Scala 2.13.
We should do the similar change as Spark did.
Refer to Spark code:

private[spark] object ArrayImplicits {

  implicit class SparkArrayOps[T](xs: Array[T]) {

    /**
     * Wraps an Array[T] as an immutable.ArraySeq[T] without copying.
     */
    def toImmutableArraySeq: immutable.ArraySeq[T] =
      immutable.ArraySeq.unsafeWrapArray(xs)
  }
}

Describe the solution you'd like
Maybe:

  • Copy Spark code: ArrayImplicits, we can not refer to this utility directly because it's only in Spark 4.0
  • search all the toSeq
  • Change from toSeq to toImmutableArraySeq
@res-life res-life added ? - Needs Triage Need team to review and classify audit_4.0.0 Audit related tasks for 4.0.0 feature request New feature or request labels Dec 9, 2024
@mattahrens mattahrens added performance A performance related task/issue and removed ? - Needs Triage Need team to review and classify labels Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
audit_4.0.0 Audit related tasks for 4.0.0 feature request New feature or request performance A performance related task/issue
Projects
None yet
Development

No branches or pull requests

2 participants